Sara Bernardini

ECHOES: An intelligent serious game for fostering social communication in children with autism

This paper presents ECHOES, a serious game built to help young children with autism spectrum conditions practice social communication skills. We focus on the design and implementation of the interactive learning activities, which take place in a two-dimensional sensory garden, and the autonomous virtual agent, which acts as a credible social partner to children with autism. Both the activities and the agent are based on principles of best autism practice and input from users. Specification guidelines are given for building an autonomous socially competent agent that supports learning in this context. We present experimental results pertaining to the effectiveness of the agent based on an extensive evaluation of the ECHOES platform, which show encouraging tendencies for a number of children.

The TARDIS Framework: Intelligent Virtual Agents for Social Coaching in Job Interviews

The TARDIS project aims to build a scenario-based serious-game simulation platform for NEETs and job-inclusion associations that supports social training and coaching in the context of job interviews. This paper presents the general architecture of the TARDIS job interview simulator, and the serious game paradigm that we are developing.

Social communication between virtual characters and children with autism

Children with ASD have difficulty with social communication, particularly joint attention. Interaction in a virtual environment (VE) may be a means for both understanding these difficulties and addressing them. It is first necessary to discover how this population interacts with virtual characters, and whether they can follow joint attention cues in a VE. This paper describes a study in which 32 children with ASD used the ECHOES VE to assist a virtual character in selecting objects by following the characters gaze and/or pointing. Both accuracy and reaction time data suggest that children were able to successfully complete the task, and qualitative data further suggests that most children perceived the character as an intentional being with relevant, mutually directed behaviour.

Automated planning of simple persuasion dialogues

We take a simple form of non-adversarial persuasion dialogue in which one participant (the persuader) aims to convince the other (the responder) to accept the topic of the dialogue by asserting sets of beliefs. The responder replies honestly to indicate whether it finds the topic to be acceptable (we make no prescription as to what formalism and semantics must be used for this, only assuming some function for determining acceptable beliefs from a logical knowledge base). Our persuader has a model of the responder, which assigns probabilities to sets of beliefs, representing the likelihood that each set is the responder’s actual beliefs. The beliefs the persuader chooses to assert and the order in which it asserts them (i.e. its strategy) can impact on the success of the dialogue and the success of a particular strategy cannot generally be guaranteed (because of the uncertainty over the responder’s beliefs). We define our persuasion dialogue as a classical planning problem, which can then be solved by an automated planner to generate a strategy that maximises the chance of success given the persuader’s model of the responder; this allows us to exploit the power of existing automated planners, which have been shown to be efficient in many complex domains. We provide preliminary results that demonstrate how the efficiency of our approach scales with the number of beliefs.

Supporting children's social communication skills through interactive narratives with virtual characters

The development of social communication skills in children relies on multimodal aspects of communication such as gaze, facial expression, and gesture. We introduce a multimodal learning environment for social skills which uses computer vision to estimate the childrens gaze direction, processes gestures from a large multi-touch screen, estimates in real time the affective state of the users, and generates interactive narratives with embodied virtual characters. We also describe how the structure underlying this system is currently being extended into a general framework for the development of interactive multimodal systems.

Building an Intelligent, Authorable Serious Game for Autistic Children and Their Carers

This paper introduces the SHARE-IT project, which leverages serious games paradigm to motivate and engage children with autism diagnosis in interactive activities, based on the state-of-the-art autism intervention practices. The aim of SHARE-IT is to formulate, in partnership with schools, parents and industry, the requirements for a robust, intelligent and authorable environment for supporting children in exploring, practicing and acquiring social interaction skills. SHARE-IT focuses on two key challenges: (i) developing robust system architecture and implementation, able to support both continuing development of a serious game for children with autism and its real world use; and (ii) selecting appropriate technologies and techniques to allow for (a) multi-device and operating system deployment, (b) the development of an intelligent serious game for supporting social interaction while (c) allowing the flexibility for the environment to be authored by lay persons. SHARE-ITs architecture is presented and several considerations of importance to enabling the engineering of an intelligent and authorable serious game are discussed. Examples of technologies developed to date are given throughout and a discussion of future challenges offered.

Incremental compilation-to-SAT procedures

We focus on incremental compilation-to-SAT procedures (iCTS), a promising way to push standard SAT-based approaches beyond their limits. We propose the first comprehensive framework that encompasses all the aspects of an incremental decision procedure, from the encoding to the incremental solver. We apply our guidelines to a real-world CTS approach (Bounded Model Checking) and show how to modify both the generation mechanism of a real BMC tool (NuSMV) and the solving engine of a public-domain SAT solver (SIM). Related approaches and experimental results are discussed as well.

Building autonomous social partners for autistic children

We present the design and implementation of an autonomous virtual agent that acts as a credible social partner for children with Autism Spectrum Conditions and supports them in acquiring social communication skills. The agents design is based on principles of best autism practice and input from users. Initial experimental results on the efficacy of the agent show encouraging tendencies for a number of children.

Automated Planning of Simple Persuasion Dialogues

We take a simple form of non-adversarial persuasion dialogue in which one participant (the persuader) aims to convince the other (the responder) to accept the topic of the dialogue by asserting sets of beliefs. The responder replies honestly to indicate whether it finds the topic to be acceptable (we make no prescription as to what formalism and semantics must be used for this, only assuming some function for determining acceptable beliefs from a logical knowledge base). Our persuader has a model of the responder, which assigns probabilities to sets of beliefs, representing the likelihood that each set is the responder’s actual beliefs. The beliefs the persuader chooses to assert and the order in which it asserts them (i.e. its strategy) can impact on the success of the dialogue and the success of a particular strategy cannot generally be guaranteed (because of the uncertainty over the responder’s beliefs). We define our persuasion dialogue as a classical planning problem, which can then be solved by an automated planner to generate a strategy that maximises the chance of success given the persuader’s model of the responder; this allows us to exploit the power of existing automated planners, which have been shown to be efficient in many complex domains. We provide preliminary results that demonstrate how the efficiency of our approach scales with the number of beliefs.

Modelling users' affect in job interviews : Technological demo

This demo presents an approach to recognising and interpreting social cues-based interactions in computer-enhanced job interview simulations. We show what social cues and complex mental states of the user are relevant in this interaction context, how they can be interpreted using static Bayesian Networks, and how they can be recognised automatically using state-of-the-art sensor technology in real-time.