Agata Rozga

Decoding Children's Social Behavior

We introduce a new problem domain for activity recognition: the analysis of childrens social and communicative behaviors based on video and audio data. We specifically target interactions between children aged 1-2 years and an adult. Such interactions arise naturally in the diagnosis and treatment of developmental disorders such as autism. We introduce a new publicly-available dataset containing over 160 sessions of a 3-5 minute child-adult interaction. In each session, the adult examiner followed a semi-structured play interaction protocol which was designed to elicit a broad range of social behaviors. We identify the key technical challenges in analyzing these behaviors, and describe methods for decoding the interactions. We present experimental results that demonstrate the potential of the dataset to drive interesting research questions, and show preliminary results for multi-modal activity recognition.

Imitation from 12 to 24 months in autism and typical development: A longitudinal Rasch analysis

The development of imitation during the second year of life plays an important role in domains of sociocognitive development such as language and social learning. Deficits in imitation ability in persons with autism spectrum disorder (ASD) from toddlerhood into adulthood have also been repeatedly documented, raising the possibility that early disruptions in imitation contribute to the onset of ASD and the deficits in language and social interaction that define the disorder. This study prospectively examined the development of imitation between 12 and 24 months of age in 154 infants at familial risk for ASD and 78 typically developing infants who were all later assessed at 36 months for ASD or other developmental delays. The study established a developmental measure of imitation ability and examined group differences over time, using an analytic Rasch measurement model. Results revealed a unidimensional latent construct of imitation and verified a reliable sequence of imitation skills that was invariant over time for all outcome groups. Results also showed that all groups displayed similar significant linear increases in imitation ability between 12 and 24 months and that these increases were related to individual growth in both expressive language and ratings of social engagement but not in fine motor development. The group of children who developed ASD by age 3 years exhibited delayed imitation development compared with the low-risk typical outcome group across all time-points, but were indistinguishable from other high-risk infants who showed other cognitive delays not related to ASD.

Automatic assessment of problem behavior in individuals with developmental disabilities

Severe behavior problems of children with developmental disabilities often require intervention by specialists. These specialists rely on direct observation of the behavior, usually in a controlled clinical environment. In this paper, we present a technique for using on-body accelerometers to assist in automated classification of problem behavior during such direct observation. Using simulated data of episodes of severe behavior acted out by trained specialists, we demonstrate how machine learning techniques can be used to segment relevant behavioral episodes from a continuous sensor stream and to classify them into distinct categories of severe behavior (aggression, disruption, and self-injury). We further validate our approach by demonstrating it produces no false positives when applied to a publicly accessible dataset of activities of daily living. Finally, we show promising classification results when our sensing and analysis system is applied to data from a real assessment session conducted with a child exhibiting problem behaviors.

Detecting eye contact using wearable eye-tracking glasses

We describe a system for detecting moments of eye contact between an adult and a child, based on a single pair of gaze-tracking glasses which are worn by the adult. Our method utilizes commercial gaze tracking technology to determine the adults point of gaze, and combines this with computer vision analysis of video of the childs face to determine their gaze direction. Eye contact is then detected as the event of simultaneous, mutual looking at faces by the dyad. We report encouraging findings from an initial implementation and evaluation of this approach.

Using electrodermal activity to recognize ease of engagement in children during social interactions

The recent emergence of comfortable wearable sensors has focused almost entirely on monitoring physical activity, ignoring opportunities to monitor more subtle phenomena, such as the quality of social interactions. We argue that it is compelling to address whether physiological sensors can shed light on quality of social interactive behavior. This work leverages the use of a wearable electrodermal activity (EDA) sensor to recognize ease of engagement of children during a social interaction with an adult. In particular, we monitored 51 child-adult dyads in a semi-structured play interaction and used Support Vector Machines to automatically identify children who had been rated by the adult as more or less difficult to engage. We report on the classification value of several features extracted from the childs EDA responses, as well as several other features capturing the physiological synchrony between the child and the adult.

Undifferentiated facial electromyography responses to dynamic, audio‐visual emotion displays in individuals with autism spectrum disorders

We examined facial electromyography (fEMG) activity to dynamic, audio-visual emotional displays in individuals with autism spectrum disorders (ASD) and typically developing (TD) individuals. Participants viewed clips of happy, angry, and fearful displays that contained both facial expression and affective prosody while surface electrodes measured corrugator supercilli and zygomaticus major facial muscle activity. Across measures of average and peak activity, the TD group demonstrated emotion-selective fEMG responding, with greater relative activation of the zygomatic to happy stimuli and greater relative activation of the corrugator to fearful stimuli. In contrast, the ASD group largely showed no significant differences between zygomatic and corrugator activity across these emotions. There were no group differences in the magnitude and timing of fEMG response in the muscle congruent to the stimuli. This evidence that fEMG responses in ASD are undifferentiated with respect to the valence of the stimulus is discussed in light of potential underlying neurobiological mechanisms.

Behavioral Imaging and Autism

Behavioral imaging encompasses the use of computational sensing and modeling techniques to measure and analyze human behavior. This article discusses a research program focused on the study of dyadic social interactions between children and their caregivers and peers. The study has resulted in a dataset containing semi-structured play interactions between children and adults. Behavioral imaging could broadly affect the quality of care for individuals with a developmental or behavioral disorder.

Detecting bids for eye contact using a wearable camera

We propose a system for detecting bids for eye contact directed from a child to an adult who is wearing a point-of-view camera. The camera captures an egocentric view of the child-adult interaction from the adults perspective. We detect and analyze the childs face in the egocentric video in order to automatically identify moments in which the child is trying to make eye contact with the adult. We present a learning-based method that couples a pose-dependent appearance model with a temporal Conditional Random Field (CRF). We present encouraging findings from an experimental evaluation using a newly collected dataset of 12 children. Our method outperforms state-of-the-art approaches and enables measuring gaze behavior in naturalistic social interactions.

Supporting parents for in-home capture of problem behaviors of children with developmental disabilities

Ubiquitous computing has shown promise in applications for health care in the home. In this paper, we focus on a study of how a particular ubicomp capability, selective archiving, can be used to support behavioral health research and practice. Selective archiving technology, which allows the capture of a window of data prior to and after an event, can enable parents of children with autism and related disabilities to record video clips of events leading up to and following an instance of problem behavior. Behavior analysts later view these video clips to perform a functional assessment. In contrast to the current practice of direct observation, a powerful method to gather data about child problem behaviors but costly in terms of human resources and liable to alter behavior in the subjects, selective archiving is cost effective and has the potential to provide rich data with minimal instructions to the natural environment. To assess the effectiveness of parent data collection through selective archiving in the home, we developed a research tool, CRAFT (Continuous Recording And Flagging Technology) and conducted a study by installing CRAFT in eight households of children with developmental disabilities and severe behavior concerns. The results of this study show the promise and remaining challenges for this technology. We have also shown that careful attention to the design of a ubicomp system for use by other domain specialists or non-technical users is key to moving ubicomp research forward.

A Novel System for Supporting Autism Diagnosis Using Home Videos: Iterative Development and Evaluation of System Design

Background Observing behavior in the natural environment is valuable to obtain an accurate and comprehensive assessment of a child’s behavior, but in practice it is limited to in-clinic observation. Research shows significant time lag between when parents first become concerned and when the child is finally diagnosed with autism. This lag can delay early interventions that have been shown to improve developmental outcomes. Objective To develop and evaluate the design of an asynchronous system that allows parents to easily collect clinically valid in-home videos of their child’s behavior and supports diagnosticians in completing diagnostic assessment of autism. Methods First, interviews were conducted with 11 clinicians and 6 families to solicit feedback from stakeholders about the system concept. Next, the system was iteratively designed, informed by experiences of families using it in a controlled home-like experimental setting and a participatory design process involving domain experts. Finally, in-field evaluation of the system design was conducted with 5 families of children (4 with previous autism diagnosis and 1 child typically developing) and 3 diagnosticians. For each family, 2 diagnosticians, blind to the child’s previous diagnostic status, independently completed an autism diagnosis via our system. We compared the outcome of the assessment between the 2 diagnosticians, and between each diagnostician and the child’s previous diagnostic status. Results The system that resulted through the iterative design process includes (1) NODA smartCapture, a mobile phone-based application for parents to record prescribed video evidence at home; and (2) NODA Connect, a Web portal for diagnosticians to direct in-home video collection, access developmental history, and conduct an assessment by linking evidence of behaviors tagged in the videos to the Diagnostic and Statistical Manual of Mental Disorders criteria. Applying clinical judgment, the diagnostician concludes a diagnostic outcome. During field evaluation, without prior training, parents easily (average rating of 4 on a 5-point scale) used the system to record video evidence. Across all in-home video evidence recorded during field evaluation, 96% (26/27) were judged as clinically useful, for performing an autism diagnosis. For 4 children (3 with autism and 1 typically developing), both diagnosticians independently arrived at the correct diagnostic status (autism versus typical). Overall, in 91% of assessments (10/11) via NODA Connect, diagnosticians confidently (average rating 4.5 on a 5-point scale) concluded a diagnostic outcome that matched with the child’s previous diagnostic status. Conclusions The in-field evaluation demonstrated that the system’s design enabled parents to easily record clinically valid evidence of their child’s behavior, and diagnosticians to complete a diagnostic assessment. These results shed light on the potential for appropriately designed telehealth technology to support clinical assessments using in-home video captured by families. This assessment model can be readily generalized to other conditions where direct observation of behavior plays a central role in the assessment process.