Thi V. Duong

Activity recognition and abnormality detection with the switching hidden semi-Markov model

This paper addresses the problem of learning and recognizing human activities of daily living (ADL), which is an important research issue in building a pervasive and smart environment. In dealing with ADL, we argue that it is beneficial to exploit both the inherent hierarchical organization of the activities and their typical duration. To this end, we introduce the switching hidden semi-markov model (S-HSMM), a two-layered extension of the hidden semi-Markov model (HSMM) for the modeling task. Activities are modeled in the S-HSMM in two ways: the bottom layer represents atomic activities and their duration using HSMMs; the top layer represents a sequence of high-level activities where each high-level activity is made of a sequence of atomic activities. We consider two methods for modeling duration: the classic explicit duration model using multinomial distribution, and the novel use of the discrete Coxian distribution. In addition, we propose an effective scheme to detect abnormality without the need for training on abnormal data. Experimental results show that the S-HSMM performs better than existing models including the flat HSMM and the hierarchical hidden Markov model in both classification and abnormality detection tasks, alleviating the need for presegmented training data. Furthermore, our discrete Coxian duration model yields better computation time and generalization error than the classic explicit duration model.

Efficient duration and hierarchical modeling for human activity recognition

A challenge in building pervasive and smart spaces is to learn and recognize human activities of daily living (ADLs). In this paper, we address this problem and argue that in dealing with ADLs, it is beneficial to exploit both their typical duration patterns and inherent hierarchical structures. We exploit efficient duration modeling using the novel Coxian distribution to form the Coxian hidden semi-Markov model (CxHSMM) and apply it to the problem of learning and recognizing ADLs with complex temporal dependencies. The Coxian duration model has several advantages over existing duration parameterization using multinomial or exponential family distributions, including its denseness in the space of nonnegative distributions, low number of parameters, computational efficiency and the existence of closed-form estimation solutions. Further we combine both hierarchical and duration extensions of the hidden Markov model (HMM) to form the novel switching hidden semi-Markov model (SHSMM), and empirically compare its performance with existing models. The model can learn what an occupant normally does during the day from unsegmented training data and then perform online activity classification, segmentation and abnormality detection. Experimental results show that Coxian modeling outperforms a range of baseline models for the task of activity segmentation. We also achieve a recognition accuracy competitive to the current state-of-the-art multinomial duration model, while gaining a significant reduction in computation. Furthermore, cross-validation model selection on the number of phases K in the Coxian indicates that only a small K is required to achieve the optimal performance. Finally, our models are further tested in a more challenging setting in which the tracking is often lost and the activities considerably overlap. With a small amount of labels supplied during training in a partially supervised learning mode, our models are again able to deliver reliable performance, again with a small number of phases, making our proposed framework an attractive choice for activity modeling.

Human Behavior Recognition with Generic Exponential Family Duration Modeling in the Hidden Semi-Markov Model

The ability to learn and recognize human activities of daily living (ADLs) is important in building pervasive and smart environments. In this paper, we tackle this problem using the hidden semi-Markov model. We discuss the state-of-the-art duration modeling choices and then address a large class of exponential family distributions to model state durations. Inference and learning are efficiently addressed by providing a graphical representation for the model in terms of a dynamic Bayesian network (DBN). We investigate both discrete and continuous distributions from the exponential family (Poisson and inverse Gaussian respectively) for the problem of learning and recognizing ADLs. A full comparison between the exponential family duration models and other existing models including the traditional multinomial and the new Coxian are also presented. Our work thus completes a thorough investigation into the aspect of duration modeling and its application to human activities recognition in a real-world smart home surveillance scenario

Topic transition detection using hierarchical hidden Markov and semi-Markov models

In this paper we introduce a probabilistic framework to exploit hierarchy, structure sharing and duration information for topic transition detection in videos. Our probabilistic detection framework is a combination of a shot classification step and a detection phase using hierarchical probabilistic models. We consider two models in this paper: the extended Hierarchical Hidden Markov Model (HHMM) and the Coxian Switching Hidden semi-Markov Model (S-HSMM) because they allow the natural decomposition of semantics in videos, including shared structures, to be modeled directly, and thus enable efficient inference and reduce the sample complexity in learning. Additionally, the S-HSMM allows the duration information to be incorporated, consequently the modeling of long-term dependencies in videos is enriched through both hierarchical and duration modeling. Furthermore, the use of Coxian distribution in the S-HSMM makes it tractable to deal with long sequences in video. Our experimentation of the proposed framework on twelve educational and training videos shows that both models outperform the baseline cases (flat HMM and HSMM) and performances reported in earlier work in topic detection. The superior performance of the S-HSMM over the HHMM verifies our belief that the duration information is an important factor in video content modeling.

TOBY: early intervention in autism through technology

We describe TOBY Playpad, an early intervention program for children with Autism Spectrum Disorder (ASD). TOBY teaches the teacher -- the parent -- during the crucial period following diagnosis, which often coincides with no access to formal therapy. We reflect on TOBYs evolution from table-top aid for flashcards to an iPad app covering a syllabus of 326 activities across 51 skills known to be deficient for ASD children, such imitation, joint attention and language. The design challenges unique to TOBY are the need to adapt to marked differences in each childs skills and rate of development (a trait of ASD) and teach parents unfamiliar concepts core to behavioural therapy, such as reinforcement, prompting, and fading. We report on three trials that successively decrease oversight and increase parental autonomy, and demonstrate clear evidence of learning. TOBYs uniquely intertwined Natural Environment Tasks are found to be effective for children and popular with parents.

Pervasive multimedia for autism intervention

There is a growing gap between the number of children with autism requiring early intervention and available therapy. We present a portable platform for pervasive delivery of early intervention therapy using multi-touch interfaces and principled ways to deliver stimuli of increasing complexity and adapt to a childs performance. Our implementation weaves Natural Environment Tasks with iPad tasks, facilitating a learning platform that integrates early intervention in the childs daily life. The systems construction of stimulus complexity relative to task is evaluated by therapists, together with field trials for evaluating both the integrity of the instructional design and goal of stimulus presentation and adjustment relative to performance for learning tasks. We show positive results across all our stakeholders-children, parents and therapists. Our results have implications for other early learning fields that require principled ways to construct lessons across skills and adjust stimuli relative to performance.

TOBY play-pad application to teach children with ASD - A pilot trial

Abstract Purpose: To investigate use patterns and learning outcomes associated with the use of Therapy Outcomes By You (TOBY. Playpad, an early intervention iPad application. Methods: Participants were 33 families with a child with an autism spectrum disorder (ASD) aged 16 years or less, and with a diagnosis of autism or pervasive developmental disorder – not otherwise specified, and no secondary diagnoses. Families were provided with TOBY and asked to use it for 4–6 weeks, without further prompting or coaching. Dependent variables included participant use patterns and initial indicators of child progress. Results: Twenty-three participants engaged extensively with TOBY, being exposed to at least 100 complete learn units and completing between 17% and 100% of the curriculum. Conclusions: TOBY may make a useful contribution to early intervention programming for children with ASD delivering high rates of appropriate learning opportunities. Further research evaluating the efficacy of TOBY in relation to independent indicators of functioning is warranted.

Autism Blogs: Expressed Emotion, Language Styles and Concerns in Personal and Community Settings

The Internet has provided an ever increasingly popular platform for individuals to voice their thoughts, and like-minded people to share stories. This unintentionally leaves characteristics of individuals and communities, which are often difficult to be collected in traditional studies. Individuals with autism are such a case, in which the Internet could facilitate even more communication given its social-spatial distance being a characteristic preference for individuals with autism. Previous studies examined the traces left in the posts of online autism communities (Autism) in comparison with other online communities (Control). This work further investigates these online populations through the contents of not only their posts but also their comments. We first compare the Autism and Control blogs based on three features: topics, language styles and affective information. The autism groups are then further examined, based on the same three features, by looking at their personal (Personal) and community (Community) blogs separately. Machine learning and statistical methods are used to discriminate blog contents in both cases. All three features are found to be significantly different between Autism and Control, and between autism Personal and Community. These features also show good indicative power in prediction of autism blogs in both personal and community settings.

Nonparametric Discovery of Learning Patterns and Autism Subgroups from Therapeutic Data

Autism Spectrum Disorder (ASD) is growing at a staggering rate, but, little is known about the cause of this condition. Inferring learning patterns from therapeutic performance data, and subsequently clustering ASD children into subgroups, is important to understand this domain, and more importantly to inform evidence-based intervention. However, this data-driven task was difficult in the past due to insufficiency of data to perform reliable analysis. For the first time, using data from a recent application for early intervention in autism (TOBY Play pad), whose download count is now exceeding 4500, we present in this paper the automatic discovery of learning patterns across 32 skills in sensory, imitation and language. We use unsupervised learning methods for this task, but a notorious problem with existing methods is the correct specification of number of patterns in advance, which in our case is even more difficult due to complexity of the data. To this end, we appeal to recent Bayesian nonparametric methods, in particular the use of Bayesian Nonparametric Factor Analysis. This model uses Indian Buffet Process (IBP) as prior on a binary matrix of infinite columns to allocate groups of intervention skills to children. The optimal number of learning patterns as well as subgroup assignments are inferred automatically from data. Our experimental results follow an exploratory approach, present different newly discovered learning patterns. To provide quantitative results, we also report the clustering evaluation against K-means and Nonnegative matrix factorization (NMF). In addition to the novelty of this new problem, we were able to demonstrate the suitability of Bayesian nonparametric models over parametric rivals.

Identification and outcomes of clinical phenotypes in amyotrophic lateral sclerosis/motor neuron disease: Australian National Motor Neuron Disease observational cohort

Objective To capture the clinical patterns, timing of key milestones and survival of patients presenting with amyotrophic lateral sclerosis/motor neuron disease (ALS/MND) within Australia. Methods Data were prospectively collected and were timed to normal clinical assessments. An initial registration clinical report form (CRF) and subsequent ongoing assessment CRFs were submitted with a completion CRF at the time of death. Design Prospective observational cohort study. Participants 1834 patients with a diagnosis of ALS/MND were registered and followed in ALS/MND clinics between 2005 and 2015. Results 5 major clinical phenotypes were determined and included ALS bulbar onset, ALS cervical onset and ALS lumbar onset, flail arm and leg and primary lateral sclerosis (PLS). Of the 1834 registered patients, 1677 (90%) could be allocated a clinical phenotype. ALS bulbar onset had a significantly lower length of survival when compared with all other clinical phenotypes (p<0.004). There were delays in the median time to diagnosis of up to 12 months for the ALS phenotypes, 18 months for the flail limb phenotypes and 19 months for PLS. Riluzole treatment was started in 78–85% of cases. The median delays in initiating riluzole therapy, from symptom onset, varied from 10 to 12 months in the ALS phenotypes and 15–18 months in the flail limb phenotypes. Percutaneous endoscopic gastrostomy was implemented in 8–36% of ALS phenotypes and 2–9% of the flail phenotypes. Non-invasive ventilation was started in 16–22% of ALS phenotypes and 21–29% of flail phenotypes. Conclusions The establishment of a cohort registry for ALS/MND is able to determine clinical phenotypes, survival and monitor time to key milestones in disease progression. It is intended to expand the cohort to a more population-based registry using opt-out methodology and facilitate data linkage to other national registries.