Jesse Hoey

Sensor-Based Activity Recognition

Research on sensor-based activity recognition has, recently, made significant progress and is attracting growing attention in a number of disciplines and application domains. However, there is a lack of high-level overview on this topic that can inform related communities of the research state of the art. In this paper, we present a comprehensive survey to examine the development and current status of various aspects of sensor-based activity recognition. We first discuss the general rationale and distinctions of vision-based and sensor-based activity recognition. Then, we review the major approaches and methods associated with sensor-based activity monitoring, modeling, and recognition from which strengths and weaknesses of those approaches are highlighted. We make a primary distinction in this paper between data-driven and knowledge-driven approaches, and use this distinction to structure our survey. We also discuss some promising directions for future research.

An analytic solution to discrete Bayesian reinforcement learning

Reinforcement learning (RL) was originally proposed as a framework to allow agents to learn in an online fashion as they interact with their environment. Existing RL algorithms come short of achieving this goal because the amount of exploration required is often too costly and/or too time consuming for online learning. As a result, RL is mostly used for offline learning in simulated environments. We propose a new algorithm, called BEETLE, for effective online learning that is computationally efficient while minimizing the amount of exploration. We take a Bayesian model-based approach, framing RL as a partially observable Markov decision process. Our two main contributions are the analytical derivation that the optimal value function is the upper envelope of a set of multivariate polynomials, and an efficient point-based value iteration algorithm that exploits this simple parameterization.

Automated handwashing assistance for persons with dementia using video and a partially observable Markov decision process

This paper presents a real-time vision-based system to assist a person with dementia wash their hands. The system uses only video inputs, and assistance is given as either verbal or visual prompts, or through the enlistment of a human caregivers help. The system combines a Bayesian sequential estimation framework for tracking hands and towel, with a decision-theoretic framework for computing policies of action. The decision making system is a partially observable Markov decision process, or POMDP. Decision policies dictating system actions are computed in the POMDP using a point-based approximate solution technique. The tracking and decision making systems are coupled using a heuristic method for temporally segmenting the input video stream based on the continuity of the belief state. A key element of the system is the ability to estimate and adapt to user psychological states, such as awareness and responsiveness. We evaluate the system in three ways. First, we evaluate the hand-tracking system by comparing its outputs to manual annotations and to a simple hand-detection method. Second, we test the POMDP solution methods in simulation, and show that our policies have higher expected return than five other heuristic methods. Third, we report results from a ten-week trial with seven persons moderate-to-severe dementia in a long-term care facility in Toronto, Canada. The subjects washed their hands once a day, with assistance given by our automated system, or by a human caregiver, in alternating two-week periods. We give two detailed case study analyses of the system working during trials, and then show agreement between the system and independent human raters of the same trials.

Ambient kitchen: designing situated services using a high fidelity prototyping environment

The Ambient Kitchen is a high fidelity prototype for exploring the design of pervasive computing algorithms and applications for everyday environments. The environment integrates data projectors, cameras, RFID tags and readers, object mounted accelerometers, and under-floor pressure sensing using a combination of wired and wireless networks. The Ambient Kitchen is a lab-based replication of a real kitchen where careful design has hidden the additional technology, and allows both the evaluation of pervasive computing prototypes and the simultaneous capture of the multiple synchronized streams of sensor data. Previous work exploring the requirements for situated support for people with cognitive impairments motivated the design of the physical and technical infrastructure and we describe both our motivations and previous work on interaction design in kitchen environments. Finally, we describe how our lab-based prototype has been put to use as: a design tool for designers; a design tool for users; an observatory to collect sensor data for activity recognition algorithm development, and an evaluation test bed. The limitations and advantages of lab-based, as opposed to in situ home-based testing, are discussed

Rapid specification and automated generation of prompting systems to assist people with dementia

Activity recognition in intelligent environments could play a key role for supporting people in their activities of daily life. Partially observable Markov decision process (POMDP) models have been used successfully, for example, to assist people with dementia when carrying out small multistep tasks such as hand washing. POMDP models are a powerful, yet flexible framework for modeling assistance that can deal with uncertainty and utility in a theoretically well-justified manner. Unfortunately, POMDPs usually require a very labor-intensive, manual set-up procedure. This paper describes a knowledge-driven method for automatically generating POMDP activity recognition and context-sensitive prompting systems for complex tasks. It starts with a psychologically justified description of the task and the particular environment in which it is to be carried out that can be generated from empirical data. This is then combined with a specification of the available sensors and effectors to build a working prompting system. The method is illustrated by building a system that prompts through the task of making a cup of tea in a real-world kitchen. The case is made that, with further development and tool support, the method could feasibly be used in a clinical or industrial setting.

Hierarchical unsupervised learning of facial expression categories

We consider the problem of unsupervised classification of temporal sequences of facial expressions in video. This problem arises in the design of an adaptive visual agent, which must be capable of identifying appropriate classes of visual events without supervision to effectively complete its tasks. We present a multilevel dynamic Bayesian network that learns the high-level dynamics of facial expressions simultaneously, with models of the expressions themselves. We show how the parameters of the model can be learned in a scalable and efficient way. We present preliminary results using real video data and a class of simulated dynamic event models. The results show that our model correctly classifies the input data comparably to a standard event classification approach, while also learning the high-level model parameters.

Body Movements for Affective Expression: A Survey of Automatic Recognition and Generation

Body movements communicate affective expressions and, in recent years, computational models have been developed to recognize affective expressions from body movements or to generate movements for virtual agents or robots which convey affective expressions. This survey summarizes the state of the art on automatic recognition and generation of such movements. For both automatic recognition and generation, important aspects such as the movements analyzed, the affective state representation used, and the use of notation systems is discussed. The survey concludes with an outline of open problems and directions for future work.

The use of an intelligent prompting system for people with dementia

perform routine activities: They cannot remember the proper sequence of steps or how to use the necessary tools. Strategies commonly used by caregivers involve continually providing reminders or cues. Family caregivers find assisting their loved ones to be particularly upsetting and embarrassing, as it necessitates invasion of privacy and role reversal. This difficult situation often results in the family caregiver not being able to cope, and the affected person being placed in a care facility. In response to the unique needs of older adults with dementia, we have been developing a new prompting device that uses artificial intelligence (AI) to automatically monitor an older adult during a common self-care activity (i.e., hand washing) and provide prompts as needed.

Representation and recognition of complex human motion

The quest for a vision system capable of representing and recognizing arbitrary motions benefits from a low dimensional, non-specific representation of flow fields, to be used in high level classification tasks. We present Zernike polynomials as an ideal candidate for such a representation. The basis of Zernike polynomials is complete and orthogonal and can be used for describing many types of motion at many scales. Starting from image sequences, locally smooth image velocities are derived using a robust estimation procedure, from which are computed compact representations of the flow using the Zernike basis. Continuous density hidden Markov models are trained using the temporal sequences of vectors thus obtained, and are used for subsequent classification. We present results of our method applied to image sequences of facial expressions both with and without significant rigid head motion and to sequences of lip motion from a known database. We demonstrate that the Zernike representation yields results competitive with those obtained using principal components, while not committing to specific types of motion. It is therefore ideal as a fundamental building block for a vision system capable of classifying arbitrary motion types.

The development of an adaptive upper-limb stroke rehabilitation robotic system

BackgroundStroke is the primary cause of adult disability. To support this large population in recovery, robotic technologies are being developed to assist in the delivery of rehabilitation. This paper presents an automated system for a rehabilitation robotic device that guides stroke patients through an upper-limb reaching task. The system uses a decision theoretic model (a partially observable Markov decision process, or POMDP) as its primary engine for decision making. The POMDP allows the system to automatically modify exercise parameters to account for the specific needs and abilities of different individuals, and to use these parameters to take appropriate decisions about stroke rehabilitation exercises.MethodsThe performance of the system was evaluated by comparing the decisions made by the system with those of a human therapist. A single patient participant was paired up with a therapist participant for the duration of the study, for a total of six sessions. Each session was an hour long and occurred three times a week for two weeks. During each session, three steps were followed: (A) after the system made a decision, the therapist either agreed or disagreed with the decision made; (B) the researcher had the device execute the decision made by the therapist; (C) the patient then performed the reaching exercise. These parts were repeated in the order of A-B-C until the end of the session. Qualitative and quantitative question were asked at the end of each session and at the completion of the study for both participants.ResultsOverall, the therapist agreed with the system decisions approximately 65% of the time. In general, the therapist thought the system decisions were believable and could envision this system being used in both a clinical and home setting. The patient was satisfied with the system and would use this system as his/her primary method of rehabilitation.ConclusionsThe data collected in this study can only be used to provide insight into the performance of the system since the sample size was limited. The next stage for this project is to test the system with a larger sample size to obtain significant results.