Hesam Sagha

Collecting complex activity datasets in highly rich networked sensor environments

We deployed 72 sensors of 10 modalities in 15 wireless and wired networked sensor systems in the environment, in objects, and on the body to create a sensor-rich environment for the machine recognition of human activities. We acquired data from 12 subjects performing morning activities, yielding over 25 hours of sensor data. We report the number of activity occurrences observed during post-processing, and estimate that over 13000 and 14000 object and environment interactions occurred. We describe the networked sensor setup and the methodology for data acquisition, synchronization and curation. We report on the challenges and outline lessons learned and best practice for similar large scale deployments of heterogeneous networked sensor systems. We evaluate data acquisition quality for on-body and object integrated wireless sensors; there is less than 2.5% packet loss after tuning. We outline our use of the dataset to develop new sensor network self-organization principles and machine learning techniques for activity recognition in opportunistic sensor configurations. Eventually this dataset will be made public.

The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition

There is a growing interest on using ambient and wearable sensors for human activity recognition, fostered by several application domains and wider availability of sensing technologies. This has triggered increasing attention on the development of robust machine learning techniques that exploits multimodal sensor setups. However, unlike other applications, there are no established benchmarking problems for this field. As a matter of fact, methods are usually tested on custom datasets acquired in very specific experimental setups. Furthermore, data is seldom shared between different groups. Our goal is to address this issue by introducing a versatile human activity dataset recorded in a sensor-rich environment. This database was the basis of an open challenge on activity recognition. We report here the outcome of this challenge, as well as baseline performance using different classification techniques. We expect this benchmarking database will motivate other researchers to replicate and outperform the presented results, thus contributing to further advances in the state-of-the-art of activity recognition methods.

A hybrid brain?computer interface based on the fusion of electroencephalographic and electromyographic activities

Hybrid brain-computer interfaces (BCIs) are representing a recent approach to develop practical BCIs. In such a system disabled users are able to use all their remaining functionalities as control possibilities in parallel with the BCI. Sometimes these people have residual activity of their muscles. Therefore, in the presented hybrid BCI framework we want to explore the parallel usage of electroencephalographic (EEG) and electromyographic (EMG) activity, whereby the control abilities of both channels are fused. Results showed that the participants could achieve a good control of their hybrid BCI independently of their level of muscular fatigue. Thereby the multimodal fusion approach of muscular and brain activity yielded better and more stable performance compared to the single conditions. Even in the case of an increasing muscular fatigue a good control (moderate and graceful degradation of the performance compared to the non-fatigued case) and a smooth handover could be achieved. Therefore, such systems allow the users a very reliable hybrid BCI control although they are getting more and more exhausted or fatigued during the day.

Benchmarking classification techniques using the Opportunity human activity dataset

Human activity recognition is a thriving research field. There are lots of studies in different sub-areas of activity recognition proposing different methods. However, unlike other applications, there is lack of established benchmarking problems for activity recognition. Typically, each research group tests and reports the performance of their algorithms on their own datasets using experimental setups specially conceived for that specific purpose. In this work, we introduce a versatile human activity dataset conceived to fill that void. We illustrate its use by presenting comparative results of different classification techniques, and discuss about several metrics that can be used to assess their performance. Being an initial benchmarking, we expect that the possibility to replicate and outperform the presented results will contribute to further advances in state-of-the-art methods.

Multimodal Fusion of Muscle and Brain Signals for a Hybrid-BCI

Practical Brain-Computer Interfaces (BCIs) for disabled people should allow them to use all their remaining functionalities as control possibilities. Sometimes these people have residual activity of their muscles, most likely in the morning when they are not exhausted. In this work we fuse electromyographic (EMG) with electroencephalographic (EEG) activity in the framework of a so called “Hybrid-BCI” (hBCI) approach. Thereby, subjects could achieve a good control of their hBCI independently of their level of muscular fatigue. Furthermore, although EMG alone yields good performance, it is outperformed by the hybrid fusing of EEG and EMG. Two different fusion techniques are explored showing graceful performance degradation in the case of signal attenuation. Such a system allows a very reliable control and a smooth handover if the subjects get exhausted or fatigued during the day.

The OPPORTUNITY Framework and Data Processing Ecosystem for Opportunistic Activity and Context Recognition

Opportunistic sensing can be used to obtain data from sensors that just happen to be present in the user’s surroundings. By harnessing these opportunistic sensor configurations to infer activity or context, ambient intelligence environments become more robust, have improved user comfort thanks to reduced requirements on body-worn sensor deployment and they are not limited to a predefined and restricted location, defined by sensors specifically deployed for an application. We present the OPPORTUNITY Framework and Data Processing Ecosystem to recognize human activities or contexts in such opportunistic sensor configurations. It addresses the challenge of inferring human activities with limited guarantees about placement, nature and run-time availability of sensors. We realize this by a combination of: (i) a sensing/context framework capable of coordinating sensor recruitment according to a high level recognition goal, (ii) the corresponding dynamic instantiation of data processing elements to infer activities, (iii) a tight interaction between the last two elements in an “ecosystem” allowing to autonomously discover novel knowledge about sensor characteristics that is reusable in subsequent recognition queries. This allows the system to operate in open-ended environments. We demonstrate OPPORTUNITY on a large-scale dataset collected to exhibit the sensor richness and related characteristics, typical of opportunistic sensing systems. The dataset comprises 25 hours of activities of daily living, collected from 12 subjects. It contains data of 72 sensors covering 10 modalities and 15 networked sensor systems deployed in objects, on the body and in the environment. We show the mapping from a recognition goal to an instantiation of the recognition system. We also show the knowledge acquisition and reuse of the autonomously discovered semantic meaning of a new unknown sensor, the autonomous update of the trust indicator of a sensor due to unforeseen deteriorations, and the autonomous discovery of the on-body sensor placement.

Kinect=IMU? Learning MIMO Signal Mappings to Automatically Translate Activity Recognition Systems across Sensor Modalities

We propose a method to automatically translate a preexisting activity recognition system, devised for a source sensor domain S, so that it can operate on a newly discovered target sensor domain T, possibly of different modality. First, we use MIMO system identification techniques to obtain a function that maps the signals of S to T. This mapping is then used to translate the recognition system across the sensor domains. We demonstrate the approach in a 5-class gesture recognition problem translating between a vision-based skeleton tracking system (Kinect), and inertial measurement units (IMUs). An adequate mapping can be learned in as few as a single gesture (3 seconds) in this scenario. The accuracy after Kinect → IMU or IMU → Kinect translation is 4% below the baseline for the same limb. Translating across modalities and also to an adjacent limb yields an accuracy 8% below baseline. We discuss the sources of errors and means for improvement. The approach is independent of the sensor modalities. It supports multimodal activity recognition and more flexible real-world activity recognition system deployments.

On-line anomaly detection and resilience in classifier ensembles

Detection of anomalies is a broad field of study, which is applied in different areas such as data monitoring, navigation, and pattern recognition. In this paper we propose two measures to detect anomalous behaviors in an ensemble of classifiers by monitoring their decisions; one based on Mahalanobis distance and another based on information theory. These approaches are useful when an ensemble of classifiers is used and a decision is made by ordinary classifier fusion methods, while each classifier is devoted to monitor part of the environment. Upon detection of anomalous classifiers we propose a strategy that attempts to minimize adverse effects of faulty classifiers by excluding them from the ensemble. We applied this method to an artificial dataset and sensor-based human activity datasets, with different sensor configurations and two types of noise (additive and rotational on inertial sensors). We compared our method with two other well-known approaches, generalized likelihood ratio (GLR) and One-Class Support Vector Machine (OCSVM), which detect anomalies at data/feature level. We found that our method is comparable with GLR and OCSVM. The advantages of our method compared to them is that it avoids monitoring raw data or features and only takes into account the decisions that are made by their classifiers, therefore it is independent of sensor modality and nature of anomaly. On the other hand, we found that OCSVM is very sensitive to the chosen parameters and furthermore in different types of anomalies it may react differently. In this paper we discuss the application domains which benefit from our method.

Detecting and Rectifying Anomalies in Body Sensor Networks

Activity recognition using on body sensors are prone to degradation due to changes on sensor readings. The changes can occur because of degradation or alteration in the behaviour of the sensor with respect to the others. In this paper we propose a method which detects anomalous nodes in the network and takes compensatory actions to keep the performance of the system as high as possible while the system is running. We show on two activity datasets with different configurations of on body sensors that detection and compensation of anomalies make the system more robust against the changes.

Ensemble creation and reconfiguration for activity recognition: An information theoretic approach

Advances in sensing, portable computing devices, and wireless communication has lead to an increase in the number and variety of sensing enabled devices (e.g. smartphones or sensing garments). Pervasive computing and activity recognition systems should be able to take advantage of these sensors, even if they are not always available or appear in runtime. These sensors can be integrated into an ensemble that fuse their information to obtain the final decision. There is therefore a need for mechanisms to select which sensors should compose the ensemble, as well as techniques for dynamically reconfigure the ensemble so as to integrate new sensors. Sensors can be integrated into an ensemble where information from each of them is fused to obtain the final decisions. From the machine learning point of view, this corresponds to the combination of classifiers where measures of the accuracy and diversity of the ensemble are used to select the elements that may lead to the highest performance. Recent works propose measures of accuracy and diversity based on an information theoretical approach. In this paper we study the use of these measures for selecting ensembles in activity recognition based on body sensor networks. Besides a comparison with traditional diversity measures (e.g., Q-, κ-statistics), we also present mechanisms to exploit these measures for the dynamic reconfiguration of the ensemble and detection of changes in the network.