David Bannach

Collecting complex activity datasets in highly rich networked sensor environments

We deployed 72 sensors of 10 modalities in 15 wireless and wired networked sensor systems in the environment, in objects, and on the body to create a sensor-rich environment for the machine recognition of human activities. We acquired data from 12 subjects performing morning activities, yielding over 25 hours of sensor data. We report the number of activity occurrences observed during post-processing, and estimate that over 13000 and 14000 object and environment interactions occurred. We describe the networked sensor setup and the methodology for data acquisition, synchronization and curation. We report on the challenges and outline lessons learned and best practice for similar large scale deployments of heterogeneous networked sensor systems. We evaluate data acquisition quality for on-body and object integrated wireless sensors; there is less than 2.5% packet loss after tuning. We outline our use of the dataset to develop new sensor network self-organization principles and machine learning techniques for activity recognition in opportunistic sensor configurations. Eventually this dataset will be made public.

Using Wearable Sensors for Real-Time Recognition Tasks in Games of Martial Arts - An Initial Experiment

Beside their stunning graphics, modern entertainment systems feature ever-higher levels of immersive user-interaction. Today, this is mostly achieved by virtual (VR) and augmented reality (AK) setups. On top of these, we envision to add ambient intelligence and context awareness to gaming applications in general and games of martial arts in particular. To this end, we conducted an initial experiment with inexpensive body-worn gyroscopes and acceleration sensors for the chum kiu motion sequence in wing tsun (a popular form of kung fu). The resulting data confirm the feasibility of our vision. Fine-tuned adaptations of various thresholding and pattern-matching techniques known from the fields of computational intelligence and signal processing should suffice to automate the analysis and recognition of important wing tsun movements in real time. Moreover, the data also seem to allow for the possibility of automatically distinguishing between certain levels of expertise and quality in executing the movements.

OPPORTUNITY: Towards opportunistic activity and context recognition systems

Opportunistic sensing allows to efficiently collect information about the physical world and the persons behaving in it. This may mainstream human context and activity recognition in wearable and pervasive computing by removing requirements for a specific deployed infrastructure. In this paper we introduce the newly started European research project OPPORTUNITY within which we develop mobile opportunistic activity and context recognition systems. We outline the projects objective, the approach we follow along opportunistic sensing, data processing and interpretation, and autonomous adaptation and evolution to environmental and user changes, and we outline preliminary results.

The OPPORTUNITY Framework and Data Processing Ecosystem for Opportunistic Activity and Context Recognition

Opportunistic sensing can be used to obtain data from sensors that just happen to be present in the user’s surroundings. By harnessing these opportunistic sensor configurations to infer activity or context, ambient intelligence environments become more robust, have improved user comfort thanks to reduced requirements on body-worn sensor deployment and they are not limited to a predefined and restricted location, defined by sensors specifically deployed for an application. We present the OPPORTUNITY Framework and Data Processing Ecosystem to recognize human activities or contexts in such opportunistic sensor configurations. It addresses the challenge of inferring human activities with limited guarantees about placement, nature and run-time availability of sensors. We realize this by a combination of: (i) a sensing/context framework capable of coordinating sensor recruitment according to a high level recognition goal, (ii) the corresponding dynamic instantiation of data processing elements to infer activities, (iii) a tight interaction between the last two elements in an “ecosystem” allowing to autonomously discover novel knowledge about sensor characteristics that is reusable in subsequent recognition queries. This allows the system to operate in open-ended environments. We demonstrate OPPORTUNITY on a large-scale dataset collected to exhibit the sensor richness and related characteristics, typical of opportunistic sensing systems. The dataset comprises 25 hours of activities of daily living, collected from 12 subjects. It contains data of 72 sensors covering 10 modalities and 15 networked sensor systems deployed in objects, on the body and in the environment. We show the mapping from a recognition goal to an instantiation of the recognition system. We also show the knowledge acquisition and reuse of the autonomously discovered semantic meaning of a new unknown sensor, the autonomous update of the trust indicator of a sensor due to unforeseen deteriorations, and the autonomous discovery of the on-body sensor placement.

Distributed modular toolbox for multi-modal context recognition

We present a GUI-based C++ toolbox that allows for building distributed, multi-modal context recognition systems by plugging together reusable, parameterizable components. The goals of the toolbox are to simplify the steps from prototypes to online implementations on low-power mobile devices, facilitate portability between platforms and foster easy adaptation and extensibility. The main features of the toolbox we focus on here are a set of parameterizable algorithms including different filters, feature computations and classifiers, a runtime environment that supports complex synchronous and asynchronous data flows, encapsulation of hardware-specific aspects including sensors and data types (e.g., int vs. float), and the ability to outsource parts of the computation to remote devices. In addition, components are provided for group-wise, event-based sensor synchronization and data labeling. We describe the architecture of the toolbox and illustrate its functionality on two case studies that are part of the downloadable distribution.

Towards wearable sensing-based assessment of fluid intake

Fluid intake is an important information for many health and assisted living applications. At the same time it is inherently difficult to monitor. Existing reliable solutions require augmented drinking containers, which severely limits the applicability of such systems. In this paper we investigate two key components of an unobtrusive, wearable solution that is independent of a particular drinking container or environment.

Integrated tool chain for recording and handling large, multimodal context recognition data sets

The demo will present a tool chain for recording, monitoring, labeling, and manipulation of complex multimodal data sets for activity recognition. The tool chain is comprehensive (going from logging, through labeling, monitoring to post processing and managing the data), integrated (with all tools being able to cooperate on joint data sets), and build around comfortable graphical user interfaces.

Activity Recognition in Opportunistic Sensor Environments

OPPORTUNITY is project under the EU FET-Open funding1 in which we develop mobile systems to recognize human activity in dynamically varying sensor setups. The system autonomously discovers available sensors around the user and self-configures to recognize desired activities. It reconfigures itself as the environment changes, and encompasses principles supporting autonomous operation in open-ended environments. OPPORTUNITY mainstreams ambient intelligence and improves user acceptance by relaxing constraints on body-worn sensor characteristics, and eases the deployment in real-world environments. We summarize key achievements of the project so far. The project outcomes are robust activity recognition systems. This may enable smarter activity-aware energy-management in buildings, and advanced activity-aware health assistants.

Can a Mobile Phone in a Pocket Reliably Recognize Ambient Sounds

We investigate how different locations inside clothing influence the ability of a system to recognize activity relevant sounds. Specifically, we consider the recognition of sounds from 9 household and office appliances recorded using an iPhone placed in 2 trouser pockets, 2 jacket pockets, a belt holster and the users’ hand. The aim is not to demonstrate good recognition rates on the above sounds (which has been done many times before) but to compare recognition rates from the individual locations and to understand how to best train the system to be location invariant.

On body capacitive sensing for a simple touchless user interface

In this paper we describe the use of textile, multi electrode capacitive on body sensing for contact less detection of simple control gestures. The work is motivated by a hospital ward rounds scenario of the in European Union sponsored WearIT@Work project. The focus of this work is not on human computer interface aspects but on the design, implementation, and evaluation of the sensing system and in particular on issues specific to on body placement and reliable detection of gestures with the multi-electrode design.