Jonathan Lester

The Mobile Sensing Platform: An Embedded Activity Recognition System

Activity-aware systems have inspired novel user interfaces and new applications in smart environments, surveillance, emergency response, and military missions. Systems that recognize human activities from body-worn sensors can further open the door to a world of healthcare applications, such as fitness monitoring, eldercare support, long-term preventive and chronic care, and cognitive assistance. Wearable systems have the advantage of being with the user continuously. So, for example, a fitness application could use real-time activity information to encourage users to perform opportunistic activities. Furthermore, the general public is more likely to accept such activity recognition systems because they are usually easy to turn off or remove.

“Are You with Me?” – Using Accelerometers to Determine If Two Devices Are Carried by the Same Person

As the proliferation of pervasive and ubiquitous computing devices continues, users will carry more devices. Without the ability for these devices to unobtrusively interact with one another, the user’s attention will be spent on coordinating, rather than using, these devices. We present a method to determine if two devices are carried by the same person, by analyzing walking data recorded by low-cost MEMS accelerometers using the coherence function, a measure of linear correlation in the frequency domain. We also show that these low-cost sensors perform similarly to more expensive accelerometers for the frequency range of human motion, 0 to 10Hz. We also present results from a large test group illustrating the algorithm’s robustness and its ability to withstand real world time delays, crucial for wireless technologies like Bluetooth and 802.11. We present results that show that our technique is 100% accurate using a sliding window of 8 seconds of data when the devices are carried in the same location on the body, is tolerant to inter-device communication latencies, and requires little communication bandwidth. In addition we present results for when devices are carried on different parts of the body.

BALANCE: towards a usable pervasive wellness application with accurate activity inference

Technology offers the potential to objectively monitor peoples eating and activity behaviors and encourage healthier lifestyles. BALANCE is a mobile phone-based system for long term wellness management. The BALANCE system automatically detects the users caloric expenditure via sensor data from a Mobile Sensing Platform unit worn on the hip. Users manually enter information on foods eaten via an interface on an N95 mobile phone. Initial validation experiments measuring oxygen consumption during treadmill walking and jogging show that the systems estimate of caloric output is within 87% of the actual value. Future work will refine and continue to evaluate the systems efficacy and develop more robust data input and activity inference methods.

New horizons in sensor development

BACKGROUND Accelerometry and other sensing technologies are important tools for physical activity measurement. Engineering advances have allowed developers to transform clunky, uncomfortable, and conspicuous monitors into relatively small, ergonomic, and convenient research tools. New devices can be used to collect data on overall physical activity and, in some cases, posture, physiological state, and location, for many days or weeks from subjects during their everyday lives. In this review article, we identify emerging trends in several types of monitoring technologies and gaps in the current state of knowledge. BEST PRACTICES The only certainty about the future of activity-sensing technologies is that researchers must anticipate and plan for change. We propose a set of best practices that may accelerate adoption of new devices and increase the likelihood that data being collected and used today will be compatible with new data sets and methods likely to appear on the horizon. FUTURE DIRECTIONS We describe several technology-driven trends, ranging from continued miniaturization of devices that provide gross summary information about activity levels and energy expenditure to new devices that provide highly detailed information about the specific type, amount, and location of physical activity. Some devices will take advantage of consumer technologies, such as mobile phones, to detect and respond to physical activity in real time, creating new opportunities in measurement, remote compliance monitoring, data-driven discovery, and intervention.

Mobile context inference using low-cost sensors

In this paper, we introduce a compact system for fusing location data with data from simple, low-cost, non-location sensors to infer a users place and situational context. Specifically, the system senses location with a GSM cell phone and a WiFi-enabled mobile device (each running Place Lab), and collects additional sensor data using a 2” x 1” sensor board that contains a set of common sensors (e.g. accelerometers, barometric pressure sensors) and is attached to the mobile device. Our chief contribution is a multi-sensor system design that provides indoor-outdoor location information, and which models the capabilities and form factor of future cell phones. With two basic examples, we demonstrate that even using fairly primitive sensor processing and fusion algorithms we can leverage the synergy between our location and non-location sensors to unlock new possibilities for mobile context inference. We conclude by discussing directions for future work.

VoiceLabel: using speech to label mobile sensor data

Many mobile machine learning applications require collecting and labeling data, and a traditional GUI on a mobile device may not be an appropriate or viable method for this task. This paper presents an alternative approach to mobile labeling of sensor data called VoiceLabel. VoiceLabel consists of two components: (1) a speech-based data collection tool for mobile devices, and (2) a desktop tool for offline segmentation of recorded data and recognition of spoken labels. The desktop tool automatically analyzes the audio stream to find and recognize spoken labels, and then presents a multimodal interface for reviewing and correcting data labels using a combination of the audio stream, the systems analysis of that audio, and the corresponding mobile sensor data. A study with ten participants showed that VoiceLabel is a viable method for labeling mobile sensor data. VoiceLabel also illustrates several key features that inform the design of other data labeling tools.

Automatic classification of daily fluid intake

Despite the potential health benefits of being able to monitor and log ones food and drink intake, manually performing this task is notoriously hard. While researchers are still exploring methods of automating this process for food, less work has been done in automatically classifying beverage intake. In this paper, we present a novel method that utilizes optical, ion selective electrical pH, and conductivity sensors in order to sense and classify liquid in a cup in a practical way. We describe two experiments, one that uses a high end commercial off-the-shelf spectrometer, and the other which uses a cheap sensor package that we engineered. Results show both that this method is feasible and relatively accurate (up to 79% classification for 68 different drinks), but also that we would be able to build this in such a way as to make it practical for real-world deployment. We describe the vision for building a sensor rich cup capable of determining the kind of liquid a person is drinking, as well as the opportunities that the success of such sensors may open.

An RFID based system for monitoring free weight exercises

In this paper we present preliminary results and future directions of work for a project in which we are building an RFID based system to sense and monitor free weight exercises.

Context to Make You More Aware

The goal of our work is to help users make more informed choices about what physical activities they undertake. One example is to provide relevant information to help someone choose whether to wait at the closest bus stop, or walk a few minutes to the next stop, without missing the bus. In this paper, we report on our development of a platform which collects relevant context information for five different scenarios, processes this data, and presents relevant information to the user at the right time, in an unobtrusive way. The personal area network we use consists of a sensor platform with wireless capabilities, a Bluetooth-enabled wristwatch, and a mobile phone. We are also able to communicate with the users PCs and various web services through multiple wireless channels. In future work, we will integrate the scenarios into a customizable set of applications, and evaluate the complete system.

An ecosystem of platforms to support sensors for personal fitness

We have developed a collection of portable platforms to enable context-aware applications to help users with their personal fitness. Our approach has been to focus on established form-factors such as cellphones and wrist-watches for the user interfaces. A variety of sensors are used to infer aspects of the users context and proactively gather and display information that is likely to be useful at that time. In this paper, we highlight some of our platforms, the roles they can play, some of the capabilities we have already implemented, and highlight some of the applications we are currently developing