Christopher R. Wren

Toward scalable activity recognition for sensor networks

Sensor networks hold the promise of truly intelligent buildings: buildings that adapt to the behavior of their occupants to improve productivity, efficiency, safety, and security. To be practical, such a network must be economical to manufacture, install and maintain. Similarly, the methodology must be efficient and must scale well to very large spaces. Finally, be be widely acceptable, it must be inherently privacy-sensitive. We propose to address these requirements by employing networks of passive infrared (PIR) motion detectors. PIR sensors are inexpensive, reliable, and require very little bandwidth. They also protect privacy since they are neither capable of directly identifying individuals nor of capturing identifiable imagery or audio. However, with an appropriate analysis methodology, we show that they are capable of providing useful contextual information. The methodology we propose supports scalability by adopting a hierarchical framework that splits computation into localized, distributed tasks. To support our methodology we provide theoretical justification for the method that grounds it in the action recognition literature. We also present quantitative results on a dataset that we have recorded from a 400 square meter wing of our laboratory. Specifically, we report quantitative results that show better than 90% recognition performance for low-level activities such as walking, loitering, and turning. We also present experimental results for mid-level activities such as visiting and meeting.

The MERL motion detector dataset

Looking into the future of residential and office building Mitsubishi Electric Research Labs (MERL) has been collecting motion sensor data from a network of over 200 sensors for a year. The data is the residual traces of year in the life of a research laboratory. It contains interesting spatio-temporal structure ranging all the way from the seconds of individuals walking down hallways, the minutes in lobbies chatting with colleagues, the hours of dozens of people attending talks and meetings, the days and weeks that drive the patterns of life, to the months and seasons with their ebb and flow of visiting employees. This document describes that dataset, which contains well over 30 million raw motion records, spanning a calendar year and two floors of our research laboratory, as well as calender, weather, and some intermediate analytic results. The dataset was originally released as part of the 2007 Workshop on Massive Datasets. The dataset can be obtained from http://www.merl.com/wmd.

Similarity-based analysis for large networks of ultra-low resolution sensors

By analyzing the similarities between bit streams coming from a network of motion detectors, we can recover the network geometry and discover structure in the human behavior being observed. This means that a low-cost network of sensors can provide powerful contextual information to building systems: improving the efficiency of elevators, lighting, heating, and cooling; enhancing safety and security; and opening up new opportunities for human-centered information systems. This paper will show how signal similarity can be used to calibrate a sensor network to accuracies below the resolution of the individual sensors. This is done by analyzing the similarity structures in the unconstrained movement of people in the observed space. We will also present our efficient behavior-learning algorithm that yields 90% correct behavior-detection in data from a sensor network comprised of motion detectors by employing similarity-based clustering to automatically decompose complex activities into detectable sub-classes.

Tracking people in mixed modality systems

In traditional surveillance systems tracking of objects is achieved by means of image and video processing. The disadvantages of such surveillance systems is that if an object needs to be tracked - it has to be observed by a video camera. However, geometries of indoor spaces typically require a large number of video cameras to provide the coverage necessary for robust operation of video-based tracking algorithms. Increased number of video streams increases the computational burden on the surveillance system in order to obtain robust tracking results. In this paper we present an approach to tracking in mixed modality systems, with a variety of sensors. The system described here includes over 200 motion sensors as well as 6 moving cameras. We track individuals in the entire space and across cameras using contextual information available from the motion sensors. Motion sensors allow us to almost instantaneously find plausible tracks in a very large volume of data, ranging in months, which for traditional video search approaches could be virtually impossible. We describe a method that allows us to evaluate when the tracking system is unreliable and present the data to a human operator for disambiguation.

Socialmotion: measuring the hidden social life of a building

In this paper we present an approach to analyzing the social behaviors that occur in a typical office space. We describe a system consisting of over 200 motion sensors connected in a wireless network observing a medium-sized office space populated with almost 100 people for a period of almost a year. We use a tracklet graph representation of the data in the sensor network, which allows us to efficiently evaluate gross patterns of office-wide social behavior of its occupants during expected seasonal changes in the workforce as well as unexpected social events that affect the entire population of the space. We present our experiments with a method based on Kullback-Leibler metric applied to the office activity modelled as a Markov process. Using this approach we detect gross deviations of short term office-wide behavior patterns from previous long-term patterns spanning various time intervals. We compare detected deviations to the company calendar and find and provide some quantitative analysis of the relative impact of those disruptions across a range of temporal scales. We also present a favorable comparison to results achieved by applying the same analysis to email logs.

Finding temporal patterns by data decomposition

We present a new unsupervised learning technique for the discovery of temporal clusters in large data sets. Our method performs hierarchical decomposition of the data to find structure at many levels of detail and to reduce the overall computational cost of pattern discovery. We present a comparison to related methods on synthetic data sets and on real gestural and pedestrian flow data.

Browsing 3-D spaces with 3-D vision: body-driven navigation through the internet city

This paper presents a computer vision stereo based interface to navigate inside a 3-D Internet city, using body gestures. A wide-baseline stereo pair of cameras is used to obtain 3-D body models of the user’s hands and head in a small desk-area environment. The interface feeds this information to an HMM gesture classifier to reliably recognize the user’s browsing commands. To illustrate the features of this interface we describe its application to our 3-D Internet browser which facilitates the recollection of information by organizing and embedding it inside a virtual city through which the user navigates.

Minimalism in ubiquitous interface design

Minimalism in ubiquitous interface design allows computational augmentations to coexist with unmodified artifacts and the constellations of task behaviors surrounding them. By transparently integrating aspects of the digital world into real artifacts, we strive to provide ubiquitous interfaces to computation that do not obscure or destroy the highly refined interaction modalities of everyday artifacts in the physical world.

Buzz: measuring and visualizing conference crowds

This exhibition explores the idea of using technology to understand the movement of people. Not just on a small stage, but in an expansive environment. Not the fine details of movement of individuals, but the gross patterns of a population. Not the identifying biometrics, but patterns of group behavior that evolve from the structure of the environment and the points of interest embedded in that structure. In this instance: a marketplace, and in particular, the marketplace of ideas called SIGGRAPH 2007 Emerging Technologies (ETech).

Workshop on massive datasets

Are the tools we use to understand our data scalable to the tens of millions of records, huge spans of time, minute details of behavior, and large geographic extent that future sensor networks will generate? In the future buildings will be studded with sensors. Every movement will generate a few bits of data. Every fluctuation in temperature will be recorded. Every deviation in lighting will be noticed. These large and complex datasets will challenge the tools we use today. Looking into the future of residential and office building Mitsubishi Electric Research Labs (MERL) has been collecting motion sensor data from a network of over 200 sensors for a year. The data is the residual traces of year in the life of a research laboratory. It contains interesting spatiotemporal structure ranging all the way from the seconds of individuals walking down hallways, the minutes in lobbies chatting with colleagues, the hours of dozens of people attending talks and meetings, the days and weeks that drive the patterns of life, to the months and seasons with their ebb and flow of visiting employees. The dataset contains well over 30 million raw motion records, spanning a calendar year and two floors of our research laboratory. As such it presents a significant challenge for behavior analysis, search, manipulation and visualization of the data. We have also prepared accompanying analytics such as partial tracks and behavior detections, as well as map data and anonymous calendar data marking the pattern of meetings, vacations and holidays. Please see the technical report for more information [1].