Sougata Sen

The case for smartwatch-based diet monitoring

We explore the use of gesture recognition on a wrist-worn smartwatch as an enabler of an automated eating activity (and diet monitoring) system. We show, using small-scale user studies, how it is possible to use the accelerometer and gyroscope data from a smartwatch to accurately separate eating episodes from similar non-eating activities, and to additionally identify the mode of eating (i.e., using a spoon, bare hands or chopsticks). Additionally, we investigate the likelihood of automatically triggering the smartwatchs camera to capture clear images of the food being consumed, for possible offline analysis to identify what (and how much) the user is eating. Our results show both the promise and challenges of this vision: while opportune moments for capturing such useful images almost always exist in an eating episode, significant further work is needed to both (a) correctly identify the appropriate instant when the camera should be triggered and (b) reliably identify the type of food via automated analyses of such images.

Accommodating user diversity for in-store shopping behavior recognition

This paper explores the possibility of using mobile sensing data to detect certain in-store shopping intentions or behaviours of shoppers. We propose a person-independent activity recognition technique called CROSDAC, which captures the diversity in the manifestation of such intentions or behaviours in a heterogeneous set of users in a data-driven manner via a 2-stage clustering-cum-classification technique. Using smartphone based sensor data (accelerometer, compass and Wi-Fi) from a directed, but real-life study involving 86 shopping episodes from 30 users in a malls food court, we show that CROSDACs mobile sensing-based approach can offer reasonably high accuracy (77:6% for a 2-class identification problem) and outperforms the traditional community-driven approaches that unquestioningly segment users on the basis of underlying demographic or lifestyle attributes.

The case for cloud-enabled mobile sensing services

We make the case for cloud-enabled mobile sensing services that support an emerging application class, one which infers near-real time collective context using sensor data obtained continuously from a large set of consumer mobile devices. We present the high-level architecture and functional requirements for such a mobile sensing service, and argue that such a service can significantly improve the scalability and energy-efficiency of large-scale mobile sensing by coordinating the sensing & processing tasks across multiple devices. We then focus specifically on the problem of energy-efficiency and provide early exemplars of how optimizing query execution jointly over multiple phones can lead to substantial energy savings.

Did you take a break today? Detecting playing foosball using your smartwatch

Prolonged working hours are a primary cause of stress, work related injuries (e.g, RSIs), and work-life imbalance in employees at a workplace. As reported by some studies, taking timely breaks from continuous work not only reduces stress and exhaustion but also improves productivity, employee bonding, and camaraderie. Our goal is to build a system that automatically detects breaks thereby assisting in maintaining healthy work-break balance. In this paper, we focus on detecting foosball breaks of employees at a workplace using a smartwatch. We selected foosball as it is one of the most commonly played games at many workplaces in the United States. Since playing foosball involves wrist and hand movement, a wrist-worn device (e.g., a smartwatch), due to its position, has a clear advantage over a smartphone for detecting foosball activity. Our evaluation using data collected from real workplace shows that we can identify with more than 95% accuracy whether a person is playing foosball or not. We also show that we can determine how long a foosball session lasted with an error of less than 3% in the best case.

IoT+Small Data: Transforming in-store shopping analytics & services

We espouse a vision of small data-based immersive retail analytics, where a combination of sensor data, from personal wearable-devices and store-deployed sensors & IoT devices, is used to create real-time, individualized services for in-store shoppers. Key challenges include (a) appropriate joint mining of sensor & wearable data to capture a shoppers product-level interactions, and (b) judicious triggering of power-hungry wearable sensors (e.g., camera) to capture only relevant portions of a shoppers in-store activities. To explore the feasibility of our vision, we conducted experiments with 5 smartwatch-wearing users who interacted with objects placed on cupboard racks in our lab (to crudely mimic corresponding grocery store interactions). Initial results show significant promise: 94% accuracy in identifying an item-picking gesture, 85% accuracy in identifying the shelf-location from where the item was picked and 61% accuracy in identifying the exact item picked (via analysis of the smartwatch camera data).

Cloud-Based Query Evaluation for Energy-Efficient Mobile Sensing

In this paper, we reduce the energy overheads of continuous mobile sensing for context-aware applications that are interested in collective context or events. We propose a cloud-based query management and optimization framework, called CloQue, which can support concurrent queries, executing over thousands of individual smartphones. CloQue exploits correlation across context of different users to reduce energy overheads via two key innovations: i) Dynamically reordering the order of predicate processing to preferentially select predicates with not just lower sensing cost and higher selectivity, but that maximally reduce the uncertainty about other context predicates, and ii) intelligently propagating the query evaluation results to dynamically update the uncertainty of other correlated, but yet-to-be evaluated, context predicates. An evaluation, using real cell phone traces from a real world dataset shows significant energy savings (between 30 to 50% compared with traditional short-circuit systems) with little loss in accuracy (5% at most).

Experiences in Building a Real-World Eating Recogniser

In this paper, we describe the progressive design of the gesture recognition module of an automated food journaling system -- Annapurna. Annapurna runs on a smartwatch and utilises data from the inertial sensors to first identify eating gestures, and then captures food images which are presented to the user in the form of a food journal. We detail the lessons we learnt from multiple in-the-wild studies, and show how eating recognizer is refined to tackle challenges such as (i) high gestural diversity, and (ii) non-eating activities with similar gestural signatures. Annapurna is finally robust (identifying eating across a wide diversity in food content, eating styles and environments) and accurate (false-positive and false-negative rates of 6.5% and 3.3% respectively)

Inferring smartphone keypress via smartwatch inertial sensing

Due to numerous benefits, sensor-rich smartwatches and wrist-worn wearable devices are quickly gaining popularity. The popularity of these devices also raises privacy concerns. In this paper we explore one such privacy concern: the possibility of extracting the location of a users touch-event on a smartphone, using the inertial sensor data of a smartwatch worn by the user on the same arm. This is a major concern not only because it might be possible for an attacker to extract private and sensitive information from the inputs provided but also because the attack mode utilises a device (smartwatch) that is distinct from the device being attacked (smartphone). Through a user study we find that such attacks are possible. Specifically, we can infer the users entry pattern on a qwerty keyboard, with an error bound of ±2 neighboring keys, with 73.85% accuracy. As a possible preventive mechanism, we also show that adding a little white noise to inertial sensor data can reduce the inference accuracy by almost 30%, without affecting the accuracy of macro-gesture recognition.

Pervasive physical analytics using multi-modal sensing

With the gradual increase in the number of individually owned mobile and wearable devices, as well as increase in the number of publicly available sensing devices, automatic & unobtrusive monitoring of Activities of Daily Living (ADLs) is gradually becoming possible. In this work, we discuss about the important trade-off between energy, accuracy and non-personalization that has to be considered while building commercially successful ADL monitoring systems. We then describe two ADL monitoring systems that we have built which addresses technical challenges pertaining to building ADL monitoring systems. We also outline our proposed next steps in this research.

Opportunities and challenges in multi-modal sensing for regular lifestyle tracking

With the availability of various publicly available and personal sensors, recording and profiling of activities of daily living (ADL) is becoming a reality. The sensors are omnipresent - in smartphones, smartwatches, and smartglasses and even in the environment around us in the form of peer smartphones or even infrastructure sensors such as bluetooth low energy beacons. However, there are various challenges pertaining to the sensor data processing, which makes creation of activities of daily life tracker challenging. In this work, we discuss about some of these challenges. We also discuss about some ADL tracking systems that we have developed and how we have addressed some of the challenges in building these systems. We further discuss about how various ADL trackers can be combined into a framework which can allow individuals to select a custom set of ADLs for self-tracking.