Joy Zhang

ACCessory: password inference using accelerometers on smartphones

We show that accelerometer readings are a powerful side channel that can be used to extract entire sequences of entered text on a smart-phone touchscreen keyboard. This possibility is a concern for two main reasons. First, unauthorized access to ones keystrokes is a serious invasion of privacy as consumers increasingly use smartphones for sensitive transactions. Second, unlike many other sensors found on smartphones, the accelerometer does not require special privileges to access on current smartphone OSes. We show that accelerometer measurements can be used to extract 6-character passwords in as few as 4.5 trials (median).

ACComplice: Location inference using accelerometers on smartphones

The security and privacy risks posed by smartphone sensors such as microphones and cameras have been well documented. However, the importance of accelerometers have been largely ignored. We show that accelerometer readings can be used to infer the trajectory and starting point of an individual who is driving. This raises concerns for two main reasons. First, unauthorized access to an individuals location is a serious invasion of privacy and security. Second, current smartphone operating systems allow any application to observe accelerometer readings without requiring special privileges. We demonstrate that accelerometers can be used to locate a device owner to within a 200 meter radius of the true location. Our results are comparable to the typical accuracy for handheld global positioning systems.

SenSec: Mobile security through passive sensing

We introduce a new mobile system framework, SenSec, which uses passive sensory data to ensure the security of applications and data on mobile devices. SenSec constantly collects sensory data from accelerometers, gyroscopes and magnetometers and constructs the gesture model of how a user uses the device. SenSec calculates the sureness that the mobile device is being used by its owner. Based on the sureness score, mobile devices can dynamically request the user to provide active authentication (such as a strong password), or disable certain features of the mobile devices to protect users privacy and information security. In this paper, we model such gesture patterns through a continuous n-gram language model using a set of features constructed from these sensors. We built mobile application prototype based on this model and use it to perform both user classification and user authentication experiments. User studies show that SenSec can achieve 75% accuracy in identifying the users and 71.3% accuracy in detecting the non-owners with only 13.1% false alarms.

KeySens: Passive User Authentication through Micro-behavior Modeling of Soft Keyboard Interaction

Mobile devices have become almost ever-present in our daily lives and increasingly so in the professional workplace. Applications put company data, personal information and sensitive documents in the hands of busy nurses at hospitals, company employees on business trips and government workers at large conferences. Smartphones and tablets also not only store data on-device, but users are frequently authorized to access sensitive information in the cloud. Protecting the sensitivity of mobile devices yet not burdening users with complicated and cumbersome active authentication methods is of great importance to the security and convenience of mobile computing. In this paper, we propose a novel passive authentication method; we model the micro-behavior of mobile users’ interaction with their devices’ soft keyboard. We show that the way a user types—the specific location touched on each key, the drift from finger down to finger up, the force of touch, the area of press—reflects their unique physical and behavioral characteristics. We demonstrate that using these micro-behavior features without any contextual information, we can passively identify that a mobile device is being used by a non-authorized user within 5 keypresses 67.7% of the time. This comes with a False Acceptance Rate (FAR) of 32.3% and a False Rejection Rate (FRR) of only 4.6%. Our detection rate after 15 keypresses is 86% with a FAR of 14% and a FRR of only 2.2%.

MobiSens: A Versatile Mobile Sensing Platform for Real-World Applications

We present the design, implementation and evaluation of MobiSens, a versatile mobile sensing platform for a variety of real-life mobile sensing applications. MobiSens addresses common requirements of mobile sensing applications on power optimization, activity segmentation, recognition and annotation, interaction between mobile client and server, motivating users to provide activity labels with convenience and privacy concerns. After releasing three versions of MobiSens to the Android Market with evolving UI and increased functionalities, we have collected 13,993 h of data from 310 users over five months. We evaluate and compare the user experience and the sensing efficiency in each release. We show that the average number of activities annotated by a user increases from 0.6 to 6. This result indicates the activity auto-segmentation/recognition feature and certain UI design changes significantly improve the user experience and motivate users to use MobiSens more actively. Based on the MobiSens platform, we have developed a range of mobile sensing applications including Mobile Lifelogger, SensCare for assisted living, Ground Reporting for soldiers to share their positions and actions horizontally and vertically, and CMU SenSec, a behavior-driven mobile Security system.

Recognizing new activities with limited training data

Activity recognition (AR) systems are typically built to recognize a predefined set of common activities. However, these systems need to be able to learn new activities to adapt to a users needs. Learning new activities is especially challenging in practical scenarios when a user provides only a few annotations for training an AR model. In this work, we study the problem of recognizing new activities with a limited amount of labeled training data. Due to the shortage of labeled data, small variations of the new activity will not be detected resulting in a significant degradation of the systems recall. We propose the FE-AT (Feature-based and Attribute-based learning) approach, which leverages the relationship between existing and new activities to compensate for the shortage of the labeled data. We evaluate FE-AT on three public datasets and demonstrate that it outperforms traditional AR approaches in recognizing new activities, especially when only a few training instances are available.

Mobile Lifelogger – Recording, Indexing, and Understanding a Mobile User’s Life

Lifelog system involves capturing personal experiences in the form of digital multimedia during an entire lifespan. Recent advancements in mobile sensor technologies have helped to develop these systems using commercial smart phones. These systems have the potential to act as a secondary memory and also aid people who struggle with episodic memory impairment (EMI). Despite their huge potential, there are major challenges that need to be addressed to make them useful. One of them is how to index the inherently large lifelog data so that the person can efficiently retrieve the log segments that interest him / her most. In this paper, we present an ongoing research of using mobile phones to record and index lifelogs using activity language. By converting sensory data such as accelerometer and GPS readings into activity language, we are able to apply statistical natural language processing techniques to index, recognize, segment, cluster, retrieve, and infer high-level semantic meanings of the collected lifelogs. Based on this indexing approach, our lifelog system supports easy retrieval of log segments representing past similar activities and automatic lifelog segmentation for efficient browsing and activity summarization.

Wi-Fi fingerprinting through active learning using smartphones

Indoor positioning is one of the key components enabling retail-related services such as location-based product recommendations or in-store navigation. In the recent years, active research has shown that indoor positioning systems based on Wi-Fi fingerprints can achieve a high positioning accuracy. However, the main barrier of broad adoption is the labor-intensive process of collecting labeled fingerprints. In this work, we propose an approach for reducing the amount of labeled data instances required for training a Wi-Fi fingerprint model. The reduction of the labeling effort is achieved by leveraging dead reckoning and an active learning-based approach for selecting data instances for labeling. We demonstrate through experiments that we can construct a Wi-Fi fingerprint database with significantly less labels while achieving a high positioning accuracy.

Helix: Unsupervised Grammar Induction for Structured Activity Recognition

The omnipresence of mobile sensors has brought tremendous opportunities to ubiquitous computing systems. In many natural settings, however, their broader applications are hindered by three main challenges: rarity of labels, uncertainty of activity granularities, and the difficulty of multi-dimensional sensor fusion. In this paper, we propose building a grammar to address all these challenges using a language-based approach. The proposed algorithm, called Helix, first generates an initial vocabulary using unlabeled sensor readings, followed by iteratively combining statistically collocated sub-activities across sensor dimensions and grouping similar activities together to discover higher level activities. The experiments using a 20-minute ping-pong game demonstrate favorable results compared to a Hierarchical Hidden Markov Model (HHMM) baseline. Closer investigations to the learned grammar also shows that the learned grammar captures the natural structure of the underlying activities.

SuperAD: supervised activity discovery

Activity recognition (AR) has become an essential component of many applications present in our everyday lives such as life-logging, fitness tracking, health and wellbeing monitoring. To build an AR system, one needs to first identify a set of activities of interest and collect labeled training data for these activities. However, activities of interest are not often known in advance. For example, a system designed to monitor a users life style for potential diabetes risk needs to recognize all physical activities a user performs in her daily life. Given the large number of possible human activities, many of them cannot be foreseen during the model training time. In this work, we study the problem of discovering these unknown activities after the system is deployed by asking users to provide additional labels. Our goal is to discover all the unknown activities (i.e., obtain at least one label per class) while minimizing the amount of labels a user needs to provide. We propose SuperAD (Supervised Activity Discovery) approach, which combines active learning, semi-supervised learning and generative modeling to discover new unknown activities. We show that the proposed approach is especially effective when discovering activities with imbalance class distribution.