Wenjie Ruan

AudioGest: enabling fine-grained hand gesture detection by decoding echo signal

Hand gesture is becoming an increasingly popular means of interacting with consumer electronic devices, such as mobile phones, tablets and laptops. In this paper, we present AudioGest, a device-free gesture recognition system that can accurately sense the hand in-air movement around users devices. Compared to the state-of-the-art, AudioGest is superior in using only one pair of built-in speaker and microphone, without any extra hardware or infrastructure support and with no training, to achieve fine-grained hand detection. Our system is able to accurately recognize various hand gestures, estimate the hand in-air time, as well as average moving speed and waving range. We achieve this by transforming the device into an active sonar system that transmits inaudible audio signal and decodes the echoes of hand at its microphone. We address various challenges including cleaning the noisy reflected sound signal, interpreting the echo spectrogram into hand gestures, decoding the Doppler frequency shifts into the hand waving speed and range, as well as being robust to the environmental motion and signal drifting. We implement the proof-of-concept prototype in three different electronic devices and extensively evaluate the system in four real-world scenarios using 3,900 hand gestures that collected by five users for more than two weeks. Our results show that AudioGest can detect six hand gestures with an accuracy up to 96%, and by distinguishing the gesture attributions, it can provide up to 162 control commands for various applications.

Service Recommendation for Mashup Composition with Implicit Correlation Regularization

In this paper, we explore service recommendation and selection in the reusable composition context. The goal is to aid developers finding the most appropriate services in their composition tasks. We specifically focus on mashups, a domain that increasingly targets people without sophisticated programming knowledge. We propose a probabilistic matrix factorization approach with implicit correlation regularization to solve this problem. In particular, we advocate that the co-invocation of services in mashups is driven by both explicit textual similarity and implicit correlation of services, and therefore develop a latent variable model to uncover the latent connections between services by analyzing their co-invocation patterns. We crawled a real dataset from Programmable Web, and extensively evaluated the effectiveness of our proposed approach.

Exploring Tag-Free RFID-Based Passive Localization and Tracking via Learning-Based Probabilistic Approaches

RFID-based localization and tracking has some promising potentials. By combining localization with its identification capability, existing applications can be enhanced and new applications can be developed. In this paper, we investigate a tag-free indoor localizing and tracking problem (e.g., people tracking) without requiring subjects to carry any tags or devices in a pure passive environment. We formulate localization as a classification task. In particular, we model the received signal strength indicator (RSSI) of passive tags using multivariate Gaussian Mixture Model (GMM), and use the Expectation Maximization (EM) to learn the maximum likelihood estimates of the model parameters. Several other learning-based probabilistic approaches are also explored in the localization problem. To track a moving subject, we propose GMM based Hidden Markov Model (HMM) and k Nearest Neighbor (kNN) based HMM approaches. We conduct extensive experiments in a testbed formed by passive RFID tags, and the experimental results demonstrate the effectiveness and accuracy of our approach.

RF-Care: Device-Free Posture Recognition for Elderly People Using A Passive RFID Tag Array

Activity recognition is a fundamental research topic for a wide range of important applications such as fall detection for elderly people. Existing techniques mainly rely on wearable sensors, which may not be reliable and practical in real-world situations since people often forget to wear these sensors. For this reason, device-free activity recognition has gained the popularity in recent years. In this paper, we propose an RFID (radio frequency identification) based, device-free posture recognition system. More specifically, we analyze Received Signal Strength Indicator (RSSI) signal patterns from an RFID tag array, and systematically examine the impact of tag configuration on system performance. On top of selected optimal subset of tags, we study the challenges on posture recognition. Apart from exploring posture classification, we specially propose to infer posture transitions via Dirichlet Process Gaussian Mixture Model (DPGMM) based Hidden Markov Model (HMM), which effectively captures the nature of uncertainty caused by signal strength varieties during posture transitions. We run a pilot study to evaluate our system with 12 orientation-sensitive postures and a series of posture change sequences. We conduct extensive experiments in both lab and real-life home environments. The results demonstrate that our system achieves high accuracy in both environments, which holds the potential to support assisted living of elderly people.

Unobtrusive human localization and activity recognition for supporting independent living of the elderly

Indoor localization and activity recognition is a fundamental research topic for a wide range of important applications such as fall detection of elderly people. It usually requires an intelligent environment to successfully infer where and what a person is doing. However, many of the existing techniques on localization and activity recognition rely heavily on peoples involvement such as wearing battery-powered sensors, which might not be practical in real-world situations (e.g., people may forget to wear sensors). In this project, we propose a device-free localization and activity recognition approach using passive RFID tags. It is achieved by learning how the Received Signal Strength Indicator (RSSI) from the passive RFID tag array is distributed when a person performs different activities in different locations. After activity patterns are discovered for a particular individual, we will also develop a context-aware, common-sense based activity reasoning engine that assists applications to make appropriate interpretation of detected activities. We believe the proposed system has the potential to better support the independent living of elderly people considering the continuously increased aging population.

Freedom: Online Activity Recognition via Dictionary-Based Sparse Representation of RFID Sensing Data

Understanding and recognizing the activities performed by people is a fundamental research topic for a wide range of important applications such as fall detection of elderly people. In this paper, we present the technical details behind Freedom, a low-cost, unobtrusive system that supports independent livingof the older people. The Freedom system interprets what aperson is doing by leveraging machine learning algorithmsand radio-frequency identification (RFID) technology. To dealwith noisy, streaming, unstable RFID signals, we particularlydevelop a dictionary-based approach that can learn dictionariesfor activities using an unsupervised sparse coding algorithm. Our approach achieves efficient and robust activity recognitionvia a more compact representation of the activities. Extensiveexperiments conducted in a real-life residential environmentdemonstrate that our proposed system offers a good overallperformance (e.g., achieving over 96% accuracy in recognizing23 activities) and has the potential to be further developed tosupport the independent living of elderly people.

Device-free indoor localization and tracking through Human-Object Interactions

Device-free indoor localization aims to localize people without requiring them to carry any devices or being actively involved in the localizing process. It underpins a wide range of applications including older people surveillance, intruder detection and indoor navigation. However, in a cluttered environment such as a residential home, the Received Signal Strength Indicator (RSSI) is heavily obstructed by furniture or metallic appliances, thus reducing the localization accuracy. This environment is important to observe as human-object interaction (HOI) events, detected by pervasive sensors, can potentially reveal peoples interleaved locations during daily living activities, such as watching TV, opening the fridge door. This paper aims to enhance the performance of commercial off-the-shelf (COTS) RFID-based localization system by leveraging HOI contexts in a furnished home. Specifically, we propose a general Bayesian probabilistic framework to integrate both RSSI signals and HOI events to infer the most likely location and trajectory. Experiments conducted in a residential house demonstrate the effectiveness of our proposed method, in which we can localize a resident with average 95% accuracy and track a moving subject with 0.58m mean error distance.

Compressive Representation for Device-Free Activity Recognition with Passive RFID Signal Strength

Understanding and recognizing human activities is a fundamental research topic for a wide range of important applications such as fall detection and remote health monitoring and intervention. Despite active research in human activity recognition over the past years, existing approaches based on computer vision or wearable sensor technologies present several significant issues such as privacy (e.g., using video camera to monitor the elderly at home) and practicality (e.g., not possible for an older person with dementia to remember wearing devices). In this paper, we present a low-cost, unobtrusive, and robust system that supports independent living of older people. The system interprets what a person is doing by deciphering signal fluctuations using radio-frequency identification (RFID) technology and machine learning algorithms. To deal with noisy, streaming, and unstable RFID signals, we develop a compressive sensing, dictionary-based approach that can learn a set of compact and informative dictionaries of activities using an unsupervised subspace decomposition. In particular, we devise a number of approaches to explore the properties of sparse coefficients of the learned dictionaries for fully utilizing the embodied discriminative information on the activity recognition task. Our approach achieves efficient and robust activity recognition via a more compact and robust representation of activities. Extensive experiments conducted in a real-life residential environment demonstrate that our proposed system offers a good overall performance and shows the promising practical potential to underpin the applications for the independent living of the elderly.

Unobtrusive Posture Recognition via Online Learning of Multi-dimensional RFID Received Signal Strength

Activity recognition is a core component of ubiquitous computing applications (e.g., fall detection of elder people) since many of such applications require an intelligent environment to infer what a person is doing or attempting to do. Unfortunately, the success of existing approaches on activity recognition relies heavily on peoples involvement such as wearing battery-powered sensors, which might not be practical in real-world situations (e.g., people may forget to wear sensors). In this paper, we propose a device-free, real-time posture recognition technique using an array of pure passive RFID tags. In particular, posture recognition is treated as a machine learning problem where a series of probabilistic model is built via learning how the Received Signal Strength Indicator (RSSI) from the tag array is distributed when a person performs different postures. We also design a segmentation algorithm to divide the continuous, multidimensional RSSI data stream into a set of individual segments by analyzing the shape of the RSSI data. Our approach for posture recognition eliminates the need for the monitored subjects to wear any devices. To the best of our knowledge, this work is the first on device-free posture recognition using low cost, unobtrusive RFID technology. Our experimental studies demonstrate the feasibility of the proposed approach for posture recognition.

Reachability Analysis of Deep Neural Networks with Provable Guarantees

Verifying correctness of deep neural networks (DNNs) is challenging. We study a generic reachability problem for feed-forward DNNs which, for a given set of inputs to the network and a Lipschitz-continuous function over its outputs, computes the lower and upper bound on the function values. Because the network and the function are Lipschitz continuous, all values in the interval between the lower and upper bound are reachable. We show how to obtain the safety verification problem, the output range analysis problem and a robustness measure by instantiating the reachability problem. We present a novel algorithm based on adaptive nested optimisation to solve the reachability problem. The technique has been implemented and evaluated on a range of DNNs, demonstrating its efficiency, scalability and ability to handle a broader class of networks than state-of-the-art verification approaches.