Surjya Ghosh

TapSense: combining self-report patterns and typing characteristics for smartphone based emotion detection

Typing based communication applications on smartphones, like WhatsApp, can induce emotional exchanges. The effects of an emotion in one session of communication can persist across sessions. In this work, we attempt automatic emotion detection by jointly modeling the typing characteristics, and the persistence of emotion. Typing characteristics, like speed, number of mistakes, special characters used, are inferred from typing sessions. Self reports recording emotion states after typing sessions capture persistence of emotion. We use this data to train a personalized machine learning model for multi-state emotion classification. We implemented an Android based smartphone application, called TapSense, that records typing related metadata, and uses a carefully designed Experience Sampling Method (ESM) to collect emotion self reports. We are able to classify four emotion states - happy, sad, stressed, and relaxed, with an average accuracy (AUCROC) of 84% for a group of 22 participants who installed and used TapSense for 3 weeks.

Unsupervised annotated city traffic map generation

Public bus services in many cities in countries like India are controlled by private owners, hence, building up a database for all the bus routes is non-trivial. In this paper, we leverage smart-phone based sensing to crowdsource and populate the information repository for bus routes in a city. We have developed an intelligent data logging module for smart-phones and a server side processing mechanism to extract roads and bus routes information. From a 3 month long study involving more than 30 volunteers in 3 different cities in India, we found that the developed system, CrowdMap, can annotate bus routes with a mean error of 10m, while consuming 80% less energy compared to a continuous GPS based system.

Towards designing an intelligent experience sampling method for emotion detection

Experience Sampling Method (ESM) is widely used in idiographic approaches to collect within-person patterns. Planning a suitable survey schedule while designing an ESM based experiment is challenging as it must balance between survey fatigue of users, and the timeliness and accuracy of the responses provided by users. Even with the proliferation of ESM experiments, survey scheduling typically remains confined to use of fixed schedules, where periodic probes are sent to user, or event-based schedules, where depending on the number of events, large number of probes may interrupt user frequently. We propose a novel survey scheduling scheme, Low-Interference High-fidelity (LIHF) ESM schedule, which is designed to reduce interference while retaining fidelity of user response. We integrated LIHF into an ESM application, called TapSense, that is used to infer users emotion from typing characteristics on smartphone keypad. Conducting a 2-week field study involving 9 users, using proposed metrics we show that using LIHF there is 26% reduction in survey fatigue, 50% improvement in triggering survey probes in timely manner, and 8% improvement in predicting emotion states based on typing patterns compared to typical ESM scheduling techniques.

Impact of experience sampling methods on tap pattern based emotion recognition

Smartphone based emotion recognition uses predictive modeling to recognize users mental states. In predictive modeling, determining ground truth plays a crucial role in labeling and training the model. Experience Sampling Method (ESM) is widely used in behavioral science to gather user responses about mental states. Smartphones equipped with sensors provide new avenues to design Experience Sampling Methods. Sensors provide multiple contexts that can be used to trigger collection of user responses. However, subsampling of sensor data can bias the inference drawn from trigger based ESM. We investigate whether continuous sensor data simplify the design of ESM. We use the typing pattern of users on smartphone as the context that can trigger response collection. We compare the context based and time based ESM designs to determine the impact of ESM strategies on emotion modeling. The results indicate how different ESM designs compare against each other.

Poster: Effectiveness of Deep Neural Network Model in Typing-based Emotion Detection on Smartphones

Typing characteristics on smartphones can provide clues for emotion detection. Collecting large volumes of typing data is also easy on smartphones. This motivates the use of Deep Neural Network (DNN) to determine emotion states from smartphone typing. In this work, we developed a DNN model based on typing features to predict four emotion states (happy, sad, stressed, relaxed) and investigate its performance on a smartphone. The evaluation of the model in a 3-week study with 15 participants reveals that it can reliably detect emotions with an average accuracy of 80% with peak CPU utilization less than 15%.

Emotion-aware computing using smartphone

In this project, we address the problem to determine human emotion states automatically using modern day smartphones. Sensor-rich smartphones have opened up the opportunity to unobtrusively collect user behaviour patterns, activity details and infer information about emotion states. Determining human emotion accurately and efficiently to build a scalable system is the major objective of the project. Towards that goal, we plan to develop an emotion detection model leveraging on different information sources present on smartphone.