Jennifer Healey

Toward machine emotional intelligence: analysis of affective physiological state

The ability to recognize emotion is one of the hallmarks of emotional intelligence, an aspect of human intelligence that has been argued to be even more important than mathematical and verbal intelligences. This paper proposes that machine intelligence needs to include emotional intelligence and demonstrates results toward this goal: developing a machines ability to recognize the human affective state given four physiological signals. We describe difficult issues unique to obtaining reliable affective data and collect a large set of data from a subject trying to elicit and experience each of eight emotional states, daily, over multiple weeks. This paper presents and compares multiple algorithms for feature-based recognition of emotional state from this data. We analyze four physiological signals that exhibit problematic day-to-day variations: The features of different emotions on the same day tend to cluster more tightly than do the features of the same emotion on different days. To handle the daily variations, we propose new features and algorithms and compare their performance. We find that the technique of seeding a Fisher Projection with the results of sequential floating forward search improves the performance of the Fisher Projection and provides the highest recognition rates reported to date for classification of affect from physiology: 81 percent recognition accuracy on eight classes of emotion, including neutral.

Detecting stress during real-world driving tasks using physiological sensors

This paper presents methods for collecting and analyzing physiological data during real-world driving tasks to determine a drivers relative stress level. Electrocardiogram, electromyogram, skin conductance, and respiration were recorded continuously while drivers followed a set route through open roads in the greater Boston area. Data from 24 drives of at least 50-min duration were collected for analysis. The data were analyzed in two ways. Analysis I used features from 5-min intervals of data during the rest, highway, and city driving conditions to distinguish three levels of driver stress with an accuracy of over 97% across multiple drivers and driving days. Analysis II compared continuous features, calculated at 1-s intervals throughout the entire drive, with a metric of observable stressors created by independent coders from videotapes. The results show that for most drivers studied, skin conductivity and heart rate metrics are most closely correlated with driver stress level. These findings indicate that physiological signals can provide a metric of driver stress in future cars capable of physiological monitoring. Such a metric could be used to help manage noncritical in-vehicle information systems and could also provide a continuous measure of how different road and traffic conditions affect drivers.

Augmented reality through wearable computing

Wearable computing moves computation from the desktop to the user. We are forming a community of networked, wearable-computer users to explore, over a long period, the augmented realities that these systems can provide. By adapting its behavior to the users changing environment, a body-worn computer can assist the user more intelligently, consistently, and continuously than a desktop system. A text-based augmented reality, the Remembrance Agent, is presented to illustrate this approach. Video cameras are used both to warp the visual input (mediated reality) and to sense the users world for graphical overlay. With a camera, the computer could track the users finger to act as the systems mouse; perform face recognition; and detect passive objects to overlay 2.5D and 3D graphics onto the real world. Additional apparatus such as audio systems, infrared beacons for sensing location, and biosensors for learning about the wearers affect are described. With the use of input from these interface devices and sensors, a long-term goal of this project is to model the users actions, anticipate his or her needs, and perform a seamless interaction between the virtual and physical environments.

SmartCar: detecting driver stress

Smart physiological sensors embedded in an automobile afford a novel opportunity to capture naturally occurring episodes of driver stress. In a series of ten ninety minute drives on public roads and highways, ECG, EMG, respiration and skin conductance sensors were used to measure the autonomic nervous system activation. The signals were digitized in real time and stored on the SmartCars Pentium class computer. Each drive followed a pre-specified route through fifteen different events, from which four stress level categories were created according to the results of the subjects self report questionnaires. In total, 545 one minute segments were classified. A linear discriminant function was used to rank each feature individually based on the recognition performance, and a sequential forward floating selection algorithm was used to find an optimal set of features for recognizing patterns of driver stress. Using multiple features improved performance significantly over the best single feature performance.

Digital processing of affective signals

Affective signal processing algorithms were developed to allow a digital computer to recognize the affective state of a user who is intentionally expressing that state. This paper describes the method used for collecting the training data, the feature extraction algorithms used and the results of pattern recognition using a Fisher linear discriminant and the leave one out test method. Four physiological signals, skin conductivity, blood volume pressure, respiration and an electromyogram (EMG) on the masseter muscle were analyzed. It was found that anger was well differentiated from peaceful emotions (90%-100%), that high and low arousal states were distinguished (80%-88%), but positive and negative valence states were difficult to distinguish (50%-82%). Subsets of three emotion states could be well separated (75%-87%) and characteristic patterns for single emotions were found.

Wearable wellness monitoring using ECG and accelerometer data

This paper presents a prototype wearable wellness monitoring system capable of recording, transmitting and analyzing continuous ECG and accelerometer data. The system also provides an application for recording activities, events and potentially important medical symptoms. The hardware allows data to be transmitted wirelessly from on-body sensors to a handheld device using Bluetooth. Data is then transmitted to a back-end server for analysis using either a wireless Internet connection, if available, or a cellular phone service. We conducted experiments using the system for activity monitoring, exercise monitoring and medical screening tests and present preliminary data and results.

BioStream: a system architecture for real-time processing of physiological signals

Continuous monitoring of physiological signals has the potential to greatly improve the quality of life of patients with chronic diseases. Recent advances in sensor technology make the capture of such signals possible. In this paper, we present BioStream, a real-time, and operator-based software solution for managing physiological sensor streams. It is built on top of a general-purpose stream processing software architecture. The system processes data using plug-in analysis components that can be easily composed into plans using a graphical programming environment. The architecture is scalable, allowing implementation on systems ranging from desktops to server farms. It guarantees real-time response and data persistence in a distributed environment. We apply this architecture to the problem of multipatient, real-time, physiological signal monitoring, analysis, indexing and visualization.

Adaptive Content for Device Independent Multi-modal Browser Applications

Adapting content appropriate to the device and modality of a users preference becomes more important as users begin to expect universal access to information, whether they are on the phone, on a desktop or using a PDA. This paper outlines the design of a travel application authored using an XForms compliant language and deployed using a DOM-based MVC multi-modal browser. The travel application authored in a language of conversational gestures that can be transcoded into multiple synchronized views for access via a variety of devices.

GSR Sock: A New e-Textile Sensor Prototype

Galvanic Skin Response (GSR) is a measure of skin conductivity that has been extensively linked to emotional stress and activation. Emotional reactions often cause increased sweat gland activity in the palms of the hands and the soles of the feet, making skin more conductive in these areas. This paper presents a novel e-textile prototype for measuring GSR from the foot using conductive fabric electrodes embedded in a standard sock. The prototype was tested using both a standard psycho-physiological monitoring system and a wearable sensing unit. The results show that the sock prototype provides a meaningful measure of GSR activity that can be used unobtrusively in daily monitoring.