Stephen Whitlow

Considering Etiquette in the Design of an Adaptive System

In this article, the authors empirically assess the costs and benefits of designing an adaptive system to follow social conventions regarding the appropriateness of interruptions. Interruption management is one area within the larger topic of automation etiquette. The authors tested these concepts in an outdoor environment using the Communications Scheduler, a wearable adaptive system that classifies users’ cognitive state via brain and heart sensors and adapts its interactions. Designed to help dismounted soldiers, it manages communications in much the same way as a good administrative assistant. Depending on a combination of message priority, user workload, and system state, it decides whether to interrupt the user’s current tasks. The system supports decision makers in two innovative ways: It reliably measures a mobile user’s cognitive workload to adapt its behavior, and it implements rules of etiquette adapted from human-human interactions to improve human-computer interactions. Results indicate costs and benefits to both interrupting and refraining from interrupting. When users were overloaded, primary task performance was improved by managing interruptions. However, overall situation awareness on secondary tasks suffered. This work empirically quantifies costs and benefits of “appropriate” interruption behaviors, demonstrating the value of designing adaptive agents that follow social conventions for interactions with humans.

Rapid image analysis using neural signals

The problem of extracting information from large collections of imagery is a challenge with few good solutions. Computers typically cannot interpret imagery as effectively as humans can, and manual analysis tools are slow. The research reported here explores the feasibility of speeding up manual image analysis by tapping into split second perceptual judgments using electroencephalograph sensors. Experimental results show that a combination of neurophysiological signals and overt physical responses--detected while a user views imagery in high speed bursts of approximately 10 images per second--provide a basis for detecting targets within large image sets. Results show an approximately six-fold, statistically significant, reduction in the time required to detect targets at high accuracy levels compared to conventional broad-area image analysis.

Cognitive State Estimation Based on EEG for Augmented Cognition

Augmented cognition is an emerging concept that aims to enhance user performance and cognitive capabilities on the basis of adaptive assistance. An integral part of such systems is the automatic assessment of the instantaneous cognitive state of the user. This paper describes an automatic cognitive state estimation methodology based on the use of EEG measurements with ambulatory users. The required robustness in this context is achieved through the use of a mutual information based dimensionality reduction approach in conjunction with a committee of classifiers, and median filter outlier rejection element. We present classification results associated with cognitive tasks performed in mobile and stationary modalities

Supporting Real-time Cognitive State Classification on a Mobile Individual

The effectiveness of neurophysiologically triggered adaptive systems hinges on reliable and effective signal processing and cognitive state classification. Although this presents a difficult technical challenge in any context, these concerns are particularly pronounced in a system designed for mobile contexts. This paper describes a neurophysiologically derived cognitive state classification approach designed for ambulatory task contexts. We highlight signal processing and classification components that render the electroencephalogram (EEG) -based cognitive state estimation system robust to noise. Field assessments show classification performance that exceeds 70% for all participants in a context that many have regarded as intractable for cognitive state classification using EEG.

Neurophysiologically driven image triage: a pilot study

Effective analysis of complex imagery is a vital aspect of important domains such as intelligence image analysis. As technological developments lower the cost of gathering and storing imagery, the cost of searching through large image sets for important information has been growing substantially. This paper demonstrates the feasibility of using neurophysiological signals associated with early perceptual processing to identify critical information within large image sets efficiently. Brain signals called evoked response potentials, detected in conjunction with rapid serial presentation of images, show promise as a human computer interaction modality for screening high volumes of imagery accurately and efficiently.

A joint human-automation cognitive system to support rapid decision-making in hostile environments

Honeywell has designed a joint human-computer cognitive system to support rapid decision making demands of dismounted soldiers. In highly networked environments the sheer magnitude of communication amid multiple tasks could overwhelm individual soldiers. Key cognitive bottlenecks constrain information flow and the performance of decision-making, especially under stress. The adaptive decision-support system mitigates non-optimal human performance via automation when the system detects a breakdown in the humans cognitive state. The humans cognitive state is assessed in real-time via a suite of neuro-physiological and physiological sensors. Adaptive mitigation strategies can include task management, optimizing information presentation via modality management, task sharing, and task loading. Mitigations are designed with consideration for both the costs and benefits of intermittent augmentation. The paper describes the system development and evolution, explorations of usable cognitive mitigation strategies, and four evaluations that show adaptive automaton can effectively, mitigate human decision-making performance at extremes (overload and underload) of workload.

Closing the Loop of an Adaptive System with Cognitive State

This paper describes an adaptive system that “closes the loop” by utilizing a real-time, directly sensed measure of cognitive state of the human operator. The Honeywell Augmented Cognition team has developed a Closed Loop Integrated Prototype (CLIP) of a Communications Scheduler, for application to the U.S. Armys Future Force Warrior (FFW) program. It is expected that in a highly networked environment the sheer magnitude of communication traffic could overwhelm the individual soldier. The CLIP exploits real-time neurophysiological and physiological measurements of the human operator in order to create a cognitive state profile, which is used to augment the work environment to improve human-automation joint performance. An experiment showed that the Communications Scheduler enabled higher situation awareness and message comprehension in high workload conditions. Based solely on cognitive state, the system inferred a subjects message comprehension and repeated unattended messages in the majority of cases, without yielding an unacceptably high false alarm rate.

Neuro-Physiologically-Driven Adaptive Automation to Improve Decision Making Under Stress

The advent of netted communications and a wide array of battlefield sensors is enabling real-time information streaming and asset management. However, the burden of information management is placed solely on the receiver of the information. Honeywell Laboratories developed a Communications Scheduler (CoS), an adaptive information management system for the dismounted Soldier, driven by an assessment of the individuals current cognitive capacity to process incoming information, in order to improve decision making under high task load conditions. An evaluation was conducted to demonstrate whether cognitive capacity to perform under differing task loads could be detected using neuro-physiological sensors and then if the adaptive automation would appropriately regulate information flow. Results revealed an improvement in primary task performance, no degradation in concurrent secondary tasks, and lower subjective workload ratings while performing cognitively challenging working memory tasks with the CoS, although a slight loss in situation awareness of lower priority message was found. The appropriate allocation of cognitive resources is key to managing multiple tasks, focusing on the most important ones, and maintaining overall situation awareness.

A superior tool for airline operations

The job of the airline dispatcher can be greatly enhanced with a new tool for making diversion decisions that satisfy safety and schedule needs.

Providing appropriate situation awareness within a mixed-initiative control system

The future of air combat relies on humans controlling large teams of unmanned combat air vehicles (UCAVs) within a dynamic battle environment. Under the DARPA Mixed Initiative Control of Automata (MICA) program, we have been challenged to design a system that empowers a human operator to control teams of up to thirty UCAVs. To address these challenges we are designing an interaction system that defines and provides adequate situation and automation awareness without overloading human operators to the point where their performance degrades gracelessly. The proposed mixed initiative system is situated within a complex and highly dynamic information space that could easily overload the multi-tasking human operators. Dozens of system parameters could be updated thousands of times during a typical mission so it is neither feasible nor prudent for human operators to maintain complete situation and automation awareness. This interaction system will provide appropriately abstracted situation awareness and notification capability that includes: general mission monitoring and automation awareness; task specific information requirements; and user initiated information requests. Our approach involves defining adequate situation awareness as a function of mission phase, human operator role, and abstracted information required to oversee tasked UCAV.