Sven Fuchs

Augmented Cognition: An Overview

Augmented cognition is a form of human-systems interaction in which a tight coupling between user and computer is achieved via physiological and neurophysiological sensing of a users cognitive state. This interactive paradigm seeks to revolutionize the manner in which humans engage with computers by leveraging this knowledge of cognitive state to precisely adapt user-system interaction in real time. This review provides an overview of contemporary works in the field of augmented cognition and details regarding the three main components of an augmented cognition system: cognitive state sensors, adaptation strategies, and control systems. The review provides a perspective on the field as well as insights into the many challenges that lie ahead for those who endeavor to realize the full potential of augmented cognition.

Enhancing Mitigation in Augmented Cognition

In augmented cognition (AugCog), mitigation strategies are used as real-time intervention techniques that are triggered by the outcome of cognitive state assessment and context to significantly improve human-systems performance. Yet, no common ground has been established regarding best practices and what aspects to consider during implementation. This paper discusses mitigation strategies currently used in AugCog systems and provides insights into their strengths and weaknesses. An event-based conceptual framework is introduced that aids real-time mitigation strategy selection by linking system events to real-time cognitive state indicators, which together determine when, what, and how to mitigate. Insights from the implementation of this framework in an AugCog system designed to optimize situation awareness are presented, which support the architecture of the framework and identify further challenges to mitigation. Future work should focus on further validating the proposed framework and leveraging techniques from other domains (e.g., film, theater) to create more effective mitigation concepts in AugCog systems.

Construction and validation of a neurophysio-technological framework for imagery analysis

Intelligence analysts are bombarded with enormous volumes of imagery, which they must visually filter through to identify relevant areas of interest. Interpretation of such data is subject to error due to (1) large data volumes, implying the need for faster and more effective processing, and (2) misinterpretation, implying the need for enhanced analyst/system effectiveness. This paper outlines the Revolutionary Accelerated Processing Image Detection (RAPID) System, designed to significantly improve data throughput and interpretation by incorporating advancing neurophysiological technology to monitor processes associated with detection and identification of relevant target stimuli in a non-invasive and temporally precise manner. Specifically, this work includes the development of innovative electroencephalographic (EEG) and eye tracking technologies to detect and flag areas of interest, potentially without an analysts conscious intervention or motor responses, while detecting and mitigating problems with tacit knowledge, such as anchoring bias in real-time to reduce the possibility of human error.

Towards a Dynamic Selection and Configuration of Adaptation Strategies in Augmented Cognition

Most Augmented Cognition systems use physiological measures to detect critical cognitive states and trigger adaptation strategies to address the problem state and restore or augment performance. Without accounting for context, however, it is likely that adaptations are triggered or withdrawn at inopportune moments, potentially disrupting or confusing the user. We have developed an approach to dynamic adaptation management that processes task and operator state indicators to dynamically select and configure context-sensitive adaptation strategies in real time. This dynamic approach is expected to avoid much of the potential cognitive cost associated with adaptations. We provide an overview of our conceptual approach, describe a proof-of-concept implementation, and summarize user feedback and initial lessons-learned from a small survey.

A Scale for Assessing Human Factors Readiness Levels

This paper describes a proposed scale of Human Factors Readiness Levels (HFRL) that provides a method for standardizing Human Factors (HF) readiness assessment. This scale can be used by HF decision makers in the acquisition, project management, or implementation phases in conjunction with Technological Readiness Levels, and includes HF-specific level descriptions and evaluation requirements. To determine HF readiness, information about the risks, processes, and quality of conducted R&D with respect to 24 HF study areas must be gathered. Gathered information is then used to determine individual HFRL scores for each area, and an overall HFRL for the evaluated system. Using HFRLs, researchers or decision makers can identify several categories of research issues. Specifically, an HFRL analysis can help them assess whether HF R&D resources are optimally allocated, whether HF area interdependencies are considered, whether gaps in the HF R&D process exist, or whether there are problems with HF R&D quality. The use of such a process will enable the standardization of HF R&D metrics across participating organizations to ensure quality of research, and facilitate sharing HF R&D efforts and outcomes across agencies.

Towards a more holistic view on user state assessment in adaptive human-computer interaction

User state assessment in adaptive human-computer-interaction has often been equaled to the assessment of workload. More recently, approaches have surfaced that also focus on other state dimensions such as fatigue, situation awareness, and the emotional state. However, interrelations and dependencies between these state dimensions are not considered if each is assessed in isolation. Furthermore, it is often neglected that individual factors, factors of the work environment, and self-regulation-strategies of the operator strongly influence these state dimensions. We suppose that adaptation based on user state assessment is more successful if this set of interactions is integrated into a more holistic assessment approach. We therefore propose a model that aims to provide a more holistic view on user state assessment. The model also aims to be generic in the sense that it can be applied to diverse operational conditions. Application is illustrated by a recent aviation incident.

Augmented Cognition can increase human performance in the control room

Many approaches have attempted to address a truly symbiotic relationship between human and machine but, thus far, a critical shortcoming has been the computer’s inability to account for human information processing (HIP) limitations. The field of Augmented Cognition (AugCog) capitalizes on recent advances in the areas of neuroscience, cognitive science and human-computer interaction to create closed-loop systems that can measure HIP and account for problems in real-time. The closed-loop architecture is achieved by employing neuro- physiological sensors that monitor operators’ cognitive activity and respond to indicators of non-optimal information processing. Upon indication of a problem, mitigation strategies are employed in real-time to counteract the problem. Examples of HIP parameters investigated by existing AugCog systems include sensory bottlenecks, cognitive workload, alertness, arousal, and situation awareness. Obtained benefits with regard to information throughput, error reduction and operator performance have consistently reached orders of magnitude. The cognitive challenges of power plant control room operators are similar to those encountered in the originally investigated military settings. Thus, by adopting an AugCog closed-loop approach, similar benefits may be realized within the control room domain. After an introduction of the general concept, this paper outlines two distinct approaches to AugCog systems and their respective applicability to the specific needs of the control room environment. It is illustrated how a closed-loop AugCog system could substantially enhance operator performance in next- generation power plant control rooms.

Using Physiological Measures to Discriminate Signal Detection Outcome during Imagery Analysis

An experiment was conducted to explore the feasibility of using physiological indicators (i.e. eye-tracking and electroencephalography [EEG]) to drive identification of relevant areas of interest during imagery analysis. Results indicate that ocular fixations are longer when a target is believed to be present. Furthermore, the accuracy of correct identification of targets could be identified based on fixation duration, given that fixations were significantly longer when a target was actually present. In addition, by synching eye-tracking fixation points to EEG, fixation-locked event-related potentials (FLERPs) show potential for detecting distinctive patterns and scalp distributions for various types of fixations, which may be used to classify fixation points based on level of interest. This paper reports findings from a study and summarizes challenges and implications for constructing a system where eye tracking is used to drive EEG ERP evaluation of interest during a defined search task within complex static images.

Multidimensional Real-Time Assessment of User State and Performance to Trigger Dynamic System Adaptation

In adaptive human-machine interaction, technical systems adapt their behavior to the current state of the human operator to mitigate critical user states and performance decrements. While many researchers use measures of workload as triggers for adjusting levels of automation, we have proposed a more holistic approach to adaptive system design that includes a multidimensional assessment of user state. This paper outlines the design requirements, conceptual framework, and proof-of-concept implementation of a Real-time Assessment of Multidimensional User State (RASMUS). RASMUS diagnostics provide information on user performance, potentially critical user states, and their related impact factors on a second-by-second-basis in real-time. Based on these diagnoses adaptive systems are enabled to infer when the user needs support and to dynamically select and apply an appropriate adaptation strategy for a given situation. While the conceptual framework is generic, the implementation has been applied to an air surveillance task, providing real-time diagnoses for high workload, passive task-related fatigue and incorrect attentional focus.

Results from Pilot Testing a System for Tactile Reception of Advanced Patterns (STRAP)

This paper presents pilot study results on the learnability and effectiveness of the System for Tactile Reception of Advanced Patterns (STRAP) that is capable of displaying complex information through tactile actuators on the users torso. Information requirements from dismounted soldier communications and tactile design guidelines resulted in 56 distinct tactile symbols. To facilitate cognitive demands for decoding, information presentation was formalized by developing construction rules for tactile symbols and a context-free grammar for compilation of tactile sentences. The pilot study outlined trained two participants on the tactile language. Results showed they were able to reach 90% criterion in less than 3.5 hours. Furthermore, once learned, participants were able to receive and comprehend complex commands comprised of multiple tactile symbols under varying levels of workload with some success.