Dylan D. Schmorrow

Foundations of Augmented Cognition

This book constitutes the proceedings of the 9th International Conference on the Foundations of Augmented Cognition, AC 2015, held as part of the 17th International Conference on Human-Computer Interaction, HCII 2015, which took place in Los Angeles, CA, USA, in August 2015. HCII 2015 received a total of 4843 submissions, of which 1462 papers and 246 posters were accepted for publication after a careful reviewing process. These papers address the latest research and development efforts and highlight the human aspects of design and use of computing systems. The papers thoroughly cover the entire field of Human-Computer Interaction, addressing major advances in knowledge and effective use of computers in a variety of application areas. The 78 papers presented in the AC 2015 proceedings address the following major topics: cognitive performance and work load, BCI and operational neuroscience, cognition, perception and emotion measurement, adaptive and tutoring training, applications of augmented cognition

Augmented Cognition: An Overview

Augmented cognition is a form of human-systems interaction in which a tight coupling between user and computer is achieved via physiological and neurophysiological sensing of a users cognitive state. This interactive paradigm seeks to revolutionize the manner in which humans engage with computers by leveraging this knowledge of cognitive state to precisely adapt user-system interaction in real time. This review provides an overview of contemporary works in the field of augmented cognition and details regarding the three main components of an augmented cognition system: cognitive state sensors, adaptation strategies, and control systems. The review provides a perspective on the field as well as insights into the many challenges that lie ahead for those who endeavor to realize the full potential of augmented cognition.

Enhancing Mitigation in Augmented Cognition

In augmented cognition (AugCog), mitigation strategies are used as real-time intervention techniques that are triggered by the outcome of cognitive state assessment and context to significantly improve human-systems performance. Yet, no common ground has been established regarding best practices and what aspects to consider during implementation. This paper discusses mitigation strategies currently used in AugCog systems and provides insights into their strengths and weaknesses. An event-based conceptual framework is introduced that aids real-time mitigation strategy selection by linking system events to real-time cognitive state indicators, which together determine when, what, and how to mitigate. Insights from the implementation of this framework in an AugCog system designed to optimize situation awareness are presented, which support the architecture of the framework and identify further challenges to mitigation. Future work should focus on further validating the proposed framework and leveraging techniques from other domains (e.g., film, theater) to create more effective mitigation concepts in AugCog systems.

Augmented cognition and cognitive state assessment technology: near-term, mid-term, and long-term research objectives

The 1st Augmented Cognition International (ACI) conference was held in July 2005 in conjunction with the HCI International conference in Las Vegas, Nevada. A full day working group session was held during this inaugural ACI conference to facilitate the development of an Augmented Cognition R&D agenda for the near- (1-2 years), medium- (within 5 years) and long-term (> 5 years). Working group attendees included scientists, developers, and practitioners from government, academia, and industry who were invited to participate based on their numerous years of experience and expertise in the Augmented Cognition and related fields. This article highlights key results of the workshop discussions that were focused on Cognitive State Assessment (CSA) R&D objectives, particularly with regard to the design and implementation of CSA tools and techniques.

An adaptive instructional architecture for training and education

Office of Naval Research (ONR) initiatives such as Human Performance Training and Education (HPT&E) as well as Virtual Technologies and Environments (VIRTE) have primarily focused on developing the strategies and technologies for creating multimodal reality or simulation based content. Resulting state-of-the-art training and education prototype simulators still rely heavily on instructors to interpret performance data, and adapt instruction via scenario generation, mitigations, feedback and after action review tools. Further research is required to fully close the loop and provide automated, adaptive instruction in these learning environments. To meet this goal, an ONR funded initiative focusing on the Training and Education arm of the HPT&E program will address the processes and components required to deliver these capabilities in the form of an Adaptive Instructional Architecture (AIA). An overview of the AIA as it applies to Marine Corps Warfighter training protocols is given as well as the theoretical foundations supporting it.

Foundations of Augmented Cognition. Directing the Future of Adaptive Systems

This book constitutes the refereed proceedings of the 6th International Conference on Augmented Cognition, FAC 2011, held in Orlando, FL, USA in July 2011, within the framework of the 14th International Conference on Human-Computer Interaction, HCII 2011, with 11 other thematically similar conferences. The 75 full papers presented were carefully reviewed and selected from numerous submissions. The papers are organized in topical parts on theories, models, and technologies for augmented cognition; neuroscience and brain monitoring; augmented cognition, social computing, and collaboration; augmented cognition for learning; augmented cognition and interaction; and augmented cognition in complex environments.

Automated SAF adaptation tool (ASAT)

The purpose of this paper is to describe a new, user-friendly tool that will enable researchers and instructors to setup and run virtual environment scenarios that adapt to the VE users real-time performance and cognitive status. This tool, the Automated SAF Adaptation Tool (ASAT), will work with existing performance and cognitive state assessment software, and with existing semi-automated forces (SAF) behavior engines. ASAT will collect processed performance and cognitive state data from the assessment software and trigger SAF behavior setting manipulations that were pre-selected by the SAF operator. A key feature of ASAT is the ability to setup and execute these real-time manipulations without the need to alter code in either the assessment software or the SAF engines.

Towards a closed-loop training system: using a physiological-based diagnosis of the Trainee's state to drive feedback delivery choices

Designers of a closed loop scenario based training systems must have specifications to drive the decisions of whether or not performance feedback is appropriate in response to student behavior, the most effective content of that feedback, and the optimal time and method of delivery. In this paper, we propose that physiological measures, when interpreted in conjunction with information about the learning objective, task environment and student performance, could provide the data necessary to inform effective, automated decision processes. In addition, we present an overview of both the relevant literature in this area and some ongoing work that is explicitly evaluating these hypotheses.

Foundations of Augmented Cognition. Advancing Human Performance and Decision-Making through Adaptive Systems

Neurophysiological compliance is a correlation of neurophysiological measures (synchronicity) between individuals. Higher compliance among team members is related to better performance, and higher synchronicity occurs during emotional moments of a stimulus. The aim of the current study is to examine whether synchrony may be observable via peripheral nervous system (PNS) activity. We used inter-subject correlation (ISC) analysis to assess whether synchronicity of PNS measures are related to stimulus emotionality or similarity in behavioral responses. Participants viewed a 100-second emotional video, followed by an appeal to donate experimental earnings to a related charity. We found high ISC for cardiac and electrodermal activity (EDA) between donors versus non-donors. For both groups, we found an association between ISC of cardiac activity and emotional moments in the stimulus. For non-donors we found an association between ISC of EDA and emotional moments. Our findings indicate that PNS measures yield similar results to neurophysiological measures.

Exploring Cognitive Readiness in Complex Operational Environments: Advances in Theory and Practice

Cognitive readiness refers to the mental preparation (including knowledge, skills, abilities, and attitudes) an individual needs to establish and sustain competent performance in complex and unpredictable operational environments (Morrison & Fletcher, 2002). Over the past decade, the scientific research community has endeavored to optimize human performance in complex domains through a better understanding of the cognitive, behavioral, and attitudinal aspects of cognitive readiness, at both the individual and team level. The goal of this symposium is to add to this growing body of work by presenting the latest advances in cognitive readiness research in terms of both theory development and practical applications across domains.