J. Chris Forsythe

Enhanced Training for Cyber Situational Awareness

A study was conducted in which participants received either tool-based or narrative-based training and then completed challenges associated with network security threats. Three teams were formed: (1) Tool-Based, for which each participant received tool-based training; (2) Narrative-Based, for which each participant received narrative-based training and (3) Combined, for which three participants received tool-based training and two received narrative-based training. Results showed that the Narrative-Based team recognized the spatial-temporal relationship between events and constructed a timeline that was a reasonable approximation of ground truth. In contrast, the Combined team produced a linear sequence of events that did not encompass the relationships between different adversaries. Finally, the Tool-Based team demonstrated little appreciation of either the spatial or temporal relationships between events. These findings suggest that participants receiving Narrative-Based training were able to use the software tools in a way that allowed them to gain a greater level of situation awareness.

Human Dimension in Cyber Operations Research and Development Priorities

Within cyber security, the human element represents one of the greatest untapped opportunities for increasing the effectiveness of network defenses. However, there has been little research to understand the human dimension in cyber operations. To better understand the needs and priorities for research and development to address these issues, a workshop was conducted August 28-29, 2012 in Washington DC. A synthesis was developed that captured the key issues and associated research questions.

Experimental Assessment of Accuracy of Automated Knowledge Capture

The U.S. armed services are widely adopting simulation-based training, largely to reduce costs associated with live training. However simulation-based training still requires a high instructor-to-student ratio which is expensive. Intelligent tutoring systems target this need, but they are often associated with high costs for knowledge engineering and implementation. To reduce these costs, we are investigating the use of machine learning to produce models of expert behavior for automated student assessment. A key concern about the expert modeling approach is whether it can provide accurate assessments on complex tasks of real-world interest. This study evaluates of the accuracy of model-based assessments on a complex task. We trained employees at Sandia National Laboratories on a Navy simulator and then compared their simulation performance to the performance of experts using both automated and manual assessment. Results show that automated assessments were comparable to the manual assessments on three metrics.

Improved Team Performance Using EEG- and Context-Based Cognitive-State Classifications for a Vehicle Crew

We present an augmented cognition (AugCog) system that utilizes two sources to assess cognitive state as a basis for actions to improve operator performance. First, continuous EEG is measured and signal processing algorithms utilized to identify patterns of activity indicative of high cognitive demand. Second, data from the automobile is used to infer the ongoing driving context. Subjects participated as eleven 2-person crews consisting of a driver/ navigator and a commander/gunner. While driving a closed-loop test route, the driver received through headphones a series of communications and had to perform two secondary tasks. Certain segments of the route were designated as threat zones. The commander was alerted when entering a threat zone and their task was to detect targets mounted on the roadside and engage those targets To determine targeting success, a photo was taken with each activation of the trigger and these photos were assessed with respect to the position of the reticle relative to the target. In a secondary task, the commander was presented a series of communications through headphones. Our results show that it is possible to reliably discriminate different cognitive states on the basis of neuronal signals. Results also confirmed our hypothesis: improved performance at the crew level in the AugCog condition for a secondary communications tasks, as compared to a control condition, with no change in performance for the primary tasks.

Measuring Expert and Novice Performance Within Computer Security Incident Response Teams

There is a great need for creating cohesive, expert cybersecurity incident response teams and training them effectively. This paper discusses new methodologies for measuring and understanding expert and novice differences within a cybersecurity environment to bolster training, selection, and teaming. This methodology for baselining and characterizing individuals and teams relies on relating eye tracking gaze patterns to psychological assessments, human-machine transaction monitoring, and electroencephalography data that are collected during participation in the game-based training platform Tracer FIRE. We discuss preliminary findings from two pilot studies using novice and professional teams.

Individual differences and the science of human performance

This study comprises the third year of the Robust Automated Knowledge Capture (RAKC) project. In the previous two years, preliminary research was conducted by collaborators at the University of Notre Dame and the University of Memphis. The focus of this preliminary research was to identify relationships between cognitive performance aptitudes (e.g., short-term memory capacity, mental rotation) and strategy selection for laboratory tasks, as well as tendencies to maintain or abandon these strategies. The current study extends initial research by assessing electrophysiological correlates with individual tendencies in strategy selection. This study identifies regularities within individual differences and uses this information to develop a model to predict and understand the relationship between these regularities and cognitive performance.

Communications-based automated assessment of team cognitive performance

In this paper we performed analysis of speech communications in order to determine if we can differentiate between expert and novice teams based on communication patterns. Two pairs of experts and novices performed numerous test sessions on the E-2 Enhanced Deployable Readiness Trainer (EDRT) which is a medium-fidelity simulator of the Naval Flight Officer (NFO) stations positioned at bank end of the E-2 Hawkeye. Results indicate that experts and novices can be differentiated based on communication patterns. First, experts and novices differ significantly with regard to the frequency of utterances, with both expert teams making many fewer radio calls than both novice teams. Next, the semantic content of utterances was considered. Using both manual and automated speech-to-text conversion, the resulting text documents were compared. For 7 of 8 subjects, the two most similar subjects (using cosine-similarity of term vectors) were in the same category of expertise (novice/expert). This means that the semantic content of utterances by experts was more similar to other experts, than novices, and vice versa. Finally, using machine learning techniques we constructed a classifier that, given as input the text of the speech of a subject, could identify whether the individual was an expert or novice with a very low error rate. By looking at the parameters of the machine learning algorithm we were also able to identify terms that are strongly associated with novices and experts.

Factors Contributing to Performance for Cyber Security Forensic Analysis

Previously, the current authors Hopkins et al. 2015 described research in which subjects provided a tool that facilitated their construction of a narrative account of events performed better in conducting cyber security forensic analysis. The narrative tool offered several distinct features. In the current paper, an analysis is reported that considered which features of the tool contributed to superior performance. This analysis revealed two features that accounted for a statistically significant portion of the variance in performance. The first feature provided a mechanism for subjects to identify suspected perpetrators of the crimes and their motives. The second feature involved the ability to create an annotated visuospatial diagram of clues regarding the crimes and their relationships to one another. Based on these results, guidance may be provided for the development of software tools meant to aid cyber security professionals in conducting forensic analysis.

Cognitive science and socio-cognitive theory for the HRI practitioner

This tutorial provides a synopsis of key findings and theoretical advances from cognitive science and socio-cognitive theory, with examples of how the results of this research can be applied to the design of human-robotic systems. Topics covered will run the gamut from basic cognitive science (e.g., perception, attention, learning and memory, information processing, multi-tasking, conscious awareness, individual differences) to socio-cognitive issues (e.g., theories of social interaction, dynamic functional allocation, mixed-initiative interaction, human-agent-robot teamwork, coactive design, theory of organizations). Additionally, the tutorial will address new technologies that attempt to leverage the current state of theory (e.g., neuroergonomics, brain-machine interfaces, detection of cognitive states, robotic prostheses and orthotics, cognitive and sensory prostheses). Throughout the tutorial, the presenters will give descriptions and demonstrations of working systems that exemplify the principles being taught. Separately, the presenters have given highly-successful tutorials on relevant subjects at workshops and conferences such as CHI and HCI International, as well as in a variety of industrial and government settings. In this tutorial, they propose to bring together their experience to bear on issues of specific interest to the HRI community.

The Next Generation of Cognitive Modeling Tools: Opportunities, Challenges and Basic Needs

In the past twenty or so years the scientific community has made impressive advancements in the modeling and simulation of general human cognition. This progress has led to the beginnings of wide-spread applications and use. In fact, we are now at a point where the community can begin to make fairly accurate predictions as to how this technology will be used in the next twenty–plus years. Accordingly, the purpose of this panel is to engage the community at large regarding the future needs and requirements associated with building cognitive models for various scientific and engineering endeavors. Specifically, this panel will discuss and make recommendations with regard to the future functionality of cognitive modeling that could be encompassed in next-generation capabilities. To do this, we will concentrate on four different domain areas. These are: academic use of cognitive modeling, cognitive model development, neuroscience-related issues, and practical applications of cognitive modeling.