James C. Forsythe

Supervised machine learning for modeling human recognition of vehicle-driving situations

A classification system is developed to identify driving situations from labeled examples of previous occurrences. The purpose of the classifier is to provide physical context to a separate system that mitigates unnecessary distractions, allowing the driver to maintain focus during periods of high difficulty. While watching videos of driving, we asked different users to indicate their perceptions of the current situation. We then trained a classifier to emulate the human recognition of driving situations using the Sandia Cognitive Framework. In unstructured conditions, such as driving in urban areas and the German autobahn, the classifier was able to correctly predict human perceptions of driving situations over 95% of the time. This paper focuses on the learning algorithms used to train the driving-situation classifier. Future work will reduce the human efforts needed to train the system.

Surety of human elements of high consequence systems: An organic model

Despite extensive safety analysis and application of safety measures, there is a frequent lament, “Why do we continue to have accidents?” Two breakdowns are prevalent in risk management and prevention. First, accidents result from human actions that engineers, analysts and management never envisioned and second, controls, intended to preclude/mitigate accident sequences, prove inadequate. This paper addresses the first breakdown, the inability to anticipate scenarios involving human action/inaction. The failure of controls has been addressed in a previous publication (Forsythe and Grose, 1998). Specifically, this paper presents an approach referred to as “surety.” The objective of this approach is to provide high levels of assurance in situations where potential system failure paths cannot be fully characterized. With regard to human elements of complex systems, traditional approaches to human reliability are not sufficient to attain surety. Consequently, an Organic Model has been developed to account for the organic properties exhibited by engineered systems that result from human involvement in those systems.

Measuring Human Performance within Computer Security Incident Response Teams

Human performance has become a pertinen t issue within cyber security. However, this research has been stymied by the limited availability of expert cyber security professionals. This is partly attributable to the ongoing workload faced by cyber security professionals, which is compound ed by the limited number of qualified personnel and turnover of p ersonnel across organizations. Additionally, it is difficult to conduct research, and particularly, openly published research, due to the sensitivity inherent to cyber ope rations at most orga nizations. As an alternative, the current research has focused on data collection during cyb er security training exercises. These events draw individuals with a range of knowledge and experience extending from seasoned professionals to recent college gradu ates to college students. The current paper describes research involving data collection at two separate cyber security exercises. This data collection involved multiple measures which included behavioral performance based on human - machine transactions and questionnaire - based assessments of cyber security experience.

Robust Automated Knowledge Capture

This report summarizes research conducted through the Sandia National Laboratories Robust Automated Knowledge Capture Laboratory Directed Research and Development project. The objective of this project was to advance scientific understanding of the influence of individual cognitive attributes on decision making. The project has developed a quantitative model known as RumRunner that has proven effective in predicting the propensity of an individual to shift strategies on the basis of task and experience related parameters. Three separate studies are described which have validated the basic RumRunner model. This work provides a basis for better understanding human decision making in high consequent national security applications, and in particular, the individual characteristics that underlie adaptive thinking.

Nested Narratives Final Report

In cybersecurity forensics and incident response, the story of what has happened is the most impor- tant artifact yet the one least supported by tools and techniques. Existing tools focus on gathering and manipulating low-level data to allow an analyst to investigate exactly what happened on a host system or a network. Higher-level analysis is usually left to whatever ad hoc tools and techniques an individual may have developed. We discuss visual representations of narrative in the context of cybersecurity incidents with an eye toward multi-scale illustration of actions and actors. We envision that this representation could smoothly encompass individual packets on a wire at the lowest level and nation-state-level actors at the highest. We present progress to date, discuss the impact of technical risk on this project and highlight opportunities for future work.

Training Adaptive Decision-Making.

Adaptive Thinking has been defined here as the capacity to recognize when a course of action that may have previously been effective is no longer effective and there is need to adjust strategy. Research was undertaken with human test subjects to identify the factors that contribute to adaptive thinking. It was discovered that those most effective in settings that call for adaptive thinking tend to possess a superior capacity to quickly and effectively generate possible courses of action, as measured using the Category Generation test. Software developed for this research has been applied to develop capabilities enabling analysts to identify crucial factors that are predictive of outcomes in fore-on-force simulation exercises.

Enhanced Training for Cyber Situational Awareness in Red versus Blue Team Exercises

This report summarizes research conducted through the Sandia National Laboratories Enhanced Training for Cyber Situational Awareness in Red Versus Blue Team Exercises Laboratory Directed Research and Development project. The objective of this project was to advance scientific understanding concerning how to best structure training for cyber defenders. Two modes of training were considered. The baseline training condition (Tool-Based training) was based on current practices where classroom instruction focuses on the functions of a software tool with various exercises in which students apply those functions. In the second training condition (Narrative-Based training), classroom instruction addressed software functions, but in the context of adversary tactics and techniques. It was hypothesized that students receiving narrative-based training would gain a deeper conceptual understanding of the software tools and this would be reflected in better performance within a red versus blue team exercise.

A Data-Driven Approach to Assessing Team Performance through Team Communication.

For teams working in complex task environments, instilling effective communication between team members is a primary goal during task training. Presently, responsibility for evaluating team communication abilities resides with instructors and outside observers who make qualitative assessments that are shared with the team following a training exercise. Constructing technologies to automate these assessments has historically been prohibitive for two reasons. First, the financial cost of instrumenting the environment to collect team communication data at the necessary fidelity has been too high for an operational setting. Second, past research on using team communication as a proxy for team performance assessment has relied on defining communication through traditional algorithmic design, an approach which does not properly capture the varied nature of communication strategies amongst different teams. Recent scientific research in team dynamics provides a theoretical framework leading to a data-driven solution for analyzing the effectiveness of team communication. By framing team communication as an emergent data stream from a complex system, one may employ machine learning or other statisticalanalysis tools to highlight communication patterns and variance, both shown as effective means for assessing team adaptability to novel scenarios. Furthermore, low-cost wearable computers have opened new possibilities for observing people’s interactions in natural settings to better analyze and improve team performance. We summarize research conducted in developing a data-driven approach to analyzing team communications within the context of Surfaced Piloting and Navigation (SPAN) training for submariners. Using Dynamic Bayesian Networks (DBNs), this approach created predictive models of communication patterns that emerge from the team in different contexts. Based upon data collection conducted in the lab and within live submarine crew training, our results demonstrate the robust nature of DBNs by still identifying key communication events even when teams altered their speaking patterns during these events to accommodate for novel changes in the scenario. Introduction Complex tasks that demand a coordinated effort benefit from the capacity of a team to pool resources via an exchange of information and coordinated action, though the effectiveness of a team may be contingent on a variety of factors [1]. Team effectiveness has particular impact within a military setting, as within combat situations the performance of a group has a direct bearing on the survival of the group and those dependent on them [2], situation that holds true when considering the success of naval operations [3]. In an attempt to determine the critical elements that make up an effective team in a military setting, variables related to team effectiveness have been examined from a variety of perspectives, including team cohesiveness (i.e., shared interpersonal closeness and group goal-orientation) [4], [5] collective orientation [1], shared mental models (i.e., synthesis of input from individual team members) [6], [7], [8], team selection and composition (e.g., the skills possessed by the individual team members, how long the members have been working together) [5], [6], [9], quality of decisions made by commanders [10], [11], cognitive readiness and adaptive decision making at the group level [12], training adequacy [5], the workload involved [13], and even neurophysiologic synchrony between team members, as assessed via electroencephalogram [14]. In the context of naval operations, assessment of the quality of teamwork has proven difficult, with such assessments relying on the observations of subject matter experts, skilled instructors, or a self-evaluation within teams during live or simulated exercises [3]. These judgments are subjective by their very nature, leading to a potential lack of consistency with regard to the quality of assessment. This issue has been recognized, and there have been attempts to resolve it, such as through outcome-based Copyright