James W. Gualtieri

Cognitive System Engineering - Based Design: Alchemy or Engineering

Cognitive Systems Engineering (CSE) techniques are widely used for the description and analysis of the sources of cognitive complexity and explicating the basis of expertise within a work domain. However, the results of the CSE techniques often focus on work analysis and are only weakly coupled to the design of decision support systems that are built based on those analyses. In fact, some within the CSE community have suggested that such a design epiphany occurs as if by magic. If CSE is to be treated as an engineering discipline, it cannot rely on magic to create systems. The approach described in this paper assumes that an explicit relationship between system design and supported cognitive work is fundamental to the designs effectiveness. The goal is a pragmatic, effective engineering process that explicitly designs systems according to relationships between cognitive work requirements and decision support concepts.

Scenario Development for Decision Support System Evaluation

This paper introduces a methodology for developing scenarios representative of the cognitive and collaborative challenges inherent in a domain of practice for evaluating Decision Support Systems (DSS). Explicit links are made between particular aspects of the DSS and specific cognitive and collaborative demands they are intended to support. The effectiveness of the DSS in supporting performance can then be systematically probed by creating scenarios that are informed by an understanding of individual and team cognitive processing factors, fundamental relationships within the domain, and known complicating factors that can arise in the domain to challenge cognitive and collaborative performance. This paper introduces a set of explicit artifacts to systematically create such scenarios to provide feedback on the viability of the DSS design concepts (e.g., are the hypothesized positive impacts of the DSS realized?), as well as feedback on additional unanticipated requirements for support.

Analysis with a Purpose: Narrowing the Gap with a Pragmatic Approach

There has been a growing interest in using Cognitive Systems Engineering (CSE) techniques to understand work domains and the cognitive demands they imposes on practitioners in order to provide a foundation for the design of decision-aids. While CSE techniques, like Cognitive Work Analysis (CWA), have been proven successful in illuminating the sources of cognitive complexity and explicating the basis of expertise, there is often still a gap between the results of the CWA and the resulting design and development of the decision support system. One way to narrow the gap is to develop an integrated set of artifacts that provide explicit links between (1) the functional goals the domain, to (2) the cognitive demands that require support, through (3) the mapping of decisions to the display space. In this paper a brief discussion of a recent example where this approach was taken is presented.

Power Tool for Countering Cyberwar: Visualizations for Information Assurance and Computer Network Defense

There has been a growing need for military decision-makers to maintain the integrity of the information contained within their computer network. Tools to support Information Assurance and Computer Network Defense (IA-CND) are needed to defend their information infrastructure and conduct Computer Network Operations with a new level of insight and understanding. This paper describes one effort to develop visualizations to aid these decision-makers in the highly abstract, complex and dynamic mission of IA-CND. This paper describes the development of a IA-CND Communications Display. Using a Cognitive Systems Engineering methodology, this project transitioned from a broad description of a work domain, to the development of decision aiding concepts for a particular portion of that domain. This methodology also provided a means to develop breakthrough support for a decision difficult domain.

Testing human-auto cyber team resilience uncovering latent failures with a novel, decision-centered approach

The increased use of automation has been necessary to keep up with the growing “cyber-pace” and complexity of threats. However, automation benefits do not come for free; the addition of these automated support agents has added additional coordination and supervisory work to the human operators domain, bringing hidden brittleness along with the benefits. This increased human-automation teamwork calls for the need to complement traditional testing approaches and holistically assess the net decision-making effectiveness and decision-making resilience of the Human-Auto cyber team. Decision Centered Testing (DCT) uncovers problems that would otherwise go unnoticed. By uncovering this joint HumanAutomation brittleness, system designers can make necessary system interface and automation design and development improvements to create a more resilient joint cognitive team. Based on Cognitive Systems Engineering (CSE), DCT uses test scenarios built around key decision-making demands of the domain, to place “pressure” on potential areas of weakness between the user and their decision support system (“edges”) to discover latent design flaws prior to being released into the fight. We present the success of DCT in the Control Domain (The Silent Automation & the “Right Sized Hole” Case), a DCT methodology introduction and how it can be used in the cyber domain.

Infochess: A Powerful Test Bed for Studying Information Operations (IO) Decision-Making

Participants will receive an introduction into the domain of Information Operations (IO) and asymmetric warfare. This interactive session will use InfoChess ™ to provide experiential learning in IO and explore decision-making. By integrating IO in the rich, yet well understood game of chess, the trainees understanding of the synergistic effects of IO and maneuver warfare is internalized to a degree not shown with traditional instruction. The audience will then be asked to collectively work together, against an asymmetrically adversary. During the course of this session, observations will be collected on the decision-making activities of the participants.