Jonathan Pfautz

Visual Representations of Meta-Information

We conducted two studies that investigated display characteristics related to color (hue, saturation, brightness, and transparency) and contrast with a background for displaying information qualifiers (termed meta-information) such as uncertainty, age, and source quality. Level of detail (or granularity) of the meta-information and task demands were also manipulated. Participants were asked to rank and rate colored regions overlaid on different map backgrounds based on the level of meta-information the regions displayed. Results from Study 1 indicated that participants could appropriately rank and rate levels of meta-information across saturation, brightness, and transparency conditions, and results from Study 1 and Study 2 showed that the natural direction of ordering is complex and dependent on the relevance of different information to the task and the contrast of the overlay region with the background.

Comparing Uncertainty Visualizations for a Dynamic Decision-Making Task:

Supporting complex decision making requires conveying relevant information characteristics or qualifiers. The authors tested transparency and numeric annotation for displaying uncertainty about object identity. Participants performed a “missile defense” game in which they decided whether to destroy moving objects (which were either threatening missiles or nonthreatening birds and planes) before they reached a city. Participants were provided with uncertain information about the objects’ classifica-tions. Uncertainty was represented through the transparency of icons representing the objects and/or with numeric annotations. Three display methods were created. Icons represented the most likely object classification (with solid icons), the most likely object classification (with icons whose transparency represented the level of uncertainty), or the probability that the icon was a missile (with transparency). In a fourth condition, participants could choose among the representations. Icons either were or were not annotated with numeric probability labels. Task performance was highest when participants could toggle the displays, with little effect of numeric annotation. In conditions in which probabilities were available graphically or numerically, participants chose to engage objects when they were farther from the city and had a lower probability of being a missile. Results provided continued support for the use of graphical uncertainty representations, even when numeric representations are present.

A Cognitive Task Analysis for Cyber Situational Awareness

Cyber Network degradation and exploitation can covertly turn an organizations technological strength into an operational weakness. It has become increasingly imperative, therefore, for an organizations personnel to have an awareness of the state of the Cyber Network that they use to carry out their mission. Recent high-level government initiatives along with hacking and exploitation in the commercial realm highlight this need for general Cyber Situational Awareness (SA). While much of the attention in both the military and commercial cyber security communities is on abrupt and blunt attacks on the network, the most insidious cyber threat to organizations are subtle and persistent attacks leading to compromised databases, processing algorithms, and displays. We recently began an effort developing software tools to support the Cyber SA of users at varying levels of responsibility and expertise (i.e., not just the network administrators). This paper presents our approach and preliminary findings from a CTA we conducted with an operational Subject Matter Expert to uncover the situational awareness requirements of such a tool. Results from our analysis indicate a list of preliminary categories of these requirements, as well as specific questions that will drive the design and development of our SA tool.

The Impact of Meta-Information on Decision-Making in Intelligence Operations

Decision-making in complex, dynamic, high-risk environments is clearly challenging. Part of this challenge is due to the presence of qualifiers of information, or meta-information (e.g., staleness, uncertainty, source), that alter a persons information processing, situational awareness, and decision-making. We investigated the influence of meta-information on decision-making in a Military Intelligence Operations (IO) environment using Cognitive Task Analysis (CTA) techniques. We performed a CTA on IO tasks surrounding the use of smart sensor webs, a relatively new technology that can be used for a variety of IO purposes. Our analysis addressed information management tasks and tactical decision-making tasks using sensor webs. We discovered that a variety of types of meta-information significantly impacted decision-making, and that the influence of meta-information was both context- and task- sensitive. In this paper, we present the results of the CTA and discuss the implications for the development of decision-aiding systems, including the design of constituent displays, interfaces, and automated systems.

Modeling human reasoning about meta-information

Information, as well as its qualifiers, or meta-information, forms the basis of human decision-making. Human behavior models (HBMs) therefore require the development of representations of both information and meta-information. However, while existing models and modeling approaches may include computational technologies that support meta-information analysis, they generally neglect its role in human reasoning. Herein, we describe the application of Bayesian belief networks to model how humans calculate, aggregate, and reason about meta-information when making decisions.

Cognitive Complexities Impacting Army Intelligence Analysis

The primary goal of this effort was to understand the problems faced by military intelligence analysis personnel as well as how, and to what degree, the identification of these problems could guide the development of computational support systems. To develop this understanding, we performed a literature review, knowledge elicitation interviews and a cognitive task analysis (CTA) in the domain of Army Intelligence Analysis at the Brigade Combat Team. This effort consisted of identifying: (1) the major functions or cognitive tasks entailed in Army Intelligence Analysis; and (2) the complexities in the domain that pose challenges to performance of these cognitive tasks. Identifying the cognitive tasks and the challenges faced in performing those tasks provided a basis for determining opportunities for more effective support of human information processing and decision-making. In this paper, we document selected results of this analysis effort.

A prototype toolkit for sensing and modeling individual and team state

Teams of individuals working together toward a common goal must be skilled at multi-tasking to perform their own work while maintaining shared attention across the team. Experimenters who study team performance can use cutting edge methods to assess physiological, neurophysiological, and behavioral underpinnings of optimal performance; however, this requires an adequate understanding of how these signals correlate with individual and team performance. We designed a toolkit to support experimenters in evaluating individual and team performance in a laboratory setting, in testing and validating models of performance, and in developing and validating augmentation strategies to improve performance. Our toolkit provides a framework that flexibly integrates current and emerging sensors. The data fusion tool fuses time-synchronized sensor data to assess performance. The model-building and execution toolset enables experimenters to choose previously entered models, adapt these models according to the current experiment, or develop new models to test. The real-time assessment tool enables experimenters to monitor the state of individual subjects and the team as a whole (e.g., stress, workload, focused attention) throughout the experiment, and how these states relate to performance. This information is then used by the real-time augmentation tool, which suggests augmentations to optimize that performance. Together, these tools provide a proof-of-concept prototype of a flexible modeling tool that would allow sensor inputs to be used to model and predict both individual and team performance.

Complexities and Challenges in the Use of Bayesian Belief Networks: Informing the Design of Causal Influence Models

Bayesian belief networks (BNs) are well-suited to capturing vague and uncertain knowledge. However, the capture of this knowledge and associated reasoning from human domain experts often requires specialized knowledge engineers responsible for translating the experts communications into BN-based models. Across application domains, we have analyzed how these models are constructed, refined, and validated with domain experts. From this analysis, we have identified key user-centered complexities and challenges that we have used to drive the selection of simplifying assumptions. This led us to develop computational techniques and user interface methods that leverage these same assumptions with the goal of improving the efficiency and ease with which expert knowledge can be expressed, verified, validated, and encoded. In this paper, we present the results of our analysis of BN construction, validation, and use. We discuss how these results motivated the design of a simplified version of BNs called Causal Influence Models (CIMs). In addition, we detail how CIMs enable the design and construction of user interface mechanisms that address complexities identified in our analysis.

Cognitive Systems Engineering for Evolvable Human-in-the-Loop Data Fusion

Data fusion systems are increasingly being used to support military planning and decision making. Typically these systems are designed around the current capabilities of particular data collectors (e.g., sensors) and processing algorithms. They incorporate an ‘ontology’ that reflects the designers perception of the key features of the world (e.g., types of threats, classes of vehicles to be tracked) and how these can be parsed by the data fusion systems. As a consequence they are limited in their ability to adapt to the dynamic changes that inevitably arise in the operational environment (e.g., new sensors, weapons, tactics). This is representative of a more generic problem with current approaches to system design that result in rigid systems that are unable to evolve to keep pace with changing operational conditions. In this paper we present the results of an analysis, design, and development effort intended to move away from traditional data fusion systems towards evolvable human-in-the-loop data fusion systems. We discuss the analysis we conducted in support of an evolvable system design and provide an overview of the prototype evolvable data fusion system architecture we are developing.

Beyond uncertainty: examining meta-information visualization techniques

The rendering of timely and accurate decisions in safety-critical systems requires that the human operator integrate information from various sources. The integration of this information in turn depends on the skill and experience level of the operator and on a thorough understanding of the qualities of that information (e.g. recency), known as meta-information. Such qualities can critically influence how an operator will process information, understand information, and make decisions based on information (Pfautz et al, 2006). In recent years, researchers have explored various visualization techniques aimed at helping operators better integrate/assimilate meta-information in the environment. However, these attempts have traditionally focused on one type of meta-information – uncertainty (Finger et al, 2002) – and empirical assessments of the benefits of the techniques were limited in scope. While consideration of uncertainty in decisionmaking is important, we postulate that other forms of metainformation (e.g. staleness) must be considered as well. We believe that empirical testing of graphical concepts is critical to being able to provide a thorough evaluation of the benefits that such visualization techniques may hold for improving decisionmaking ability. Here, we briefly summarize ongoing research initiatives where we have used Cognitive Work Analysis techniques to identify the information and meta-information requirements of operators across various safety-critical domains. These techniques have supported the development and empirical evaluation of methods for visualizing meta-information to improving decision-making.