Patricia L. McDermott

Using the Analytic Hierarchy Process to Examine Judgment Consistency in a Complex Multiattribute Task

This paper investigates the impact of framing and time pressure on human judgment performance in a complex multiattribute judgment task. We focus on the decision process of human participants who must choose between pairwise alternatives in a resource-allocation task. We used the Analytic Hierarchy Process (AHP) to calculate the relative weights of the four alternatives (i.e., C1, C2, C3, and C4) and the judgment consistency. Using the AHP, we examined two sets of hypotheses that address the impact of task conditions on the weight prioritization of choice alternatives and the internal consistency of the judgment behavior under varying task conditions. The experiment simulated the allocation of robotic assets across the battlefield to collect data about an enemy. Participants had to make a judgment about which asset to allocate to a new area by taking into account three criteria related to the likelihood of success. We manipulated the information frame and the nature of the task. We found that, in general, participants gave significantly different weights to the same alternatives under different frames and task conditions. Specifically, in terms of ln-transformed priority weights, participants gave significantly lower weights to C2 and C4 and higher weight to C3 under gain frame than under loss frame, and also, under different task conditions (i.e., Tasks #1, #2, and #3), participants gave significantly higher weight to C4 in Task #1, lower weights to C1 and C4, higher weight to C3 in Task #2, and lower weight to C3 in Task #3. Furthermore, we found that the internal consistency of the decision behavior was worse, first, in the loss frame than the gain frame and, second, under time pressure. Our methodology complements utility-theoretic frameworks by assessing judgment consistency without requiring the use of task-performance outcomes. This work is a step toward establishing a coherence criterion to investigate judgment under naturalistic conditions. The results will be useful for the design of multiattribute interfaces and decision aiding tools for real-time judgments in time-pressured task environments.

Effective Human to Human Communication of Information Provided by an Unmanned Vehicle

Much of the research on unmanned-vehicles (UVs) focuses on technology or interface design. This study however, investigated how to best support effective communication between the operator monitoring a UV and the Soldier in the field using that information to complete a mission. Several questions arise: Does the operator need to be co-located with Soldiers in the field or can he or she be in a more secure rearward location? Does the team need the capability to transmit visual images or is radio communication adequate? Is information from one type of UV better than others? Do real time mapping and tracking technologies increase situation awareness (SA)? To begin to answer these questions, military teams conducted rescue missions using the video game Raven Shield as a simulated battlefield. The analysis of performance data, self reports, and observations provide some valuable insight to these questions.

Advantages of Co-Location for Effective Human to Human Communication of Information Provided by an Unmanned Vehicle

Much of the research on unmanned-vehicles (UVs) focuses on technology or interface design. This study investigated how to best support effective communication between the UV operator and the Soldier in the field using UV-provided information to complete a mission. In a previous study investigating the impact of different team configurations and the utility of supporting communication technologies, our team found co-location of team members to be beneficial (McDermott et al., 2005). In this experiment we investigate what aspects of co-location are key to successful team performance: Is face-to-face communication vital compared to voice-only when team members are distributed? Is the ability of the UV operator to see what the Soldier performing the mission can see critical? We also seek additional insight to inconclusive results from the first study regarding the utility of image transmission and access to an electronic map displaying both the UV and Soldier locations.

The Presentation of Risk and Uncertainty in the Context of National Missile Defense Simulations

Risk perception and uncertainty management are important components of military decision making, especially in time-stressed and resource-limited environments. The purpose of this experiment was to understand the interaction of integrality of information, presentation mode, and information frame on situation awareness (SA) and decision-making (missile allocation) in a National Missile Defense (NMD) paradigm. Results of the information frame manipulation (expected gain v. expected loss) support earlier findings that subjects are loss averse. SA Accuracy was higher with graphical displays than alphanumeric displays. The implications for NMD are discussed.

Part Task Training Methods in Simulated and Realistic Tasks

Part task methods are widely used in training programs when the full target task is too complex or impractical due to cost or availability. However, part task training has had mixed success in transfer to the whole task, in comparison to whole task training. For unmanned systems, the goal is to create training that prepares trainees to control an actual vehicle once one is available. The present research compares part and whole task training options in a simulated training environment and a realistic transfer environment. The task is teleoperating a robotic device to detect and identify vehicles in an urban environment. Part task training on the real task yielded a significant benefit in first trial whole task transfer performance and in trials to achieve transfer criteria relative to training on the simulated environment only. While this gain came at a relative cost in overall training time, it produced a significant training effectiveness gain by reducing the training time required on the more expensive robotic system.

Bandwidth allocation in a military teleoperation task

The implications of bandwidth allocation are described for teleoperation in a military task that involved navigation, target detection, and target identification. Color versus grayscale imagery was manipulated. Participants themselves traded off resolution and frame rate settings. Participants minimized switching between resolution/frame rate settings and tended to use settings with high resolution/low frame rate. Courses completed with the highest resolution (and lowest frame rate) had the fastest target identification times, but no other differences were observed between settings. Color imagery offered advantages for overall course time and the time to identify a tank as friendly or enemy.

Systematic analysis of risk visualization strategies for homeland defense

An approach to conduct systematic analysis of risk visualization strategies in a homeland missile defense context is presented. The approach consists of three phases which take different perspectives toward characterizing the problems of resource management and allocation for defense against missile attacks. Decision makers in missile defense are supervisory controllers of highly automated and resource-constrained systems. Our approach is suggested as a method to assess the effectiveness of visualization strategies in a dynamic decision environment. The three phases of our approach included a test for subject utility preferences, a text for subject multi-attribute prefences using the Analytic Hierarchy Process (AHP), and a test for subject action selection using a human-in-the-loop simulation test bed. Our approach is proposed to systematically evaluate the effectiveness of multistage decision making strategies against immediate and delayed threats. The alternatives were calculated based on expected cases of lives lost as a result of attacks on defended areas. Implementation of the test bed enabled us to investigate the effect of visualization formats on decision behaviour. Some preliminary results based on a pilot study will be presented. The implications of our study extend beyond missile defense. In a world where catastrophic loss of civilian lives is seen by some as a victory, those entrusted with homeland security must effectively manage limited resources to combat terrorist activities. Effective visualization formats will help to promote decision that are informed with respect to the risks and outcomes.

A Scalable Architecture for Operational FMV Exploitation

A scalable open systems and standards derived software ecosystem is described for computer vision analytics (CVA) assisted exploitation of full motion video (FMV). The ecosystem, referred to as the Advanced Video Activity Analytics (AVAA), has two instantiations, one for size, weight, and power (SWAP) constrained conditions, and the other for large to massive cloud based configurations. The architecture is designed to meet operational analyst requirements to increase their productivity and accuracy for exploiting FMV using local cluster or scalable cloudbased computing resources. CVAs are encapsulated within a software plug-in architecture and FMV processing pipelines are constructed by combining these plug-ins to accomplish analytical tasks and manage provenance of processing history. An example pipeline for real-time motion detection and moving object characterization using the flux tensor approach is presented. An example video ingest experiment is described. Quantitative and qualitative methods for human factors engineering (HFE) assessment to evaluate cognitive loads for alternative work flow design choices are discussed. This HFE process is used for validating that an AVAA system instantiation with candidate workflow pipelines meets CVA assisted FMV exploitation operational goals for specific analyst workflows. AVAA offers a new framework for video understanding at scale for large enterprise applications in the government and commercial sectors.

Framing, Loss Aversion, and Visualization of Risk for a Dynamic Simulation Environment

The purpose was to understand the effects of loss aversion and framing on situation awareness (SA) and decision making for a dynamic missile simulation task. Whereas framing has been shown in numerous experiments that were based on the original paradigm, we hypothesized that the loss aversion effects were more general and would extend to conditions in which sure gains were not possible and would affect SA as well as decision making. Forty-eight students participated in a dynamic simulation in which operators had to decide which cities to defend and to answer SA probes while viewing 2-min scenarios wherein probabilities were assigned to a number of target cities as the missile attack unfolded. The authors varied frame in terms of lives lost or survived for a chosen allocation scheme and varied presentation mode in terms of either expected value or individual risk indices (probabilities and lives) as part of the graphic displays. There were significant effects on missile conservation decisions, coverage of small cities, and SA. The results supported theories positing more general effects of loss aversion related to negative affect. Guidelines related to presenting gain or loss information via graphic displays were suggested.

Team Communication with and without Aids for Transmitting Remote Information

The introduction of unmanned vehicles (UVs) and real-time electronic information presentation to military teams is intended to keep Soldiers safe and enable more effective performance. Two game-based experiments were conducted in which teams conducted a time-limited “Black Hawk Down” rescue mission with and without UV-provided information. When UV information was available, it was relayed between team members in face-to-face communication or remotely, with or without electronic maps, and with or without video images from the UV. Three types of analyses were performed. The communication content of the verbal protocol was analyzed and the use of information display technologies and employment of strategies was tabulated. The verbal protocol analyses revealed the push/pull of team communication and common confusions. Results of the technology use and strategy assessment form the basis for recommendations for display design and recommended practice for the use of such technologies in the field.