Jean MacMillan

Cognitive bias in software engineering

Developers thought processes are a fundamental area of concern. Cognitive scientist have discovered that peoples intiative inferences and probality judgments do not strictly conform to the laws of logic or mathematics, and that people are willing to provide plausible explanations for random events. This article examines the role these phenomena might have in software development, ultimately concluding that what are cast as one-sided software development guidelines often can be recast beneficially as two-sided trade-offs.

Questionnaires for Distributed Assessment of Team Mutual Awareness

Teams, by definition, perform interdependent tasks that require team members to coordinate their decisions and actions in order to achieve their shared goals (Orasanu and Salas, 1993). In order to successfully achieve the level of coordination that is required for successful interdependent performance, team members need a shared awareness of the situation, and of the roles, tasks, and actions of the other team members. The existence of this “shared mental model” among team members has been suggested as an explanatory mechanism for effective teams, and, as measured in various ways, has been shown to increase team performance (Cannon-Bowers, Salas, and Converse, 1993; Stout, CannonBowers, Salas, and Milanovich, 1999).

Evaluating Expertise in a Complex Domain–Measures Based on Theory

Human factors practitioners are often concerned with defining and evaluating expertise in complex domains where there may be no agreed-upon expertise levels, no single right answers to problems, and where the observation and measurement of real-world expert performance is difficult. This paper reports the results of an experiment in which expertise was assessed in an extremely complex and demanding domain–military command decision making in tactical warfare. The hypotheses of the experiment were: 1) command decisionmaking expertise can be recognized in practice by domain experts; 2) differences in the command decisionmaking expertise of individuals can be identified even under conditions that do not fully replicate the real world; and 3) observers who are not domain experts can recognize the expert behaviors predicted by a mental-model theory about the nature of expertise. In the experiment, the expertise of military officers in developing tactical plans was assessed independently by three “super-expert” judges, and these expertise-level ratings were correlated with independent theory-based measures used by observers who were not domain experts. The results suggest that experts in a domain have a shared underlying concept of expertise in that domain even if they cannot articulate that concept, that this expertise can be elicited and measured in situations that do not completely mimic the real world, and that expertise measures based on a mental-model theory can be used effectively by observers who are not experts in the domain.

Operator Reliance on Automated Support for Target Recognition

In machine-aided target recognition, human operators work with an automatic target recognition (ATR) system to locate targets in cluttered and degraded imagery. The operator must integrate his or her own visual judgment concerning whether a target is present in the image with the ATRs judgment, which is typically expressed numerically. We conducted a series of experiments in which subjects attempted to locate target shapes among non-targets based only on visual images and based on both visual images and supplementary numeric information such as an ATR might provide. Image quality was controlled as an independent variable through the use of distortion rates that randomly altered pixel values to degrade the image. We found that subjects maintained a constant false alarm rate as image distortion increased, at the expense of a lower hit rate. This result was found consistently in experiments where the subjects task was to distinguish single targets from a blank background, to distinguish single targets from single non-targets, and to locate multiple targets in a multiple-object display. We also found a bias toward over reliance on image versus numeric information. As image distortion increased, subjects failed to make optimal use of supplementary numeric information and showed an unnecessary decrease in performance. The results suggest that operators may experience difficulty in working with an ATR that has a high false alarm rate, even if the ATRs hit rate is also high, and that numeric expressions of ATR judgment may be undervalued by operators in locating targets.

From Team Structure to Team Performance: A Framework

This paper presents a theoretical framework that links the organizational structure of a team to the teams performance through intervening factors such as the need for coordination, the need for communication, the extent to which the teams mission can be pre-planned, and the teams mutual awareness of each others tasks. We suggest that the organizational structure of a team interacts with the nature of the teams mission (in particular, the interdependence among the tasks to be performed) to generate the need for coordination in order to successfully accomplish the mission. The need for coordination and the extent to which the mission can be pre-planned drive the need for communication during the mission. The efficiency of that communication is, in turn, affected by factors such as the teams level of mutual awareness. The paper presents several innovative measures for components of the suggested framework, and summarizes empirical evidence for the framework.

Team-in-the-Loop, Synthetic Simulation: Bridging the Gap between Laboratory and Field Research

The reported work demonstrates some recent results in an ongoing effort to bridge the gap between controlled laboratory investigations of team performance and more complicated but realistic research from the field. Reported work is part of a broad, multidisciplinary research program linking insights from both extremes in a middle ground of controlled experimentation with medium fidelity, team-in-the-loop synthetic tasks. Our work, focussed on the domain of AWACS command and control, has taken a multi-level, multi-method approach to task development and performance assessment. In this report, we will discuss recent data and insights from a high-fidelity DMT exercise—RoadRunner 98-and our ongoing efforts to apply these insights and results to the development of a series of experiments using medium-fidelity simulation to capture the same core constructs. Paper contains preliminary data from RoadRunner 98, a description of our measurement approach and discussion of our current research using the medium-fidelity DDD-AWACS simulator.

Capturing and Building Expertise in Virtual Worlds

Model-driven simulation can make the design and delivery of instruction more efficient and effective. We describe two computational models that support both the design and delivery of instruction. BEST (the Benchmarked Experiential System for Training) can guide experts through the space of domain problems during the knowledge engineering phase of instructional design; it can guide trainees through the space of training objectives during instruction. PRESTO (Pedagogically Relevant Engineering of Scenarios for Training Objectives) builds scenarios on the fly to elicit the knowledge of experts during instructional design, and to satisfy the instructional objectives of trainees.

16. Design of a Multi-Vehicle Control System: System Design and User Interaction

As they are currently conducted, missions by single ROVs consist of several sub-tasks. After a vehicle has been launched, a human operator or a small team is responsible for controlling the flight, navigation, status monitoring, flight and mission alteration, problem diagnosis, communication and coordination with other operators, and often data analysis and interpretation. These tasks are similar in terms of their locus of control (e.g., keyboard and mouse input, joystick, trackball, visual display).

Development of a Team-in-the-Loop, Synthetic Simulation Environment Based on Data from Cognitive Task Analysis

This abstract reports the results of a successful effort to create a synthetic task environment that captures key elements of a team task, based on Cognitive Task Analysis of the important features of the task from a teamwork and cognition viewpoint. Studying high-performance teams has traditionally been difficult in the laboratory. The complexities of inter-action, difficulties of controlling naturally-occurring performance-affecting variables, and importance of interpersonal, coordinative and communication-related behaviors all add to the difficulty. To capture the essence of team performance in an experimental setting, it is necessary to place performers in relatively rich task environments that create opportunities for coordination, communication and adaptation. In recent years, many team researchers have turned to simulation technologies in an effort to create enough realism to stress expert teams, yet control enough environmental noise to collect empirically clean team performance results. Due to recent advances in hardware, software and distributed network technologies, many of these simulations have become highfidelity copies of actual field systems, providing participants with very realistic, full immersion performance experiences. The problem is, these simulations often end up being as complicated as the environments they were meant to model. In this demonstration we report the development of a medium-fidelity simulation environment that offers many of the same cap-abilities of high-fidelity simulation in terms of richness of experience but maintains enough abstraction to allow for controlled research and training. In order to maintain cognitive fidelity in our design, our approach to simulation development began with very careful analysis of the field task and environment we had set out to emulate—AWACS Command and Control—to insure that the variables that might affect team performance were preserved while others, not likely to influence team performance, were abstracted out of the simulation. Using a blend of Cognitive Task Analysis (CTA) methods, we gained a thorough understanding of the tasks and environment of AWACS Weapons Directors (WDs). Once collected, our CTA data were interpreted through the lens of team-performance theory and models. These data and theory were used together to decide which among the many possible variables that might be modeled were the important variables required to reproduce the teamwork and coordination demands of the task under consideration. The goal of our work has been to develop and validate a medium-fidelity simulation of AWACS command and control using a process of focused CTA of the domain, theory-driven conversion of CTA data into a reasonable minimum set of requirements for a simulation, software development, and finally, validation of our medium-fidelity simulation using expert participants and high-fidelity or field-gathered data for comparison. We are demonstrating that it is possible to strike a balance between highly complex, poorly controlled exercises on the one hand and over-controlled, overly simple laboratory research on the other. To date, we have developed the simulation demonstrated here and are in the process of collecting data that can be directly compared to data from a higher fidelity simulation (C3STARS, at Brooks AFB) to establish that our approach and resulting medium-fidelity simulation can be used to capture performance data that generalize to the field.

Hybrid Team Training Testbed for Awacs Aircraft Controllers

Simulation-based team training has become an important component of preparation and readiness for a variety of military teams. However, despite extensive resource allocation to simulation-based training environments, and to computer generated forces that can act in those environments, there is relatively little research on how synthetic entities can best be designed and used to achieve team training objectives. In this paper, we outline the development of a research program that seeks to define principles of effective team training with synthetic entities. We describe our domain of application (Air Force AWACS Weapons Directors and the pilots they control), team training objectives, the components and capabilities of the simulation, and our plan for using the simulation to evaluate the effectiveness of hybrid team training. Ultimately, this work will result in a series of principles of effective team training with agents that can facilitate the development of hybrid training environments.