John S. Barnett

Who is flying this plane anyway? What mishaps tell us about crew member role assignment and air crew situation awareness

This paper reports a detailed analysis of over 300 civilian incident reports that identified whether loss of situation awareness (SA) was related to air crew role assignment. The results indicate (a) that loss of SA is responsible for an incident more often when the captain is at the controls than when the first officer (FO) is at the controls, and (b) that the pilot flying (PF) is more likely to lose situation awareness than the pilot not flying (PNF). As a result, captains lose SA more often across aircraft types, flight segments, and weather conditions when they are the PF than when they are the PNF. The results also suggest that the person who is flying commits more of the critical errors that lead to an incident. Together, the results indicate that captains lose SA more often and make more tactical errors when they are at the controls than when they are not. Applications of this research include aircrew training, procedure development, and accident/incident analysis.

The Effect of Automation Reliability on User Automation Trust and Reliance in a Search-and-Rescue Scenario

Advances in modern day technology are rapidly increasing the ability of engineers to automate ever more complicated tasks. Often these automated aids are paired with human operators who can supervise their work to ensure that it is free of errors and to even take control of the system if it malfunctions (e.g., pilots supervising an autopilot feature). The goal of this collaboration, between humans and machines, is that it can enhance performance beyond what would be possible by either alone. Arguably the success of this partnership depends in part upon attributions an operator develops that help guide their interaction with the automation. One particular factor that has been shown to guide operator reliance on an automated ‘teammate’ is trust. The following study examined 140 participants performing a simulated search-and-rescue task. The goal of this experiment was to examine the relationship between automated agents reliability, operator trust, operator reliance, and performance scores. Results indicated that greater automation reliability is positively correlated with greater user reliance (r = .66), perceived trust (r = .21), and performance scores (r = .34). These results indicate that more reliable aids are rated as significantly higher in terms of perceived trust and relied upon more than less reliable aids. Additionally, the size of the effect is much larger for operator behaviors (i.e., reliance) compared to more subjective measures (i.e., self-reported trust).

Evaluation of wearable simulation interface for military training.

Objective: This research evaluated the training effectiveness of a novel simulation interface, a wearable computer integrated into a soldier’s load-bearing equipment. Background: Military teams often use game-based simulators on desktop computers to train squad-level procedures. A wearable computer interface that mimics the soldier’s equipment was expected to provide better training through increased realism and immersion. Method: A heuristic usability evaluation and two experiments were conducted. Eight evaluators interacted with both wearable and desktop interfaces and completed a usability survey. The first experiment compared the training retention of the wearable interface with a desktop simulator and interactive training video. The second experiment compared the training transfer of the wearable and desktop simulators with a live training environment. Results: Results indicated the wearable interface was more difficult to use and elicited stronger symptoms of simulator sickness. There was no significant difference in training retention between the wearable, desktop, or interactive video training methods. The live training used in the second experiment provided superior training transfer than the simulator conditions, with no difference between the desktop and wearable. Conclusion: The wearable simulator interface did not provide better training than the desktop computer interface. It also had poorer usability and caused worse simulator sickness. Therefore, it was a less effective training tool. Application: This research illustrates the importance of conducting empirical evaluations of novel training technologies. New and innovative technologies are always coveted by users, but new does not always guarantee improvement.

Training Effectiveness of Wearable and Desktop Simulator Interfaces

The availability of increasingly advanced simulation interfaces has led researchers to question whether these newer interfaces provide a more effective means of providing simulation-based training. An empirical evaluation was conducted comparing the knowledge gained from training with three different systems: narrated computer animations (Flash videos) that are currently in use in the U.S. Army, an interactive virtual environment presented on a standard desktop PC, and the same virtual environment presented on a wearable simulator with head-mounted display. Results indicated no difference in the knowledge gained from any of the training methods, although the Flash videos were deemed less engaging, enjoyable, and elicited less presence than both of the virtual environment training methods. The wearable simulation interface was also found to cause greater levels of simulator sickness than either the desktop PC or Flash video training methods. The results of the current study show no evidence of a benefit of using the wearable system over more traditional desktop systems.

Loss of Aircrew Situation Awareness: A Cross-Validation

Loss of situation awareness (SA) among aircrews has been cited as a contributing factor in numerous aircraft accidents, although this concept is often difficult to examine in context. For the current study, we analyzed over 300 incident reports filed by commercial aircrews through NASA’s Aviation Safety Reporting System (ASRS). In doing so, we cross-validated the results of a study by Gibson, Orasanu, Veleda, and Nygren (1997), with a different sample of incident reports and across raters. Our sample of reports differed significantly from those used by Gibson et al. with respect to several characteristics (e.g., composition of aircraft types and phase of flight during which SA loss occurred). Although we used different raters and a different sample of reports, we found a surprising similarity between our results and those reported by the previous study, particularly with respect to who lost SA and why. Our study and that by Gibson et al. agreed on who lost SA most often (entire crew, 61% and 63%; Captain only, 26% and 28%; and First Officer only, 13% and 9%, respectively.) Further, both studies showed similar causal factors that contributed to SA loss (high workloaddistractions, lack of crew communicatiodcoordination, improper procedure, and maintenance or equipment malfunctions, in that order). These findings indicate (a) that the study of incident reports for loss of situation awareness can be a reliable technique, and (b) that loss of SA appears to be consistently caused by a limited number of factors. The results may be used to drive more specific studies in the area that should investigate the exact mechanism by which these factors cause loss of SA.

Desktop and Wearable Game-Based Simulations: A Usability Study

Currently, there is considerable interest related to the use of computer games as simulators for training. One of the issues with using computer simulations is that the controls should be simple and natural enough that the trainee can focus on the training rather than on complicated simulator controls. The current research examined two versions of a game based simulator for training Soldier tasks in an urban environment. One version employs a common desktop computer interface, while the second version has the computer components, controls, and the display embedded into Soldier field equipment so that the simulator is wearable by the trainee. A heuristic usability evaluation of the two simulators was conducted to determine the ease of use of the controls and displays. The results found the wearable simulator to have more usability concerns than the desktop computer. These findings can be used to improve future simulator interfaces, particularly those employing novel controls that differ from traditional desktop computer interfaces.

Effect of Audio-Visual Alerts on Situation Awareness and Workload in a Net-Centric Warfare Scenario

The goal of net-centric warfare (NCW) is to give soldiers an information advantage that leads to a war-fighting advantage. However, the inherent nature of NCW systems is often quite complex and dynamic, which leads to impaired situation awareness (SA) and heightened levels of mental workload for the human operator. The following study investigated the moderating effects of automated audio-visual alerts on user SA and perceived workload while using a net-centric warfare system. Twenty-six participants observed battlefield scenarios on a simulation of the common NCW system, Force XXI Battle Command Brigade and Below (FBCB2), and were required to pay attention and remember critical events (e.g., the appearance of an enemy unit). The system was presented with or without an automated alerting aid that provided combined auditory and visual alerts when certain critical events occurred (i.e., System to Help Implement and Empower Leader Decisions; SHIELD). Results revealed that contrary to current assumptions the use of an automated alerting aid did not impact user SA; however, perceived workload was significantly lower with the addition of the alerting aid. This work demonstrates that the automated alerts used in this NCW experiment do not affect SA (either positively or negatively) and decreased perceived workload.