Brian F. Gore

Identifying Black Swans in NextGen: Predicting Human Performance in Off-Nominal Conditions

Objective: The objective is to validate a computational model of visual attention against empirical data—derived from a meta-analysis—of pilots’ failure to notice safety-critical unexpected events. Background: Many aircraft accidents have resulted, in part, because of failure to notice nonsalient unexpected events outside of foveal vision, illustrating the phenomenon of change blindness. A model of visual noticing, N-SEEV (noticing— salience, expectancy, effort, and value), was developed to predict these failures. Method: First, 25 studies that reported objective data on miss rate for unexpected events in high-fidelity cockpit simulations were identified, and their miss rate data pooled across five variables (phase of flight, event expectancy, event location, presence of a head-up display, and presence of a highway-in-the-sky display). Second, the parameters of the N-SEEV model were tailored to mimic these dichotomies. Results: The N-SEEV model output predicted variance in the obtained miss rate (r = .73). The individual miss rates of all six dichotomous conditions were predicted within 14%, and four of these were predicted within 7%. Conclusion: The N-SEEV model, developed on the basis of an independent data set, was able to successfully predict variance in this safety-critical measure of pilot response to abnormal circumstances, as collected from the literature. Applications: As new technology and procedures are envisioned for the future airspace, it is important to predict if these may compromise safety in terms of pilots’ failing to notice unexpected events. Computational models such as N-SEEV support cost-effective means of making such predictions.

Modeling Pilot Situation Awareness

Introduction The Man–machine Integration Design and Analysis (MIDAS) human performance model was augmented to improve predictions of multi-operator situation awareness (SA). In MIDAS, the environment is defined by situation elements (SE) that are processed by the modeled operator via a series of sub-models including visual attention, perception, and memory. Collectively, these sub-models represent the situation assessment process and determine which SEs are attended to, and comprehended by, the modeled operator. SA is computed as a ratio of the Actual SA (the number of SEs that are detected or comprehended) to the Optimal SA (the number of SEs that are required or desired to complete the task).

A Computational Implementation of a Human Attention Guiding Mechanism in MIDAS v5

In complex human-machine systems, the human operator is often required to intervene to detect and solve problems. Given this increased reliance on the human in these critical human-machine systems, there is an increasing need to validly predict how operators allocate their visual attention. This paper describes the information-seeking (attention-guiding) model within the Man-machine Integration Design and Analysis System (MIDAS) v5 software - a predictive model that uses the Salience, Effort, Expectancy and Value (SEEV) of an area of interest to guide a persons attention. The paper highlights the differences between using a probabilistic fixation approach and the SEEV approach in MIDAS to drive attention.

Man-machine Integration Design and Analysis System (MIDAS) v5: Augmentations, Motivations, and Directions for Aeronautics Applications

As automation and advanced technologies are introduced into transport systems ranging from the Next Generation Air Transportation System termed NextGen, to the advanced surface vehicle Intelligent Transportations Systems, to future systems designed for space exploration, there is an increased need to validly predict how the future systems will be vulnerable to error given the demands imposed by assisted technologies. One formalized method to study the impact of assisted technologies on the human operator in a safe and non-obtrusive manner is through the use of human performance models (HPMs). HPMs play an integral role when complex human–system designs are proposed, developed, and tested. One HPM tool termed the Man–machine Integration Design and Analysis System (MIDAS) is a NASA Ames Research Center HPM software tool that has been applied to predict human–system performance in various domains since 1986. MIDAS is a dynamic, integrated HPM environment that facilitates the design, visualization, and computational evaluation of complex man–machine system concepts in simulated operational environments. A range of aviation specific applications including an approach used to model human error for NASA’s Aviation Safety Program, and “what-if” analyses to evaluate flight deck technologies for NextGen operations will be discussed. This chapter will culminate by raising two challenges for the field of predictive HPMs for complex human–system designs that evaluate assisted technologies: that of (1) model transparency and (2) model validation.

Human performance cognitive-behavioral modeling: a benefit for occupational safety

Human Performance Modeling (HPM) is a computer-aided job analysis software methodology used to generate predictions of complex humanautomation integration and system flow patterns with the goal of improving operator and system safety. The use of HPM tools has recently been increasing due to reductions in computational cost, augmentations in the tool’s fidelity, and usefulness in the generated output. An examination of an Air Man-machine Integration Design and Analysis System (Air MIDAS) model evaluating complex human-automation integration currently underway at NASA Ames Research Center will highlight the importance to occupational safety of considering both cognitive and physical aspects of performance when researching human error.

Risk assessment and human performance modelling: the need for an integrated systems approach

One of the difficulties in completing scientific experiments aboard the International Space Station (ISS) is anticipating all of the effects of the operational environment on the procedural sequences. Human Performance Modelling (HPM) and Probabilistic Risk Assessment (PRA)-like approaches could together generate such predictions for the space life sciences experimentation application domain. An informal risk assessment was completed through Subject Matter Expert (SME) interviews on procedural completion and potential for erroneous performance. The results suggested there would be significant differences in the performance and risk of completing a series of experiments sequentially as compared to completing the experiments in parallel. As a result, an initial HPM was developed to test predictions of simulated operator workload for a complex space-related biological experiment and the risk of error. Two procedural sequences are analysed, which highlight different human performance profiles that raise risks, or vulnerabilities, in physical, cognitive and psychomotor performance and task times. Future efforts need to build on these findings to include a comprehensive PRA process computationally linked to performance predictions generated from HPMs.

Meaningful Assessments of Simulator Performance and Sickness: Can't Have One without the Other?

The requirement to evaluate the differential impacts of simulator sickness on performance assessments was explored. Simulator sickness and performance data were analyzed in two phases that indicated: 1) an experimental display condition by age interaction with regard to development of simulator sickness; and 2) associated detrimental effects of simulator sickness on performance. Arguably, these results may be quite disturbing to users, and past users, of simulators for system and other Development, Test and Evaluation efforts (DT&Es). The utility of simulator sickness measures as covariates, in the analysis of performance effects, is demonstrated as a means for their assessment and statistical control. It is strongly recommended that researchers explore and control the potential confounding effects of simulator sickness in order to assure meaningful performance assessments.

The Study of Distributed Cognition in Free Flight: A Human Performance Modeling Tool Structural Comparison

The Requirements and Technical Concepts for Aviation, Inc. (RTCA) has recently proposed a new concept known as “free flight” for guiding the separation of aircraft in the National Airspace System (NAS). “Free flight” in the United States is a Federal Aviation Administration (FAA) strategic goal for system capacity and for Air Traffic Services to improve accessibility, flexibility, and predictability in the national airspace in order to reduce flight times, crew resources, maintenance, and fuel costs. The scenarios in the current experiment were used to explore the farthest out parameters of “free flight” anticipated by RTCA in the year 2025. An evaluation of predicted scenario generation structures associated with “current day” and “free flight” operations under a specific experimental scenario was performed using two integrated human performance modeling tools, Air Man-machine Integration Design and Analysis System (Air MIDAS) and the Integrated Performance Modeling Environment (IPME). In analyses of a common scenario, each tool predicted a different Point of Closest Approach (PCA) distance under “current day” operations ( M Air MIDAS = 5.86 nm, M IPME = 16.67 nm) as compared with “free flight” operations ( M Air MIDAS = 6.65 nm, M IPME = 4.94 nm). This indicates that differences in the proximity between the aircraft existed between the two software tools as well as the contextually sensitive variables. This provides evidence that more focus needs to be brought to the effects of contextual effects that impact the virtual human operator of the NAS. A validation effort of these contextual findings with human-in-the-loop (HITL) data is anticipated and required prior to drawing any definite conclusions.

A methodical approach for developing valid human performance models of flight deck operations

Validation is critically important when human performance models are used to predict the effect of future system designs on human performance. A model of flight deck operations was validated using a rigorous, iterative, model validation process. The process included the validation of model inputs (task trace and model input parameters), process models (workload, perception, and visual attention) and model outputs of human performance measures (including workload and visual attention). This model will be used to evaluate proposed changes to flight deck technologies and pilot procedures in the NextGen Closely Spaced Parallel Operations concept.

Modeling Operator Performance and Cognition in Robotic Missions

Control of the robotic arm on the International Space Station is a challenging endeavor, not only due to the high consequence of failure, but also because the limited number and arrangement of cameras greatly increases the difficulty of maneuvering the arm. There is great potential for automation to reduce such effort, but developing the right kind and degree of automation is a key concern. Mismatches between the perspective of the operator and the view of the robotic arm, and between the direction of control and response of the arm, contribute to performance degradations. In this paper we describe the development of a computational structure that combines a set of existent human performance modules to address such issues. These modules include the Frame of Reference Transformation (FORT), the Basic Operational Robotic Instructional System (BORIS), the Man-machine Integration Design and Analysis System (MIDAS v5), and the Salience, Effort, Expectancy, and Value (SEEV) attention model as applied in a simulation model of a robotic operator termed MORRIS.