Jeffrey T. Hansberger

Improving Rater Calibration in Aviation: A Case Study

The assessment of pilot performance relies on systematic observation and assessment by a single trained rater or instructor/evaluator (I/E). Because of the importance of aviation safety, it is imperative that the rating and evaluation processes used by these I/Es yield reliable and valid data. This article describes a case study focused on improving the reliability and validity of crew assessment at a commercial carrier. The process for improving reliability involved the evaluation of current training methods, the construction and evaluation of 5 metrics for assessing interrater reliability, and a standardized process for using these metrics to train I/Es. A separate set of interventions was developed and implemented to improve structural validity. Data collected from 2 fleets in this airline during a 3-year period indicate aspects of reliability and validity that should be the focal points of continuing I/E training.

Strategy variability: How too much of a good thing can hurt performance

Recent research (e.g., Siegler, 1996) has discovered the important and vital role that variability plays among strategy use and development over time. However, as many researchers have pointed out (e.g., Miller, 1993), the majority of the research addressing this issue has focused on the outcomes, rather than on the potentially more informative aspects of variability, strategy development, and the process of adaptation. In this study, we examined the role of variability during strategy development, utilizing a longitudinal method. Thirteen participants were studied over 3 months as they coached a simulated football team. The results suggest that variability plays a major role in adaptive skill acquisition in a dynamic environment—in the direction, however, opposite to that predicted by previous research done with simple static tasks.

Using Cognitive Modeling to Understand Crew Behavior

The ACT-R cognitive architecture has been used successfully to model aviation crew performance during the descent phase of flight. This study extended that approach by simulating the duties of a pilot flying (PF) and Pilot Not Flying (PNF) in separate ACT-R models which then were run jointly across several executions. For each set of executions, level of expertise and taskload of the PF and PNF were varied. The models were further tested by implementing interruptions through Air Traffic Control (ATC) calls. The results showed that both expertise and taskload affected performance. Further, ATC interruptions produced crew miscommunication, differential situation awareness, and forgetting relevant goals.

The Implications of Goals and Experience for Driving Expertise

Many questions remain unanswered concerning how individuals obtain and acquire expertise in dynamic task domains. This study investigates how the type of goals used in driving situations affect driving experience and expertise. Past research suggests that the quantity of experience alone is not sufficient to accurately predict driving expertise. It is hypothesized that the type of goals utilized during this experience influences driving expertise. The measures used to assess driving expertise uncovered four distinct facets of expertise, self-reported vehicle control and visual scan patterns, driving knowledge, and decision-making. Each of the driving expertise facets was found to possess a different relationship with experience and goal type. Most notable was the relationship of experience with driving knowledge, moderated by learning goal orientation. Further research could have implications for training in the driving domain as well as other dynamic task domains.

The Effects of Prior Knowledge on Goal Variability & Learning: The More Goals the Merrier

Prior knowledge and experience influence many of our everyday task interactions and knowledge acquisition. One way this happens is by goal choice and goal-oriented behavior. This study focused on the effects of goal variability with repeated interactions with a complex, dynamic task over several months. The effect of goal variability on knowledge acquisition mechanisms and learning of task domain knowledge was observed through the use of the mircogenetic approach. Prior knowledge had strong implications for the profile of active goals, interactions with the task and ultimately, learning. Goal variability influenced the use of specific knowledge-acquisition mechanisms. By the final session, goal variability, regardless of the content of the goals, heavily influenced increments in domain knowledge. Variability may have beneficial effects when applied to goal selection, knowledge-acquisition mechanisms, task strategies, and learning over time. Implications of this study may extend to training and other learning and teaching environments.

Proceduralizing Teamwork in Aviation Crews

This project investigated whether appropriate procedural interventions can facilitate team process and outcomes. Specific crew procedures were designed to fit general Crew Resource Management (CRM) principles and the specific needs of a commercial airline. The procedures were trained and implemented as Standard Operating Procedure (SOP) for one fleet while a control fleet did not have the training or SOP interventions. Multiple evaluation methods used different samples of evaluators and different evaluation instruments. Evaluation data converged to confirm the positive effects of the interventions in the experimental fleet.

Cognitive Modeling of Airline Crew Automation Errors

Cockpit automation has changed the roles, responsibilities, and activities of pilots, leading to new types of errors on the flight deck. This research is focused on understanding those errors through the development of a computational cognitive model that describes how pilots interact with automated systems. The cognitive model under development is based on a cognitive task analysis supplemented with eye tracking data collected from commercial pilots flying a low-fidelity simulator.