Sandra G. Hart

Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research

Abstract The results of a multi-year research program to identify the factors associated with variations in subjective workload within and between different types of tasks are reviewed. Subjective evaluations of 10 workload-related factors were obtained from 16 different experiments. The experimental tasks included simple cognitive and manual control tasks, complex laboratory and supervisory control tasks, and aircraft simulation. Task-, behavior-, and subject-related correlates of subjective workload experiences varied as a function of difficulty manipulations within experiments, different sources of workload between experiments, and individual differences in workload definition. A multi-dimensional rating scale is proposed in which information about the magnitude and sources of six workload-related factors are combined to derive a sensitive and reliable estimate of workload.

Workload Assessment and Prediction

The concept of workload and its relationship to performance is introduced in this chapter. Four categories of workload measurement techniques (ratings, primary and secondary task measures, and physiological indices) are reviewed, examples of each category are described, and their strengths and weaknesses are summarized. The importance of carefully formulating the question which a measure is to address is emphasized, and it is argued that the question should guide the selection of measures. Issues relevant to implementing and interpreting workload measures are discussed and some of the reasons that different measures provide apparently conflicting information about the same situation (i. e., dissociation) are addressed. Finally, the chapter concludes with a brief description of models that can be used to predict workload.

Measuring pilot workload in a motion base trainer. a comparison of four techniques

Various techniques have been developed to predict and measure pilot workload. This simulation was conducted in order to compare four widely used methods: a visual two-and four-choice reaction time task, time production, retrospective multi-dimensional subjective ratings and in-flight verbal workload estimates. Two scenarios with different levels of difficulty determined by preliminary research were designed to test these techniques. The insertion of the secondary tasks did not significantly affect flight performance. All four techniques were able to distinguish between the overall levels of scenario complexity. In addition, the three secondary tasks and workload ratings obtained in-flight were generally able to distinguish among levels of difficulty for different segments within the scenarios.

Pilot errors as a source of workload

A pilot opinion survey was conducted to develop a database for creating simulation scenarios that impose predetermined levels of pilot workload. Twelve pilots estimated the effect of 163 events and activities (which they had encountered during their previous flying experiences) on performance, effort, workload, and stress. The events, described in the context of flight scenario segments, included control, navigation and communications activities, aircraft and system failures, and pilot errors. In general, workload, stress, and effort ratings were significantly correlated with each other but not with performance ratings; however, some different response patterns were found as a function of flight segment (e.g., workload, stress, and performance, but not effort, ratings varied with flight phase) and type of event. Errors were rated as a significant source of change for workload, stress, and performance, suggesting that errors could be conceptualized as a cause of workload rather than as a symptom.

Field Test of Video Game Trainer

A field study was conducted at the US Army Aviation Center to determine whether workload-coping and attention-management skills developed through structured video game experience would generalize to flight training. Three groups of 24 trainees were compared: (1) One received 10 hours of training on an IBM-PC version of Space Fortress, replicating an earlier study; (2) The second played a commerical video game (Apache Strike) for 10 hours which also required tracking, monitoring, situation assessment, and memory; (3) The third matched group receive no game training. Flight school records were monitored during the next 18 mos to compare performance of the three groups during initial flight training. Check ride ratings began to show an advantage for the group trained with Space Fortress by the Instrument stage of training, as predicted. Furthermore, attrition rates were lower for this group, replicating the results of an earlier study conducted by Gopher (1990) in the Israeli Air Force Flight School.

Defining the subjective experience of workload

Flight scenarios that represent different types and levels of pilot workload are needed in order to conduct research about, and develop measures of, pilot workload. In order to be useful, however, the workload associated with such scenarios and the component tasks must be determined independently. An initial study designed to provide such information was conducted by asking a panel of general aviation pilots to evaluate flight-related tasks for the overall, perceptual, physical, and cognitive workload they impose. These ratings will provide the nucleus for a data base of flight-related primary tasks that have been independently rated for workload to use in workload assessment research.

Inflight Evaluation of Four Measures of Pilot Workload

Four measures of pilot workload were tested in the NASA C-141 Kuiper Airborne Observatory. The measures included a communications analysis, subjective ratings of workload, subjective ratings of additional factors related to workload, and heart rate. Data were collected for 11 flights, each of which lasted approximately seven hours. Heart rate was found to be significantly higher for the pilot who was flying than for the pilots who was not flying and it varied significantly across flight segments, peaking during landing and take off, particularly for the pilot in the left seat who was responsible for aircraft control. For both left and right seats, the subjective assessment of stress rather than the subjective assessment of workload was significantly correlated with variation in heart rate. Frequencies of different types of communications varied significantly across segments of flight, however, they were not correlated with subjective ratings of workload. There was a significant difference between the left and right seats in the types of activities that contributed to their workload, however, workload was considered to be equivalent for the two

Measuring Pilot Workload in a Moving-Base Simulator: I. Asynchronous Secondary Choice-Reaction Task:

The de facto method for measuring airplane pilot workload is based upon subjective ratings. While researchers agree that such subjective data should be bolstered by using objective behavioral measures, results to date have been mixed. No clear objective technique has surfaced as the metric of choice. We believe this difficulty is in part due to neglect of theoretical work in psychology that predicts some of the difficulties that are inherent in a futile search for the one and only best secondary task to measure workload. An initial study that used both subjective ratings and an asynchronous choice-reaction secondary task was conducted to determine if such a secondary task could indeed meet the methodological constraints imposed by current theories of attention. Two variants of a flight scenario were combined with two levels of the secondary task. Appropriate single-task control conditions were also included. Results give grounds for cautious optimism but indicate that future research should use synchronous secondary tasks where possible.

Step tracking shrinking targets

Four models describing how people might acquire targets that dynamically vary in size were examined; two that described movement speed as a simple function of target size (either initial or final) and two that described movement speed as a function of the predicted size of the targets at a fixed time in the future (one was referenced to the beginning of the reaction time phase, and the other to the end of this phase). It was found that movement time was best described as a function of a size prediction made at the end, rather than the start, of the reaction time phase. Subjective workload ratings primarily reflected the total amount of time needed to acquire the targets rather than the time pressure imposed by the diminishing size of these targets.

Decision support for workload assessment - Introducing WC FIELDE

Currently there is a great demand for mental workload evaluation in the course of system design and modification. In light of this demand, a microprocessor-based decision support system has been created called WC FIELDE: Workload Consultant for FIELD Evaluation. The system helps the user select workload measures appropriate to his or her application from the large pool of currently available techniques. Both novices and those with some workload experience may benefit from using WC FIELDE, since the systems operation is entirely transparent and all rules involved in the decision process are available for the user to examine. WC FIELDE recommends several assessment methodologies in decreasing order of appropriateness, and provides additional information on each measure at the end of the program in the form of text files.