Mark S. Crabtree

Subjective Workload Assessment in a Memory Update Task

Twelve subjects performed a short-term memory task under several difficulty levels and rated the workload associated with each condition using the Subjective Workload Assessment Technique (SWAT). SWAT ratings proved more sensitive than memory error to task difficulty variations in all but one of the most difficult memory conditions. Most importantly, SWAT ratings demonstrated their greatest relative sensitivity at the lowest levels of workload. The results are interpreted as supporting the applicability of SWAT as a sensitive workload index.

The Effect of Delayed Report on Subjective Ratings of Mental Workload

Forty-eight subjects performed a short-term memory task with several difficulty levels and provided either immediate or delayed ratings of workload via the Subjective Workload Assessment Technique (SWAT). Mean SWAT ratings did not vary significantly as a function of delayed report, but a substantial number of subjects gave delayed ratings that were discrepant from their immediate ratings. A counterbalancing effect in delayed ratings appears to have been a factor in the failure of the delay effect to reach significance. A secondary objective of this study was to examine the sensitivity of SWAT in a between-subjects design. SWAT ratings varied significantly as a function of task difficulty manipulations, supporting the sensitivity of SWAT to the workload of the conditions used.

Workload Measurement in System Design and Evaluation

Because of its central role in system development, workload measurement has been extensively researched. These efforts have produced a variety of workload assessment techniques, many of which can be classified as either subjective, physiological, or behavioral measures. These categories of measures can vary along several dimensions that can be used as criteria in selection of a technique for a particular application. The proposed selection criteria include the sensitivity, diagnosticity, and intrusiveness associated with a technique. Different stages of system design can require techniques that differ on the noted dimensions. Since no technique is capable of meeting all of the applicable criteria, a comprehensive approach to workload assessment will require a battery of subjective, physiological, and behavioral measures. Future research dealing with comparative evaluation of the various assessment techniques along the noted dimensions will be required in order to refine workload metric selection criteria.

The Effect of Intervening Task Performance on Subjective Workload Ratings

Thirty subjects performed a short-term memory task and used the Subjective Workload Assessment Technique (SWAT) to provide workload ratings under one of five conditions. Ratings were provided either immediately following task performance, after a delay period during which no additional tasks were performed, or after a delay period during which an additional set of memory tasks at one of three levels of difficulty was performed. Neither the delay interval nor the requirement to perform a set of intervening tasks significantly affected mean SWAT ratings relative to the immediate rating control condition. Patterns in the data suggested that performance of a set of difficult intervening tasks had the greatest tendency to affect memory task ratings, and indicate that the potential influence of intervening task performance should not be completely discounted in workload rating scale applications.

Standardized Tests for the Evaluation and Classification of Workload Metrics

Although a variety of metrics are available for workload assessment, little research has been conducted to construct guidelines for optimally matching them to individual workload questions or test environments. In an effort to remedy this problem, AFAMRL is currently developing a standardized methodology which will be used to evaluate candidate workload measures on several theoretical and pragmatic criteria. The central feature of this methodology is a set of representative loading tasks. This paper outlines the analyses that were conducted to screen potential components of the Criterion Task Set (CTS) and describes the tasks which were selected to produce demands on a number of functional dimensions of information processing.

Benefits of Using Objective and Subjective Workload Measures

This paper describes the results of two measures of workload, one objective, one subjective, that were applied to a series of switch setting exercises. The objective measure was an Interval Production Task (IPT), based upon a secondary task paradigm, in which the subject is required to maintain a constant rate of tapping with one hand while performing the primary task with the other hand. According to the supporting theory, variations in workload on the primary task will be reflected in variations in the tapping rate. Previous studies have found the IPT to be particularly sensitive to changes in the psychomotor workload. The subjective measures used were the Subjective Workload Assessment Technique (SWAT). Although the two workload measures did not produce the same level of significance for workload differences, it should be noted that they did produce the same rank order for the three tasks. It was concluded that the use of both subjective and objective measures of workload could produce increased confidence in the results, as well as insight into the nature of the task loading.

Methods and Systems for Measuring Human Performance Capabilities

Human factors specialists require practical efficient data collection technologies which can be used to answer questions about the performance capabilities of the human operator. This paper considers several factors which contribute to the interpretability, sensitivity, and practical utility of standardized task batteries designed to assess the effects on human performance of work environments, stressors, biochemically active agents and performance enhancement interventions. The procedures employed in the development of a task battery by the U.S. Air Force Aerospace Medical Research Laboratory are used to illustrate both the theoretical and pragmatic issues which are discussed.

Quantification of Crew Workload Imposed by Communications-Related Tasks in Commercial Transport Aircraft*

Information theoretic analysis and subjective paired-comparison and task ranking techniques were employed in order to scale the workload of 20 communications-related tasks frequently performed by the captain and first officer of transport category aircraft. Tasks were drawn from taped conversations between aircraft and air traffic controllers (ATC). Twenty crewmembers performed subjective message comparisons and task rankings on the basis of workload. Information theoretic results indicated a broad range of task difficulty levels, and substantial differences between captain and first officer workload levels. Preliminary subjective data tended to corroborate these results. A hybrid scale reflecting the results of both the analytical and the subjective techniques is currently being developed. The findings will be used to select representative sets of communications for use in high fidelity simulation.

Workload and Performance: Orthogonal Measures

During an attempt to corroborate historical error data with recently gathered data on subjective workload values, it was noted that the data not only failed to show any degree of correlation, but gave strong evidence to support arguments that the workload and error measures are orthogonal (totally independent). This paper discusses several explanations of the observed phenomenon. A hypothesis is advanced that error free performance is strongly dependent upon task procedure construction, while workload is dependent upon subtask difficulty and situational stresses.