Shaun Hutchins

False Alerts in Air Traffic Control Conflict Alerting System: Is There a “Cry Wolf” Effect?

Objective: The aim is to establish the extent to which the high false-alarm rate of air traffic control midair conflict alerts is responsible for a “cry wolf” effect—where true alerts are not responded to and all alerts are delayed in their response. Background: Some aircraft collisions have been partly attributed to the cry wolf effect, and in other domains (health care and systems monitoring), there is a causal connection between false-alarm rate and cry wolf behavior. We hypothesized that a corresponding relationship exists in air traffic control (ATC). Method: Aircraft track and alert system behavior data surrounding 495 conflict alerts were analyzed to identify true and false alerts, trajectory type, and controller behavior. Forty-five percent of the alerts were false, ranging from 0.28 to 0.58. Results: Although centers with more false alerts contributed to more nonresponses, there was no evidence that these were nonresponses to true alerts or that response times were delayed in those centers. Instead, controllers showed desirable anticipatory behavior by issuing trajectory changes prior to the alert. Those trajectory pairs whose conflicts were more difficult to visualize induced more reliance on, and less compliance with, the alerting system. Conclusion: The high false-alarm rate does not appear to induce cry wolf behavior in the context of en route ATC conflict alerts. Application: There is no need to substantially modify conflict alert algorithms, but the conflict alert system may be modified to address difficult-to-visualize conflicts.

Effectiveness of Part-Task Training and Increasing-Difficulty Training Strategies A Meta-Analysis Approach

Objective: The objective was to conduct meta-analyses that investigated the effects of two training strategies, increasing difficulty (ID) and part-task training (PTT), on transfer of skills and the variables that moderate effectiveness of the strategies. Background: Cognitive load theory (CLT) provides a basis for predicting that training strategies reducing the intrinsic load of a task during training avail more resources to be devoted to learning. Two strategies that accomplish this goal, by dividing tasks in parts or by simplifying tasks in early training trials, have offered only mixed success. Method: A pair of complementary effect size measures were used in the meta-analyses conducted on 37 transfer studies employing the two training strategies: (a) a transfer ratio analysis on the ratio of treatment transfer performance to control transfer performance and (b) a Hedges’ g analysis on the standardized difference between treatment and control group means. Results: PTT generally produced negative transfer when the parts were performed concurrently in the whole transfer task but not when the parts were performed in sequence. Variable-priority training of the whole task was a successful technique. ID training was successful when the increases were implemented adaptively but not when increased in fixed steps. Both strategies provided evidence that experienced learners benefited less, or suffered more, from the strategy, consistent with CLT. Conclusion: PTT can be successful if the integrated parts are varied in the priority they are given to the learner. ID training is successful if the increases are adaptive. The fundamental elements of CLT are confirmed.

The Impact of Sleep Disruption on Complex Cognitive Tasks A Meta-Analysis

Objective: We aimed to build upon the state of knowledge about the impacts of sleep disruption into the domain of complex cognitive task performance for three types of sleep disruption: total sleep deprivation, sleep restriction, and circadian cycle. Background: Sleep disruption affects human performance by increasing the likelihood of errors or the time it takes to complete tasks, such as the Psychomotor Vigilance Task. It is not clear whether complex tasks are affected in the same way. Understanding the impact of sleep disruption on complex cognitive tasks is important for, and in some instances more relevant to, professional workers confronted with unexpected, catastrophic failures following a period of disrupted sleep. Method: Meta-analytic review methods were applied to each of the three different areas of sleep disruption research. Results: Complex cognitive task performance declines over consecutive hours of continuous wakefulness as well as consecutive days of restricted sleep, is worse for severely restricted sleep (4 or fewer hours in bed), is worse during the circadian nadir than apex, and appears less degraded than simple task performance. Conclusion: The reviews suggest that complex cognitive task performance may not be impacted by disrupted sleep as severely as simple cognitive task performance. Application: Findings apply to predicting effects of sleep disruption on workers in safety-critical environments, such as health care, aviation, the military, process control, and in particular, safety-critical environments involving shiftwork or long-duration missions.

Supporting Interruption Management and Multimodal Interface Design Three Meta-Analyses of Task Performance as a Function of Interrupting Task Modality

Objective: The aim of this study was to integrate empirical data showing the effects of interrupting task modality on the performance of an ongoing visual-manual task and the interrupting task itself. The goal is to support interruption management and the design of multimodal interfaces. Background: Multimodal interfaces have been proposed as a promising means to support interruption management. To ensure the effectiveness of this approach, their design needs to be based on an analysis of empirical data concerning the effectiveness of individual and redundant channels of information presentation. Method: Three meta-analyses were conducted to contrast performance on an ongoing visual task and interrupting tasks as a function of interrupting task modality (auditory vs. tactile, auditory vs. visual, and single modality vs. redundant auditory-visual). In total, 68 studies were included and six moderator variables were considered. Results: The main findings from the meta-analyses are that response times are faster for tactile interrupting tasks in case of low-urgency messages. Accuracy is higher with tactile interrupting tasks for low-complexity signals but higher with auditory interrupting tasks for high-complexity signals. Redundant auditory-visual combinations are preferable for communication tasks during high workload and with a small visual angle of separation. Conclusion: The three meta-analyses contribute to the knowledge base in multimodal information processing and design. They highlight the importance of moderator variables in predicting the effects of interruption task modality on ongoing and interrupting task performance. Applications: The findings from this research will help inform the design of multimodal interfaces in data-rich, event-driven domains.

Auditory-Visual Redundancy in Vehicle Control Interruptions Two Meta-analyses

Two novel versions of a meta analysis were employed to assess the conditions of ongoing vehicle control task simulations in which (1) auditory presentation of an interrupting task were beneficial over visual presentations and (2) redundant (av) presentation was better than single modality presentation (providing redundancy gain). Altogether 29 studies were identified. The results revealed that the interrupting task benefited from auditory presentation, but the ongoing task (visual vehicle control task) generally did not. Performance of the visual interrupting task was slightly hindered by separation from the ongoing task. The redundancy analysis revealed that the interrupting task benefited from redundancy when it involved spatial localization and alerting and the accuracy of verbal communications; but suffered when speed of the verbal communications response was measured, and when the two visual channels were separated. Implications for multi-modal presentation of information on vehicle workstations are discussed.

Costs and benefits of more learner freedom: meta-analyses of exploratory and learner control training methods.

Objective: Individual meta-analyses were conducted for six training methods as part of a U.S. Army basic research project. The objective was to identify evidence-based guidelines for the effectiveness of each training method, under different moderating conditions, for cognitive skill transfer in adult learning. Results and implications for two of these training methods, learner control (LC) and exploratory learning (EL), are discussed. LC provides learners with active control over training variables. EL requires learners to discover relationships and interactions between variables. Background: There is mixed evidence on the effectiveness of both LC and EL learning methods on transfer relative to more guided training methods. Cognitive load theory (CLT) provides a basis for predicting that training strategies that manage intrinsic load of a task during training and minimize extraneous load will avail more resources that can be devoted to learning. Method: Meta-analyses were conducted using a Hedges’s g analysis of effect sizes. Control conditions with little to no learner freedom were contrasted with treatment conditions manipulating more learner freedom. Results: Overall more LC was no different from training with limited or no learner control, and more EL was less effective than limited or no exploration; however, each can be effective under certain conditions. Both strategies have been more effective for cognitive skill learning than for knowledge recall tasks. LC exhibited more benefit to very near transfer, whereas EL’s benefit was to far transfer. Conclusion: Task type, transfer test, and transfer distance moderate the overall transfer cost of more learner freedom. Application: The findings are applicable to the development of instructional design guidelines for the use of LC and EL in adult skill training.

The Influence of Cognitive Load on Transfer With Error Prevention Training Methods A Meta-Analysis

Objective: The objective was to conduct research synthesis for the U.S. Army on the effectiveness of two error prevention training strategies (training wheels and scaffolding) on the transfer of training. Background: Motivated as part of an ongoing program of research on training effectiveness, the current work presents some of the program’s research into the effects on transfer of error prevention strategies during training from a cognitive load perspective. Based on cognitive load theory, two training strategies were hypothesized to reduce intrinsic load by supporting learners early in acquisition during schema development. Method: A transfer ratio and Hedges’ g were used in the two meta-analyses conducted on transfer studies employing the two training strategies. Moderators relevant to cognitive load theory and specific to the implemented strategies were examined. The transfer ratio was the ratio of treatment transfer performance to control transfer. Hedges’ g was used in comparing treatment and control group standardized mean differences. Both effect sizes were analyzed with versions of sample weighted fixed effect models. Results: Analysis of the training wheels strategy suggests a transfer benefit. The observed benefit was strongest when the training wheels were a worked example coupled with a principle-based prompt. Analysis of the scaffolding data also suggests a transfer benefit for the strategy. Conclusion: Both training wheels and scaffolding demonstrated positive transfer as training strategies. As error prevention techniques, both support the intrinsic load–reducing implications of cognitive load theory. Application: The findings are applicable to the development of instructional design guidelines in professional skill-based organizations such as the military.

Investigating the Impact of Training on Transfer A Meta-Analytic Approach

Do training methods differentially impact task performance? This paper discusses a set of meta-analytic studies being conducted as part of an ongoing research project to identify evidence-based guidelines for training the complex cognitive skills involved in technology related task domains. Given an emphasis on complex cognitive tasks and training transfer, our current focus is on training methods for effectively managing learner effort and workload during training. In this paper we identify six prominent training methods and present a deeper look at the results for two different meta-analysis strategies. Small to moderate overall beneficial effects were found for three of the six training strategies (adaptive training, error prevention (training wheels) and scaffolding). No benefit was found for learner control and exploratory learning. An overall cost for part task training was associated with its failure to train a time sharing skill. The modest benefit for error prevention was amplified for cognitive skills.

The Presentation of Risk and Uncertainty in the Context of National Missile Defense Simulations

Risk perception and uncertainty management are important components of military decision making, especially in time-stressed and resource-limited environments. The purpose of this experiment was to understand the interaction of integrality of information, presentation mode, and information frame on situation awareness (SA) and decision-making (missile allocation) in a National Missile Defense (NMD) paradigm. Results of the information frame manipulation (expected gain v. expected loss) support earlier findings that subjects are loss averse. SA Accuracy was higher with graphical displays than alphanumeric displays. The implications for NMD are discussed.

Systematic analysis of risk visualization strategies for homeland defense

An approach to conduct systematic analysis of risk visualization strategies in a homeland missile defense context is presented. The approach consists of three phases which take different perspectives toward characterizing the problems of resource management and allocation for defense against missile attacks. Decision makers in missile defense are supervisory controllers of highly automated and resource-constrained systems. Our approach is suggested as a method to assess the effectiveness of visualization strategies in a dynamic decision environment. The three phases of our approach included a test for subject utility preferences, a text for subject multi-attribute prefences using the Analytic Hierarchy Process (AHP), and a test for subject action selection using a human-in-the-loop simulation test bed. Our approach is proposed to systematically evaluate the effectiveness of multistage decision making strategies against immediate and delayed threats. The alternatives were calculated based on expected cases of lives lost as a result of attacks on defended areas. Implementation of the test bed enabled us to investigate the effect of visualization formats on decision behaviour. Some preliminary results based on a pilot study will be presented. The implications of our study extend beyond missile defense. In a world where catastrophic loss of civilian lives is seen by some as a victory, those entrusted with homeland security must effectively manage limited resources to combat terrorist activities. Effective visualization formats will help to promote decision that are informed with respect to the risks and outcomes.