Benoît R. Vallières

Nonexplicit Change Detection in Complex Dynamic Settings What Eye Movements Reveal

Objective: We employed a computer-controlled command-and-control (C2) simulation and recorded eye movements to examine the extent and nature of the inability to detect critical changes in dynamic displays when change detection is implicit (i.e., requires no explicit report) to the operator’s task. Background: Change blindness—the failure to notice significant changes to a visual scene—may have dire consequences on performance in C2 and surveillance operations. Method: Participants performed a radar-based risk-assessment task involving multiple subtasks. Although participants were not required to explicitly report critical changes to the operational display, change detection was critical in informing decision making. Participants’ eye movements were used as an index of visual attention across the display. Results: Nonfixated (i.e., unattended) changes were more likely to be missed than were fixated (i.e., attended) changes, supporting the idea that focused attention is necessary for conscious change detection. The finding of significant pupil dilation for changes undetected but fixated suggests that attended changes can nonetheless be missed because of a failure of attentional processes. Conclusion: Change blindness in complex dynamic displays takes the form of failures in establishing task-appropriate patterns of attentional allocation. Application: These findings have implications in the design of change-detection support tools for dynamic displays and work procedure in C2 and surveillance.

Decision support and vulnerability to interruption in a dynamic multitasking environment

Using a microworld simulation of maritime decision making, we compared two decision support systems (DSS) in their impact upon recovery from interruption. The Temporal Overview Display (TOD) and Change History Table (CHT) - designed to support temporal awareness and change detection, respectively - have previously proven useful in improving situation awareness; however, evaluation of support tools for multitasking environments should not be limited to the specific aspects of the task that they were designed to augment. Using a combination of performance, self-report, and eye-tracking measures, we find that both DSS counter-intuitively have a negative effect on performance. Resumption lags are increased, elevated post-interruption decision-making times persist for longer, and defensive effectiveness is impaired relative to No-DSS. Eye-tracking measures indicate that in the baseline condition, participants tend to encode the visual display more broadly, where as those in the two DSS conditions may have experienced a degree of attentional tunnelling due to high workload. We suggest that for a support tool to be beneficial it should ease the burden on attentional resources so that these can be used for reconstructing a mental model of the post-interruption scene. We use a maritime microworld to study multitasking and interruption recovery.We use a holistic approach in the assessment of two DSS.We use a combination of performance, self-report and eye tracking measures.Both DSS impair defensive effectiveness and prolong interruption recovery.

Supporting situation awareness: A tradeoff between benefits and overhead

The prevalence of surveillance and information collection technologies provides decision-makers with greater volume and complexity of information to monitor and on which to base decisions than ever before. In this ever increasing dynamic and information rich environment, the role for decision support systems (DSS) to augment cognition and situation awareness (SA) is becoming crucial. However, unless a better understanding is gained of the factors that promote SA without interfering with other critical cognitive functions, the design and development of such technology may serve only to exacerbate rather than enhance the desired effect. The present study investigates how DSS designed to support particular aspects of SA may affect task performance. In the context of a functional computer-controlled simulation of single ship naval anti-air warfare, a baseline condition was compared to two conditions in which a DSS was integrated to the original interface to support different facets of SA. Participants in the two DSS conditions showed an increased SA level, as measured by the QUASA technique, compared to those in the control condition. Despite this benefit, the two DSS actually lead to a reduced performance, as indexed by defense effectiveness. These findings suggest that the benefits of DSS in terms of SA may be accompanied by an overhead with adverse effects on task performance, particularly in situations of high cognitive load and time constraints. This calls for more holistic evaluations of the cognitive impacts of decision support technologies and the development of methods to simultaneously address competing constraints when designing DSS.

Supporting dynamic change detection: using the right tool for the task

Detecting task-relevant changes in a visual scene is necessary for successfully monitoring and managing dynamic command and control situations. Change blindness—the failure to notice visual changes—is an important source of human error. Change History EXplicit (CHEX) is a tool developed to aid change detection and maintain situation awareness; and in the current study we test the generality of its ability to facilitate the detection of changes when this subtask is embedded within a broader dynamic decision-making task. A multitasking air-warfare simulation required participants to perform radar-based subtasks, for which change detection was a necessary aspect of the higher-order goal of protecting one’s own ship. In this task, however, CHEX rendered the operator even more vulnerable to attentional failures in change detection and increased perceived workload. Such support was only effective when participants performed a change detection task without concurrent subtasks. Results are interpreted in terms of the NSEEV model of attention behavior (Steelman, McCarley, & Wickens, Hum. Factors 53:142–153, 2011; J. Exp. Psychol. Appl. 19:403–419, 2013), and suggest that decision aids for use in multitasking contexts must be designed to fit within the available workload capacity of the user so that they may truly augment cognition.

Supporting Change Detection in Complex Dynamic Situations: Does the CHEX Serve its Purpose?

Change detection is required in monitoring and managing complex situations such as air traffic control. Considering that change blindness—the incapacity to detect changes in a visual scene—is a considerable source of human errors and that most studies on change detection have involved static visual scenes, it is crucial to evaluate existing tools designed to help this cognitive function in complex dynamic situations. The goal of the present study was to determine the efficacy of the Change History Explicit (CHEX)—a tool already proven effective when explicit change detection is the only task to execute—when change detection is implicit and intrinsic to a more complex task. Results revealed that the CHEX failed to improve implicit change detection when this task was embedded in a threat-evaluation and weapon-assignment (TEWA) task. Moreover, TEWA performance was hindered and mental workload was perceived as higher when the CHEX was available. Even when the information load imposed by the CHEX was reduced, the tool remained ineffective. This suggests that the nature of the change detection task should be taken into account when designing a decision support system.

The CSSS Microworld: A Gateway to Understanding and Improving CCTV Security Surveillance

The CSSS microworld simulates the task of a closed-circuit television (CCTV) operator responsible for monitoring multiple CCTV feeds in the context of security surveillance. Operators must manage the display of multiple CCTV feeds, monitor the feeds for critical incidents, and then report detected incidents. The microworld can be used for human factors research, interface design, training and personnel selection, and systems engineering. We present a use case of the CSSS microworld to identify the best predictors of CCTV performance. Our results show that the Automated Operation Span and inattentional blindness tests can predict both CSSS detection rate and false alarms, suggesting that these instruments have the potential for quickly assessing the surveillance competency of candidates. This use case illustrates how the CSSS platform may prove to be useful in the selection of personnel for CCTV operator roles.

Pupil Dilation and Eye Movements Can Reveal Upcoming Choice in Dynamic Decision-Making

In dynamic environments such as air-traffic control, emergency response and security surveillance, there are severe constraints to information processing and decision-making. Human operators must constantly monitor, assess, and integrate incoming information in order to make optimal decisions in such complex environments. In order to maximize operators’ performance, there is a need for effective technological support for dynamic decision-making. Eye tracking is one promising avenue that can provide online, non-obtrusive indices of cognitive functioning. Using a simulated maritime decision-making environment, we evaluated whether oculometry may be exploited to foretell the decision made by the operator beforehand. Our results showed that pupil dilation and fixation transitions can reveal the upcoming judgment of the human operator by about half a second before the decision. This finding can be useful to design adaptive support tools for dynamic decision-making by integrating the operator’s cognitive state.

Insights from Eye Movement into Dynamic Decision-Making Research and Usability Testing

This study shows how the use of various measures of eye movement can serve to portray dynamic decision-making (DDM) in a coherent fashion. We extracted eye movement metrics relative to 1) scanpath, 2) eye fixations, and 3) pupillary response, to characterize DDM during the process of risk assessment. Results from Experiment 1 revealed that incorrect classifications were associated with 1) less efficient information search, 2) difficulties in making sense of critical information, and 3) a low level of cognitive load. In Experiment 2, we used eye tracking to assess the impact on DDM of introducing a decision support system. The addition of a temporal-overview display seems to affect processing time in DDM as indexed by shorter scanpaths and fixations during classifications. These findings illustrate how event-based eye movement measures can reveal characteristics and limitations of the ongoing cognitive processing involved in DDM and also contribute to usability testing.

The Benefits and the Costs of Using Auditory Warning Messages in Dynamic Decision Making Settings

The failure to notice critical changes in both visual and auditory scenes may have important consequences for performance in complex dynamic environments, especially those related to security, such as aviation, surveillance during major events, and command and control of emergency response. Previous work has shown that a significant number of situation changes remain undetected by operators in such environments. In the current study, we examined the impact of using auditory warning messages to support the detection of critical situation changes and to a broader extent the decision making required by the environment. Twenty-two participants performed a radar operator task involving multiple subtasks while detecting critical task-related events that were cued by a specific type of audio message. Results showed that about 22% of the critical changes remained undetected by participants, a percentage similar to that found in previous work using visual cues to support change detection. However, we found that audio messages tended to bias threat evaluation toward perceiving objects as more threatening than they were in reality. Such findings revealed both benefits and costs associated with using audio messages to support change detection in complex dynamic environments.

Comparing Naval Decision Support Technologies Using Decision Models, Process Tracing and Error Analysis

Providing decision support to operators in command and control contexts requires careful assessment of its impacts on task performance. Here we describe a human-in-the-loop experiment using a naval air defense testbed to compare three conditions: 1) a baseline interface (DSSBASE; 2) one that displays the temporal proximity of radar aircraft (DSSTEMP); and 3) one which adds a change history panel (DSSCHEX). Threatevaluation accuracy and cognitive models of participants’ judgments did not significantly differ across conditions. Eye-tracking data showed that DSSTEMP lead to a reduced verification of track attributes. Confusion matrices also differed across conditions: DSSTEMP lead to more errors that are “two-categoriesaway” when erroneously classifying hostile aircraft as non-hostile or uncertain. We conclude that in addition to looking at standard metrics of task accuracy and response times, critical insights can be obtained by assessing how different designs for decision support may alter strategies and cognitive processes.