Alain Chavaillaz

Experience of automation failures in training: effects on trust, automation bias, complacency and performance.

Abstract This work examined the effects of operators’ exposure to various types of automation failures in training. Forty-five participants were trained for 3.5 h on a simulated process control environment. During training, participants either experienced a fully reliable, automatic fault repair facility (i.e. faults detected and correctly diagnosed), a misdiagnosis-prone one (i.e. faults detected but not correctly diagnosed) or a miss-prone one (i.e. faults not detected). One week after training, participants were tested for 3 h, experiencing two types of automation failures (misdiagnosis, miss). The results showed that automation bias was very high when operators trained on miss-prone automation encountered a failure of the diagnostic system. Operator errors resulting from automation bias were much higher when automation misdiagnosed a fault than when it missed one. Differences in trust levels that were instilled by the different training experiences disappeared during the testing session. Practitioner Summary: The experience of automation failures during training has some consequences. A greater potential for operator errors may be expected when an automatic system failed to diagnose a fault than when it failed to detect one.

The use of adaptable automation: Effects of extended skill lay-off and changes in system reliability

This experiment aimed to examine how skill lay-off and system reliability would affect operator behaviour in a simulated work environment under wide-range and large-choice adaptable automation comprising six different levels. Twenty-four participants were tested twice during a 2-hr testing session, with the second session taking place 8 months after the first. In the middle of the second testing session, system reliability changed. The results showed that after the retention interval trust increased and self-confidence decreased. Complacency was unaffected by the lay-off period. Diagnostic speed slowed down after the retention interval but diagnostic accuracy was maintained. No difference between experimental conditions was found for automation management behaviour (i.e. level of automation chosen and frequency of switching between levels). There were few effects of system reliability. Overall, the findings showed that subjective measures were more sensitive to the impact of skill lay-off than objective behavioural measures.

AniSAM & AniAvatar: Animated Visualizations of Affective States

Tools that provide visual feedback about emotions to the user in the form of an avatar or an emoticon have become increasingly important. While a great deal of effort has already been put into the reliable and accurate automatic detection of emotions, only very little is known about how this information about affective states should be displayed in a comprehensible way to the user. In the present study, three newly developed feedback tools were evaluated. The tools were developed on the basis of an existing non-verbal questionnaire to represent two dimensions of emotion (i.e. valence and arousal) based on the circumplex model of affect. A total number of 826 participants were tested, using different vignettes that describe situations with specific affective content. Employing three newly developed affective feedback tools (AniSAM, AniAvatar and MergedSAM), the ratings obtained were compared to ratings using the original SAM instrument, a well-established questionnaire to measure affect. Results indicated that the animated feedback increased the accuracy of the arousal representation. Furthermore, valence feedback was more accurate when provided with an animated manikin-based tool rather than an avatar-based tool. This provided first evidence for the usefulness of animated tools offering visual feedback on user emotion. All instruments need to undergo further development. AniSAM and AniAvatar can be downloaded for purposes of practical applications and further research.

Effects of extended lay-off periods on performance and operator trust under adaptable automation.

Little is known about the long-term effects of system reliability when operators do not use a system during an extended lay-off period. To examine threats to skill maintenance, 28 participants operated twice a simulation of a complex process control system for 2.5 h, with an 8-month retention interval between sessions. Operators were provided with an adaptable support system, which operated at one of the following reliability levels: 60%, 80% or 100%. Results showed that performance, workload, and trust remained stable at the second testing session, but operators lost self-confidence in their system management abilities. Finally, the effects of system reliability observed at the first testing session were largely found again at the second session. The findings overall suggest that adaptable automation may be a promising means to support operators in maintaining their performance at the second testing session.

How operators make use of wide-choice adaptable automation: observations from a series of experimental studies

ABSTRACT The present paper is concerned with the design of adaptable automation. It analyses the aggregated data from five experiments (N = 154) to examine which automation levels operators prefer, how often they switch between them, and whether performance is associated with frequent switching or the automation levels being used. Using wide-choice adaptable automation (i.e. up to six levels were offered), the experiments were conducted using a PC-based simulation of a complex work environment. The results showed that about 95% of operators had a clearly preferred automation level, which they used for more than 50% of the time. They strongly preferred intermediate automation levels over levels at the higher and lower ends of the scale. Most operators switched rarely between levels and when they did, they usually made small adjustment rather than large changes. Several implications for the design of adaptable automation were derived from the empirical data.

Operator adaptation to changes in system reliability under adaptable automation

Abstract This experiment examined how operators coped with a change in system reliability between training and testing. Forty participants were trained for 3 h on a complex process control simulation modelling six levels of automation (LOA). In training, participants either experienced a high- (100%) or low-reliability system (50%). The impact of training experience on operator behaviour was examined during a 2.5 h testing session, in which participants either experienced a high- (100%) or low-reliability system (60%). The results showed that most operators did not often switch between LOA. Most chose an LOA that relieved them of most tasks but maintained their decision authority. Training experience did not have a strong impact on the outcome measures (e.g. performance, complacency). Low system reliability led to decreased performance and self-confidence. Furthermore, complacency was observed under high system reliability. Overall, the findings suggest benefits of adaptable automation because it accommodates different operator preferences for LOA. Practitioner Summary: The present research shows that operators can adapt to changes in system reliability between training and testing sessions. Furthermore, it provides evidence that each operator has his/her preferred automation level. Since this preference varies strongly between operators, adaptable automation seems to be suitable to accommodate these large differences.

On the effectiveness of performance-based adaptive automation

ABSTRACT This article examines the effectiveness of different forms of performance-based adaptive automation (PBAA). Using data from three experiments (N = 10, N = 38, N = 40), different models of algorithm design were compared for their effectiveness in driving PBAA. The following components were varied: type of task (i.e. primary or secondary tasks), baseline of performance data (e.g. moving average), and triggering criterion (i.e. level of deviation from standard performance). The data were generated by operators working with a computer-based simulation of a process control environment. The results showed that none of the models enjoyed a convincing level of effectiveness. The automation algorithms generally achieved higher levels of miss prevention than false alarm prevention. Surprisingly, primary task performance was generally better at driving PBAA than secondary task performance. The results suggest that it may be difficult to design an effective algorithm of PBAA if the work environment is highly complex.

Automation in visual inspection tasks: X-ray luggage screening supported by a system of direct, indirect or adaptable cueing with low and high system reliability

Abstract The present study evaluated three automation modes for improving performance in an X-ray luggage screening task. One hundred and forty participants were asked to detect the presence of prohibited items in X-ray images of cabin luggage. Twenty participants conducted this task without automatic support (control group), whereas the others worked with either indirect cues (system indicated the target presence without specifying its location), or direct cues (system pointed out the exact target location) or adaptable automation (participants could freely choose between no cue, direct and indirect cues). Furthermore, automatic support reliability was manipulated (low versus high). The results showed a clear advantage for direct cues regarding detection performance and response time. No benefits were observed for adaptable automation. Finally, high automation reliability led to better performance and higher operator trust. The findings overall confirmed that automatic support systems for luggage screening should be designed such that they provide direct, highly reliable cues.Practitioner summary: The present study confirmed previous findings showing better detection performance in X-ray images of luggage when supported by automation providing direct, highly reliable cues. Furthermore, participants used adaptable automation only to select their preferred level of automation. This behaviour did not provide the benefits expected under adaptable automation.