Jürgen Sauer

Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions

We present in this paper a new multimodal corpus of spontaneous collaborative and affective interactions in French: RECOLA, which is being made available to the research community. Participants were recorded in dyads during a video conference while completing a task requiring collaboration. Different multimodal data, i.e., audio, video, ECG and EDA, were recorded continuously and synchronously. In total, 46 participants took part in the test, for which the first 5 minutes of interaction were kept to ease annotation. In addition to these recordings, 6 annotators measured emotion continuously on two dimensions: arousal and valence, as well as social behavior labels on live dimensions. The corpus allowed us to take self-report measures of users during task completion. Methodologies and issues related to affective corpus construction are briefly reviewed in this paper. We further detail how the corpus was constructed, i.e., participants, procedure and task, the multimodal recording setup, the annotation of data and some analysis of the quality of these annotations.

Effects of sleep deprivation and user interface on complex performance: a multilevel analysis of compensatory control.

This study was carried out to test the compensatory control model, which predicts performance maintenance under stress at the expense of effort and increased selectivity. It examined the effects of sleep deprivation on performance in an automated process control task based on a simplified life support system with two types of operator control panel interface: machine centered (M-C), in which access to the system was scheduled by the computer, and human-centered (H-C), in which access was ad-lib. The task environment also permitted the analysis of changes in strategy and in subsidiary activities (alarm reaction time, prospective memory). In a 2 × 2 repeated-measures design, 16 participants carried out the task with each interface after both normal sleep and one night of sleep deprivation (SD). No effects of SD were observed on primary task performance. As predicted, SD effects were confined to strategy changes and subsidiary task impairment and occurred only under the (low control) M-C interface. Subjective effort was increased under SD, with greater increases of effort associated with high levels of performance protection. The findings provide strong evidence in favor of the compensatory control model and argue for the use of complex, multilevel tasks in the analysis of performance under stress. Actual or potential applications include the development of more sensitive performance-testing systems based on multilevel analysis of decrement, and the design of interfaces for shift work and other suboptimal work conditions.

The influence of user expertise and prototype fidelity in usability tests

An empirical study examined the impact of user expertise and prototype fidelity on the outcomes of a usability test. User expertise (expert vs. novice) and prototype fidelity (paper prototype, 3D mock-up, and fully operational appliance) were manipulated as independent variables in a 2x3 between-subjects design. Employing a floor scrubber as a model product, 48 users carried out several cleaning tasks. Usability problems identified by participants were recorded. Furthermore, performance, system management strategies and perceived usability were measured. The results showed that experts reported more usability problems than novices but these were considered to be less severe than those reported by novices. Reduced fidelity prototypes were generally suitable to predict product usability of the real appliance. The implications for the running of usability tests are specific to the fidelity of the prototype.

Designing training for process control simulators: a review of empirical findings and current practices

Although simulators have been widely used as training environments in different industries (e.g. oil and nuclear power), there is little rigorous empirical work evaluating the effectiveness of the training methods employed. This article examines the use of simulator training in process control environments. The results of an exploratory field study are reported and the current practices of simulator training are described. The study revealed that simulator training varied considerably across organisations, often with little theoretical or empirical work to guide training design. To evaluate the utility and effectiveness of different methods of simulator training in process control environments, the article also presents a literature review of the research on laboratory- and field-based training. Several training methods are identified as having particular potential for temporal and adaptive transfer and are to be empirically tested in future studies.

The influence of product aesthetics and user state in usability testing

An empirical study examined the effects of two influencing factors in usability tests on user performance and usability ratings. Product aesthetics (high vs. low) as the main independent factor and prior usage event (positive vs. negative) as a subsidiary independent factor were varied in a between-participants design. 60 participants took part in the experiment, completing a number of typical tasks of mobile phone users. The results showed that increased product aesthetics had a positive effect on perceived usability and led to longer task completion times. After a negative usage event had been experimentally induced, usability ratings dropped as expected but user performance on subsequent tasks remained unaffected. An important implication of the study is that the aesthetic properties of a product may have multiple effects that go beyond perceived product attractiveness.

Adaptability of training in simulated process control : Knowledge-versus rule-based guidance under task changes and environmental stress

Objective: The study examined the adaptability of different types of process control training across changes in task and environmental stress. Background: The literature on training leads us to expect greater flexibility for system-based training, as opposed to procedure-based training. However, the stress literature suggests that knowledge-based strategies (making use of executive control) may be more vulnerable under stress conditions. Method: Two groups were given 6 hr of training on the Cabin Air Management System (CAMS), a complex, multilevel, PC-based process control task, emphasizing either system knowledge or use of procedures. They were then required to carry out the task for 3 hr (with noise during the middle 1 hr) across a range of both familiar and unfamiliar fault scenarios. Results: For the primary control task, the system-trained group performed better, especially for less familiar and complex faults. However, for lower priority tasks requiring executive control, procedure-trained operators performed better and were less impaired by noise. Conclusion: System training was more effective for managing unexpected task events, whereas procedural training was better under noise. The results are interpreted in terms of the rationale for instructing operators in the range of strategies required for effective process skills in complex work environments. Application: Training methodologies for safety critical applications should aim to develop skill in the use of both procedural and system knowledge strategies. Operators should be trained in the most effective deployment of these strategies during unfamiliar task events and environmental stress and given stress exposure training.

The effects of heuristic rule training on operator performance in a simulated process control environment

In complex work environments, the occurrence of novel system states represents a particular challenge for the design of training. This article is concerned with the use of heuristic rules to prepare operators for the management of unfamiliar fault states. An experiment was carried out to examine the effects of heuristic rule training on operator performance and system management behaviour. Thirty-nine trainee operators from the chemical industry took part in the study. They were trained for 4 h on a PC-based simulation of a process control task. Operators in the experimental group received training on heuristic rules while operators in the control group did not. One week later the operators participated in a 70-min testing session. While the results showed that heuristic rules training led to better diagnostic performance, it was also associated with increased operator fatigue and impaired secondary task performance. The implications of the findings for using heuristic rule training are discussed.

Cognitive diversity and team performance in a complex multiple task environment

This article examines the multiple effects of cognitive diversity in teams operating complex human-machine-systems. The study employed a PC-based multiple-task environment, called the Cabin Air Management System, which models a process control task in the operational context of a spacecrafts life support system. Two types of cognitive diversity were examined: system understanding and team specialization. System understanding referred to the depth of understanding team members were given during training (low-level procedure-oriented vs. high level knowledge-oriented training). Team specialization referred to the degree to which knowledge about system fault scenarios was distributed between team members (specialized vs. non-specialized). A total of 72 participants took part in the study. After having received 4.5 h of training on an individual basis, participants completed a 1-h experimental session, in which they worked in two-person teams on a series of fault scenarios of varying difficulty. Measures were taken of primary and secondary task performance, system intervention and information sampling strategies, system knowledge, subjective operator state, communication patterns and conflict. The results provided evidence for the benefits of cognitive diversity with regard to system understanding. This manifested itself in better primary task performance and more efficient manual system control. No advantages were found for cognitive diversity with regard to specialization. There was no effect of cognitive diversity on intra-team conflict, with conflict levels generally being very low. The article concludes with a discussion of the implications of the findings for the engineering of cognitive diversity in teams operating complex human-machine-systems.

Multiple-task performance on a computer-simulated life support system during a space mission simulation.

This paper presents an experiment which examined the effects of isolation and confinement during a simulation of a short-term space mission. During the 7-day spaceflight simulation, four Canadian astronauts were tested daily on a 30-min performance task. The task, CAMS (Cabin Air Management System), represents a computer-based simulation of a generic life support system. As a multiple-task environment, it allows the measurement of a wide range of task management variables such as primary and secondary task performance, and system control activities. Measures of subjective state variables were also taken. The results did not show any evidence of serious performance decrements for any crew member. The analysis revealed different adjustment patterns with which crew members responded as a function of mission duration and variations in workload. Among the secondary tasks employed, prospective memory was found to be more sensitive than reaction time to increases in workload. The paper concludes with a discussion of the utility of spaceflight simulations and computer-based simulations of space work.

Explicit control of adaptive automation under different levels of environmental stress

This article examines the effectiveness of three different forms of explicit control of adaptive automation under low- and high-stress conditions, operationalised by different levels of noise. In total, 60 participants were assigned to one of three types of automation design (free, prompted and forced choice). They were trained for 4 h on a highly automated simulation of a process control environment, called AutoCAMS. This was followed by a 4-h testing session under noise exposure and quiet conditions. Measures of performance, psychophysiology and subjective reactions were taken. The results showed that all three modes of explicit control of adaptive automation modes were able to attenuate the negative effects of noise. This was partly due to the fact that operators opted for higher levels of automation under noise. It also emerged that forced choice showed marginal advantages over the two other automation modes. Statement of Relevance: This work is relevant to the design of adaptive automation since it emphasises the need to consider the impact of work-related stressors during task completion. During the presence of stressors, different forms of operator support through automation may be required than under more favourable working conditions.