Pavle Mijović

Towards continuous and real-time attention monitoring at work: reaction time versus brain response

Abstract Continuous and objective measurement of the user attention state still represents a major challenge in the ergonomics research. Recently available wearable electroencephalography (EEG) opens new opportunities for objective and continuous evaluation of operators’ attention, which may provide a new paradigm in ergonomics. In this study, wearable EEG was recorded during simulated assembly operation, with the aim to analyse P300 event-related potential component, which provides reliable information on attention processing. In parallel, reaction times (RTs) were recorded and the correlation between these two attention-related modalities was investigated. Negative correlation between P300 amplitudes and RTs has been observed on the group level (p < .001). However, on the individual level, the obtained correlations were not consistent. As a result, we propose the P300 amplitude for accurate attention monitoring in ergonomics research. On the other hand, no significant correlation between RTs and P300 latency was found on group, neither on individual level. Practitioner Summary: Ergonomic studies of assembly operations mainly investigated physical aspects, while mental states of the assemblers were not sufficiently addressed. Presented study aims at attention tracking, using realistic workplace replica. It is shown that drops in attention could be successfully traced only by direct brainwave observation, using wireless electroencephalographic measurements.

Benefits of Instructed Responding in Manual Assembly Tasks: An ERP Approach

The majority of neuroergonomics studies are focused mainly on investigating the interaction between operators and automated systems. Far less attention has been dedicated to the investigation of brain processes in more traditional workplaces, such as manual assembly, which are still ubiquitous in industry. The present study investigates whether assembly workers’ attention can be enhanced if they are instructed with which hand to initiate the assembly operation, as opposed to the case when they can commence the operation with whichever hand they prefer. For this aim, we replicated a specific workplace, where 17 participants in the study simulated a manual assembly operation of the rubber hoses that are used in vehicle hydraulic brake systems, while wearing wireless electroencephalography (EEG). The specific EEG feature of interest for this study was the P300 components’ amplitude of the event-related potential (ERP), as it has previously been shown that it is positively related to human attention. The behavioral attention-related modality of reaction times (RTs) was also recorded. Participants were presented with two distinct tasks during the simulated operation, which were counterbalanced across participants. In the first task, digits were used as indicators for the operation initiation (Numbers task), where participants could freely choose with which hand they would commence the action upon seeing the digit. In the second task, participants were presented with arrows, which served as instructed operation initiators (Arrows task), and they were instructed to start each operation with the hand that corresponded to the arrow direction. The results of this study showed that the P300 amplitude was significantly higher in the instructed condition. Interestingly, the RTs did not differ across any task conditions. This, together with the other findings of this study, suggests that attention levels can be increased using instructed responses without compromising work performance or operators’ well-being, paving the way for future applications in manual assembly task design.

Towards Creation of Implicit HCI Model for Prediction and Prevention of Operators’ Error

This paper describes development of a new generation of the interactive industrial workplace, through introduction of a novel implicit Human Computer Interaction (HCI) model. Proposed framework aims at being a foundation of a computer-based system that enables an increase of workers safety and well-being in industrial environments. Further aim is to enable an increase in production levels, together with improvement of ergonomics of the workplace. Specifically targeted environments are industrial workplaces that include repetitive tasks, which are in most of the cases monotonic in nature. Implicit HCI model could enable development of a specific technical solution that is meant to be an integral and inseparable part of a future workplace and should serve to predict human errors and communicate a warning to a worker. As such, system is meant to increase situational awareness of the workers and prevent errors in operating that would otherwise lead to work-related injuries (including causalities).

Investigating Brain Dynamics in Industrial Environment – Integrating Mobile EEG and Kinect for Cognitive State Detection of a Worker

In the present work we used wearable EEG sensor for recording brain activity during simulated assembly work, in replicated industrial environment. We investigated attention related modalities of P300 ERP component and engagement index (EI), which is extracted from signal power ratios of α, β and θ frequency bands. Simultaneously, we quantified the task unrelated movements, which are previously reported to be related to attention level, in an automated way employing kinectTM sensor. Reaction times were also recorded and investigated. We found that during the monotonous task, both the P300 amplitude and EI decreased as the time of the task progressed. On the other hand, the increase of the task unrelated movement quantity was observed, together with the increase in RTs. These findings lead to conclusion that the monotonous assembly work induces the decrease of attention and engagement of the workers as the task progresses, which is observable in both neural (EEG) and behavioral (RT and unrelated movements) signal modalities. Apart from observing how the attention-related modalities are changing over time, we investigated the functional relationship between the neural and behavioral modalities by using Pearson’s correlation. Since the Person’s correlation coefficients showed the functional relationship between the attention-related modalities, we proposed the creation of the multimodal implicit Human-Computer Interaction (HCI) system, which could acquire and process neural and behavioral data in real-time, with the aim of creating the system that could be aware of the operator’s mental states during the industrial work, consequently improving the operator’s well-being.

Neuroergonomics Method for Measuring the Influence of Mental Workload Modulation on Cognitive State of Manual Assembly Worker

In this study, we simulated a manual assembly operation, where participants were exposed to two distinct ways of information presentation, reflecting two task conditions (monotonous and more demanding task condition). We investigated how changes in mental workload (MWL) modulate the P300 component of event-related potentials (ERPs), recorded from wireless electroencephalography (EEG), reaction times (RTs) and quantity of task unrelated movements (retrieved from Kinect). We found a decrease in P300 amplitude and an increase in the quantity of the task unrelated movements, both indicating a decrease in attention level during a monotonous task (lower MWL). During the more demanding task, where a slightly higher MWL was imposed, these trends were not obvious. RTs did not show any dependency on the level of workload applied. These results suggest that a wireless EEG, but also Kinect, can be used to measure the influence of MWL variation on the cognitive state of the workers.

A Tool for Neuroergonomic Study of Repetitive Operational Tasks

The objective of the present work in progress, in the field of neuroergonomics, is to find psychophysiological correlates for vigilance decrement and mental fatigue in industrial environments as well as the measurements of postural loads using multimodal system. This system is composed of (wirelessly obtained) electroencephalographic measurements (EEG), electrodermal activity (EDA) and commercial 3D motion capture sensor kinect™. The aim of this study is to decrease potential operational error due to slips in attention and mental fatigue of a worker and consequently lead to more reliable and safer production. Further, postural loads could be assessed on-line, thus preventing the over-load of the workers and, consequently, possible work-related MSDs.