Vanja Ković

Balloons and bavoons versus spikes and shikes: ERPs reveal shared neural processes for shape-sound-meaning congruence in words, and shape-sound congruence in pseudowords.

There is something about the sound of a pseudoword like takete that goes better with a spiky, than a curvy shape (Köhler, 1929:1947). Yet despite decades of research into sound symbolism, the role of this effect on real words in the lexicons of natural languages remains controversial. We report one behavioural and one ERP study investigating whether sound symbolism is active during normal language processing for real words in a speakers native language, in the same way as for novel word forms. The results indicate that sound-symbolic congruence has a number of influences on natural language processing: Written forms presented in a congruent visual context generate more errors during lexical access, as well as a chain of differences in the ERP. These effects have a very early onset (40-80 ms, 100-160 ms, 280-320 ms) and are later overshadowed by familiar types of semantic processing, indicating that sound symbolism represents an early sensory-co-activation effect.

Towards continuous and real-time attention monitoring at work: reaction time versus brain response

Abstract Continuous and objective measurement of the user attention state still represents a major challenge in the ergonomics research. Recently available wearable electroencephalography (EEG) opens new opportunities for objective and continuous evaluation of operators’ attention, which may provide a new paradigm in ergonomics. In this study, wearable EEG was recorded during simulated assembly operation, with the aim to analyse P300 event-related potential component, which provides reliable information on attention processing. In parallel, reaction times (RTs) were recorded and the correlation between these two attention-related modalities was investigated. Negative correlation between P300 amplitudes and RTs has been observed on the group level (p < .001). However, on the individual level, the obtained correlations were not consistent. As a result, we propose the P300 amplitude for accurate attention monitoring in ergonomics research. On the other hand, no significant correlation between RTs and P300 latency was found on group, neither on individual level. Practitioner Summary: Ergonomic studies of assembly operations mainly investigated physical aspects, while mental states of the assemblers were not sufficiently addressed. Presented study aims at attention tracking, using realistic workplace replica. It is shown that drops in attention could be successfully traced only by direct brainwave observation, using wireless electroencephalographic measurements.

Benefits of Instructed Responding in Manual Assembly Tasks: An ERP Approach

The majority of neuroergonomics studies are focused mainly on investigating the interaction between operators and automated systems. Far less attention has been dedicated to the investigation of brain processes in more traditional workplaces, such as manual assembly, which are still ubiquitous in industry. The present study investigates whether assembly workers’ attention can be enhanced if they are instructed with which hand to initiate the assembly operation, as opposed to the case when they can commence the operation with whichever hand they prefer. For this aim, we replicated a specific workplace, where 17 participants in the study simulated a manual assembly operation of the rubber hoses that are used in vehicle hydraulic brake systems, while wearing wireless electroencephalography (EEG). The specific EEG feature of interest for this study was the P300 components’ amplitude of the event-related potential (ERP), as it has previously been shown that it is positively related to human attention. The behavioral attention-related modality of reaction times (RTs) was also recorded. Participants were presented with two distinct tasks during the simulated operation, which were counterbalanced across participants. In the first task, digits were used as indicators for the operation initiation (Numbers task), where participants could freely choose with which hand they would commence the action upon seeing the digit. In the second task, participants were presented with arrows, which served as instructed operation initiators (Arrows task), and they were instructed to start each operation with the hand that corresponded to the arrow direction. The results of this study showed that the P300 amplitude was significantly higher in the instructed condition. Interestingly, the RTs did not differ across any task conditions. This, together with the other findings of this study, suggests that attention levels can be increased using instructed responses without compromising work performance or operators’ well-being, paving the way for future applications in manual assembly task design.

Towards Creation of Implicit HCI Model for Prediction and Prevention of Operators’ Error

This paper describes development of a new generation of the interactive industrial workplace, through introduction of a novel implicit Human Computer Interaction (HCI) model. Proposed framework aims at being a foundation of a computer-based system that enables an increase of workers safety and well-being in industrial environments. Further aim is to enable an increase in production levels, together with improvement of ergonomics of the workplace. Specifically targeted environments are industrial workplaces that include repetitive tasks, which are in most of the cases monotonic in nature. Implicit HCI model could enable development of a specific technical solution that is meant to be an integral and inseparable part of a future workplace and should serve to predict human errors and communicate a warning to a worker. As such, system is meant to increase situational awareness of the workers and prevent errors in operating that would otherwise lead to work-related injuries (including causalities).

In the absence of animacy: superordinate category structure affects subordinate label verification.

Theoretical accounts as well as behavioral studies reporting animacy effects offer inconsistent and sometimes contradictory results. A possible explanation for these inconsistencies may be inadvertent biases in the stimuli selected for test – with category-specific effects driven by characteristics of test stimuli other than animacy per se. In this study, we pit animacy against feature structure (intra-item variability), in a picture-word matching task. For unimpaired adults, regardless of whether objects were from animate (mammals; insects) or inanimate (clothes; musical instruments) superordinate categories, participants were faster to match basic level labels with objects from categories with low intra-item variability (mammals; clothes) than from categories with high intra-item variability (insects; instruments). Thus, pitting animacy against variability allowed us to clarify that observable differences in processing speed between animals and instruments are systematically driven by the intra-item variability of the superordinate categories, and not by animacy itself.

To call a cloud ‘cirrus’: sound symbolism in names for categories or items

The aim of the present paper is to experimentally test whether sound symbolism has selective effects on labels with different ranges-of-reference within a simple noun-hierarchy. In two experiments, adult participants learned the make up of two categories of unfamiliar objects (‘alien life forms’), and were passively exposed to either category-labels or item-labels, in a learning-by-guessing categorization task. Following category training, participants were tested on their visual discrimination of object pairs. For different groups of participants, the labels were either congruent or incongruent with the objects. In Experiment 1, when trained on items with individual labels, participants were worse (made more errors) at detecting visual object mismatches when trained labels were incongruent. In Experiment 2, when participants were trained on items in labelled categories, participants were faster at detecting a match if the trained labels were congruent, and faster at detecting a mismatch if the trained labels were incongruent. This pattern of results suggests that sound symbolism in category labels facilitates later similarity judgments when congruent, and discrimination when incongruent, whereas for item labels incongruence generates error in judgements of visual object differences. These findings reveal that sound symbolic congruence has a different outcome at different levels of labelling within a noun hierarchy. These effects emerged in the absence of the label itself, indicating subtle but pervasive effects on visual object processing.

Investigating Brain Dynamics in Industrial Environment – Integrating Mobile EEG and Kinect for Cognitive State Detection of a Worker

In the present work we used wearable EEG sensor for recording brain activity during simulated assembly work, in replicated industrial environment. We investigated attention related modalities of P300 ERP component and engagement index (EI), which is extracted from signal power ratios of α, β and θ frequency bands. Simultaneously, we quantified the task unrelated movements, which are previously reported to be related to attention level, in an automated way employing kinectTM sensor. Reaction times were also recorded and investigated. We found that during the monotonous task, both the P300 amplitude and EI decreased as the time of the task progressed. On the other hand, the increase of the task unrelated movement quantity was observed, together with the increase in RTs. These findings lead to conclusion that the monotonous assembly work induces the decrease of attention and engagement of the workers as the task progresses, which is observable in both neural (EEG) and behavioral (RT and unrelated movements) signal modalities. Apart from observing how the attention-related modalities are changing over time, we investigated the functional relationship between the neural and behavioral modalities by using Pearson’s correlation. Since the Person’s correlation coefficients showed the functional relationship between the attention-related modalities, we proposed the creation of the multimodal implicit Human-Computer Interaction (HCI) system, which could acquire and process neural and behavioral data in real-time, with the aim of creating the system that could be aware of the operator’s mental states during the industrial work, consequently improving the operator’s well-being.

Comparing the temporal dynamics of thematic and taxonomic processing using event-related potentials

We report the results of a study comparing the temporal dynamics of thematic and taxonomic knowledge activation in a picture-word priming paradigm using event-related potentials. Although we found no behavioral differences between thematic and taxonomic processing, ERP data revealed distinct patterns of N400 and P600 amplitude modulation for thematic and taxonomic priming. Thematically related target stimuli elicited less negativity than taxonomic targets between 280–460 ms after stimulus onset, suggesting easier semantic processing of thematic than taxonomic relationships. Moreover, P600 mean amplitude was significantly increased for taxonomic targets between 520–600 ms, consistent with a greater need for stimulus reevaluation in that condition. These results offer novel evidence in favor of a dissociation between thematic and taxonomic thinking in the early phases of conceptual evaluation.

Neuroergonomics Method for Measuring the Influence of Mental Workload Modulation on Cognitive State of Manual Assembly Worker

In this study, we simulated a manual assembly operation, where participants were exposed to two distinct ways of information presentation, reflecting two task conditions (monotonous and more demanding task condition). We investigated how changes in mental workload (MWL) modulate the P300 component of event-related potentials (ERPs), recorded from wireless electroencephalography (EEG), reaction times (RTs) and quantity of task unrelated movements (retrieved from Kinect). We found a decrease in P300 amplitude and an increase in the quantity of the task unrelated movements, both indicating a decrease in attention level during a monotonous task (lower MWL). During the more demanding task, where a slightly higher MWL was imposed, these trends were not obvious. RTs did not show any dependency on the level of workload applied. These results suggest that a wireless EEG, but also Kinect, can be used to measure the influence of MWL variation on the cognitive state of the workers.