Dan Conway

Robust Multimodal Cognitive Load Measurement

This book explores robust multimodal cognitive load measurement with physiological and behavioural modalities, which involve the eye, Galvanic Skin Response, speech, language, pen input, mouse movement and multimodality fusions. Factors including stress, trust, and environmental factors such as illumination are discussed regarding their implications for cognitive load measurement. Furthermore, dynamic workload adjustment and real-time cognitive load measurement with data streaming are presented in order to make cognitive load measurement accessible by more widespread applications and users. Finally, application examples are reviewed demonstrating the feasibility of multimodal cognitive load measurement in practical applications. This is the first book of its kind to systematically introduce various computational methods for automatic and real-time cognitive load measurement and by doing so moves the practical application of cognitive load measurement from the domain of the computer scientist and psychologist to more general end-users, ready for widespread implementation. Robust Multimodal Cognitive Load Measurement is intended for researchers and practitioners involved with cognitive load studies and communities within the computer, cognitive, and social sciences. The book will especially benefit researchers in areas like behaviour analysis, social analytics, human-computer interaction (HCI), intelligent information processing, and decision support systems.

Speech Signal Based Measures

This chapter reviews cognitive load measurement methods via speech with typical experiments to induce different cognitive load levels via speech being introduced. Various speech features are then investigated, and a comparison of cognitive load level classification methods is conducted.

Multimodal Measures and Data Fusion

This chapter presents a model for multimodal cognitive load. The features extracted from speech, pen input and GSR in a user study are fused using the AdaBoost boosting algorithm to demonstrate the methods advantages.

Dynamic Cognitive Load Adjustments in a Feedback Loop

This chapter presents a cognitive load adaptation model that dynamically adjusts workload during human-machine interaction, in order to keep the task demands at an appropriate level. Physiological signals such as GSR are collected to evaluate human workload in real-time and the task difficulty levels are adjusted in real-time to better fit the user.

Trust and Cognitive Load

This chapter investigates the relationship between trust perception and cognitive load. An experimental platform is designed and employed to collect multimodal data and different types of analyses are conducted.

Eye-Based Measures

This chapter investigates the measurement of cognitive load through eye tracking under the influence of varying luminance conditions. We demonstrate how reliable measurements can be achieved via this non-intrusive approach. We also discuss the characteristics of pupillary response and their association with different stages of cognitive processes when performing arithmetic tasks. The experimental results presented demonstrate the feasibility of a comparatively fine-grained method of cognitive load measurement in dynamic workplace environments.

Stress and Cognitive Load

This chapter investigates the effect of stress on cognitive load measurement using GSR as a physiological index of CL. The experiment utilises feelings of lack of control, task failure and social-evaluation to induce stress. The experiment described demonstrates that mean GSR can be used as an index of cognitive load during task execution, but that this relationship is obfuscated when test subjects experience fluctuating levels of stress. Alternate analysis methods are then presented that show how this confounding factor can be overcome.

Emotion and Cognitive Load

This chapter investigates pupillary response and GSR as a cognitive load measure under the influence of such confounding factors. A video-based eye tracker is used to record pupillary response and GSR is recorded during arithmetic tasks under both luminance and emotional changes. The mean-difference feature and its extension (Haar-like features) are used to characterise physiological responses of cognitive load under these context effects. Boosting based feature selection and classification are employed that successfully classify workload even under the influence of those noisy factors.

Pen Input Based Measures

This chapter introduces methods to examine cognitive load via writing and pen-based features, including writing velocity, pen pressure, writing gestures and additional features derived from these signals. Based on the examination of different types of script, including text, digits and sketches, we show that cognitive load with behavioural features are affected by both text content and writing direction.

Galvanic Skin Response-Based Measures

This chapter focuses on the use of Galvanic Skin Response (GSR) for cognitive load measurement. GSR is a measure of conductivity of human skin, and provides an indication of changes within the human sympathetic nervous system.