Kun Yu

Robust Multimodal Cognitive Load Measurement

This book explores robust multimodal cognitive load measurement with physiological and behavioural modalities, which involve the eye, Galvanic Skin Response, speech, language, pen input, mouse movement and multimodality fusions. Factors including stress, trust, and environmental factors such as illumination are discussed regarding their implications for cognitive load measurement. Furthermore, dynamic workload adjustment and real-time cognitive load measurement with data streaming are presented in order to make cognitive load measurement accessible by more widespread applications and users. Finally, application examples are reviewed demonstrating the feasibility of multimodal cognitive load measurement in practical applications. This is the first book of its kind to systematically introduce various computational methods for automatic and real-time cognitive load measurement and by doing so moves the practical application of cognitive load measurement from the domain of the computer scientist and psychologist to more general end-users, ready for widespread implementation. Robust Multimodal Cognitive Load Measurement is intended for researchers and practitioners involved with cognitive load studies and communities within the computer, cognitive, and social sciences. The book will especially benefit researchers in areas like behaviour analysis, social analytics, human-computer interaction (HCI), intelligent information processing, and decision support systems.

User Trust Dynamics: An Investigation Driven by Differences in System Performance

Trust is a key factor affecting the way people rely on automated systems. On the other hand, system performance has comprehensive implications on a users trust variations. This paper examines systems of varied levels of accuracy, in order to reveal the relationship between system performance, a users trust and reliance on the system. In particular, it is identified that system failures have a stronger effect on trust than system successes. We also describe how patterns of trust change according to a number of consecutive system failures or successes. Importantly, we show that increasing user familiarity with the system decreases the rate of trust change, which provides new insights on the development of user trust. Finally, our analysis established a correlation between a users reliance on a system and their trust level. Combining all these findings can have important implications in general system design and implementation, by predicting how trust builds and when it stabilizes, as well as allowing for indirectly reading a users trust in real time based on system reliance.

Human-centric analysis of driver inattention

Driver distraction is an important risk factor for road traffic injuries, and has been the focus of a number of empirical studies aiming to raise awareness about the risks of distracted driving and to promote countermeasures. While some of the recorded road incidents in these studies have their roots in distracting events (such as mobile phone usage) a large proportion of recorded road incidents can be attributed to more elusive driver inattention factors not linked to specific trigger events. These distraction categories are especially challenging and currently not in focus of current research as they are difficult to detect and address by suitable prognostic measures, in order to improve road safety. To contribute to this issue, this paper presents research into monitoring drivers mental states in real-time, using objective measurements. We propose an iterative research methodology where specific mental states are elicited, user response captured experimentally, and interaction models built using advanced machine learning techniques. Behavioral measures such as speech, eye activity or posture, and physiological measures such as galvanic skin response or heart rate provide input features for the models. This driver-centric approach addresses the complex issue of driver inattention, and can help improve road safety through active monitoring of road users, customized decision support in the vehicle, and objective training feedback. Low-fidelity simulators we have built allowed us to roll out some preliminary tasks prompting encouraging feedback from subjects during informal testing.

Multimodal behavioral and physiological signals as indicators of cognitive load

1.

Trust and Reliance Based on System Accuracy

Trust plays an important role in various user-facing systems and applications. It is particularly important in the context of decision support systems, where the systems output serves as one of the inputs for the users decision making processes. In this work, we study the dynamics of explicit and implicit user trust in a simulated automated quality monitoring system, as a function of the system accuracy. We establish that users correctly perceive the accuracy of the system and adjust their trust accordingly.

Elicitation of mental states and user experience factors in a driving simulator

It has been previously established that high cognitive load influences driving performance, but is the drivers perception of their experience while driving also influenced by cognitive load? To our knowledge, little evaluation has taken place regarding the investigation of such an effect of cognitive load on user experience, especially in the automotive domain. This paper introduces motivation and background of our current research on real-time monitoring of drivers mental states aiming to explore the previously mentioned relations between mental states and UX in a driving simulator environment. Furthermore, the paper presents initial ideas of task designs that are targeted to elicit and induce mental states like cognitive load and selected user experience factors we consider being interesting in the automotive domain and that we aim to discuss with other researchers and practitioners in the workshop.

Indexing Cognitive Load using Blood Volume Pulse Features

Physiological responses contain rich affective information even when humans are not expressing any external signs. In this paper, we investigate the use of the Blood Volume Pulse (BVP) signals for indexing cognitive load. An experiment, which introduced cognitive load as a secondary task in a decision making context was conducted in the study. BVP signals were analyzed in order to establish relationships between BVP and cognitive load levels. A set of features (e.g. peak and max features) was found to be significantly distinctive across different cognitive load levels. The identified BVP features can be used to set up machine learning models for the automatic classification of CL levels in intelligent systems.

Dynamic Handwriting Signal Features Predict Domain Expertise

As commercial pen-centric systems proliferate, they create a parallel need for analytic techniques based on dynamic writing. Within educational applications, recent empirical research has shown that signal-level features of students’ writing, such as stroke distance, pressure and duration, are adapted to conserve total energy expenditure as they consolidate expertise in a domain. The present research examined how accurately three different machine-learning algorithms could automatically classify users’ domain expertise based on signal features of their writing, without any content analysis. Compared with an unguided machine-learning classification accuracy of 71%, hybrid methods using empirical-statistical guidance correctly classified 79–92% of students by their domain expertise level. In addition to improved accuracy, the hybrid approach contributed a causal understanding of prediction success and generalization to new data. These novel findings open up opportunities to design new automated learning analytic systems and student-adaptive educational technologies for the rapidly expanding sector of commercial pen systems.

Correlation for user confidence in predictive decision making

Despite the recognized value of Machine Learning (ML) techniques and high expectation of applying ML techniques within various applications, significant barriers to the widespread adoption and local implementation of ML approaches still exist in the areas of trust (of ML results), comprehension (of ML processes), as well as confidence (in decision making) by users. This paper investigates the effects of correlation between features and target values on user confidence in data analytics-driven decision making. Our user study found that revealing the correlation between features and target variables affected user confidence in decision making significantly. Moreover, users felt more confident in decision making when correlation shared the same trend with the prediction model performance. These findings would help design intelligent user interfaces and evaluate the effectiveness of machine learning models in applications.

Do I Trust a Machine? Differences in User Trust Based on System Performance

Trust plays an important role in various user-facing systems and applications. It is particularly important in the context of decision support systems, where the system’s output serves as one of the inputs for the users’ decision making processes. In this chapter, we study the dynamics of explicit and implicit user trust in a simulated automated quality monitoring system, as a function of the system accuracy. We establish that users correctly perceive the accuracy of the system and adjust their trust accordingly. The results also show notable differences between two groups of users and indicate a possible threshold in the acceptance of the system. This important learning can be leveraged by designers of practical systems for sustaining the desired level of user trust.