Natalie Ruiz

Galvanic skin response (GSR) as an index of cognitive load

Multimodal user interfaces (MMUI) allow users to control computers using speech and gesture, and have the potential to minimise users. experienced cognitive load, especially when performing complex tasks. In this paper, we describe our attempt to use a physiological measure, namely Galvanic Skin Response (GSR), to objectively evaluate users. stress and arousal levels while using unimodal and multimodal versions of the same interface. Preliminary results show that users. GSR readings significantly increase when task cognitive load level increases. Moreover, users. GSR readings are found to be lower when using a multimodal interface, instead of a unimodal interface. Cross-examination of GSR data with multimodal data annotation showed promising results in explaining the peaks in the GSR data, which are found to correlate with sub-task user events. This interesting result verifies that GSR can be used to serve as an objective indicator of user cognitive load level in real time, with a very fine granularity.

Speech-based cognitive load monitoring system

Monitoring cognitive load is important for the prevention of faulty errors in task-critical operations, and the development of adaptive user interfaces, to maintain productivity and efficiency in work performance. Speech, as an objective and non-intrusive measure, is a suitable method for monitoring cognitive load. Existing approaches for cognitive load monitoring are limited in speaker-dependent recognition and need manually labeled data. We propose a novel automatic, speaker-independent classification approach to monitor, in real-time, the persons cognitive load level by using speech features. In this approach, a Gaussian mixture model (GMM) based classifier is created with unsupervised training. Channel and speaker normalization are deployed for improving robustness. Different delta techniques are investigated for capturing temporal information. And a background model is introduced to reduce the impact of insufficient training data. The final system achieves 71.1% and 77.5% accuracy on two different tasks, each of which has three discrete cognitive load levels. This performance shows a great potential in real-world applications.

Eye activity as a measure of human mental effort in HCI

The measurement of a users mental effort is a problem whose solutions may have important applications to adaptive interfaces and interface evaluation. Previous studies have empirically shown links between eye activity and mental effort; however these have usually investigated only one class of eye activity on tasks atypical of HCI. This paper reports on research into eight eye activity based features, spanning eye blink, pupillary response and eye movement information, for real time mental effort measurement. Results from an experiment conducted using a computer-based training system show that the three classes of eye features are capable of discriminating different cognitive load levels. Correlation analysis between various pairs of features suggests that significant improvements in discriminating different effort levels can be made by combining multiple features. This shows an initial step towards a real-time cognitive load measurement system in human-computer interaction.

Automatic cognitive load detection from speech features

Cognitive load variations have been found to impact multimodal behaviour, in particular, features of spoken input. In this paper, we present a design and implementation of a user study aimed at soliciting natural speech at three different levels of cognitive load. Some of the speech data produced was then used to train a number of models to automatically detect cognitive load. We describe a classification approach, the cognitive load levels were detected and output as discrete level ranges. The final system achieved a 71.1% accuracy for 3 levels classification in a speaker-independent setting. The ability to detect and manage a users cognitive load can help us to adapt intelligent interfaces that ensure optimal user performance

Measuring cognitive workload with low-cost electroencephalograph

Electroencephalography (EEG) is an important physiological index of cognitive workload. While previous research has employed high-end EEG devices, this work investigates the feasibility of measuring cognitive workload with a low-cost EEG system. In our experiment, EEG signals are recorded from subjects performing silent reading tasks under different difficulty levels. Experimental results demonstrate the effectiveness of cognitive workload evaluation even with low-cost EEG equipment.

Using pen input features as indices of cognitive load

Multimodal interfaces are known to be useful in map-based applications, and in complex, time-pressure based tasks. Cognitive load variations in such tasks have been found to impact multimodal behaviour. For example, users become more multimodal and tend towards semantic complementarity as cognitive load increases. The richness of multimodal data means that systems could monitor particular input features to detect experienced load variations. In this paper, we present our attempt to induce controlled levels of load and solicit natural speech and pen-gesture inputs. In particular, we analyse for these features in the pen gesture modality. Our experimental design relies on a map-based Wizard of Oz, using a tablet PC. This paper details analysis of pen-gesture interaction across subjects, and presents suggestive trends of increases in the degree of degeneration of pen-gestures in some subjects, and possible trends in gesture kinematics, when cognitive load increases.

Examining the redundancy of multimodal input

Speech and gesture modalities can allow users to interact with complex applications in novel ways. Often users will adapt their multimodal behaviour to cope with increasing levels of domain complexity. These strategies can change how multimodal constructions are planned and executed by users. In the frame of Baddeleys Theory of Working Memory, we present some of the results from an empirical study conducted with users of a multimodal interface, under varying levels of cognitive load. In particular, we examine how multimodal behavioural features are sensitive to cognitive load variations. We report significant decreases in multimodal redundancy (33.6%) and trends of increased multimodal complementarity, as cognitive load increases.

Think before you talk: an empirical study of relationship between speech pauses and cognitive load

Measuring a users level of cognitive load while they are interacting with the system could offer another dimension to the development of adaptable user interfaces. High levels of cognitive load affect performance and efficiency. However, current methods of measuring cognitive load are physically intrusive and interrupt the task flow. Certain speech features have been shown to change under high levels of load and are good candidates for cognitive load indices for usability evaluation and automatic adaptation of an interface or work environment. A speech-based dual-task user study is presented in which we explore the behaviour of speech pause features in natural speech. The experiment yielded new results confirming that speech pauses are useful indicators of high load versus low load speech. We report an increase in the percentage of time spent pausing from low load to high load tasks. We interpret these results within the framework of Baddeleys modal model of working memory and detail how such a measure could be utilized in the cognitive load measurement.

Investigating speech features and automatic measurement of cognitive load

The ability to measure cognitive load level in real time is extremely useful for improving the efficiency of interfaces and contents delivering, especially when interfaces and contents get complex in a multimedia environment. Speech is highly suitable for measuring cognitive load due to its non-intrusive nature and ease of collection. In this paper, we investigated the patterns of prosodic features and confirmed it is relevant to cognitive load. We also explored varied classification techniques to capture those relevant patterns of speech features. Gaussian Mixture Model (GMM), Support Vector Machine (SVM), and a hybrid SVM-GMM based classifiers were investigated with MFCC and pitch features. Individual systems and a fusion based system were evaluated on two different task scenarios - reading comprehension and Stroop test. The SVM-GMM based system achieved the highest performance on both tasks and improved the accuracy of three levels classification to 75.6% and 82.2%, respectively.

Potential speech features for cognitive load measurement

Intelligent user interfaces with an awareness of a users experienced level of cognitive load have the potential to change the way output strategies are implemented and executed. However, current methods of measuring cognitive load are intrusive and unsuitable in real-time scenarios. Certain speech features have been shown to change under high levels of load. We present a dual-task speech based user study in which we explore three speech features: pause length, pause frequency and latency to response. These features are evaluated for their diagnostic capacity. Pause length and latency to response are shown to be useful indicators of high load versus low load speech.