Eric H. C. Choi

Galvanic skin response (GSR) as an index of cognitive load

Multimodal user interfaces (MMUI) allow users to control computers using speech and gesture, and have the potential to minimise users. experienced cognitive load, especially when performing complex tasks. In this paper, we describe our attempt to use a physiological measure, namely Galvanic Skin Response (GSR), to objectively evaluate users. stress and arousal levels while using unimodal and multimodal versions of the same interface. Preliminary results show that users. GSR readings significantly increase when task cognitive load level increases. Moreover, users. GSR readings are found to be lower when using a multimodal interface, instead of a unimodal interface. Cross-examination of GSR data with multimodal data annotation showed promising results in explaining the peaks in the GSR data, which are found to correlate with sub-task user events. This interesting result verifies that GSR can be used to serve as an objective indicator of user cognitive load level in real time, with a very fine granularity.

Investigation of spectral centroid features for cognitive load classification

Speech is a promising modality for the convenient measurement of cognitive load, and recent years have seen the development of several cognitive load classification systems. Many of these systems have utilised mel frequency cepstral coefficients (MFCC) and prosodic features like pitch and intensity to discriminate between different cognitive load levels. However, the accuracies obtained by these systems are still not high enough to allow for their use outside of laboratory environments. One reason for this might be the imperfect acoustic description of speech provided by MFCCs. Since these features do not characterise the distribution of the spectral energy within subbands, in this paper, we investigate the use of spectral centroid frequency (SCF) and spectral centroid amplitude (SCA) features, applying them to the problem of automatic cognitive load classification. The effect of varying the number of filters and the frequency scale used is also evaluated, in terms of the effectiveness of the resultant spectral centroid features in discriminating between cognitive loads. The results of classification experiments show that the spectral centroid features consistently and significantly outperform a baseline system employing MFCC, pitch, and intensity features. Experimental results reported in this paper indicate that the fusion of an SCF based system with an SCA based system results in a relative reduction in error rate of 39% and 29% for two different cognitive load databases.

Using pen input features as indices of cognitive load

Multimodal interfaces are known to be useful in map-based applications, and in complex, time-pressure based tasks. Cognitive load variations in such tasks have been found to impact multimodal behaviour. For example, users become more multimodal and tend towards semantic complementarity as cognitive load increases. The richness of multimodal data means that systems could monitor particular input features to detect experienced load variations. In this paper, we present our attempt to induce controlled levels of load and solicit natural speech and pen-gesture inputs. In particular, we analyse for these features in the pen gesture modality. Our experimental design relies on a map-based Wizard of Oz, using a tablet PC. This paper details analysis of pen-gesture interaction across subjects, and presents suggestive trends of increases in the degree of degeneration of pen-gestures in some subjects, and possible trends in gesture kinematics, when cognitive load increases.

Pupillary response based cognitive workload measurement under luminance changes

Pupillary response has been widely accepted as a physiological index of cognitive workload. It can be reliably measured with remote eye trackers in a non-intrusive way. However, pupillometric measurement might fail to assess cognitive workload due to the variation of luminance conditions. To overcome this problem, we study the characteristics of pupillary responses at different stages of cognitive process when performing arithmetic tasks, and propose a fine-grained approach for cognitive workload measurement. Experimental results show that cognitive workload could be effectively measured even under luminance changes.

Glottal features for speech-based cognitive load classification

Cognitive load measurement is important when designing adaptive interfaces that optimize the performance of users working on high mental load tasks. Recent research on automatic speech-based measurement system indicates that cognitive load information is more prominent in the frequency region below 1 kHz. This study investigates the effects of cognitive load on glottal parameters (open quotient, normalized amplitude quotient and speed quotient), and proposes a system employing these parameters as features for cognitive load classification. Analysis of the glottal parameter distributions suggests that an increase in cognitive load can be related to a more creaky voice quality. Additionally, three-class classification results show that score-level fusion of systems based on the glottal features and baseline features (MFCCs, pitch, intensity and shifted delta cepstra) improves the baseline accuracy from 79% to 84%.

Formant frequencies under cognitive load: effects and classification

Cognitive load measurement systems measure the mental demand experienced by human while performing a cognitive task, which is useful in monitoring and enhancing task performance. Various speech-based systems have been proposed for cognitive load classification, but the effect of cognitive load on the speech production system is still not well understood. In this work, we study formant frequencies under different load conditions and utilize formant frequency-based features for automatic cognitive load classification. We find that the slope, dispersion, and duration of vowel formant trajectories exhibit changes under different load conditions; slope and duration are found to be useful features in vowel-based classification. Additionally, 2-class and 3-class utterance-based classification results, evaluated on two different databases, show that the performance of frame-based formant features was comparable, if not better than, baseline MFCC features.

A Study of Voice Source and Vocal Tract Filter Based Features in Cognitive Load Classification

Speech has been recognized as an attractive method for the measurement of cognitive load. Previous approaches have used mel frequency cepstral coefficients (MFCCs) as discriminative features to classify cognitive load. The MFCCs contain information from both the voice source and the vocal tract, so that the individual contributions of each to cognitive load variation are unclear. This paper aims to extract speech features related to either the voice source or the vocal tract and use them to discriminate between cognitive load levels in order to identify the individual contribution of each for cognitive load measurement. Voice source-related features are then used to improve the performance of current cognitive load classification systems, using adapted Gaussian mixture models. Our experimental result shows that the use of voice source feature could yield around 12% reduction in relative error rate compared with the baseline system based on MFCCs, intensity, and pitch contour.

An improved soft threshold method for DCT speech enhancement

An improved soft threshold method for speech enhancement in the discrete cosine transform (DCT) domain is proposed in this paper. Rather than apply a threshold only to noise-dominant frames, as per traditional DCT-based approaches, our proposed approach also applies the threshold process appropriately in signal-dominant frames. Experimental results show a quality improvement with our proposed method compared to traditional soft threshold methods.

Multi-lingual Phoneme Recognition and Language Identification Using Phonotactic Information

Previous research indicates that automatic language identification systems based on phonotactic information produce the best results compared with other systems based on acoustic or prosodic information. This paper investigates two different approaches that use phonotactic information: parallel phoneme recognition followed by language modeling (PPRLM) and multi-lingual PRLM. In the PPRLM approach, we have modified the system by using four different language models with different discounting methods, including the linear, absolute, good-turning and Witten-Bell. Our results show that the modified PPRLM system with the Witten-Bell discounting outperforms other systems and achieves 75.5% language identification accuracy for the OGI-TS speech corpus

A non-uniform subband approach to speech-based cognitive load classification

Speech has recently been recognized as an attractive method for the measurement of cognitive load. Current speech-based cognitive load measurement systems utilize acoustic features derived from auditory-motivated frequency scales. This paper aims to investigate the distribution of speech information specific to cognitive load discrimination as a function of frequency. We found that this distribution is neither uniform nor very similar to the Mel auditory scale and based on our experiments, we propose a novel non-uniform filterbank for acoustic feature extraction to classify cognitive load. Experimental results showed that the use of the proposed filterbank provided a relative improvement of about 10%, compared with the classification accuracy of the traditional cognitive load classification system based on a Mel-scale filterbank.