Yujia Cao

Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI

In this study, we argue that multimodal presentations should be created in a cognitive-aware manner, especially in a high-load HCI situation where the user task challenges the full capacity of the human cognition. An experiment was conducted to investigate the cognitive effects of modalities, using a high-load task. The performance measurements and subjective reports consistently confirm a significant modality impact on cognitive workload, stress and performance. A relation between modality usage and physiological states was not found, due to the insufficient sensitivity and individual differences of the physiological measurements. The findings of this experiment can be well explained by several modality-related cognitive theories. We further integrate these theories into a suitability prediction model, which can systematically predict how suitable a certain modality usage is for this presentation task. The model demonstrates a possible approach towards cognitive-aware modality planning and can be modified for other applications.

Manipulating stress and cognitive load in conversational interactions with a multimodal system for crisis management support

The quality assessment of multimodal conversational interfaces is influenced by many factors. Stress and cognitive load are two of most important. In the literature, these two factors are considered as being related and accordingly summarized under the single concept of ‘cognitive demand’. However, our assumption is that even if they are related, these two factors can still occur independently. Therefore, it is essential to control their levels during the interaction in order to determine the impact that each factor has on the perceived conversational quality. In this paper we present preliminary experiments in which we tried to achieve a factor separation by inducing alternating low/high levels of both stress and cognitive load. The stress/cognitive load levels were manipulated by varying task difficulty, information presentation and time pressure. Physiological measurements, performance metrics, as well as subjective reports were deployed to validate the induced stress and cognitive load levels. Results showed that our manipulations were successful for the cognitive load and partly for the stress. The levels of both factors were better indicated by subjective reports and performance metrics than by physiological measurements.

Multimodal presentation of local danger warnings for drivers: A situation-dependent assessment of usability

This study addresses how advanced driver assistance systems (ADAS) should communicate with drivers, focusing on the local danger warning function. To achieve high-quality assistance, the communication mode needs to be adaptive to changes in driving situation (drivers state, workload and environment). In a user study investigating different warning communication modes, drivers were required to assess the usability of each communication mode in five different driving situations. Results revealed that the driving situation significantly affected the perceived usefulness of each warning communication mode, mainly due to the use of modality. Moreover, regardless of communication mode, the local danger warning function was considered as the most useful in the low visibility situation and the least useful in the highly demanding driving situation. Findings of this study can be applied to the design of ADAS in general.

Cognitive-Aware Modality Allocation in Intelligent Multimodal Information Presentation

Intelligent multimodal presentation (IMMP) systems are able to generate multimodal presentations adaptively, based on the run-time requirements of user-computer interaction. Modality allocation in IMMP system needs to adapt the modality choice to changes in various relevant factors, such as the type of the information to be conveyed, the presentation goal, the characteristics of the available modalities, the user profile, the condition of the environment and the type of user task. In this study, we emphasize that modality allocation in IMMP systems should also take into account the cognitive impacts of modality on human information processing. We first describe several modality-related cognitive and neuropsychological findings. Then a user study is presented to demonstrate the effects of modality on performance, cognitive load and stress, using a high-load and time-critical user task. Finally, we show a possible way to integrate relevant cognitive theories into a computational model that can systematically predict the suitability of a modality choice for a given presentation task.