Narada D. Warakagoda

Multimodal Interfaces to Mobile Terminals - A Design-For-All Approach

Multimodal human-computer user interfaces are able to combine different input signals, extract the combined meaning from them, find requested information and present the response in the most appropriate format. Hence, a multimodal human-computer interface offers the users an opportunity to choose the most natural interaction pattern for the actual application and context of use. If the preferred mode fails in a certain context or task, users may switch to a more appropriate mode or they can combine modalities. Around thirty years ago Bolt presented the “Put That There” concept demonstrator, which processed speech in parallel with manual pointing during object manipulation (Bolt, 1980). Since then major advances have been made in automatic speech recognition (ASR) algorithms and natural language processing (NLP), in handwriting and gesture recognition, as well as in speed, processing power and memory capacity of computers. Today’s multimodal systems are capable of recognizing and combining a wide variety of signals such as speech, touch, manual gestures, gaze tracking, facial expressions, head and body movements. The response can be presented by e.g. facial animation in the form of human-like presentation agents on the screen in a multimedia system. These advanced systems need various sensors and a lot of processing power and memory. They are therefore best suited for interaction with computers and in kiosk applications, as demonstrated in e.g. (Oviatt, 2000); (Gustafson et al., 2000); (Wahlster, 2001); (Beskow, et al. 2002); (Karpov, 2006); (Smartkom, 2007). Modern mobile terminals are now portable computers where the traditional audio user interfaces, microphones and speakers, are accompanied with touch screens, cameras, accelerometers and gyroscopes etc. These enriched user interfaces combined with the ever increasing capacity of processors, access to mobile networks with increasing bandwidths and functionality as global positioning system (GPS) and near field communication (NFC) will make mobile terminals well suited for developing user-friendly multimodal interfaces in the years to come. However, the multimodal functionality on mobile terminals is still restricted to two input modes: speech (audio) and touch, and two output modes: audio and vision. This type of multimodality, sometimes called tap & talk (or point & speak), is essentially