Implementation of multi-modal interface for VR application

Abstract

Recently, some Head Mount Displays (HMD) are released for consumers. A general VR system is provided to a virtual experience with the virtual world according to user’s responses organized by three types of components such as an input system, an output system and a simulation system. The input system is used as a controller, a mouse, a keyboard and head tracking device. These devices are used to physical operations on the real world, but these devices are invisible on the virtual world during using the HMD. In this paper, we introduce a multi-modal interface to VR application supporting both voice and gesture interface on a general HMD. We also discuss about a prototype system to use low cost devices such as the HMD, a gesture input device, a general PC and a USB microphone.