Teruhisa Misu

Situated multi-modal dialog system in vehicles

In this paper, we address Townsurfer, a situated multi-modal dialog system in vehicles. The system integrates multi-modal inputs of speech, geo-location, gaze (face direction) and dialog history to answer drivers queries about their surroundings. To select appropriate data source used to answer queries, we apply belief tracking across the above modalities. We conducted a preliminary data collection and an evaluation focusing on the effect of gaze (head irection) and geo-location estimations. We report the result and analysis on the data.

Non-monologue HMM-based speech synthesis for service robots: A cloud robotics approach

Robot utterances generally sound monotonous, unnatural, and unfriendly because their Text-to-Speech (TTS) systems are not optimized for communication but for text-reading. Here we present a non-monologue speech synthesis for robots. We collected a speech corpus in a non-monologue style in which two professional voice talents read scripted dialogues. Hidden Markov models (HMMs) were then trained with the corpus and used for speech synthesis. We conducted experiments in which the proposed method was evaluated by 24 subjects in three scenarios: text-reading, dialogue, and domestic service robot (DSR) scenarios. In the DSR scenario, we used a physical robot and compared our proposed method with a baseline method using the standard Mean Opinion Score (MOS) criterion. Our experimental results showed that our proposed methods performance was (1) at the same level as the baseline method in the text-reading scenario and (2) exceeded it in the DSR scenario. We deployed our proposed system as a cloud-based speech synthesis service so that it can be used without any cost.

Situated language understanding for a spoken dialog system within vehicles

HighlightsWe implemented and analyzed issues in situated language understanding in moving car.We analyzed timing of utterances, spatial relationships between the car and targets.Our algorithms improved the target identification rate by 24.1%. In this paper, we address issues in situated language understanding in a moving car, which has the additional challenge of being a rapidly changing environment. More specifically, we propose methods for understanding user queries regarding specific target buildings in their surroundings. Unlike previous studies on physically situated interactions, such as interactions with mobile robots, the task at hand is very time sensitive because the spatial relationship between the car and target changes while the user is speaking. We collected situated utterances from drivers using our research system called Townsurfer, which was embedded in a real vehicle. Based on this data, we analyzed the timing of user queries, the spatial relationships between the car and the targets, the head pose of the user, and linguistic cues. Based on this analysis, we further propose methods to optimize timing and spatial distances and to make use of linguistic cues. Finally, we demonstrate that our algorithms improved the target identification rate by 24.1% absolute.

Look at Me: Augmented Reality Pedestrian Warning System Using an In-Vehicle Volumetric Head Up Display

Current pedestrian collision warning systems use either auditory alarms or visual symbols to inform drivers. These traditional approaches cannot tell the driver where the detected pedestrians are located, which is critical for the driver to respond appropriately. To address this problem, we introduce a new driver interface taking advantage of a volumetric head-up display (HUD). In our experimental user study, sixteen participants drove a test vehicle in a parking lot while braking for crossing pedestrians using different interface designs on the HUD. Our results showed that spatial information provided by conformal graphics on the HUD resulted in not only better driver performance but also smoother braking behavior as compared to the baseline.

Investigating Critical Speech Recognition Errors in Spoken Short Messages

Understanding dictated short-messages requires the system to perform speech recognition on the user’s speech. This speech recognition process is prone to errors. If the system can automatically detect the presence of an error, it can use dialog to clarify or correct its transcript. In this work, we present our analysis on what types of errors a recognition system makes, and propose a method to detect these critical errors. In particular, we distinguish between simple and critical errors, where the meaning in the transcript is not the same as the user dictated. We show that our method outperforms standard baseline techniques by 2 % absolute F-score.

Visual Saliency and Crowdsourcing-based Priors for an In-car Situated Dialog System

This paper addresses issues in situated language understanding in a moving car. We propose a reference resolution method to identify user queries about specific target objects in their surroundings. We investigate methods of predicting which target object is likely to be queried given a visual scene and what kind of linguistic cues users naturally provide to describe a given target object in a situated environment. We propose methods to incorporate the visual saliency of the visual scene as a prior. Crowdsourced statistics of how people describe an object are also used as a prior. We have collected situated utterances from drivers using our research system, which was embedded in a real vehicle. We demonstrate that the proposed algorithms improve target identification rate by 15.1%.

Identification of the Driver's Interest Point using a Head Pose Trajectory for Situated Dialog Systems

This paper addresses issues existing in situated language understanding in a moving car. Particularly, we propose a method for understanding user queries regarding specific target buildings in their surroundings based on the drivers head pose and speech information. To identify a meaningful head pose motion related to the user query that is among spontaneous motions while driving, we construct a model describing the relationship between sequences of a drivers head pose and the relative direction to an interest point using the Gaussian process regression. We also consider time-varying interest point using kernel density estimation. We collected situated queries from subject drivers by using our research system embedded in a real car. The proposed method achieves an improvement in the target identification rate by 14% in the user-independent training condition and 27% in the user-dependent training condition over the method that uses the head motion at the start-of-speech timing.

Driving maneuver prediction using car sensor and driver physiological signals

This study presents the preliminary attempt to investigate the usage of driver physiology signals, including electrocardiography (ECG) and respiration wave signals, to predict driving maneuvers. While most studies on driving maneuver prediction uses direct measurements from vehicle or road scene, we believe the mental state changes from the driver when making plans for maneuver can be reflected from the physiological signals. We extract both time and frequency domain features from the physiological signals, and use them as the features to predict the drivers future maneuver. We formulate the prediction of driver maneuver as a multi-class classification problem by using the features extracted from signal before the driving maneuvers. The multi classes correspond to various types of driving maneuvers including Start, Stop, Lane Switch and Turn. We use the support vector machine (SVM) as the classifier, and compare the performance of using both physiological and car signals (CAN bus) with the baseline classifier that is trained with only car signal. An improved performance is observed when using the physiological features with 0.04 in F-score on average. This improvement is more obvious as the prediction is made earlier.

Predicting unexpected maneuver while approaching intersection

Unfamiliar urban intersections pose high demand on drivers. They are not only engaged in correctly assessing large amount of visual stimuli, including multiple diverse moving objects (e.g. other vehicles, pedestrians, cyclists) but also actively processing instructions provided by navigation system, either in-car or on other devices such as smart-phones. In such a highly dynamic and engaging situation, drivers are prone to make a mistake. In this paper, we look into the intersection behavior of the driver to predict an unexpected maneuver that would cause deviation from the planned path such as missing an upcoming turn or make a last minute aggressive maneuver. We conduct an on-road test of naturally following planned route as suggested by the navigation system. Our ultimate goal is to develop a Advanced Driver Assistance System (ADAS) that can predict unexpected maneuver and help driver in timely manner to correct those mistakes e.g. by providing detail navigation instructions for the driver to better orient himself or herself in a challenging situation. We propose an unexpected maneuver detection framework that can utilize vehicle, map as well as driver information to predict ahead in time. We further illustrate the benefit of utilizing driver information for early prediction.

Driver Behavior and Performance with Augmented Reality Pedestrian Collision Warning: An Outdoor User Study

This article investigates the effects of visual warning presentation methods on human performance in augmented reality (AR) driving. An experimental user study was conducted in a parking lot where participants drove a test vehicle while braking for any cross traffic with assistance from AR visual warnings presented on a monoscopic and volumetric head-up display (HUD). Results showed that monoscopic displays can be as effective as volumetric displays for human performance in AR braking tasks. The experiment also demonstrated the benefits of conformal graphics, which are tightly integrated into the real world, such as their ability to guide drivers attention and their positive consequences on driver behavior and performance. These findings suggest that conformal graphics presented via monoscopic HUDs can enhance driver performance by leveraging the effectiveness of monocular depth cues. The proposed approaches and methods can be used and further developed by future researchers and practitioners to better understand driver performance in AR as well as inform usability evaluation of future automotive AR applications.