Eduardo Dias

Interactive Displays in Vehicles: Improving Usability with a Pointing Gesture Tracker and Bayesian Intent Predictors

Interactive displays are becoming an integrated part of the modern vehicle environment. Their use typically entails dedicating a considerable amount of attention and undertaking a pointing gesture to select an interface item/icon displayed on a touchscreen. This can have serious safety implications for the driver. The pointing gesture can also be highly perturbed due to the road and driving conditions, resulting in erroneous selections. In this paper, we propose a probabilistic intent prediction approach that facilitates establishing the targeted icon on the interface early in the pointing gesture. It employs a 3D vision sensory device to continuously track the pointing hand/finger in conjunction with suitable Bayesian prediction algorithms. The introduced technique can significantly reduce the pointing task completion time, the necessary associated visual, cognitive and movement efforts as well as enhance the selection accuracy. The substantial furnished gains and the pointing gesture characteristics are demonstrated using data collected in an instrumented vehicle.

Visual Monitoring of Driver and Passenger Control Panel Interactions

Advances in vehicular technology have resulted in more controls being incorporated into cabin designs. We present a system to determine which vehicle occupant is interacting with a control on the center console when it is activated, enabling the full use of dual-view touchscreens and the removal of duplicate controls. The proposed method relies on a background subtraction algorithm incorporating information from a superpixel segmentation stage. A manifold generated via the diffusion maps process handles the large variation in hand shapes, along with determining which part of the hand interacts with controls for a given gesture. We demonstrate superior results compared with other approaches on a challenging dataset.

Automatic Detection of a Driver’s Complex Mental States

Automatic classification of drivers’ mental states is an important yet relatively unexplored topic. In this paper, we define a taxonomy of a set of complex mental states that are relevant to driving, namely: Happy, Bothered, Concentrated and Confused. We present our video segmentation and annotation methodology of a spontaneous dataset of natural driving videos from 10 different drivers. We also present our real-time annotation tool used for labelling the dataset via an emotion perception experiment and discuss the challenges faced in obtaining the ground truth labels. Finally, we present a methodology for automatic classification of drivers’ mental states. We compare SVM models trained on our dataset with an existing nearest neighbour model pre-trained on posed dataset, using facial Action Units as input features. We demonstrate that our temporal SVM approach yields better results. The dataset’s extracted features and validated emotion labels, together with the annotation tool, will be made available to the research community.

Detection of valuable left-behind items in vehicle cabins

We propose a method for detecting valuable left-behind items in vehicle cabins which uses a single overhead camera. An additional sub-network is incorporated into the Faster R-CNN framework in order to allow it to estimate item value based on visual properties, as well as to perform detection. A loss function which contains a user-specified minimum-value threshold is introduced, which enables warnings to be given if a detected item is above this threshold. As a significant amount of real data is time consuming to collect on the scale necessary for (deep) learning-based methods, an ImageNet model is first retrained on synthetic data to adapt it to our environment, before training on some real data. The effectiveness of this detection and validation approach is demonstrated by integrating additional valuation subnetworks into two convolutional neural network detection architectures.

Intelligent scheduling for in-car notifications

The process of driving a car involves a cognitive load that varies over time. Additional load comes from secondary factors not directly associated with the driving process, including navigation devices, entertainment systems and the cars own warnings. In this paper, we present a framework for intelligent scheduling of in-car notifications based on the drivers estimated cognitive load. As the single channel for communication, it reschedules the notifications using a priority queue, and relays them to the driver based on the urgency of the notification and the overall estimated cognitive load being experienced by the driver at any given moment. We evaluate our system using a dataset collected from a cars CAN bus during multiple on-road trials and show that our proposed approach reduces the number of simultaneous calls on the drivers attention during the driving task. We also demonstrate that our intelligent scheduling significantly reduces the maximum cognitive load experienced by the driver and the frequency with which high loads occur.