Michael Barz

Prediction of gaze estimation error for error-aware gaze-based interfaces

Gaze estimation error is inherent in head-mounted eye trackers and seriously impacts performance, usability, and user experience of gaze-based interfaces. Particularly in mobile settings, this error varies constantly as users move in front and look at different parts of a display. We envision a new class of gaze-based interfaces that are aware of the gaze estimation error and adapt to it in real time. As a first step towards this vision we introduce an error model that is able to predict the gaze estimation error. Our method covers major building blocks of mobile gaze estimation, specifically mapping of pupil positions to scene camera coordinates, marker-based display detection, and mapping of gaze from scene camera to on-screen coordinates. We develop our model through a series of principled measurements of a state-of-the-art head-mounted eye tracker.

Evaluating Remote and Head-worn Eye Trackers in Multi-modal Speech-based HRI (Demo)

Gaze is known to be a dominant modality for conveying spatial information, and it has been used for grounding in human-robot dialogues. In this work, we present the prototype of a gaze-supported multi-modal dialogue system that enhances two core tasks in human-robot collaboration: 1) our robot is able to learn new objects and their location from user instructions involving gaze, and 2) it can instruct the user to move objects and passively track this movement by interpreting the users gaze. We performed a user study to investigate the impact of different eye trackers on user performance. In particular, we compare a head-worn device and an RGB-based remote eye tracker. Our results show that the head-mounted eye tracker outperforms the remote device in terms of task completion time and the required number of utterances due to its higher precision.

Multimodal multisensor activity annotation tool

In this paper we describe a multimodal-multisensor annotation tool for physiological computing; for example mobile gesture-based interaction devices or health monitoring devices can be connected. It should be used as an expert authoring tool to annotate multiple video-based sensor streams for domain-specific activities. Resulting datasets can be used as supervised datasets for new machine learning tasks. Our tool provides connectors to commercially available sensor systems (e.g., Intel RealSense F200 3D camera, Leap Motion, and Myo) and a graphical user interface for annotation.

Gaze-guided object classification using deep neural networks for attention-based computing

Recent advances in eye tracking technologies opened the way to design novel attention-based user interfaces. This is promising for pro-active and assistive technologies for cyber-physical systems in the domains of, e.g., healthcare and industry 4.0. Prior approaches to recognize a users attention are usually limited to the raw gaze signal or sensors in instrumented environments. We propose a system that (1) incorporates the gaze signal and the egocentric camera of the eye tracker to identify the objects the user focuses at; (2) employs object classification based on deep learning which we recompiled for our purposes on a GPU-based image classification server; (3) detects whether the user actually draws attention to that object; and (4) combines these modules for constructing episodic memories of egocentric events in real-time.

Speech-based Medical Decision Support in VR using a Deep Neural Network (Demonstration)

We present a speech dialogue system that facilitates medical decision support for doctors in a virtual reality (VR) application. The therapy prediction is based on a recurrent neural network model that incorporates the examination history of patients. A central supervised patient database provides input to our predictive model and allows us, first, to add new examination reports by a pen-based mobile application on-the-fly, and second, to get therapy prediction results in real-time. This demo includes a visualisation of patient records, radiology image data, and the therapy prediction results in VR.

Medical 3D Images in Multimodal Virtual Reality

We present a multimodal medical 3D image system for radiologists in an virtual reality (VR) environment. Users can walk freely inside the virtual room and interact with the system using speech, going through patient records, and manipulate 3D image data with hand gestures. Medical images are retrieved from the hospitals Picture and Archiving System (PACS) and displayed as 3D objects inside VR. Our system incorporates a dialogue-based decision support system for treatments. A central supervised patient database provides input to our predictive model and allows us, first, to add new examination reports by a pen-based mobile application on-the-fly, and second, to get therapy prediction results in real-time. This demo includes a visualisation of real patient records, 3D DICOM radiology image data, and real-time therapy predictions in VR.

Error-aware gaze-based interfaces for robust mobile gaze interaction

Gaze estimation error can severely hamper usability and performance of mobile gaze-based interfaces given that the error varies constantly for different interaction positions. In this work, we explore error-aware gaze-based interfaces that estimate and adapt to gaze estimation error on-the-fly. We implement a sample error-aware user interface for gaze-based selection and different error compensation methods: a naïve approach that increases component size directly proportional to the absolute error, a recent model by Feit et al. that is based on the two-dimensional error distribution, and a novel predictive model that shifts gaze by a directional error estimate. We evaluate these models in a 12-participant user study and show that our predictive model significantly outperforms the others in terms of selection rate, particularly for small gaze targets. These results underline both the feasibility and potential of next generation error-aware gaze-based user interfaces.

Visual Search Target Inference Using Bag of Deep Visual Words

Visual Search target inference subsumes methods for predicting the target object through eye tracking. A person intents to find an object in a visual scene which we predict based on the fixation behavior. Knowing about the search target can improve intelligent user interaction. In this work, we implement a new feature encoding, the Bag of Deep Visual Words, for search target inference using a pre-trained convolutional neural network (CNN). Our work is based on a recent approach from the literature that uses Bag of Visual Words, common in computer vision applications. We evaluate our method using a gold standard dataset.

Human-in-the-Loop Control Processes in Gas Turbine Maintenance

In this applied research paper, we describe an architecture for seamlessly integrating factory workers in industrial cyber-physical production environments. Our human-in-the-loop control process uses novel input techniques and relies on state-of-the-art industry standards. Our architecture allows for real-time processing of semantically annotated data from multiple sources (e.g., machine sensors, user input devices) and real-time analysis of data for anomaly detection and recovery. We use a semantic knowledge base for storing and querying data (http://www.metaphacts.com) and the Business Process Model and Notation (BPMN) for modelling and controlling the process. We exemplify our industrial solution in the use case of the maintenance of a Siemens gas turbine. We report on this case study and show the advantages of our approach for smart factories. An informal evaluation in the gas turbine maintenance use case shows the utility of automated anomaly detection and handling: workers can fill in paper-based incident reports by using a digital pen; the digitised version is stored in metaphacts and linked to semantic knowledge sources such as process models, structure models, business process models, and user models. Subsequently, automatic maintenance and recovery processes that involve human experts are triggered.