Sebastian Feuerstack

Model-Based Design of Interactions That can Bridge Realities - The Augmented "Drag-and-Drop"

The development of Augmented Reality (AR) applications is still driven by development at source-code level. Although recent approaches focus to standardize AR functionality, to our knowledge a declarative and model-driven design (MDD) has not been applied for AR development so far. MDD approaches have been successfully applied to model user interfaces for a wide spectrum of modes (such as speech-command interfaces, or remote controls) and media such as HTML, 3D, and smart phone interfaces for instance. In this paper we propose a MDD approach for modeling seamless interaction between Web and AR interfaces. Therefore we implemented a prototype of a web furniture shop that supports arranging furniture in an augmented reality to prove our approach. We then present our approach of modeling a reality spanning Drag-and-Drop interaction between a 2D browser and an AR scene. Finally, we discuss the issues that we were confronted with to support reality spanning interaction like switching interaction modes and coordinate systems and present limitations that we have experienced with using other control modes, like a Wii-Controller or glove-based gesture detection.

Revealing Differences in Designers’ and Users’ Perspectives

Monitoring complex systems includes scanning, aggregating and processing data from various sources. The design of graphical interfaces for monitoring tasks involves a fine-grained exploration of the importance and expected frequency of events that an operator needs to be informed about.

Model-based design of multimodal interaction for augmented reality web applications

Despite the increasing use of Augmented Reality (AR) in many different application areas, implementation support is limited and still driven by development at source-code level. Although efforts have been made to overcome these limitations, there is a clear gap between authoring environments and source code level framework approaches for creating AR interfaces for the web with multimodal control. Model-based design for interaction can offer support to fill this gap between authoring environments and frameworks. However, to the best of our knowledge, a declarative and model-driven design (MDD) has not yet been applied to model AR interfaces for a wide spectrum of modes. Thus, this paper presents an extension of the model-driven design to cope with interactors, whose novelty lies on the introduction of a modeling approach targeted at AR developers and designers in their task to design new forms of interactions that can be later used in authoring environments. To validate our approach, we demonstrate how a reality spanning Drag-and-Drop interaction can be modeled for an online furniture shop. And we implemented a gesture based control to show how new control modes can be added to an existing MDD-based design to extend the interaction capabilities.

A Heuristic for relative Perception Accuracy and Reaction Time Estimation for HMI Designs

A human operator monitoring a safety-critical system has to gather information fast and accurate to detect problems and execute countermeasures in time. So far testing such HMIs is a complex task, since it requires HMI design prototypes embedded into simulated environments to perform tests with professional operators. We propose Konect Value, a heuristic to estimate the relative perception accuracy and operator reaction time already in the HMI design phase. The model-based estimation heuristic solely requires a task model and HMI design sketches as an input. The evaluation metric was applied to seven different HMIs, which were designed by Human Factor experts to support truck platooning. A comparison of the estimated accuracy and reaction times of Konect Value to a lab study (n=33) revealed high correlations for the relative reaction time (r=0.83, p<0.05) and also the relative perception accuracy (r=-0.90, p<0.01). This indicates that Konect Value is a promising heuristic for early HMI design evaluation in the safety-critical system domain.

Tutorial: How Does Your HMI Design Affect the Visual Attention of the Driver

The consideration of drivers visual attention for Human Machine Interface (HMI) design is critical to ensure fast reaction times in unexpected situations and to promote situation awareness in hand-over situations. The effect of an HMI to the attention distribution of the driver can be measured by performing eye-tracking studies in a driving simulator. Performing eye-tracking studies requires functional HMI prototypes but give no insights on the underlying mechanisms for the measured behavior. In the tutorial we introduce a tool-driven and model-based approach to visual attention prediction, which can be performed already based on early HMI mockup ideas and with less effort compared to eye-tracking studies. The tutorial starts with an introduction to the theories of model-based visual attention prediction. Thereafter, participants are invited to either predict the visual attention for their own HMI design ideas or conduct an evaluation of an exemplary use case with the software tools that the participants can install on their computers or use in our lab.

Tool-supported comparative visualizations to reveal the difference between ‘what has been designed’ and ‘how it is perceived’ for monitoring interface design

Monitoring is one of the most important tasks for an operator of a complex safety-critical system like a ship bridge or air traffic control. It is a prerequisite for good situation awareness. Designing an interface for such environments requires optimizing what is presented to the most limited resource: the operators visual attention. But the real operators attention distribution is hard to anticipate for a designer. We apply cognitive attention prediction methods to predict attention based on information gained on the one hand by the HMI designer and on the other hand the future user of an HMI. This contribution proposes and evaluates a set of comparative visualizations that support elaborating the differences between what has been designed and how it is perceived. An initial study indicated that a visually supported comparative analysis supports a designer in identifying differences and also seems to stimulate the designer to reason about the design.

Engineering automotive HMIs that are optimized for correct and fast perception

Automotive HMI design is driven by systematic model-based user engineering methods focusing on traceability and functional validation. User-centered design processes can open up the design space to discover new creative design solutions whereas model-based engineering methods offer a rigorous design derivation process. We present Konect, a user-centered design derivation process that fosters creativity and also ensures a systematic design derivation to end up with new interface designs that are optimized for correct and fast readability. Five Human Factor Experts applied the method and ended up with 5 creative and very different design results. In a follow-up experiment with 33 car drivers, we figured out that all designs could be perceived faster and more frequently correct than automotive interfaces for the same assistant system that have been designed with other design methods.

Comparing the multimodal interaction technique design of MINT with NiMMiT

With new sensors that can capture hand and body movements in 3D, novel interaction techniques gain importance. But development of new forms of interaction is highly iterative, depends on extensive user testing and therefore is expensive. We propose a model-based notation using statecharts and mappings to ease multimodal interaction technique design. This model-based specification can be used to communicate designs, for evaluation and to enable re-use. Our contribution continues previous research on model-based interaction technique design considers multimodal interaction and addresses problems like the state explosion, error management and consideration of output modalities mentioned by earlier research. We evaluate our notation by comparing it with NiMMiT referring to the same use case to identify similarities, strength and problems.