Frank Flemisch

Towards a dynamic balance between humans and automation: authority, ability, responsibility and control in shared and cooperative control situations

Progress enables the creation of more automated and intelligent machines with increasing abilities that open up new roles between humans and machines. Only with a proper design for the resulting cooperative human–machine systems, these advances will make our lives easier, safer and enjoyable rather than harder and miserable. Starting from examples of natural cooperative systems, the paper investigates four cornerstone concepts for the design of such systems: ability, authority, control and responsibility, as well as their relationship to each other and to concepts like levels of automation and autonomy. Consistency in the relations between these concepts is identified as an important quality for the system design. A simple graphical tool is introduced that can help to visualize the cornerstone concepts and their relations in a single diagram. Examples from the automotive domain, where a cooperative guidance and control of highly automated vehicles is under investigation, demonstrate the application of the concepts and the tool. Transitions in authority and control, e.g. initiated by changes in the ability of human or machine, are identified as key challenges. A sufficient consistency of the mental models of human and machines, not only in the system use but also in the design and evaluation, can be a key enabler for a successful dynamic balance between humans and machines.

The theater-system technique: agile designing and testing of system behavior and interaction, applied to highly automated vehicles

In this paper, the theater-system technique, a method for agile designing and testing of system behavior and interaction concepts is described. The technique is based on the Wizard-of-Oz approach, originally used for emulating automated speech recognition, and is extended towards an interactive, user-centered design technique. The paper describes the design process using the theater-system technique, the technical build-up of the theater-system, and an application of the technique: the design of a haptic-multimodal interaction strategy for highly automated vehicles. The use of the theater-system in the design process is manifold: It is used for the concrete design work of the design team, for the assessment of user expectations as well as for early usability assessments, extending the principles of user-centered design towards a dynamically balanced design.

Shared and cooperative movement control of intelligent technical systems: Sketch of the design space of haptic-multimodal coupling between operator, co-automation, base system and environment

Abstract This paper sketches the concept of haptic-multimodal coupling between operator, co-automation, base system and environment. Haptic-multimodal couplings use mainly the haptic interaction resource, e.g. the combination of hands and feet with active inceptors like active sidesticks or steering wheels and complement this with e.g. visual and acoustic feedback. Haptic-multimodal couplings can serve as a base for shared control, and, if the co-automation has a minimum of understanding of and reactivity to the human operator, for a cooperative control between operator and automation. The paper gives a brief introduction of shared and cooperative control, starting with examples in the non-technical world and sketches the basic structure the couplings and coupling schemes. While much of the design space is yet to be explored and described more systematically, some combinations of haptic-multimodal couplings can already be applied, for example to the cooperative control of an intelligent ground vehicle or in telerobotics. The paper briefly describes examples of an automation-initiated de-coupling of a driver and of a helicopter pilot in case of an emergency maneuver and the coupling between an operator and a satellite control for a berthing maneuver.

Uncanny and Unsafe Valley of Assistance and Automation: First Sketch and Application to Vehicle Automation

Progress in sensors, computer power and increasing connectivity allow to build and operate more and more powerful assistance and automation systems, e.g. in aviation, cars and manufacturing. Besides many benefits, new problems occur e.g. in human-machine-interaction. In the field of automation, e.g. vehicle automation, a comparable, metaphorical design correlation is implied, an unsafe valley e.g. between partially- and highly-automated automation levels, in which due to misperceptions a loss of safety could occur. This contribution sketches the concept of the (uncanny and) unsafe valley of automation, summarizes early affirmative studies, gives first hints towards an explanation of the valley, outlines the design space how to secure the borders of the valley, and how to bridge the valley.

Joint Decision Making and Cooperative Driver-Vehicle Interaction during Critical Driving Situations

Abstract In automated driving, the human driver and an automation form a joint human-machine system. In this system, each partner has her own individual cognitive as well as perceptual processes, which enable them to perform the complex task of driving. On different layers of the driving task, both, drivers and automation systems, assess the situation and derive action decisions. Although the processes can be divided between human and machine, and are sometimes very elaborate, the outcome should be a joint one because it affects the entire driver-vehicle system. In this paper, the individual processes for decision-making are defined and a framework for joint decision-making is proposed. Joint decision-making relies on common goals and norms of the two subsystems, human and automation, and evolves with experience.

Behaviour Adaptation Using Interaction Patterns with Augmented Reality Elements

This publication describes a systematic approach for behaviour adaptations of humans, based on interaction patterns as a fundamental way to design and describe human machine interaction, and on image schemas as the basic elements of the resulting interaction. The natural learning path since childhood involves getting knowledge by experience; it is during this process that image schemas are built. The approach described in this paper was developed in close interplay with the concepts of cooperative guidance and control (CGC), where a cooperative automation and a human control a machine together, and of augmented reality (AR), where a natural representation of the world, e.g. in form of a video stream, is enriched with dynamic symbology. The concept was instantiated as interaction patterns “longitudinal and lateral collision avoidance”, implemented in a fix based simulator, and tested with professional operators whether driving performance and safety in a vehicle with restricted vision could be improved. Furthermore, it was tested whether interaction patterns could be used to adapt the current driver behaviour towards better performance while reducing the task load. Using interaction patterns that escalated according to the drivers actions and the current environmental state, lead to a reduction of temporal demand, effort and frustration. Furthermore less collisions were counted and the overall lateral displacement of the vehicle was reduced. The results were a good mix of encouragement and lessons learned, both for the methodical approach of pattern based human machine interaction, and for the application of AR-based cooperative guidance and control.

Derivation of a Model of Safety Critical Transitions between Driver and Vehicle in Automated Driving

In automated driving, there is the risk that users must take over the vehicle guidance despite a potential lack of involvement in the driving task. This publication presents an initial model of control distribution between users and the automated system. In this model, the elements of the control distribution in automated driving are addressed together with possible and safe transitions between different driving modes. Furthermore, the approach is initially empirically validated. In a driving study, in which participants operated both driving and a non-driving related task, objective driving data as well as eye-tracking parameters are used to estimate the model’s accuracy. Such an explanatory model can serve as a first approach to describe potential concepts of cooperation between users and automated vehicles. In this way, prospective road traffic concepts could be improved by preventing safety critical transitions between the driver and the vehicle.

Automotive UI for Controllability and Safe Transitions of Control

Highly automated vehicles require users to take over control when they reach the limits of the automation. These control transitions can lead to hazardous accidents if the human and automation do not have a consistent mental model of the abilities, authorities, and responsibilities of each other. In this workshop, we aim to apply existing knowledge to identify issues in control transitions in co-operative human-machine systems and propose solutions for them. Concrete focus points concern controllability, driving mode awareness, interaction design problems such as conveying the state of driver, automation, and vehicle, and evaluation methods.

Kooperativ interagierende Automobile

Zusammenfassung Die Verknüpfung von Automatisierung mit Vernetzung von Fahrzeugen weist in langfristiger Betrachtung eine kaum überschätzbare ökonomische, ökologische und gesellschaftliche Relevanz auf. In der langfristigen Vision wird die vernetzte Automatisierung unseren Straßenverkehr grundlegend verändern. Kooperatives Verhalten zwischen Verkehrsteilnehmern wird dadurch automatisiert erfolgen und somit das Regelverhalten bilden. Außerdem eröffnet technische Kooperation im Vergleich zur heutigen manuellen Kooperation zwischen Verkehrsteilnehmern qualitativ verbesserte und neuartige funktionale Möglichkeiten aufgrund der erzielbaren Latenzzeit, Genauigkeit, Eindeutigkeit und Verlässlichkeit automatischen Verhaltens. Entsprechend aufeinander abgestimmte Trajektorien weisen eine höhere erreichbare Dichte, Geschwindigkeit und Sicherheit auf. Bei langfristig zu erwartendem hohem Durchdringungsgrad des Straßenverkehrs mit vernetzten automatischen Verkehrsteilnehmern sind sichere und hocheffiziente Verkehrsflüsse denkbar, die biologischen Schwarmbewegungen nahe kommen. Dieser Überblicksbeitrag beschreibt den Stand der Forschung in den wesentlichen Forschungsfeldern und das von der DFG eingerichtete Schwerpunktprogramm Kooperativ interagierende Automobile.

Vision and Driving Support for Shielded Vehicles – Implementation and Test of an Electronic Vision Replacement System with Augmented Reality

The compromises encountered in the design of shielded and armored vehicles regarding the trade-offs between outside visibility and crew protection often result in the decrease of situational awareness. Drivers using busy roads face the challenge of maneuvering large vehicles, which, due to their safety requirements, have large blind spots that make it difficult to detect elements that could pose a threat to the safety of the crew and the vehicle. This paper presents the implementation of a camera-monitor system used as an electronic vision replacement and its evaluation results from a test trial with soldiers from the German Bundeswehr. Additionally, elements of Augmented Reality were incorporated to further assist the driver while navigating through the simulated urban scenario.