Mark A. Neerincx

Experience in System Design for Human-Robot Teaming in Urban Search and Rescue

The paper describes experience with applying a user-centric design methodology in developing systems for human-robot teaming in Urban Search & Rescue. A human-robot team consists of several robots (rovers/UGVs, microcopter/UAVs), several humans at an off-site command post (mission commander, UGV operators) and one on-site human (UAV operator). This system has been developed in close cooperation with several rescue organizations, and has been deployed in a real-life tunnel accident use case. The human-robot team jointly explores an accident site, communicating using a multi-modal team interface, and spoken dialogue. The paper describes the development of this complex socio-technical system per se, as well as recent experience in evaluating the performance of this system.

Situated cognitive engineering for crew support in space

Space crews are in need for excellent cognitive support to perform nominal and off-nominal actions. This paper presents a coherent cognitive engineering methodology for the design of such support, which may be used to establish adequate usability, context-specific support that is integrated into astronaut’s task performance and/or electronic partners who enhance human–machine team’s resilience. It comprises (a) usability guidelines, measures and methods, (b) a general process guide that integrates task procedure design into user interface design and a software framework to implement such support and (c) theories, methods and tools to analyse, model and test future human–machine collaborations in space. In empirical studies, the knowledge base and tools for crew support are continuously being extended, refined and maintained.

Unobtrusive monitoring of knowledge workers for stress self-regulation

In our connected workplaces it can be hard to work calm and focused. In a simulated work environment we manipulated the stressors time pressure and email interruptions. We found effects on subjective experience and working behavior. Initial results indicate that the sensor data that we collected is suitable for user state modeling in stress related terms.

Interaction Design Patterns for Adaptive Human-Agent-Robot Teamwork in High-Risk Domains

Integrating cognitive agents and robots into teams that operate in high-demand situations involves mutual and context-dependent behaviors of the human and agent/robot team-members. We propose a cognitive engineering method that includes the development of Interaction Design patterns for such systems as re-usable, theoretically and empirically founded, design solutions. This paper presents an overview of the background, the method and three example patterns.

ICM: An Intuitive Model Independent and Accurate Certainty Measure for Machine Learning.

End-users of machine learning-based systems benefit from measures that quantify the trustworthiness of the underlying models. Measures like accuracy provide for a general sense of model performance, but offer no detailed information on specific model outputs. Probabilistic outputs, on the other hand, express such details, but they are not available for all types of machine learning, and can be heavily influenced by bias and lack of representative training data. Further, they are often difficult to understand for non-experts. This study proposes an intuitive certainty measure (ICM) that produces an accurate estimate of how certain a machine learning model is for a specific output, based on errors it made in the past. It is designed to be easily explainable to non-experts and to act in a predictable, reproducible way. ICM was tested on four synthetic tasks solved by support vector machines, and a real-world task solved by a deep neural network. Our results show that ICM is both more accurate and intuitive than related approaches. Moreover, ICM is neutral with respect to the chosen machine learning model, making it widely applicable

Free look UGV teleoperation control tested in game environment: Enhanced performance and reduced workload

Concurrent telecontrol of the chassis and camera of an Unmanned Ground Vehicle (UGV) is a demanding task for Urban Search and Rescue (USAR) teams. The standard way of controlling UGVs is called Tank Control (TC), but there is reason to believe that Free Look Control (FLC), a control mode used in games, could reduce this load substantially by decoupling, and providing separate controls for, camera translation and rotation. The general hypothesis is that FLC (1) reduces robot operators workload and (2) enhances their performance for dynamic and time-critical USAR scenarios. A game-based environment was set-up to systematically compare FLC with TC in two typical search and rescue tasks: navigation and exploration. The results show that FLC improves mission performance in both exploration (search) and path following (navigation) scenarios. In the former, more objects were found, and in the latter shorter navigation times were achieved. FLC also caused lower workload and stress levels in both scenarios, without inducing a significant difference in the number of collisions. Finally, FLC was preferred by 75% of the subjects for exploration, and 56% for path following.

Dynamic Task Allocation for Human-robot Teams

Artificial agents, such as robots, are increasingly deployed for teamwork in dynamic, high-demand environments. This paper presents a framework, which applies context information to establish task (re)allocations that improve human-robot teamâ??s performance. Based on the framework, a model for adaptive automation was designed that takes the cognitive task load (CTL) of a human team member and the coordination costs of switching to a new task allocation into account. Based on these two context factors, it tries to optimize the level of autonomy of a robot for each task. The model was instantiated for a single human agent cooperating with a single robot in the urban search and rescue domain. A first experiment provided encouraging results: the cognitive task load of participants mostly reacted to the model as intended. Recommendations for improving the model are provided, such as adding more context information.

Human-Robot Teamwork in USAR Environments: the TRADR Project

The TRADR project aims at developing methods and models for human-robot teamwork, enabling robots to operate in search & rescue environments alongside humans as teammates, rather than as tools. Through a user-centered cognitive engineering method, human-robot teamwork is analyzed, modeled, implemented and evaluated in an iterative fashion. Important is the notion of persistence: rather than treating each sortie as a separate instance for which the build-up of situation awareness and exploration starts from scratch, the objective for the TRADR project is to provide robotic support in an ongoing, fluent manner. This paper provides a short overview of important aspects for human-robot teaming, such as human-robot teamwork coordination and joint situation awareness.

Measuring emotions of robot operators in urban search and rescue missions

This paper evaluates the feasibility and reliability of measuring the (emotional) state of the robot operators in urban search and rescue missions in real-time. An experiment has been conducted, in which a high-fidelity team task in a realistic urban search and rescue setting was executed by fire fighters in cooperation with robots. During the task, several emotion-eliciting events were triggered. In addition, the heart rate variability, skin conductance and facial expressions were monitored. After the scenario execution, the fire fighters were asked to describe their emotional state during task execution. We found that the facial expressions were not reliably recognized, but that heart rate variability and skin conductance measured a higher arousal during (some of) the emotion-eliciting events. However, the different measures still have shortcomings regarding use in complex and dynamic environments.

Estimating an operator's cognitive state in real time: A user modeling approach

This paper presents a Cognitive Task Load (CTL) model designed to keep track of an operators mental workload, both quantitatively (amount of workload) and qualitatively (cognitive state). Every second, the CTL-model updates a diagnosis of the operators cognitive state; by integrating this model in a (semi-)autonomous robot, the robots level of automation and user interface can be attuned to the operators state. The CTL-models predictions were tested in an Urban Search And Rescue (USAR) setting. The test showed insufficient workload variations to validate the model. This indicates that participants should be subjected to more “high-pressure” conditions in future trials. These results also suggest that in a realistic environment, an operators mental workload is affected by high-level coping strategies.