Nathan Schurr

Using multiagent teams to improve the training of incident commanders

The DEFACTO system is a multiagent based tool for training incident commanders for large scale disasters. In this paper, we highlight some of the lessons that we have learned from our interaction with the Los Angeles Fire Department (LAFD) and how they have affected the way that we continued the design of our training system. These lessons were gleaned from LAFD feedback and initial training exercises and they include: system design, visualization, improving trainee situational awareness, adjusting training level of difficulty and situation scale. We have taken these lessons and used them to improve the DEFACTO systems training capabilities. We have conducted initial training exercises to illustrate the utility of the system in terms of providing useful feedback to the trainee.

Conflicts in teamwork: hybrids to the rescue

Today within the AAMAS community, we see at least four competing approaches to building multiagent systems: belief-desire-intention (BDI), distributed constraint optimization (DCOP), distributed POMDPs, and auctions or game-theoretic approaches. While there is exciting progress within each approach, there is a lack of cross-cutting research. This paper highlights hybrid approaches for multiagent teamwork. In particular, for the past decade, the TEAMCORE research group has focused on building agent teams in complex, dynamic domains. While our early work was inspired by BDI, we will present an overview of recent research that uses DCOPs and distributed POMDPs in building agent teams. While DCOP and distributed POMDP algorithms provide promising results, hybrid approaches help us address problems of scalability and expressiveness. For example, in the BDI-POMDP hybrid approach, BDI team plans are exploited to improve POMDP tractability, and POMDPs improve BDI team plan performance. We present some recent results from applying this approach in a Disaster Rescue simulation domain being developed with help from the Los Angeles Fire Department.

Team Oriented Programming and Proxy Agents: The Next Generation

Coordination between large teams of highly heterogeneous entities will change the way complex goals are pursued in real world environments. One approach to achieving the required coordination in such teams is to give each team member a proxy that assumes routine coordination activities on behalf of its team member. Despite that approach’s success, as we attempt to apply this first generation of proxy architecture to larger teams in more challenging environments, some limitations become clear. In this paper, we present initial efforts on the next generation of proxy architecture and Team Oriented Programming (TOP), called Machinetta. Machinetta aims to overcome the limitations of the previous generation of proxies and allow effective coordination between very large teams of highly heterogeneous agents. We describe the principles underlying the design of the Machinetta proxies and present initial results from two domains.

The Defacto System: Coordinating Human-Agent Teams for the Future of Disaster Response

Enabling effective interactions between agent teams and humans for disaster response is a critical area of research, with encouraging progress in the past few years. However, previous work suffers from two key limitations: (i) limited human situational awareness, reducing human effectiveness in directing agent teams and (ii) the agent team’s rigid interaction strategies that limit team performance. This paper presents a software prototype called DEFACTO (Demonstrating Effective Flexible Agent Coordination of Teams through Omnipresence). DEFACTO is based on a software proxy architecture and 3D visualization system, which addresses the two limitations described above. First, the 3D visualization interface enables human virtual omnipresence in the environment, improving human situational awareness and ability to assist agents. Second, generalizing past work on adjustable autonomy, the agent team chooses among a variety of “team-level” interaction strategies, even excluding humans from the loop in extreme circumstances.

The DEFACTO system for human omnipresence to coordinate agent teams: the future of disaster response

Enabling interactions of agent-teams and humans is a critical area of research, with encouraging progress in the past few years. However, previous work suffers from three key limitations: (i) limited human situational awareness, reducing human effectiveness in directing agent teams, (ii) the agent teams rigid interaction strategies that limit team performance, and (iii) lack of formal tools to analyze the impact of such interaction strategies. This paper presents a software prototype called DEFACTO (Demonstrating Effective Flexible Agent Coordination of Teams through Omnipresence). DEFACTO is based on a software proxy architecture and 3D visualization system, which addresses the three limitations mentioned above.

Towards flexible coordination of human-agent teams

Enabling interactions of agent-teams and humans is a critical area of research, with encouraging progress in the past few years. However, previous work suffers from three key limitations: (i) limited human situational awareness, reducing human effectiveness in directing agent teams, (ii) the agent teams rigid interaction strategies that limit team performance, and (iii) lack of formal tools to analyze the impact of such interaction strategies. This article presents a software prototype called DEFACTO (Demonstrating Effective Flexible Agent Coordination of Teams through Omnipresence). DEFACTO is based on a software proxy architecture and 3D visualization system, which addresses the three limitations mentioned above. First, the 3D visualization interface enables human virtual omnipresence in the environment, improving human situational awareness and ability to assist agents. Second, generalizing past work on adjustable autonomy, the agent team chooses among a variety of team-level interaction strategies, even excluding humans from the loop in extreme circumstances. Third, analysis tools help predict the performance of (and choose among) different interaction strategies. DEFACTO is illustrated in a future disaster response simulation scenario, and extensive experimental results are presented.

Exploring human error in an RPA target detection task

Intelligence, Surveillance, and Reconnaissance (ISR) sensing platforms are becoming increasingly complex. Consequently, the fidelity of collected data is continuing to increase, along with the number of deployable sensors that retrieve these data, such as those found on Remotely Piloted Aircraft (RPAs). There are numerous, critical challenges when designing ISR systems because the technology and the human are tightly integrated, resulting in interdependent performance and behaviors. Predicting operator error can inform more effective means of managing erroneous decisions, but current methods of doing so are impractical because of the effort required to construct operator models. We explored human performance in a target detection task by conducting a human-in-the-loop experiment that examined the performance of operators who simultaneously monitored four simulated RPA video feeds and determined the presence of targets at points of interest (POIs). The results of this experiment confirm that performance varies significantly across certain flight conditions (e.g., combinations of altitude, speed, aspect angle). A statistical model was constructed from the human data to predict operator error in new situations. In future work, the model predictions will be integrated with an automated flight planner that will adjust RPA air tasking orders in real time and intelligently revisit POIs when human error is likely.

A General Context-Aware Framework for Improved Human-System Interactions

For humans and automation to effectively collaborate and perform tasks, all participants need access to a common representation of potentially relevant situational information, or context. This article describes a general framework for building context-aware interactive intelligent systems that comprises three major functions: (1) capture human-system interactions and infer implicit context; (2) analyze and predict user intent and goals; and (3) provide effective augmentation or mitigation strategies to improve performance, such as delivering timely, personalized information and recommendations, adjusting levels of automation, or adapting visualizations. Our goal is to develop an approach that enables humans to interact with automation more intuitively and naturally that is reusable across domains by modeling context and algorithms at a higher-level of abstraction. We first provide an operational definition of context and discuss challenges and opportunities for exploiting context. We then describe our current work towards a general platform that supports developing context-aware applications in a variety of domains. We then explore an example use case illustrating how our framework can facilitate personalized collaboration within an information management and decision support tool. Future work includes evaluating our framework.

Multimodal interaction for human-robot teams

Unmanned ground vehicles have the potential for supporting small dismounted teams in mapping facilities, maintaining security in cleared buildings, and extending the team’s reconnaissance and persistent surveillance capability. In order for such autonomous systems to integrate with the team, we must move beyond current interaction methods using heads-down teleoperation which require intensive human attention and affect the human operator’s ability to maintain local situational awareness and ensure their own safety. This paper focuses on the design, development and demonstration of a multimodal interaction system that incorporates naturalistic human gestures, voice commands, and a tablet interface. By providing multiple, partially redundant interaction modes, our system degrades gracefully in complex environments and enables the human operator to robustly select the most suitable interaction method given the situational demands. For instance, the human can silently use arm and hand gestures for commanding a team of robots when it is important to maintain stealth. The tablet interface provides an overhead situational map allowing waypoint-based navigation for multiple ground robots in beyond-line-of-sight conditions. Using lightweight, wearable motion sensing hardware either worn comfortably beneath the operator’s clothing or integrated within their uniform, our non-vision-based approach enables an accurate, continuous gesture recognition capability without line-of-sight constraints. To reduce the training necessary to operate the system, we designed the interactions around familiar arm and hand gestures.

Demonstration of DEFACTO: training tool for incident commanders

In the wake of large-scale national and international terrorist incidents, it is critical to provide first responders and rescue personnel with tools and techniques that will enable them to evaluate response readiness and tactics, measure inter-agency coordination and improve training and decision making capability. We focus in particular on building tools for training and tactics evaluation for incident commanders, who are in charge of managing teams of fire fighters at critical incidents. Such tools would provide intelligent software agents that simulate first responder tactics. decisions, and behaviors in simulated urban areas and allow the incident commander (human) to interact. These agents form teams, where each agent simulates a fire engine, which plans and acts autonomously in a simulated environment. Through interactions with these software agents, an incident commander can evaluate tactics and realize the consequences of key decisions, while responding to such disasters.