Phillip M. Walker

Human Interaction With Robot Swarms: A Survey

Recent advances in technology are delivering robots of reduced size and cost. A natural outgrowth of these advances are systems comprised of large numbers of robots that collaborate autonomously in diverse applications. Research on effective autonomous control of such systems, commonly called swarms, has increased dramatically in recent years and received attention from many domains, such as bioinspired robotics and control theory. These kinds of distributed systems present novel challenges for the effective integration of human supervisors, operators, and teammates that are only beginning to be addressed. This paper is the first survey of human-swarm interaction (HSI) and identifies the core concepts needed to design a human-swarm system. We first present the basics of swarm robotics. Then, we introduce HSI from the perspective of a human operator by discussing the cognitive complexity of solving tasks with swarm systems. Next, we introduce the interface between swarm and operator and identify challenges and solutions relating to human-swarm communication, state estimation and visualization, and human control of swarms. For the latter, we develop a taxonomy of control methods that enable operators to control swarms effectively. Finally, we synthesize the results to highlight remaining challenges, unanswered questions, and open problems for HSI, as well as how to address them in future works.

Neglect benevolence in human control of swarms in the presence of latency

Autonomous swarm algorithms have been studied extensively in the past several years. However, there is little research on the effect of injecting human influence into a robot swarm-whether it be to update the swarms current goals or reshape swarm behavior. While there has been growing research in the field of human-swarm interaction (HSI), no previous studies have investigated how humans interact with swarms under communication latency.We investigate the effects of latency both with and without a predictive display in a basic swarm foraging task to see if such a display can help mitigate the effects of delayed feedback of the swarm state. Furthermore, we introduce a new concept called neglect benevolence to represent how a human operator may need to give time for swarm algorithms to stabilize before issuing new commands, and we investigate it with respect to task performance. Our study shows that latency did affect a users ability to control a swarm to find targets in the foraging task, and that the predictive display helped to remove these effects. We also found evidence for neglect benevolence, and that operators exploited neglect benevolence in different ways, leading to two different, but equally successful strategies in the target-searching task.

Human influence of robotic swarms with bandwidth and localization issues

Swarm robots use simple local rules to create complex emergent behaviors. The simplicity of the local rules allows for large numbers of low-cost robots in deployment, but the same simplicity creates difficulties when deploying in many applicable environments. These complex missions sometimes require human operators to influence the swarms towards achieving the mission goals. Human swarm interaction (HSI) is a young field with few user studies exploring operator behavior. These studies all assume perfect information between the operator and the swarm, which is unrealistic in many applicable scenarios. Indoor search and rescue or underwater exploration may present environments where radio limitations restrict the bandwidth of the robots. This study explores this bandwidth restriction in a user study. Three levels of bandwidth are explored to determine what amount of information is necessary to accomplish a swarm foraging task. The lowest bandwidth condition performs poorly, but the medium and high bandwidth condition both perform well. The medium bandwidth condition does so by aggregating useful swarm information to compress the state information. Further, the study shows operators preferences that should have hindered task performance, but operator adaptation allowed for error correction.

Human control of robot swarms with dynamic leaders

Controlling a swarm of robots after deployment is difficult, due to the unpredictable and emergent behavior of swarm algorithms. Past work has focused on influencing the swarm via statically selected leaders-swarm members that the operator directly controls-that are pre-selected and remain leaders throughout the scenario execution. This paper investigates the use of dynamically selected leaders that are directly controlled by the human operator to guide the rest of the swarm, which is operating under a flocking-style algorithm. The goal of the operator is to move the swarm to goal regions that arise dynamically in the environment. We experimentally investigated (a) the effect of density of leaders on the ease of human control and system performance, and (b) how restriction of information communicated to the human operator affects the ability to guide the swarm to goal regions. The density of leaders is computed based on an extension of the random competition clustering (RCC) algorithm used in wireless sensor networks to select cluster heads. In particular, we studied the effect of different guarantees of the maximum number of hops in the communication graph from any robot to the nearest leader. Increasing the maximum hop guarantee effectively lowers the density of leaders in the swarm. Our results show that, while there was a large drop in the number of goals reached when moving from a 1-hop to a 2-hop guarantee, the difference between a 2-hop and 3-hop guarantee was not statistically significant. Furthermore, we found that performance was just as good when the information returned to the operator was restricted, showing that operators can still navigate a swarm even when they have imperfect information.

Using Haptic Feedback in Human Robotic Swarms Interaction

Robotic swarms display emergent behaviors that are robust to failure of individual robots, although they can not necessarily accomplish complex tasks with these behaviors. The research objective is to make use of their robust behaviors to accomplish complex tasks in many types of environments. For now, it is difficult to affect swarm “goals”, and therefore difficult them to direct to perform complex tasks. The extant literature on Human Swarm Interaction (HSI) focuses on demonstrating the usefulness of human operator inputs for swarms to accomplish complex tasks. The human typically gets visual feedback of the state of the swarm and influences the robots through a computer interface. This paper presents a user study investigating the effectiveness of haptic feedback in improving HSI. We use methods developed in studies using haptics in multi-robot systems (where the communication and structure is very rigid) and potential field algorithms developed for fully-autonomous swarms to determine the benefits of haptic feedback from the semi-autonomous control algorithm. In some environments, haptic feedback proved beneficial whereas in other environments haptic feedback did not improve performance over visual feedback alone. However, presence of haptic feedback did not degrade the performance under any of the experimental conditions. This supports our working hypothesis that haptic feedback is potentially useful in HSI.

Human Control of Leader-Based Swarms

As swarms are used in increasingly more complex scenarios, further investigation is needed to determine how to give human operators the best tools to properly influence the swarm after deployment. Previous research has focused on relaying influence from the operator to the swarm, either by broadcasting commands to the entire swarm or by influencing the swarm through the teleoperation of a leader. While these methods each have their different applications, there has been a lack of research into how the influence should be propagated through the swarm in leader-based methods. This paper focuses on two simple methods of information propagation-flooding and consensus-and compares the ability of operators to maneuver the swarm to goal points using each, both with and without sensing error. Flooding involves each robot explicitly matching the speed and direction of the leader (or matching the speed and direction of the first neighboring robot that has already done so), and consensus involves each robot matching the average speed and direction of all the neighbors it senses. We discover that the flooding method is significantly more effective, yet the consensus method has some advantages at lower speeds, and in terms of overall connectivity and cohesion of the swarm.

Using Coverage for Measuring the Effect of Haptic Feedback in Human Robotic Swarm Interaction

A robotic swarm is a decentralized group of robots which overcome failure of individual robots with robust emergent behaviors based on local interactions. These behaviors are not well built for accomplishing complex tasks, however, because of the changing assumptions required in various applications and environments. A new movement in the research field is to add human input to influence the swarm in order to help make the robots goal directed and overcome these problems. This research in Human Swarm Interaction (HSI) focuses on different control laws and ways to integrate the human intent with local control laws of the robots. Previous studies have all used visual feedback through a computer interface to give the user the swarm state information. This study adapted swarm control algorithms to give the operator hap tic feedback as well as visual feedback. The study shows the benefits of the additional feedback in a target searching class. Researchers in multi-robot systems have shown benefits of hap tic feedback in obstacle navigation before, but this study is a novel method because of the decentralized formation of the robotic swarm. In most environments, operators were able to cover significantly more area, increasing the chance of finding more targets. The other environment found no significant difference, showing that the hap tic feedback does not degrade performance in any of the tested environments. This supports our hypothesis that hap tic feedback is useful in HSI and requires further research to maximize its potential.

Control of Swarms with Multiple Leader Agents

The study of human control of robotic swarms involves designing interfaces and algorithms for allowing a human operator to influence a swarm of robots. One of the main difficulties, however, is determining how to most effectively influence the swarm after it has been deployed. Past work has focused on influencing the swarm via statically selected leaders-swarm members that the operator directly controls. This paper investigates the use of a small subset of the swarm as leaders that are dynamically selected during the scenario execution and are directly controlled by the human operator to guide the rest of the swarm, which is operating under a flocking-style algorithm. The goal of the operator in this study is to move the swarm to goal regions that arise dynamically in the environment.We experimentally investigated three different aspects of dynamic leader-based swarm control and their interactions: leader density (in terms of guaranteed hops to a leader), sensing error, and method of information propagation from leaders to the rest of the swarm. Our results show that, while there was a large drop in the number of goals reached when moving from a 1-hop to a 2-hop guarantee, the difference between a 2-hop, 3-hop, and 4-hop guarantee was not statistically significant. Furthermore, we found that sensing error impacted the explicit information-propagation method more than the tacit method conditions, and caused participants more trouble the lower the density of leaders, although the explicit method performed better overall.

Levels of Automation for Human Influence of Robot Swarms

Autonomous swarm algorithms and human-robot interaction (HRI) have both attracted increasing attention from researchers in recent years. However, HRI has rarely extended beyond single robots or small multi-robot teams. While one of the benefits of robot swarms is their robust capabilities and the ability of their distributed algorithms to deal autonomously with the complex interactions amongst swarm members, there is undoubtedly a need for humans to influence such swarms in some circumstances—especially when these swarms are operating in unknown or hostile environments. In this paper, we approach the problem of human-swarm interaction (HSI) using previous research in levels of automation (LOAs) in HRI. We create a target searching task whereby the swarm can operate at two different levels of autonomy: an autonomous dispersion algorithm, or user-defined goto points. We investigate what environmental conditions are conducive to different amounts of human influence, and at what point further human intervention has a detrimental effect on the swarm’s performance. The results show that for complex environments containing numerous obstacles and small passageways, there is indeed a need for some human influence; however, after a certain point, further influence causes performance degradation.

Neglect benevolence in human-swarm interaction with communication latency

In practical applications of robot swarms with bio-inspired behaviors, a human operator will need to exert control over the swarm to fulfill the mission objectives. In many operational settings, human operators are remotely located and the communication environment is harsh. Hence, there exists some latency in information (or control command) transfer between the human and the swarm. In this paper, we conduct experiments of human-swarm interaction to investigate the effects of communication latency on the performance of a human-swarm system in a swarm foraging task. We develop and investigate the concept of neglect benevolence, where a human operator allows the swarm to evolve on its own and stabilize before giving new commands. Our experimental results indicate that operators exploited neglect benevolence in different ways to develop successful strategies in the foraging task. Furthermore, we show experimentally that the use of a predictive display can help mitigate the adverse effects of communication latency.