Steven Nunnally

Towards human control of robot swarms

In this paper we investigate principles of swarm control that enable a human operator to exert influence on and control large swarms of robots. We present two principles, coined selection and beacon control, that differ with respect to their temporal and spatial persistence. The former requires active selection of groups of robots while the latter exerts a passive influence on nearby robots. Both principles are implemented in a testbed in which operators exert influence on a robot swarm by switching between a set of behaviors ranging from trivial behaviors up to distributed autonomous algorithms. Performance is tested in a series of complex foraging tasks in environments with different obstacles ranging from open to cluttered and structured. The robotic swarm has only local communication and sensing capabilities with the number of robots ranging from 50 to 200. Experiments with human operators utilizing either selection or beacon control are compared with each other and to a simple autonomous swarm with regard to performance, adaptation to complex environments, and scalability to larger swarms. Our results show superior performance of autonomous swarms in open environments, of selection control in complex environments, and indicate a potential for scaling beacon control to larger swarms.

Human-swarm interaction: an experimental study of two types of interaction with foraging swarms

In this paper we present the first study of human-swarm interaction comparing two fundamental types of interaction, coined intermittent and environmental. These types are exemplified by two control methods, selection and beacon control, made available to a human operator to control a foraging swarm of robots. Selection and beacon control differ with respect to their temporal and spatial influence on the swarm and enable an operator to generate different strategies from the basic behaviors of the swarm. Selection control requires an active selection of groups of robots while beacon control exerts an influence on nearby robots within a set range. Both control methods are implemented in a testbed in which operators solve an information foraging problem by utilizing a set of swarm behaviors. The robotic swarm has only local communication and sensing capabilities. The number of robots in the swarm range from 50 to 200. Operator performance for each control method is compared in a series of missions in different environments with no obstacles up to cluttered and structured obstacles. In addition, performance is compared to simple and advanced autonomous swarms. Thirty-two participants were recruited for participation in the study. Autonomous swarm algorithms were tested in repeated simulations. Our results showed that selection control scales better to larger swarms and generally outperforms beacon control. Operators utilized different swarm behaviors with different frequency across control methods, suggesting an adaptation to different strategies induced by choice of control method. Simple autonomous swarms outperformed human operators in open environments, but operators adapted better to complex environments with obstacles. Human controlled swarms fell short of task-specific benchmarks under all conditions. Our results reinforce the importance of understanding and choosing appropriate types of human-swarm interaction when designing swarm systems, in addition to choosing appropriate swarm behaviors.

Neglect benevolence in human control of swarms in the presence of latency

Autonomous swarm algorithms have been studied extensively in the past several years. However, there is little research on the effect of injecting human influence into a robot swarm-whether it be to update the swarms current goals or reshape swarm behavior. While there has been growing research in the field of human-swarm interaction (HSI), no previous studies have investigated how humans interact with swarms under communication latency.We investigate the effects of latency both with and without a predictive display in a basic swarm foraging task to see if such a display can help mitigate the effects of delayed feedback of the swarm state. Furthermore, we introduce a new concept called neglect benevolence to represent how a human operator may need to give time for swarm algorithms to stabilize before issuing new commands, and we investigate it with respect to task performance. Our study shows that latency did affect a users ability to control a swarm to find targets in the foraging task, and that the predictive display helped to remove these effects. We also found evidence for neglect benevolence, and that operators exploited neglect benevolence in different ways, leading to two different, but equally successful strategies in the target-searching task.

Human influence of robotic swarms with bandwidth and localization issues

Swarm robots use simple local rules to create complex emergent behaviors. The simplicity of the local rules allows for large numbers of low-cost robots in deployment, but the same simplicity creates difficulties when deploying in many applicable environments. These complex missions sometimes require human operators to influence the swarms towards achieving the mission goals. Human swarm interaction (HSI) is a young field with few user studies exploring operator behavior. These studies all assume perfect information between the operator and the swarm, which is unrealistic in many applicable scenarios. Indoor search and rescue or underwater exploration may present environments where radio limitations restrict the bandwidth of the robots. This study explores this bandwidth restriction in a user study. Three levels of bandwidth are explored to determine what amount of information is necessary to accomplish a swarm foraging task. The lowest bandwidth condition performs poorly, but the medium and high bandwidth condition both perform well. The medium bandwidth condition does so by aggregating useful swarm information to compress the state information. Further, the study shows operators preferences that should have hindered task performance, but operator adaptation allowed for error correction.

Using Haptic Feedback in Human Robotic Swarms Interaction

Robotic swarms display emergent behaviors that are robust to failure of individual robots, although they can not necessarily accomplish complex tasks with these behaviors. The research objective is to make use of their robust behaviors to accomplish complex tasks in many types of environments. For now, it is difficult to affect swarm “goals”, and therefore difficult them to direct to perform complex tasks. The extant literature on Human Swarm Interaction (HSI) focuses on demonstrating the usefulness of human operator inputs for swarms to accomplish complex tasks. The human typically gets visual feedback of the state of the swarm and influences the robots through a computer interface. This paper presents a user study investigating the effectiveness of haptic feedback in improving HSI. We use methods developed in studies using haptics in multi-robot systems (where the communication and structure is very rigid) and potential field algorithms developed for fully-autonomous swarms to determine the benefits of haptic feedback from the semi-autonomous control algorithm. In some environments, haptic feedback proved beneficial whereas in other environments haptic feedback did not improve performance over visual feedback alone. However, presence of haptic feedback did not degrade the performance under any of the experimental conditions. This supports our working hypothesis that haptic feedback is potentially useful in HSI.

Using Coverage for Measuring the Effect of Haptic Feedback in Human Robotic Swarm Interaction

A robotic swarm is a decentralized group of robots which overcome failure of individual robots with robust emergent behaviors based on local interactions. These behaviors are not well built for accomplishing complex tasks, however, because of the changing assumptions required in various applications and environments. A new movement in the research field is to add human input to influence the swarm in order to help make the robots goal directed and overcome these problems. This research in Human Swarm Interaction (HSI) focuses on different control laws and ways to integrate the human intent with local control laws of the robots. Previous studies have all used visual feedback through a computer interface to give the user the swarm state information. This study adapted swarm control algorithms to give the operator hap tic feedback as well as visual feedback. The study shows the benefits of the additional feedback in a target searching class. Researchers in multi-robot systems have shown benefits of hap tic feedback in obstacle navigation before, but this study is a novel method because of the decentralized formation of the robotic swarm. In most environments, operators were able to cover significantly more area, increasing the chance of finding more targets. The other environment found no significant difference, showing that the hap tic feedback does not degrade performance in any of the tested environments. This supports our hypothesis that hap tic feedback is useful in HSI and requires further research to maximize its potential.

Levels of Automation for Human Influence of Robot Swarms

Autonomous swarm algorithms and human-robot interaction (HRI) have both attracted increasing attention from researchers in recent years. However, HRI has rarely extended beyond single robots or small multi-robot teams. While one of the benefits of robot swarms is their robust capabilities and the ability of their distributed algorithms to deal autonomously with the complex interactions amongst swarm members, there is undoubtedly a need for humans to influence such swarms in some circumstances—especially when these swarms are operating in unknown or hostile environments. In this paper, we approach the problem of human-swarm interaction (HSI) using previous research in levels of automation (LOAs) in HRI. We create a target searching task whereby the swarm can operate at two different levels of autonomy: an autonomous dispersion algorithm, or user-defined goto points. We investigate what environmental conditions are conducive to different amounts of human influence, and at what point further human intervention has a detrimental effect on the swarm’s performance. The results show that for complex environments containing numerous obstacles and small passageways, there is indeed a need for some human influence; however, after a certain point, further influence causes performance degradation.

Neglect benevolence in human-swarm interaction with communication latency

In practical applications of robot swarms with bio-inspired behaviors, a human operator will need to exert control over the swarm to fulfill the mission objectives. In many operational settings, human operators are remotely located and the communication environment is harsh. Hence, there exists some latency in information (or control command) transfer between the human and the swarm. In this paper, we conduct experiments of human-swarm interaction to investigate the effects of communication latency on the performance of a human-swarm system in a swarm foraging task. We develop and investigate the concept of neglect benevolence, where a human operator allows the swarm to evolve on its own and stabilize before giving new commands. Our experimental results indicate that operators exploited neglect benevolence in different ways to develop successful strategies in the foraging task. Furthermore, we show experimentally that the use of a predictive display can help mitigate the adverse effects of communication latency.

An indirect measure of the implicit level of presence in virtual environments

Virtual Environments (VEs) are a common occurrence for many computer users. Considering their spreading usage and speedy development it is ever more important to develop methods that capture and measure key aspects of a VE, like presence. One of the main problems with measuring the level of presence in VEs is that the users may not be consciously aware of its affect. This is a problem especially for direct measures that rely on questionnaires and only measure the perceived level of presence explicitly. In this paper we develop and validate an indirect measure for the implicit level of presence of users, based on the physical reaction of users to events in the VE. The addition of an implicit measure will enable us to evaluate and compare VEs more effectively, especially with regard to their main function as immersive environments. Our approach is practical, cost-effective and delivers reliable results.