Ayumi Sugiyama

Learning and relearning of target decision strategies in continuous coordinated cleaning tasks with shallow coordination1

We propose a method of autonomous learning of target decision strategies for coordination in the continuous cleaning domain. With ongoing advances in computer and sensor technologies, we can expect robot applications for covering large areas that often require coordinated/cooperative activities by multiple robots. We focus on the cleaning tasks by multiple robots or by agents which are programs to control the robots in this paper. We assumed situations where agents did not directly exchange deep and complicated internal information and reasoning results such as plans, strategies and long-term targets for their sophisticated coordinated activities, but rather exchanged superficial information such as the locations of other agents (using the equipment de- ployed) for their shallow coordination and individually learned appropriate strategies by observing how much dirt/dust had been vacuumed up in multi-agent system environments. We will first discuss the preliminary method of improving the coordinated activities by autonomously learning to select cleaning strategies to determine which targets to move to clear them. Although we could have improved the efficiency of cleaning, we observed a phenomenon where performance degraded if agents continued to learn strategies. This is because so many agents overly selected the same strategy (over-selection) by using autonomous learning. In addition, the preliminary method assumed information given about which regions in the environment easily became dirty. Thus, we propose a method that was extended by incorporating the preliminary method with (1) environmental learning to iden- tify which places were likely to be dirty and (2) autonomous relearning through self-monitoring the amount of vacuumed dirt to avoid strategies from being over-selected. We experimentally evaluated the proposed method by comparing its performance with those obtained by the regimes of agents with a single strategy and obtained with the preliminary method. The experimental results revealed that the proposed method enabled agents to select target decision strategies and, if necessary, to abandon the current strategies from their own perspectives, resulting in appropriate combinations of multiple strategies. We also found that environmental learning on dirt accumulation was effectively learned.

Analysis of task allocation based on social utility and incompatible individual preference

This paper proposes a task allocation method in which, although social utility is attempted to be maximized, agents also give weight to individual preferences based on their own specifications and capabilities. Due to the recent advances in computer and network technologies, many services can be provided by appropriately combining multiple types of information and different computational capabilities. The tasks that are carried out to perform these services are executed by allocating them to appropriate agents, which are computational entities having specific functionalities. However, these tasks are huge and appear simultaneously, and task allocation is thus a challenging issue since it is a combinatorial problem. The proposed method, which is based on our previous work, allocates resources/tasks to the appropriate agents by taking into account both social utility and individual preferences. We experimentally demonstrate that the appropriate strategy to decide the preference depends on the type of task and the features of the reward function as well as the social utility.

Meta-strategy for cooperative tasks with learning of environments in multi-agent continuous tasks

With the development of robot technology, we can expect self-propelled robots working in large areas where cooperative and coordinated behaviors by multiple (hardware and software) robots are necessary. However, it is not trivial for agents, which are control programs running on robots, to determine the actions for their cooperative behaviors, because such strategies depend on the characteristics of the environment and the capabilities of individual agents. Therefore, using the example of continuous cleaning tasks by multiple agents, we propose a method of meta-strategy that decide the appropriate planning strategies for cooperation and coordination through with the learning of the performance of individual strategies and the environmental data in a multi-agent systems context, but without complex reasoning for deep coordination due to the limited CPU capability and battery capacity. We experimentally evaluated our method by comparing it with a conventional method that assumes that agents have knowledge on where agents visit frequently (since they are easy to become dirty). We found that agents with the proposed method could operate as effectively as and, in complex areas, outperformed those with the conventional method. Finally, we describe that the reasons for such a counterintuitive phenomenon is induced from splitting up in working by autonomous agents based on the local observations. We also discuss the limitation of the current method.

Autonomous strategy determination with learning of environments in multi-agent continuous cleaning

With the development of robot technology, we can expect self- propelled robots working in large areas where coordinated and collaborative behaviors by multiple robots are necessary. Thus, the learning appropriate strategy for coordination and cooperation in multiple autonomous agents is an important issue. However, conventional methods assumed that agents was given knowledge about the environment. This paper proposes a method of autonomous strategy learning for multiple agents coordination integrated with learning where are easy to become dirty in the environments using examples of continuous cleaning tasks. We found that agents with the proposed method could operate as effectively as those with the conventional method and we found that the proposed method often outperformed it in complex areas by splitting up in their works.

Improvement of Robustness to Environmental Changes by Autonomous Divisional Cooperation in Multi-agent Cooperative Patrol Problem

We propose a learning and negotiation method to enhance divisional cooperation and demonstrate its robustness to environmental changes in the context of the multi-agent cooperative problem. With the ongoing advances in information and communication technology, we now have access to a vast array of information, and everything has become more closely connected due to innovations such as the Internet of Things. However, this makes the tasks/problems in these environments complicated. In particular, we often require fast decision making and flexible responses to adapt to changes of environment. For these requirements, multi-agent systems have been attracting interest, but the manner in which multiple agents cooperate with each other is a challenging issue because of the computational cost, environmental complexity, and sophisticated interaction required between agents. In this work, we address a problem called the continuous cooperative patrol problem, which requires high autonomy, and propose an autonomous learning method with simple negotiation to enhance divisional cooperation for efficient work. We also investigate how this system can have high robustness, as this is one of the key elements in an autonomous distributed system. We experimentally show that agents with our method generate role sharing in a bottom-up manner for effective divisional cooperation. The results also show that two roles, specialist and generalist, emerged in a bottom-up manner, and these roles enhanced the overall efficiency and the robustness to environmental change.

Learning to Coordinate with Deep Reinforcement Learning in Doubles Pong Game

This paper discusses the emergence of cooperative and coordinated behaviors between joint and concurrent learning agents using deep Q-learning. Multi-agent systems (MAS) arise in a variety of domains. The collective effort is one of the main building blocks of many fundamental systems that exist in the world, and thus, sequential decision making under uncertainty for collaborative work is one of the important and challenging issues for intelligent cooperative multiple agents. However, the decisions for cooperation are highly sophisticated and complicated because agents may have a certain shared goal or individual goals to achieve and their behavior is inevitably influenced by each other. Therefore, we attempt to explore whether agents using deep Q-networks (DQN) can learn cooperative behavior. We use doubles pong game as an example and we investigate how they learn to divide their works through iterated game executions. In our approach, agents jointly learn to divide their area of responsibility and each agent uses its own DQN to modify its behavior. We also investigate how learned behavior changes according to environmental characteristics including reward schemes and learning techniques. Our experiments indicate that effective cooperative behaviors with balanced division of workload emerge. These results help us to better understand how agents behave and interact with each other in complex environments and how they coherently choose their individual actions such that the resulting joint actions are optimal.

Adaptive Task Allocation Based on Social Utility and Individual Preference in Distributed Environments

Abstract Recent advances in computer and network technologies enable the provision of many services combining multiple types of information and different computational capabilities. The tasks for these services are executed by allocating them to appropriate collaborative agents, which are computational entities with specific functionality. However, the number of these tasks is huge, and these tasks appear simultaneously, and appropriate allocation strongly depends on the agent’s capability and the resource patterns required to complete tasks. Thus, we first propose a task allocation method in which, although the social utility for the shared and required performance is attempted to be maximized, agents also give weight to individual preferences based on their own specifications and capabilities. We also propose a learning method in which collaborative agents autonomously decide the preference adaptively in the dynamic environment. We experimentally demonstrate that the appropriate strategy to decide the preference depends on the type of task and the features of the task reward. We then show that agents using the proposed learning method adaptively decided their preference and could maintain excellent performance in a changing environment.

Frequency-Based Multi-agent Patrolling Model and Its Area Partitioning Solution Method for Balanced Workload

Multi-agent patrolling problem has received growing attention from many researchers due to its wide range of potential applications. In realistic environment, e.g., security patrolling, each location has different visitation requirement according to the required security level. Therefore, a patrolling system with non-uniform visiting frequency is preferable. The difference in visiting frequency generally causes imbalanced workload amongst agents leading to inefficiency. This paper, thus, aims at partitioning a given area to balance agents’ workload by considering that different visiting frequency and then generating route inside each sub-area. We formulate the problem of frequency-based multi-agent patrolling and propose its semi-optimal solution method, whose overall process consists of two steps – graph partitioning and sub-graph patrolling. Our work improve traditional k-means clustering algorithm by formulating a new objective function and combine it with simulated annealing – a useful tool for operations research. Experimental results illustrated the effectiveness and reasonable computational efficiency of our approach.