Sébastien Paquet

Online planning algorithms for POMDPs

Partially Observable Markov Decision Processes (POMDPs) provide a rich framework for sequential decision-making under uncertainty in stochastic domains. However, solving a POMDP is often intractable except for small problems due to their complexity. Here, we focus on online approaches that alleviate the computational complexity by computing good local policies at each decision step during the execution. Online algorithms generally consist of a lookahead search to find the best action to execute at each time step in an environment. Our objectives here are to survey the various existing online POMDP methods, analyze their properties and discuss their advantages and disadvantages; and to thoroughly evaluate these online approaches in different environments under various metrics (return, error bound reduction, lower bound improvement). Our experimental results indicate that state-of-the-art online heuristic search methods can handle large POMDP domains efficiently.

An online POMDP algorithm for complex multiagent environments

In this paper, we present an online method for POMDPs, called RTBSS (Real-Time Belief Space Search), which is based on a look-ahead search to find the best action to execute at each cycle in an environment. We thus avoid the overwhelming complexity of computing a policy for each possible situation. By doing so, we show that this method is particularly efficient for large real-time environments where offline approaches are not applicable because of their complexity. We first describe the formalism of our online method, followed by some results on standard POMDPs. Then, we present an adaptation of our method for a complex multiagent environment and results showing its efficiency in such environments.

CHARGE TRANSPORT AND SIGNAL GENERATION IN CDTE PIXEL DETECTORS

Abstract The current and charge signals on individual pixels of CdTe γ-ray cameras are calculated. The time and position dependent free carrier densities following the absorption of a γ-ray in the detector are first calculated by solving the continuity equations. These density distributions are then combined with the physical electric field and the pixel weighting field to provide the current induced through an individual pixel from which the charge signal is also obtained. Simulations are made for various pixel sizes. These simulations confirm that the energy resolution of electron collecting pixel detectors must improve when smaller pixels are used since small pixels are less sensitive to holes.

Real-Time decision making for large POMDPs

In this paper, we introduce an approach called RTBSS (Real-Time Belief Space Search) for real-time decision making in large POMDPs The approach is based on a look-ahead search that is applied online each time the agent has to make a decision RTBSS is particularly interesting for large real-time environments where offline solutions are not applicable because of their complexity.

Comparison of different coordination strategies for the RoboCupRescue simulation

One of the difficulties of using Artificial Neural Networks (ANNs) to estimate atmospheric temperature is the large number of potential input variables available. In this study, four different feature extraction methods were used to reduce the input vector to train four networks to estimate temperature at different atmospheric levels. The four techniques used were: genetic algorithms (GA), coefficient of determination (CoD), mutual information (MI) and simple neural analysis (SNA). The results demonstrate that of the four methods used for this data set, mutual information and simple neural analysis can generate networks that have a smaller input parameter set, while still maintaining a high degree of accuracy.

Multiagent systems viewed as distributed scheduling systems: methodology and experiments

In this article, we present a design technique that facilitates the work of extracting and defining the tasks scheduling problem for a multiagent system We also compare a centralized scheduling approach to a decentralized scheduling approach to see the difference in the efficiency of the schedules and the amount of information transmitted between the agents Our experimental results show that the decentralized approach needs less messages, while being as efficient as the centralized approach.

Multi-attribute decision making in a complex multiagent environment using reinforcement learning with selective perception

Choosing between multiple alternative tasks is a hard problem for agents evolving in an uncertain real-time multiagent environment. An example of such environment is the RoboCupRescue simulation, where at each step an agent has to choose between a number of tasks. To do that, we have used a reinforcement learning technique where an agent learns the expected reward it should obtain if it chooses a particular task. Since all possible tasks can be described by a lot of attributes, we have used a selective perception technique to enable agents to narrow down the description of each task.

Learning coordination in RoboCupRescue

In this abstract, we present a complex multiagent environment, the RoboCupRescue simulation, and show some of the learning opportunities for the coordination of agents in this environment.

Task allocation learning in a multiagent environment: Application to the RoboCupRescue simulation

Coordinating agents in a complex environment is a hard problem, but it can become even harder when certain characteristics of the tasks, like the required number of agents, are unknown. In these settings, agents not only have to coordinate themselves on the different tasks, but they also have to learn how many agents are required for each task. To contribute to this problem, we present in this paper a selective perception reinforcement learning algorithm which enables agents to learn the required number of agents that should coordinate their efforts on a given task. Even though there are continuous variables in the task description, agents in our algorithm are able to learn their expected reward according to the task description and the number of agents. The results, obtained in the RoboCupRescue simulation environment, show an improvement in the agents overall performance.

Learning the required number of agents for complex tasks

Coordinating agents in a complex environment is a hard problem, but it can become even harder when certain characteristics of the tasks, like the required number of agents, are unknown. In those settings, agents not only have to coordinate themselves on the different tasks, but they also have to learn how many agents are required for each task. To achieve that, we have elaborated a selective perception reinforcement learning algorithm to enable agents to learn the required number of agents. Even though there were continuous variables in the task description, the agents were able to learn their expected reward according to the task description and the number of agents. The results, obtained in the RoboCupRescue, show an improvement in the agents overall performance.