Renê Pegoraro

Experience generalization for concurrent reinforcement learners: the minimax-QS algorithm

This paper investigates the use of experience generalization on concurrent and on-line policy learning in multi-agent scenarios, using reinforcement learning algorithms. Agents learning concurrently implies in a non-stationary scenario, since the reward received by one agent (for applying an action in a state) depends on the behavior of the other agents. Non-stationary scenarios can be viewed as a two-player game in which an agent and the other player (which represents the other agents and the environment) select actions from the available actions in the current state; these actions define the possible next state. An RL algorithm that can be applied to such a scenario is the Minimax-Q algorithm, which is known to guarantee convergence to equilibrium in the limit. However, finding optimal control policies using any RL algorithm Minimax-Q included) can be very time consuming. We investigate the use of experience generalization for increasing the rate of convergence of RL algorithms, and contribute a new learning algorithm, Minimax-QS, which incorporates experience generalization to the Minimax-Q algorithm. We also prove its convergence to Minimax-Q values under suitable conditions.

A Self-Healing Architecture for Web Service-Based Applications

Web service-based application is an architectural style, where a collection of Web services communicates to each other to execute processes. With the popularity increase of developing Web service-based application and once Web services may change, in terms of functional and non-functional quality of service (QoS), we need mechanisms to monitor, diagnose, and repair Web services into a Web application. This work presents a description of self-healing architecture that deals with these mechanisms. Other contributions of this paper are using the proxy server to measure Web service QoS values and to employ some strategies to recovery the effects from misbehaved Web services.

Use of myoelectric signals to command mobile entertainment robot by disabled children: Design and control architecture

Severely disabled children have little chance of environmental and social exploration and discovery. This lack of interaction and independency may lead to an idea that they are unable to do anything by themselves. In an attempt to help children in this situation, educational robotics can offer and aid, once it can provide them a certain degree of independency in the exploration of environment. The system developed in this work allows the child to transmit the commands to a robot through myoelectric and movement sensors. The sensors are placed on the childs body so they can obtain information from the body inclination and muscle contraction, thus allowing commanding, through a wireless communication, the mobile entertainment robot to carry out tasks such as play with objects and draw. In this paper, the details of the robot design and control architecture are presented and discussed. With this system, disabled children get a better cognitive development and social interaction, balancing in a certain way, the negative effects of their disabilities.

Environment for Teaching and Development of Mobile Robot Systems

Using robots for teaching is one approach that has gathered good results on Middle-School, High-School and Universities. Robotics gives chance to experiment concepts of a broad range of disciplines, principally those from Engineering courses and Computer Science. However, there are not many kits that enables the use of robotics in classroom. This article describes the methodologies to implement tools which serves as test beds for the use of robotics to teach Computer Science and Engineering. Therefore, it proposes the development of a flexible, low cost hardware to integrate sensors and control actuators commonly found on mobile robots, the development of a mobile robot device whose sensors and actuators allows the experimentation of different concepts, and an environment for the implementation of control algorithms through a computer network. This paper describes each one of these tools and discusses the implementation issues and future works.

Experience generalization for multi-agent reinforcement learning

On-line learning methods have been applied successfully in multi-agent systems to achieve coordination among agents. Learning in multi-agent systems implies a non-stationary scenario perceived by the agents, since the behavior of other agents may change as they simultaneously learn how to improve their actions. Non-stationary scenarios can be modeled as Markov games, which can be solved using the Minimax-Q algorithm - a combination of Q-learning (a reinforcement learning (RL) algorithm which directly learns an optimal control policy) and the Minimax algorithm. However, finding optimal control policies using any RL algorithm (Q-learning and Minimax-Q included) can be very time consuming. In an attempt to improve the learning time of Q-learning, we considered the QS-algorithm, in which a single experience can update more than a single action value by using a spreading function. In this paper, we present a Minimax-QS algorithm which combines the Minimax-Q algorithm and QS-algorithm. We conduct a series of empirical evaluations of the algorithm in a simplified simulator of the soccer domain. We show that even when using a very simple domain-dependent spreading function, the performance of the learning algorithm can be improved.

Conception of Wheeled Mobile Robots with Reconfigurable Control using Integrate Prototyping

With the fast innovation of the hardware and software technologies using rapid prototyping devices, with application in the robotics and automation, more and more it becomes necessary the development of applications based on methodologies that facilitate future modifications, updates and enhancements in the original projected system. This paper presents a conception of mobile robots using rapid prototyping, distributing the several control actions in growing levels of complexity and using resources of reconfigurable computing proposal oriented to embed systems implementation. Software and the hardware are structuralized in independents blocks, with connection through common bus. The study and applications of new structures control that permits good performance in relation to the parameter variations. This kind of controller can be tested on different platform representing the wheeled mobile robots using reprogrammable logic components (FPGA)