Murilo Fernandes Martins

Heuristically-Accelerated Multiagent Reinforcement Learning

This paper presents a novel class of algorithms, called Heuristically-Accelerated Multiagent Reinforcement Learning (HAMRL), which allows the use of heuristics to speed up well-known multiagent reinforcement learning (RL) algorithms such as the Minimax-Q. Such HAMRL algorithms are characterized by a heuristic function, which suggests the selection of particular actions over others. This function represents an initial action selection policy, which can be handcrafted, extracted from previous experience in distinct domains, or learnt from observation. To validate the proposal, a thorough theoretical analysis proving the convergence of four algorithms from the HAMRL class (HAMMQ, HAMQ(λ), HAMQS, and HAMS) is presented. In addition, a comprehensive systematical evaluation was conducted in two distinct adversarial domains. The results show that even the most straightforward heuristics can produce virtually optimal action selection policies in much fewer episodes, significantly improving the performance of the HAMRL over vanilla RL algorithms.

Heuristically-Accelerated Reinforcement Learning: A Comparative Analysis of Performance

This paper presents a comparative analysis of three Reinforcement Learning algorithms (Q-learning, Q(\(\lambda \))-learning and QS-learning) and their heuristically-accelerated variants (HAQL, HAQ(\(\lambda \)) and HAQS) where heuristics bias action selection, thus speeding up the learning. The experiments were performed in a simulated robot soccer environment which reproduces the conditions of a real competition league environment. The results clearly demonstrate that the use of heuristics substantially improves the performance of the learning algorithms.

On the construction of a RoboCup small size league team

The Robot Soccer domain has become an important artificial intelligence test bench and a widely studied research area. It is a domain with real, dynamic, and uncertain environment, where teams of robots cooperate and face adversarial competition. To build a RoboCup Small Size League (SSL) team able to compete in the world championship requires multidisciplinary research in fields like robotic hardware development, machine learning, multi-robot systems, computer vision, control theory, and mechanics, among others.This paper intends to provide insights about the aspects involved on the development of the RoboFEI RoboCup SSL robot soccer team and to present the contributions produced over its course. Among these contributions, a computer vision system employing an artificial neural network (ANN) to recognize colors, a heuristic algorithm to recognize partially detected objects, an implementation of the known rapidly-exploring random trees (RRT) path planning algorithm with additional rules, enabling the angle of approach of the robot to be controlled, and a layered strategy software system.Experimental results on real robots demonstrate the high performance of the vision system and the efficiency of the RRT algorithm implementation. Some strategy functions are also experimented, with empirical results showing their effectiveness.

A Qualitative-Probabilistic Approach to Autonomous Mobile Robot Self Localisation and Self Vision Calibration

Typically, the spatial features of a robots environment are specified using metric coordinates, and well-known mobile robot localisation techniques are used to track the exact robot position. In this paper, a qualitative-probabilistic approach is proposed to address the problem of mobile robot localisation. This approach combines a recently proposed logic theory called Perceptual Qualitative Reasoning about Shadows (PQRS) with a Bayesian filter. The approach herein proposed was systematically evaluated through experiments using a mobile robot in a real environment, where the sequential prediction and measurement steps of the Bayesian filter are used to both self-localisation and self-calibration of the robots vision system from the observation of objects and their shadows. The results demonstrate that the qualitative-probabilistic approach effectively improves the accuracy of robot localisation, keeping the vision system well calibrated so that shadows can be properly detected.

Probabilistic self-localisation on a qualitative map based on occlusions

Spatial knowledge plays an essential role in human reasoning, permitting tasks such as locating objects in the world (including oneself), reasoning about everyday actions and describing perceptual information. This is also the case in the field of mobile robotics, where one of the most basic (and essential) tasks is the autonomous determination of the pose of a robot with respect to a map, given its perception of the environment. This is the problem of robot self-localisation (or simply the localisation problem). This paper presents a probabilistic algorithm for robot self-localisation that is based on a topological map constructed from the observation of spatial occlusion. Distinct locations on the map are defined by means of a classical formalism for qualitative spatial reasoning, whose base definitions are closer to the human categorisation of space than traditional, numerical, localisation procedures. The approach herein proposed was systematically evaluated through experiments using a mobile robot equipped with a RGB-D sensor. The results obtained show that the localisation algorithm is successful in locating the robot in qualitatively distinct regions.

Towards Model-based Vision Systems for Robot Soccer Teams

Since it’s beginning, Robot Soccer has been a platform for research and development of independent mobile robots and multi-agent systems, involving the most diverse areas of engineering and computer science. There are some problems to be solved in this domain, such as mechanical construction, electronics and control of mobile robots. But the main challenge is found in the areas related to Artificial Intelligence, as multi-agent systems, machine learning and computer vision. The problems and challenges mentioned above are not trivial, since Robot Soccer is dynamic, uncertain and probabilistic. A computer vision system for a Robot Soccer team must be fast and robust, and it is desirable that it can handle noise and luminous intensity variations. A number of techniques can be applied for object recognition in the domain of Robot Soccer, as described by (Grittani et al., 2000). The research of (Grittani et al., 2000) is based only on color information, as well as the research of (Weiss & Hildebrand, 2004) that uses color information to reduce the amount of information contained in each image frame through a called “relevance point filter”. Other researches uses the shape model of the objects to detect on the image, technique generally used in local vision systems. The research of (Gonner et al., 2005), for instance, detects the ball through it’s shape model projected on the image, a circumference, but still uses color-only information to recognize the robots. No matter which technique is used to solve the Robot Soccer computer vision challenge, it must be able to determine position and angle of the robots and the ball with maximum accuracy and minimal processing time possible, because the success of the strategy and control system depends on the information given by the computer vision system. This chapter extends the work presented by (Martins et al., 2006a), which considers the use of a well known image segmentation technique – the Hough Transform – to locate the mobile robots and the ball on global vision images, taking advantage of the domain characteristics – the robots and ball shape. To implement the Hough Transform technique, which is in most cases implemented in robotic systems using special hardware, only an offthe-shelf frame grabber and a personal computer are used. A new approach to interpret the Hough space is proposed, as well as the method used to recognize objects, which is based on a constraint satisfaction approach.

Logic-probabilistic model for event recognition in a robotic search and rescue scenario.

This paper presents initial results towards the development of a logic-based probabilistic event recognition system capable of learning and inferring high-level joint actions from simultaneous task execution demonstrations on a search and rescue scenario. We adopt a probabilistic extension of the Event Calculus defined over Markov Logic Networks (MLN-EC). This formalism was applied to learn and infer the actions of human operators teleoperating robots in a real-world robotic search and rescue task. Experimental results in both simulation and real robots show that the probabilistic event logic can recognise the actions taken by the human teleoperators in real world domains containing two collaborating robots, even with uncertain and noisy data.

Probabilistic logic for multi-robot event recognition

This paper presents initial results towards the development of a logic-based event recognition system capable of handling high-level joint actions from simultaneous task execution demonstrations on a search and rescue scenario.

A fast model-based vision system for a robot soccer team

Robot Soccer is a challenging research domain for Artificial Intelligence, which was proposed in order to provide a long-term problem in which researchers can investigate the construction of systems involving multiple agents working together in a dynamic, uncertain and probabilistic environment, to achieve a specific goal. This work focuses on the design and implementation of a fast and robust computer vision system for a team of small size robot soccer players. The proposed system combines artificial intelligence and computer vision techniques to locate the mobile robots and the ball, based on global vision images. To increase system performance, this work proposes a new approach to interpret the space created by a well-known computer vision technique called Hough Transform, as well as a fast object recognition method based on constraint satisfaction techniques. The system was implemented entirely in software using an off-the-shelf frame grabber. Experiments using real time image capture allows to conclude that the implemented system are efficient and robust to noises and lighting variation, being capable of locating all objects in each frame, computing their position and orientation in less than 20 milliseconds.