Ndedi D. Monekosso

Phe-Q: A Pheromone Based Q-Learning

Biological systems have often provided inspiration for the design of artificial systems. On such example of a natural system that has inspired researchers is the ant colony. In this paper an algorithm for multi-agent reinforcement learning, a modified Q-learning, is proposed. The algorithm is inspired by the natural behaviour of ants, which deposit pheromones in the environment to communicate. The benefit besides simulating ant behaviour in a colony is to design complex multi-agent systems. Complex behaviour can emerge from relatively simple interacting agents. The proposed Q-learning update equation includes a belief factor. The belief factor reflects the confidence the agent has in the pheromone detected in its environment. Agents communicate implicitly to co-operate in learning to solve a path-planning problem. The results indicate that combining synthetic pheromone with standard Q-learning speeds up the learning process. It will be shown that the agents can be biased towards a preferred solution by adjusting the pheromone deposit and evaporation rates.

An Analysis of the Pheromone Q-Learning Algorithm

The Phe-Q machine learning technique, a modified Q-learning technique, was developed to enable co-operating agents to communicate in learning to solve a problem. The Phe-Q learning technique combines Q-learning with synthetic pheromone to improve on the speed of convergence. The Phe-Q update equation includes a belief factor that reflects the confidence the agent has in the pheromone (the communication) deposited in the environment by other agents. With the Phe-Q update equation, speed of convergence towards an optimal solution depends on a number parameters including the number of agents solving a problem, the amount of pheromone deposited, and the evaporation rate. In this paper, work carried out to optimise speed of learning with the Phe-Q technique is described. The objective was to to optimise Phe-Q learning with respect to pheromone deposition rates, evaporation rates.

Motion estimation with edge continuity constraint for crowd scene analysis

This paper presents a new motion estimation method aimed at crowd scene analysis in complex video sequences. The proposed technique makes use of image descriptors extracted from points lying at the maximum curvature on the Canny edge map of an analyzed image. Matches between two consecutive frames are then carried out by searching for descriptors that satisfy both a well-defined similarity metric and a structural constraint imposed by the edge map. A preliminary assessment using real-life video sequences gives both qualitative and quantitative results.

Autonomous Spacecraft Resource Management: A Multi-agent Approach

The paper presents a multi-agent system that learns to manage the re-sources of an unmanned spacecraft. Each agent controls a sub-system and learns to optimise its resources. The agents can coordinate their actions to satisfy user requests. Co-ordination is achieved by exchanging sched-uling information between agents. Resource management is implemented using two reinforcement learning techniques: the Monte-carlo and the Q-learning. The paper demonstrates how the approach can be used to model the imaging system of a spacecraft. The environment is represented by agents which control the spacecraft sub-systems involved in the imaging activity. The agent in charge of the resource management senses the information regarding the resource requested, the resource conflicts and the resource availability. Scheduling of resources is learnt when all subsystems are fully functional and when resources are reduced by random failures.

Self-Organizing Maps for the Automatic Interpretation of Crowd Dynamics

This paper introduces the use of self-organizing maps for the visualization of crowd dynamics and to learn models of the dominant motions of crowds in complex scenes. The self-organizing map (SOM) model is a well known dimensionality reduction method proved to bear resemblance with characteristics of the human brain, representing sensory input by topologically ordered computational maps. This paper proposes algorithms to learn and compare crowd dynamics with the SOM model. Different information is employed as input to the used SOM. Qualitative and quantitative results are presented in the paper.

A Collective Can Do Better

Can we devise simple solutions to complex problems? Is it possible to do so by making use of elemental modules, which when collaborating create emerging intelligence? The answer is yes. No complex mathematical models are required. Nature offers a variety of techniques that lend themselves well to solving complex problems making use of simpler atomic entities. Insects, for instance, as individuals are very simple, but in a collective they are powerful systems able to solve very complex tasks. This chapter describes how the insect world can inspire engineers and computer scientists to devise simple solutions to complex problems. After all the simplest solution is always the best.1

A quantitative comparison of two new motion estimation algorithms

This paper proposes a comparison of two motion estimation algorithms for crowd scene analysis in video sequences. The first method uses the local gradient supported by neighbouring topology constraints. The second method makes use of descriptors extracted from points lying at the maximum curvature along Canny edges. Performance is evaluated using real-world video sequences, providing the reader with a quantitative comparison of the two methods.

Reasoning about Dynamic Scenes Using Autonomous Agents

The scene interpretation system proposed below integrates computer vision and artificial intelligence techniques to combine the information generated by multiple cameras on typical secure sites. A multi-agent architecture is proposed as the backbone of the system within which the agents control the different components of the system and incrementally build a model of the scene by merging the information gathered over time and between cameras. The choice of a distributed artificial intelligence architecture is justified by the need for scalable designs capable of co-operating to infer an optimal interpretation of the scene. Decentralizing intelligence means creating more robust and reliable sources of interpretation, but also allows easy maintenance and updating of the system. The scene model is learned using Hidden Markov models which capture the range of possible scene behaviours. The employment of such probabilistic interpretation techniques is justified by the very nature of surveillance data, which is typically incomplete, uncertain and asynchronous.

Coupling multi-view dynamics with mixtures of Gaussians

Multi-camera firsion is rapidly becoming an emerging research area, especially for visual surveillance applications. Data fusion can be obtained with calibrated cameras, either calibrating prior use - following standard techniques (I) - or through learning mechanism in 30 Cartesian frame (2), typically the scene ground plane. In this paper we describe a method to merge video data acquired by two overlapping views, by learning the camera registration on the basis of occurring dynamics. Scene dynamics in each independent view can be modeled as a mixture of Gaussian components, and that dynamics can be coupled assuming stochastic correlation between the underlying processes.

An Improved Q-Learning Algorithm Using Synthetic Pheromones

In this paper we propose an algorithm for multi-agent Q-learning. The algorithm is inspired by the natural behaviour of ants, which deposit pheromone in the environment to communicate. The benefit besides simulating ant behaviour in a colony is to design complex multi-agent systems. Complex behaviour can emerge from relatively simple interacting agents. The proposed Q-learning update equation includes a belief factor. The belief factor reflects the confidence the agent has in the pheromone detected in its environment. Agents communicate implicitly to co-ordinate and co-operate in learning to solve a problem.