Raymond Cunningham

Using feedback in collaborative reinforcement learning to adaptively optimize MANET routing

Designers face many system optimization problems when building distributed systems. Traditionally, designers have relied on optimization techniques that require either prior knowledge or centrally managed runtime knowledge of the systems environment, but such techniques are not viable in dynamic networks where topology, resource, and node availability are subject to frequent and unpredictable change. To address this problem, we propose collaborative reinforcement learning (CRL) as a technique that enables groups of reinforcement learning agents to solve system optimization problems online in dynamic, decentralized networks. We evaluate an implementation of CRL in a routing protocol for mobile ad hoc networks, called SAMPLE. Simulation results show how feedback in the selection of links by routing agents enables SAMPLE to adapt and optimize its routing behavior to varying network conditions and properties, resulting in optimization of network throughput. In the experiments, SAMPLE displays emergent properties such as traffic flows that exploit stable routes and reroute around areas of wireless interference or congestion. SAMPLE is an example of a complex adaptive distributed system.

Supporting CORBA applications in a mobile environment

CORBA, the Common Object Request Broker Architecture, defines a framework for developing object-oriented distributed applications. Unfortunately, current implementations of CORBA have not been designed with support for mobile computers in mind. Using CORBA in a mobile environment raises a number of problems due to hardware mobility and the characteristics of wireless networks. This paper identifies and discusses these problems and presents the design and implementation of our Architecture for Location Independent CORBA Environments (ALICE). ALICE allows CORBA objects running on mobile devices to interact transparently with objects hosted by off-the-shelf CORBA implementations. Importantly, ALICE allows server as well as client objects to reside on mobile hosts without relying on a centralised location register to keep track of their whereabouts.

A Collaborative Reinforcement Learning Approach to Urban Traffic Control Optimization

The high growth rate of vehicles per capita now poses a real challenge to efficient urban traffic control (UTC). An efficient solution to UTC must be adaptive in order to deal with the highly-dynamic nature of urban traffic. In the near future, global positioning systems and vehicle-to-vehicle/infrastructure communication may provide a more detailed local view of the traffic situation that could be employed for better global UTC optimization. In this paper we describe the design of a next-generation UTC system that exploits such local knowledge about a junctions traffic in order to optimize traffic control. Global UTC optimization is achieved using a local adaptive round robin (ARR) phase switching model optimized using collaborative reinforcement learning (CRL). The design employs an ARR-CRL-based agent controller for each signalized junction that collaborates with neighbouring agents in order to learn appropriate phase timing based on the traffic pattern. We compare our approach to non-adaptive fixed-time UTC system and to a saturation balancing algorithm in a large-scale simulation of traffic in Dublins inner city centre. We show that the ARR-CRL approach can provide significant improvement resulting in up to ~57% lower average waiting time per vehicle compared to the saturation balancing algorithm.

Discovery of stable peers in a self-organising peer-to-peer gradient topology

Peer-to-peer (P2P) systems are characterised by a wide disparity in peer resources and capabilities. In particular, a number of measurements on deployed P2P systems show that peer stability (e.g. uptime) varies by several orders of magnitude between peers. In this paper, we introduce a peer utility metric and construct a self-organising P2P topology based on this metric that allows the efficient discovery of stable peers in the system. We propose and evaluate a search algorithm and we show that it achieves significantly better performance than random walking. Our approach can be used by certain classes of applications to improve the availability and performance of system services by placing them on the most stable peers, as well as to reduce the amount of network traffic required to discover and use these services. As a proof-of-concept, we demonstrate the design of a naming service on the gradient topology.

Time bounded medium access control for ad hoc networks

Most previous work on medium access control (MAC) protocols for wireless ad hoc networks has focused on the twin goals of maximising throughput and minimising average packet delay as required for general-purpose applications.In this paper we describe a new MAC protocol for use in multi-hop ad hoc networks whose goal is to provide, with high probability, time-bounded access to the wireless medium for applications with guaranteed response time requirements.The Time-Bounded Medium Access Control (TBMAC) protocol is based on time-division multiple access with dynamic but predictable slot allocation. TBMAC uses a light-weight atomic multicast protocol to achieve distributed agreement on slot allocation and employs location information to minimise contention for slots.TBMAC is the first time-bounded MAC protocol for multi-hop wireless ad hoc networks. In this paper we describe the protocol and provide a number of time bounds for the transmission of messages.

Using aggregation for adaptive super-peer discovery on the gradient topology

Peer-to-peer environments exhibit a very high diversity in individual peer characteristics ranging by orders of magnitude in terms of uptime, available bandwidth, and storage space. Many systems attempt to exploit this resource heterogeneity by using the best performing and most reliable peers, called super-peers, for hosting system services. However, due to inherent decentralisation, scale, dynamism, and complexity of P2P environments, self-managing super-peer selection is a challenging problem. In this paper, decentralised aggregation techniques are used to reduce the uncertainty about system properties by approximating the peer utility distribution allowing peers to calculate adaptive thresholds in order to discover appropriate super-peers. Furthermore, a heuristic search algorithm is described that allows super-peers, above a certain utility threshold, to be efficiently discovered and utilised by any peer in the system.

High-bandwidth mesh-based overlay multicast in heterogeneous environments

In this paper we present MeshCast, a peer-to-peer (p2p) multicast protocol for applications requiring high bandwidth (such as live video streaming) from a server to a large number of receivers. Traditional tree-based approaches to overlay multicast inefficiently utilise the outgoing bandwidth of participating nodes and poorly adapt to node membership churn. In contrast, MeshCast is based on Chainsaw mesh-based approach to data delivery that better utilises bandwidth and provides excellent adaptation properties. In this paper we identify properties that enable mesh-based overlay multicast protocols to better utilise the available bandwidth and consequently support higher data stream rates in heterogeneous environments. MeshCast uses a gossip-based algorithm to adapt the overlay to peer heterogeneity, while still preserving the advantages of a mesh-based overlay. Our experiments show that MeshCast can support 68% higher stream rates and provides a 22% improvement in buffering delay over the recently proposed Chainsaw protocol for a heterogeneous node bandwidth distribution.

Self-Adapting Context Definition

Many approaches to context-aware computing have focused on providing means for developers to define application contexts. In these approaches, correct application behavior depends on the developer defining the right contexts. The application cannot adapt incorrectly defined contexts as they are statically defined by the developer. We propose a method by which an application can use feedback to evaluate its contexts and adapt them where necessary. This results in more accurate context definitions and should lead to improved application performance. We discuss how this is achieved using Q-learning, and present a scenario and experimental results to support our approach.

Building autonomic systems using collaborative reinforcement learning

This paper presents Collaborative Reinforcement Learning (CRL), a coordination model for online system optimization in decentralized multi-agent systems. In CRL system optimization problems are represented as a set of discrete optimization problems, each of whose solution cost is minimized by model-based reinforcement learning agents collaborating on their solution. CRL systems can be built to provide autonomic behaviours such as optimizing system performance in an unpredictable environment and adaptation to partial failures. We evaluate CRL using an ad hoc routing protocol that optimizes system routing performance in an unpredictable network environment.

Towards real-time middleware for vehicular ad hoc networks

Applications of inter-vehicle and vehicle-to-roadside communication that make use of vehicular ad hoc networks (VANETs) will often require reliable communication that provides guaranteed real-time message propagation. This paper describes an event-based middleware, called RT-STEAM. Unlike other event systems, RT-STEAM does not rely on a centralized event broker or look-up service while still supporting event channels providing hard real-time event delivery. RT-STEAM event filtering can be based on subject, content and/or proximity. To guarantee real-time communication, we exploit proximity-based event propagation to guarantee real-time constraints within the defined proximities only. The proximity within which real-time guarantees are available is adapted to maintain time bounds while allowing changes to membership and topology, typical of VANETs. This Space-Elastic Model of real-time communication is the first to directly address adaptation in the space domain to guarantee real-time constraints.