Tim Clarke

Output feedback eigenstructure assignment using a new reduced orthogonality condition

This paper presents a new method of output feedback eigenstructure assignment. A new, reduced orthogonality condition is derived which is less restrictive on the design degrees of freedom than others in the literature. From this a novel, general formulation for the gain matrix is admitted which utilizes a two stage design process. In the first stage, a subset of desired eigenvectors is assigned, either left or right. In the second stage, a dual set is assigned which adheres to the necessary and sufficient conditions for output feedback eigenstructure assignment proved earlier in the paper. A numerical example is presented to illustrate the whole procedure.

MACSIM: A SIMULINK ENABLED ENVIRONMENT FOR MULTI-AGENT SYSTEM SIMULATION

Abstract Agent-based approaches to software and algorithm development have received a great deal of research attention in recent years and are becoming widely utilised in the construction of complex systems. This paper presents MACSim a novel simulation environment which allows a multi-agent system to be embedded into the industry standard system, Simulink. The architecture of MACSim is discussed and a case study is presented where a number of MACSim agents are used to control the behaviour of a Boeing 747 in simulation.

Application of reinforcement learning to medium access control for wireless sensor networks

This paper presents a novel approach to medium access control for single-hop wireless sensor networks. The ALOHA-Q protocol applies Q-Learning to frame based ALOHA as an intelligent slot selection strategy capable of migrating from random access to perfect scheduling. Results show that ALOHA-Q significantly outperforms Slotted ALOHA in terms of energy-efficiency, delay and throughput. It achieves comparable performance to S-MAC and Z-MAC with much lower complexity and overheads. A Markov model is developed to estimate the convergence time of its simple learning process and to validate the simulation results.

Distributed Heuristically Accelerated Q-Learning for Robust Cognitive Spectrum Management in LTE Cellular Systems

In this paper, we propose an algorithm for dynamic spectrum access (DSA) in LTE cellular systems-distributed ICIC accelerated Q-learning (DIAQ). It combines distributed reinforcement learning (RL) and standardized inter-cell interference coordination (ICIC) signalling in the LTE downlink, using the framework of heuristically accelerated RL (HARL). Furthermore, we present a novel Bayesian network based approach to theoretical analysis of RL based DSA. It explains a predicted improvement in the convergence behaviour achieved by DIAQ, compared to classical RL. The scheme is also assessed using large scale simulations of a stadium temporary event network. Compared to a typical heuristic ICIC approach, DIAQ provides significantly better quality of service and supports considerably higher network throughput densities. In addition, DIAQ dramatically improves initial performance, speeds up convergence, and improves steady state performance of a state-of-the-art distributed Q-learning algorithm, confirming the theoretical predictions. Finally, our scheme is designed to comply with the current LTE standards. Therefore, it enables easy implementation of robust distributed machine intelligence for full self-organisation in existing commercial networks.

Distributed Q-learning based dynamic spectrum management in cognitive cellular systems: Choosing the right learning rate

This paper presents the concept of the Win-or-Learn-Fast (WoLF) variable learning rate for distributed Q-learning based dynamic spectrum management algorithms. It demonstrates the importance of choosing the learning rate correctly by simulating a large scale stadium temporary event network. The results show that using the WoLF variable learning rate provides a significant improvement in quality of service, in terms of the probabilities of file blocking and interruption, over typical values of fixed learning rates. The results have also demonstrated that it is possible to provide a better and more robust quality of service using distributed Q-learning with a WoLF variable learning rate, than a spectrum sensing based opportunistic spectrum access scheme, but with no spectrum sensing involved.

Transfer Learning: A Paradigm for Dynamic Spectrum and Topology Management in Flexible Architectures

In this paper, we introduce a novel paradigm of transfer learning for spectrum and topology management in a rapidly deployable opportunistic network for the post disaster and temporary event scenarios. The network architecture is designed to be rapidly changing between different disaster phases, and highly flexible during the temporary event period. Transfer learning is developed to learn the dynamic radio environment from network topologies. This also allows previously learnt information in earlier phases of a deployment to be efficiently used to influence the learning process in later phases of a deployment. A Transfer Learning strategy is designed to change the knowledge base from the most recent phase via multi-agent coordination. We evaluate transfer learning paradigm in a small cell Terrestrial eNB architecture, integrated with Q-Learning and Linear Reinforcement Learning. It is demonstrated that transfer learning significantly improves the initial performance, the convergence speed and the steady state QoS, by exchanging topology information for resource prioritization.

Desirable eigenstructure for good short-term helicopter handling qualities: The attitude command response case

Abstract The eigenstructure necessary to achieve a good short-term attitude command response in a generic single-rotor helicopter is presented. It achieves appropriate mode decoupling and is consistent with the physical relationships between the state variables. This eigenstructure translates exactly into ideal transfer functions for use with a variety of control design methodologies. The paper discusses how such an eigenstructure is defined to satisfy the requirements of two important military rotorcraft specifications: US ADS 33 and UK DEF STAN 00970.

Robust eigenstructure assignment using the genetic algorithm and constrained state feedback

Abstract This paper describes a robust eigenstructure assignment methodology for a constrained state feedback problem. The method, which is based upon the linear quadratic regulator and involves the minimization, via the genetic algorithm, of a multiobjective cost function, is applied to L1011 Tristar aircraft lateral dynamics. The design example generates a fixed-gain state feedback solution which shows independent phase margins of 51· in each channel, while exhibiting an eigenstructure close to that desired, lying well within specified handling quality requirements. If two states are made unavailable for feedback, the robustness properties are seriously eroded. When a dynamic feedback compensator is then used, there is a substantial recovery of the robustness. It is concluded that the genetic algorithm approach described here is easy to use and generates good multivariable stability margins.

Multichannel non-persistent CSMA MAC schemes with reinforcement learning for cognitive radio networks

This paper presents two multichannel non-persistent CSMA (M-np-CSMA) MAC schemes using Simple Reinforcement Learning and State-Action-Reward-State-Action (SARSA) learning respectively for distributed cognitive radio networks. The two learning schemes both use reinforcement learning to help the users learn the environment and historical transmissions. Compared with M-np-CSMA MAC protocol with random channel choice, the learning schemes can help the cognitive users choose the best channels which offer more spectrum access opportunities to sense and access. The results show that both learning schemes can effectively improve the throughput and decrease the packet delay at heavy traffic loads and with a large number of cognitive users. The Simple Reinforcement Learning scheme and SARSA scheme can achieves a 15% and 25% improvement in the maximum throughput respectively, compared with the M-np-CSMA without learning.

Using feedback to regulate gene expression in a developmental control architecture

We present what we believe is the first attempt to physically reconstruct the exploratory mechanism of genetic regulatory networks. Feedback plays a crucial role during developmental processes and its mechanisms have recently become much clearer due to evidence from evolutionary developmental biology. We believe that without similar mechanisms of interaction and feedback, digital genomes cannot guide themselves across functional search spaces in a way that fully exploits a domains resources, particularly in the complex search domains of real-world physics. Our architecture is designed to let evolution utilise feedback as part of its mechanism of exploration.