Milos Rovcanin

A reinforcement learning based solution for cognitive network cooperation between co-located, heterogeneous wireless sensor networks

Due to a drastic increase in the number of wireless communication devices, these devices are forced to interfere or interact with each other. This raises the issue of possible effects this coexistence might have on the performance of these networks. Negative effects are a consequence of contention for network resources (such as free wireless communication frequencies) between different devices, which can be avoided if co-located networks cooperate with each other and share the available resources. This paper presents a self-learning, cognitive cooperation approach for heterogeneous co-located networks. Cooperation is performed by activating or deactivating services such as interference avoidance, packet sharing, various MAC protocols, etc. Activation of a cooperative service might have both positive and negative effects on a networks performance, regarding its high level goals. Such a cooperation approach has to incorporate a reasoning mechanism, centralized or distributed, capable of determining the influence of each symbiotic service on the performance of all the participating sub-networks, taking into consideration their requirements. In this paper, a cooperation method incorporating a machine learning technique, known as the Least Squares Policy Iteration (LSPI), is proposed and discussed as a novel network cooperation paradigm.

Data traffic differentiation and QoS on the train, in fast parameter varying, heterogeneous wireless networks

Although Internet on the train and train to wayside communication in general becomes more and more available for train operators, there are still a lot of challenges for future research. We previously developed a network platform that is responsible for an uninterrupted and seamless connectivity from the train to the wayside through heterogeneous wireless access technologies. This paper mainly focuses on the concept for providing sufficient Quality of Service (QoS) guarantees in a dynamic train environment. Within this network platform, IPv6 strategies are adopted for QoS, exploiting multi-homing and intelligent aggregation techniques. The implementation that has been done in the Click Modular Router programming environment will also be presented in details.

Design and prototype of a train-to-wayside communication architecture

Telecommunication has become very important in modern society and seems to be almost omnipresent, making daily life easier, more pleasant and connecting people everywhere. It does not only connect people, but also machines, enhancing the efficiency of automated tasks and monitoring automated processes. In this context the IBBT (Interdisciplinary Institute for BroadBand Technology) project TRACK (TRain Applications over an advanced Communication networK), sets the definition and prototyping of an end-to-end train-to-wayside communication architecture as one of the main research goals. The architecture provides networking capabilities for train monitoring, personnel applications and passenger Internet services. In the context of the project a prototype framework was developed to give a complete functioning demonstrator. Every aspect: tunneling and mobility, performance enhancements, and priority and quality of service were taken into consideration. In contrast to other research in this area, which has given mostly high-level overviews, TRACK resulted in a detailed architecture with all different elements present.

Coping with Network Dynamics Using Reinforcement Learning Based Network Optimization in Wireless Sensor Networks

Due to the constant increase of in density of wireless network devices, inter-network cooperation is increasingly important. To avoid conflicts, advanced algorithms for inter-network optimization utilize cognition processes such as reinforcement learning to enable networks to become capable of solving complex optimization problems on their own with minimal outside intervention. This paper investigates the inherent trade-offs that occur when using reinforcement learning techniques in dynamic networks: the need to keep the network running optimally whilst, at the same time, different (suboptimal) network settings need to be continuously investigated to cope with changing network conditions. To cope with these network dynamics, two existing algorithms, “epsilon greedy” and Softmax, are compared to a novel approach, based on a logarithmic probability distribution function. It is shown that, depending on the expected level of dynamics, the new algorithm outperforms existing solutions.

Application of the LSPI reinforcement learning technique to a co-located network negotiation problem

Optimizing multiple co-located networks, each with a variable number of network functionalities that influence each other, is a complex problem that has not yet received a lot of attention in the research community. However, since independent co-located networks increasingly influence each other, optimization solutions can no longer afford to look only at the performance of a single network. To this end, we propose a multi-tiered solution, based on Least Square Policy Improvement (LSPI), a machine learning technique.

On Suitability of the Reinforcement Learning Methodology in Dynamic, Heterogeneous, Self-optimizing Networks

An ever growing number of deployed wireless networks dictates a tempo with which the inter-network cooperation techniques are being developed. Cooperation, in this sense, can go far beyond a simple activation of an interference avoidance techniques. This paper describes and evaluates the performance of a reinforcement learning based reasoning engine, used in a self-learning, cognitively controlled cooperation between heterogeneous, co-located networks. Coupled with a concept of cooperation through the network service negotiation, this approach represents an efficient, yet scalable solution for the dynamic network self-optimization.

Traffic Differentiation - A Basic Step Towards Providing End-to-End QoS on the Train-to-Wayside Wireless Communication System

We developed a network platform that is responsible for an uninterrupted and seamless connectivity from the train to the wayside through heterogeneous wireless access technologies. However, limiting the offered services to only an onboard Internet is not a feasible business case. A viable one should extend to a broad spectrum of railway communication services like: train control, diagnostics, real time passenger information, entertainment, security CCTV surveillance etc. In a highly dynamic environment (from the communication link point of view) such a fast moving train, it is necessary to introduce prioritization among different traffic classes. This will implicitly determine under what conditions a certain flow should get suspended or dropped in order to preserve the flows of a higher priority as long as possible and to ensure that they meet their QoS demands. The first step towards this goal is data traffic differentiation.