Lorenza Giupponi

Distributed Q-Learning for Aggregated Interference Control in Cognitive Radio Networks

This paper deals with the problem of aggregated interference generated by multiple cognitive radios (CRs) at the receivers of primary (licensed) users. In particular, we consider a secondary CR system based on the IEEE 802.22 standard for wireless regional area networks (WRANs), and we model it as a multiagent system where the multiple agents are the different secondary base stations in charge of controlling the secondary cells. We propose a form of real-time multiagent reinforcement learning, which is known as decentralized Q-learning, to manage the aggregated interference generated by multiple WRAN systems. We consider both situations of complete and partial information about the environment. By directly interacting with the surrounding environment in a distributed fashion, the multiagent system is able to learn, in the first case, an efficient policy to solve the problem and, in the second case, a reasonably good suboptimal policy. Computational and memory requirement considerations are also presented, discussing two different options for uploading and processing the learning information. Simulation results, which are presented for both the upstream and downstream cases, reveal that the proposed approach is able to fulfill the primary-user interference constraints, without introducing signaling overhead in the system.

Docitive networks: an emerging paradigm for dynamic spectrum management [Dynamic Spectrum Management]

Prime design goals for next-generation wireless networks to support emerging applications are spectral efficiency and low operational cost. Among a gamut of technical solutions, cognitive approaches have long been perceived as a catalyst for the above goals by facilitating the coexistence of primary and secondary users by means of efficient dynamic spectrum management. While most available techniques today are essentially opportunistic in nature, a truly cognitive device needs to exhibit a certain degree of intelligence to draw optimum decisions based on prior observations and anticipated actions. Said intelligence however, comes along with high complexity and poor convergence, which currently prevents any viable deployment of cognitive networks. We thus introduce an emerging and largely unexplored concept of docitive networks, where nodes effectively teach other nodes with the prime aims of reducing cognitive complexity, speeding up the learning process, and drawing better and more reliable decisions. To this end, we review some important concepts borrowed from the machine learning community for both centralized and decentralized systems, in order to position the emerging docitive with known cognitive approaches. Finally, we validate introduced concepts in the context of a primary digital television system dynamically coexisting with IEEE 802.22 secondary networks. For this scenario, we demonstrate the superiority of various unprecedented docitive over known opportunistic/cognitive algorithms.

A Cell Outage Management Framework for Dense Heterogeneous Networks

In this paper, we present a novel cell outage management (COM) framework for heterogeneous networks with split control and data planes-a candidate architecture for meeting future capacity, quality-of-service, and energy efficiency demands. In such an architecture, the control and data functionalities are not necessarily handled by the same node. The control base stations (BSs) manage the transmission of control information and user equipment (UE) mobility, whereas the data BSs handle UE data. An implication of this split architecture is that an outage to a BS in one plane has to be compensated by other BSs in the same plane. Our COM framework addresses this challenge by incorporating two distinct cell outage detection (COD) algorithms to cope with the idiosyncrasies of both data and control planes. The COD algorithm for control cells leverages the relatively larger number of UEs in the control cell to gather large-scale minimization-of-drive-test report data and detects an outage by applying machine learning and anomaly detection techniques. To improve outage detection accuracy, we also investigate and compare the performance of two anomaly-detecting algorithms, i.e., k-nearest-neighbor- and local-outlier-factor-based anomaly detectors, within the control COD. On the other hand, for data cell COD, we propose a heuristic Grey-prediction-based approach, which can work with the small number of UE in the data cell, by exploiting the fact that the control BS manages UE-data BS connectivity and by receiving a periodic update of the received signal reference power statistic between the UEs and data BSs in its coverage. The detection accuracy of the heuristic data COD algorithm is further improved by exploiting the Fourier series of the residual error that is inherent to a Grey prediction model. Our COM framework integrates these two COD algorithms with a cell outage compensation (COC) algorithm that can be applied to both planes. Our COC solution utilizes an actor-critic-based reinforcement learning algorithm, which optimizes the capacity and coverage of the identified outage zone in a plane, by adjusting the antenna gain and transmission power of the surrounding BSs in that plane. The simulation results show that the proposed framework can detect both data and control cell outage and compensate for the detected outage in a reliable manner.

From cognition to docition: The teaching radio paradigm for distributed & autonomous deployments

We advocate for a novel communication paradigm of docition which facilitates distributed and autonomous networking at minimal control overhead and maximal performance. We consider that the nodes in foreseen networks are intelligent radios able to learn and thus self-adapt to prior set performance targets within a given surrounding environment. We briefly review the state-of-the-art of purely distributed learning algorithms, and we identify the most appropriate approaches allowing for self-adaptation to particular system dynamics. In such distributed settings, however, the learning is typically complex, imprecise and slow due to mutually-impacting decisions resulting in non-stationarities. The docitive paradigm proposes a timely solution which encourages more knowledgeable nodes to teach surrounding nodes to speed up the development of their cognitive state. We advocate for different degrees of docition, such as teaching at start-up or run-time, and demonstrate that this improves the convergence speed and precision of known cognitive algorithms. We evaluate the docitive paradigm in the context of a femtocell network modeled as a multi-agent system, where the agents are the femto access points, implementing a realtime multi-agent reinforcement learning technique known as decentralized Q-learning. We propose different docitive algorithms and we show their superiority to the well know paradigm of independent learning.

Aggregated interference control for cognitive radio networks based on multi-agent learning

This paper deals with the problem of aggregated interference generated by multiple cognitive radios (CR) at the receivers of primary (licensed) users. In particular, we consider a secondary CR system based on the IEEE 802.22 standard for wireless regional area networks (WRAN), and we model it as a multi-agent system where the multiple agents are the different secondary base stations in charge of controlling the different secondary cells. We propose a solution for the aggregated interference problem based on a form of real-time multi-agent reinforcement learning known as decentralized Qlearning, so that the multi-agent system is designed to learn an optimal policy by directly interacting with the surrounding environment in a distributed fashion. Simulation results reveal that the proposed approach is able to fulfil the primary users interference constraints, without introducing signalling overhead in the system.

Use of learning, game theory and optimization as biomimetic approaches for Self-Organization in macro-femtocell coexistence

In this paper, we present the use of several Biomimetic approaches for Self Organization (SO) in heterogeneous scenarios where macrocell and femtocell networks coexist. Mainly these approaches are categorized in indirect biomimetics and direct biomimetics. Under indirect biomimetics we discuss 1) emerging paradigms in learning theory and 2) game theory for their potential to enable SO solutions in heterogeneous networks. By means of numerical results we demonstrate the pros and cons of these indirect biomimetic approaches for designing SO in macro-femto coexistence scenarios. Furthermore, we demonstrate the use of direct biomimetic approaches for designing SO by exploiting one to one mapping between a natural SO system and our system model for heterogeneous networks based on Outdoor Fixed Relays (OFR). Numerical results show that the proposed analytical solution can enhance wireless backhaul capacity of the OFR based femtocells by adapting the macro base station (BS) antenna tilts in a distributed and self organizing manner.

Decentralized Q-Learning for Aggregated Interference Control in Completely and Partially Observable Cognitive Radio Networks

This paper deals with the problem of aggregated interference generated by multiple cognitive radios (CR) at the receivers of primary (licensed) users. In particular, we consider a secondary CR system based on the mEE 802.22 standard for wireless regional area networks (WRAN), and we model it as a multi-agent system where the multiple agents are the different secondary base stations in charge of controlling the secondary cells. We propose a form of real-time multi-agent reinforcement learning, known as decentralized Q-leaming, to manage the aggregated interference generated by multiple WRAN cells. We consider both situations of complete and partial information about the environment. By directly interacting with the surrounding environment in a distributed fashion, the multi-agent system is able to learn, in the first case, an optimal policy to solve the problem and, in the second case, a reasonably good suboptimal policy. Simulation results reveal that the proposed approach is able to fulfill the primary users interference constraints, without introducing signaling overhead in the system.

LTE-advanced self-organizing network conflicts and coordination algorithms

Self-organizing network (SON) functions have been introduced in the LTE and LTEAdvanced standards by the Third Generation Partnership Project as an excellent solution that promises enormous improvements in network performance. However, the most challenging issue in implementing SON functions in reality is the identification of the best possible interactions among simultaneously operating and even conflicting SON functions in order to guarantee robust, stable, and desired network operation. In this direction, the first step is the comprehensive modeling of various types of conflicts among SON functions, not only to acquire a detailed view of the problem, but also to pave the way for designing appropriate Self-Coordination mechanisms among SON functions. In this article we present a comprehensive classification of SON function conflicts, which leads the way for designing suitable conflict resolution solutions among SON functions and implementing SON in reality. Identifying conflicting and interfering relations among autonomous network management functionalities is a tremendously complex task. We demonstrate how analysis of fundamental trade-offs among performance metrics can us to the identification of potential conflicts. Moreover, we present analytical models of these conflicts using reference signal received power plots in multi-cell environments, which help to dig into the complex relations among SON functions. We identify potential chain reactions among SON function conflicts that can affect the concurrent operation of multiple SON functions in reality. Finally, we propose a selfcoordination framework for conflict resolution among multiple SON functions in LTE/LTEAdvanced networks, while highlighting a number of future research challenges for conflict-free operation of SON.

Distributed Q-learning for energy harvesting Heterogeneous Networks

We consider a two-tier urban Heterogeneous Network where small cells powered with renewable energy are deployed in order to provide capacity extension and to offload macro base stations. We use reinforcement learning techniques to concoct an algorithm that autonomously learns energy inflow and traffic demand patterns. This algorithm is based on a decentralized multi-agent Q-learning technique that, by interacting with the environment, obtains optimal policies aimed at improving the system performance in terms of drop rate, throughput and energy efficiency. Simulation results show that our solution effectively adapts to changing environmental conditions and meets most of our performance objectives. At the end of the paper we identify areas for improvement.

Switch-On/Off Policies for Energy Harvesting Small Cells through Distributed Q-Learning

The massive deployment of small cells (SCs) represents one of the most promising solutions adopted by 5G cellular networks to meet the foreseen huge traffic demand. The high number of network elements entails a significant increase in the energy consumption. The usage of renewable energies for powering the small cells can help reduce the environmental impact of mobile networks in terms of energy consumption and also save on electric bills. In this paper, we consider a two-tier cellular network architecture where SCs can offload macro base stations and solely rely on energy harvesting and storage. In order to deal with the erratic nature of the energy arrival process, we exploit an ON/OFF switching algorithm, based on reinforcement learning, that autonomously learns energy income and traffic demand patterns. The algorithm is based on distributed multi-agent Q-learning for jointly optimizing the system performance and the self-sustainability of the SCs. We analyze the algorithm by assessing its convergence time, characterizing the obtained ON/OFF policies, and evaluating an offline trained variant. Simulation results demonstrate that our solution is able to increase the energy efficiency of the system with respect to simpler approaches. Moreover, the proposed method provides an harvested energy surplus, which can be used by mobile operators to offer ancillary services to the smart electricity grid.