Kok-Lim Alvin Yau

Review: Reinforcement learning for context awareness and intelligence in wireless networks: Review, new features and open issues

In wireless networks, context awareness and intelligence are capabilities that enable each host to observe, learn, and respond to its complex and dynamic operating environment in an efficient manner. These capabilities contrast with traditional approaches where each host adheres to a predefined set of rules, and responds accordingly. In recent years, context awareness and intelligence have gained tremendous popularity due to the substantial network-wide performance enhancement they have to offer. In this article, we advocate the use of reinforcement learning (RL) to achieve context awareness and intelligence. The RL approach has been applied in a variety of schemes such as routing, resource management and dynamic channel selection in wireless networks. Examples of wireless networks are mobile ad hoc networks, wireless sensor networks, cellular networks and cognitive radio networks. This article presents an overview of classical RL and three extensions, including events, rules and agent interaction and coordination, to wireless networks. We discuss how several wireless network schemes have been approached using RL to provide network performance enhancement, and also open issues associated with this approach. Throughout the paper, discussions are presented in a tutorial manner, and are related to existing work in order to establish a foundation for further research in this field, specifically, for the improvement of the RL approach in the context of wireless networking, for the improvement of the RL approach through the use of the extensions in existing schemes, as well as for the design and implementation of RL in new schemes.

QoS in IEEE 802.11-based wireless networks: A contemporary review

Apart from mobile cellular networks, IEEE 802.11-based wireless local area networks (WLANs) represent the most widely deployed wireless networking technology. With the migration of critical applications onto data networks, and the emergence of multimedia applications such as digital audio/video and multimedia games, the success of IEEE 802.11 depends critically on its ability to provide Quality of Service (QoS). A lot of research has focused on equipping IEEE 802.11 WLANs with features to support QoS. In this survey, we provide an overview of these techniques. We discuss the QoS features incorporated by the IEEE 802.11 standard at both physical (PHY) and Media Access Control (MAC) layers, as well as other higher-layer proposals. We also focus on how the new architectural developments of software-defined networking (SDN) and cloud networking can be used to facilitate QoS provisioning in IEEE 802.11-based networks. We conclude this paper by identifying some open research issues for future consideration.

Cognitive Radio-based Wireless Sensor Networks: Conceptual design and open issues

Traditional static spectrum allocation policies have been to grant each wireless service exclusive usage of certain frequency bands, leaving several spectrum bands unlicensed for industrial, scientific and medical purposes. The rapid proliferation of low-cost wireless applications in unlicensed spectrum bands has resulted in spectrum scarcity among those bands. Since most applications in Wireless Sensor Networks (WSNs) utilize the unlicensed spectrum, network-wide performance of WSNs will inevitably degrade as their popularity increases. Sharing of under-utilized licensed spectrum among unlicensed devices is a promising solution to the spectrum scarcity issue. Cognitive Radio (CR) is a new paradigm in wireless communication that allows sensor nodes as the unlicensed users or Secondary Users (SUs) to detect and use the under-utilized licensed spectrum temporarily. Given that the licensed or Primary Users (PUs) are oblivious to the presence of SUs, the SUs access the licensed spectrum opportunistically without interfering the PUs, while improving their own performance. In this paper, we propose an approach to build Cognitive Radio-based Wireless Sensor Networks (CR-WSNs). We believe that CR-WSN is the next-generation WSN. Realizing that both WSNs and CR present unique challenges to the design of CR-WSNs, we provide an overview and conceptual design of WSNs from the perspective of CR. The open issues are discussed to motivate new research interests in this field. We also present our method to achieving context-awareness and intelligence, which are the key components in CR networks, to address an open issue in CR-WSN.

A context-aware and Intelligent Dynamic Channel Selection scheme for cognitive radio networks

The tremendous growth in ubiquitous low-cost wireless applications that utilize the unlicensed spectrum bands has laid increasing stress on the limited and scarce radio spectrum resources. Given that the licensed or Primary Users (PUs) are oblivious to the presence of unlicensed or Secondary Users (SUs), Cognitive Radio (CR) is a new paradigm in wireless communication that allows the SUs to detect and use the underutilized licensed spectrums opportunistically and temporarily. In this paper, we propose a Context-aware and Intelligent Dynamic Channel Selection scheme that helps SUs to select channel adaptively for data transmission to enhance QoS, particularly throughput and delay. Our scheme is suitable for CR networks with mobile hosts. We formulate and design our scheme using Reinforcement Learning that offers a simple and yet practical solution. Channel heterogeneity, which is a feature unique to CR networks that has been ignored in previous studies, is considered in this paper. Simulation results reveal that the proposed scheme achieves very good performance.

Routing in Distributed Cognitive Radio Networks: A Survey

Cognitive Radio Networks (CRNs) have been receiving significant research attention recently due to their ability to solve issues associated with spectrum congestion and underutilization. In a CRN, unlicensed users (or Secondary Users, SUs) are able to exploit and use underutilized licensed channels, but they must evacuate the channels if any interference is caused to the licensed users (or Primary Users, PUs) who own the channels. Due to the dynamicity of spectrum availability in CRNs, design of protocols and schemes at different layers of the SU’s network stack has been challenging. In this article, we focus on routing and discuss the challenges and characteristics associated with it. Subsequently, we provide an extensive survey on existing routing schemes in CRNs. Generally speaking, there are three categories of challenges, namely channel-based, host-based, and network-based. The channel-based challenges are associated with the operating environment, the host-based with the SUs, and the network-based with the network-wide SUs. Furthermore, the existing routing schemes in the literature are segregated into three broad categories based on the relationship between PUs and SUs in their investigation, namely intra-system, inter-system, and hybrid-system; and within each category, they are further categorized based on their types, namely Proactive, Reactive, Hybrid, and Adaptive Per-hop. Additionally, we present performance enhancements achieved by the existing routing schemes in CRNs. Finally, we discuss various open issues related to routing in CRNs in order to establish a foundation and to spark new interests in this research area.

Exploiting the Power of Multiplicity: A Holistic Survey of Network-Layer Multipath

The Internet is inherently a multipath network: For an underlying network with only a single path, connecting various nodes would have been debilitatingly fragile. Unfortunately, traditional Internet technologies have been designed around the restrictive assumption of a single working path between a source and a destination. The lack of native multipath support constrains network performance even as the underlying network is richly connected and has redundant multiple paths. Computer networks can exploit the power of multiplicity, through which a diverse collection of paths is resource pooled as a single resource, to unlock the inherent redundancy of the Internet. This opens up a new vista of opportunities, promising increased throughput (through concurrent usage of multiple paths) and increased reliability and fault tolerance (through the use of multiple paths in backup/redundant arrangements). There are many emerging trends in networking that signify that the Internets future will be multipath, including the use of multipath technology in data center computing; the ready availability of multiple heterogeneous radio interfaces in wireless (such as Wi-Fi and cellular) in wireless devices; ubiquity of mobile devices that are multihomed with heterogeneous access networks; and the development and standardization of multipath transport protocols such as multipath TCP. The aim of this paper is to provide a comprehensive survey of the literature on network-layer multipath solutions. We will present a detailed investigation of two important design issues, namely, the control plane problem of how to compute and select the routes and the data plane problem of how to split the flow on the computed paths. The main contribution of this paper is a systematic articulation of the main design issues in network-layer multipath routing along with a broad-ranging survey of the vast literature on network-layer multipathing. We also highlight open issues and identify directions for future work.

Applications of Reinforcement Learning to Cognitive Radio Networks

Cognitive Radio (CR) enables an unlicensed user to change its transmission and reception parameters adaptively according to spectrum availability in a wide range of licensed channels. The concept of a Cognition Cycle (CC) is the key element of CR to provide context awareness and intelligence so that each unlicensed user is able to observe and carry out an optimal action on its operating environment for performance enhancement. The CC can be applied in various application schemes in CR networks such as Dynamic Channel Selection (DCS), topology management, congestion control, and scheduling. In this paper, Reinforcement Learning (RL) is applied to implement the conceptual of the CC. We provide an extensive overview of our work including single-agent and multi-agent approaches to show that RL is a promising technique. Our contribution in this paper is to propose various application schemes using our RL approach to warrant further research on RL in CR networks.

Application of reinforcement learning to routing in distributed wireless networks: a review

The dynamicity of distributed wireless networks caused by node mobility, dynamic network topology, and others has been a major challenge to routing in such networks. In the traditional routing schemes, routing decisions of a wireless node may solely depend on a predefined set of routing policies, which may only be suitable for a certain network circumstances. Reinforcement Learning (RL) has been shown to address this routing challenge by enabling wireless nodes to observe and gather information from their dynamic local operating environment, learn, and make efficient routing decisions on the fly. In this article, we focus on the application of the traditional, as well as the enhanced, RL models, to routing in wireless networks. The routing challenges associated with different types of distributed wireless networks, and the advantages brought about by the application of RL to routing are identified. In general, three types of RL models have been applied to routing schemes in order to improve network performance, namely Q-routing, multi-agent reinforcement learning, and partially observable Markov decision process. We provide an extensive review on new features in RL-based routing, and how various routing challenges and problems have been approached using RL. We also present a real hardware implementation of a RL-based routing scheme. Subsequently, we present performance enhancements achieved by the RL-based routing schemes. Finally, we discuss various open issues related to RL-based routing schemes in distributed wireless networks, which help to explore new research directions in this area. Discussions in this article are presented in a tutorial manner in order to establish a foundation for further research in this field.

Context-awareness and intelligence in Distributed Cognitive Radio Networks: A Reinforcement Learning approach

Cognitive Radio (CR) is a next-generation wireless communication system that exploits underutilized licensed spectrum to improve the utilization of the overall radio spectrum. A Distributed Cognitive Radio Network (DCRN) is a distributed wireless network established by a number of CR hosts in the absence of fixed network infrastructure. Context-awareness and intelligence are key characteristics of CR networks that enable the CR hosts to be aware of their operating environment in order to achieve a joint action that improves network-wide performance in a distributed manner through learning. In this paper, we advocate the use of Reinforcement Learning (RL) in application schemes that require context-awareness and intelligence such as the Dynamic Channel Selection (DCS), scheduling, and congestion control. We investigate the performance of the RL in respect to DCS. We show that RL and our enhanced RL approach are able to converge to a joint action that provide better network-wide performance. We also show the effects of network density and various essential parameters in RL on the performance.

SMART: A SpectruM-Aware clusteR-based rouTing scheme for distributed cognitive radio networks

Cognitive radio (CR) is the next-generation wireless communication system that allows unlicensed users (or secondary users, SUs) to exploit the underutilized spectrum (or white spaces) in licensed spectrum while minimizing interference to licensed users (or primary users, PUs). This article proposes a SpectruM-Aware clusteR-based rouTing (SMART) scheme that enables SUs to form clusters in a cognitive radio network (CRN) and enables each SU source node to search for a route to its destination node on the clustered network. An intrinsic characteristic of CRNs is the dynamicity of operating environment in which network conditions (i.e., PUs’ activities) change as time goes by. Based on the network conditions, SMART enables SUs to adjust the number of common channels in a cluster through cluster merging and splitting, and searches for a route on the clustered network using an artificial intelligence approach called reinforcement learning. Simulation results show that SMART selects stable routes and significantly reduces interference to PUs, as well as routing overhead in terms of route discovery frequency, without significant degradation of throughput and end-to-end delay.