Jan Oksanen

Reinforcement learning based sensing policy optimization for energy efficient cognitive radio networks

This paper introduces a machine learning based collaborative multi-band spectrum sensing policy for cognitive radios. The proposed sensing policy guides secondary users to focus the search of unused radio spectrum to those frequencies that persistently provide them high data rate. The proposed policy is based on machine learning, which makes it adaptive with the temporally and spatially varying radio spectrum. Furthermore, there is no need for dynamic modeling of the primary activity since it is implicitly learned over time. Energy efficiency is achieved by minimizing the number of assigned sensors per each subband under a constraint on miss detection probability. It is important to control the missed detections because they cause collisions with primary transmissions and lead to retransmissions at both the primary and secondary user. Simulations show that the proposed machine learning based sensing policy improves the overall throughput of the secondary network and improves the energy efficiency while controlling the miss detection probability.

Diversity-based spectrum sensing policy for detecting primary signals over multiple frequency bands

Cognitive radios and flexible spectrum use (FSU) provide an efficient way to exploit underutilized radio spectrum by allowing secondary users to access licensed frequencies in an agile manner with the constraint that the licensed user will not be interfered. In order to identify such spectral opportunities, spectrum sensing is needed by the secondary users. In this paper a cooperative spectrum sensing policy employed by spatially displaced multiple cognitive radios is proposed. It enables sensing of multiple potentially discontinuous frequency bands simultaneously and facilitates mitigating the effects of shadowing and fading through spatial diversity.

A sensing policy based on confidence bounds and a restless multi-armed bandit model

A sensing policy for the restless multi-armed bandit problem with stationary but unknown reward distributions is proposed. The work is presented in the context of cognitive radios in which the bandit problem arises when deciding which parts of the spectrum to sense and exploit. It is shown that the proposed policy attains asymptotically logarithmic weak regret rate when the rewards are bounded independent and identically distributed or finite state Markovian. Simulation results verifying uniformly logarithmic weak regret are also presented. The proposed policy is a centrally coordinated index policy, in which the index of a frequency band is comprised of a sample mean term and a confidence term. The sample mean term promotes spectrum exploitation whereas the confidence term encourages exploration. The confidence term is designed such that the time interval between consecutive sensing instances of any suboptimal band grows exponentially. This exponential growth between suboptimal sensing time instances leads to logarithmically growing weak regret. Simulation results demonstrate that the proposed policy performs better than other similar methods in the literature.

Reinforcement learning method for energy efficient cooperative multiband spectrum sensing

Cognitive radios (CR) and dynamic spectrum access (DSA) attempt to exploit the underutilized radio spectrum by allowing secondary users to access the licensed frequencies in an opportunistic manner. In order to avoid collisions with the primary user the secondary users need to sense the spectrum, and to mitigate the effects of channel fading on sensing cooperative schemes have been proposed in the literature. In this paper a multiband spectrum sensing policy for coordinating cooperative sensing is proposed. The proposed policy employs the ∈-greedy reinforcement learning method to prioritize the sensing of different subbands and to assign those secondary users to sense them that are able to provide a desired level of miss detection probability. In order to improve the energy efficiency, the number of assigned sensors per subband is minimized.

Design of spectrum sensing policy for multi-user multi-band cognitive radio network

Finding an optimal sensing policy for a particular access policy and sensing scheme is a laborious combinatorial problem that requires the system model parameters to be known. In practise the parameters or the model itself may not be completely known making reinforcement learning methods appealing. In this paper a non-parametric reinforcement learning-based method is developed for sensing and accessing multi-band radio spectrum in multi-user cognitive radio networks. A suboptimal sensing policy search algorithm is proposed for a particular multi-user multi-band access policy and the randomized Chair-Varshney rule. The randomized Chair-Varshney rule is used to reduce the probability of false alarms under a constraint on the probability of detection that protects the primary user. The simulation results show that the proposed method achieves a sum profit (e.g. data rate) close to the optimal sensing policy while achieving the desired probability of detection.

Characterization of spatial diversity in cooperative spectrum sensing

Cognitive radios (CR) and dynamic spectrum access (DSA) provide an efficient way to exploit underutilized radio spectrum by allowing secondary users to access licensed frequencies in an agile manner. This principle allows secondary users to access a licensed frequency band when their transmission will not interfere the primary receiver. Consequently, spectrum sensing is needed by the secondary users to describe the state of a licensed frequency band as vacant or occupied, which can be for a single user a challenging task due to the random nature of the wireless channel. In order to mitigate the effects of channel fading on detection, cooperative detection algorithms have been proposed in the literature. The gain from cooperation comes from spatial diversity as the signal is observed via multiple independent channels. In order to characterize spatial diversity in multichannel signal detection we propose a measure for the spatial diversity. We define spatial diversity as the maximum slope of the probability of detection curve in the logarithmic SNR scale. This definition looks promising for three reasons: 1) it shows that diversity is obtained with diminishing returns as the number of sensors is increased, 2) the number of samples has very little effect and 3) it shows very little gain for correlated channels.

An Order Optimal Policy for Exploiting Idle Spectrum in Cognitive Radio Networks

In this paper, a spectrum sensing policy employing recency-based exploration is proposed for cognitive radio networks. We formulate the problem of finding a spectrum sensing policy for multiband dynamic spectrum access as a stochastic restless multiarmed bandit problem with stationary unknown reward distributions. In cognitive radio networks, the multiarmed bandit problem arises when deciding where in the radio spectrum to look for idle frequencies that could be efficiently exploited for data transmission. We consider two models for the dynamics of the frequency bands: 1) the independent model where the state of the band evolves randomly independently from the past and 2) the Gilbert-Elliot model, where the states evolve according to a two-state Markov chain. It is shown that, in these conditions, the proposed sensing policy attains asymptotically logarithmic weak regret. The policy proposed in this paper is an index policy, in which the index of a frequency band comprises a sample mean term and a recency-based exploration bonus term. The sample mean promotes spectrum exploitation, whereas the exploration bonus encourages further exploration for idle bands providing high data rates. The proposed recency-based approach readily allows constructing the exploration bonus such that it will grow the time interval between consecutive sensing time instants of a suboptimal band exponentially, which then leads to logarithmically increasing weak regret. Simulation results confirming logarithmic weak regret are presented, and it is found that the proposed policy provides often improved performance at low complexity over other state-of-the-art policies in the literature.

Robotics Inspired Opportunistic Routing for Cognitive Radio Using Potential Fields

Potential field based routing is proposed for cognitive radio networks. This approach is inspired by the analogy between packet routing in cognitive radio networks and robot navigation in the presence of obstacles. Using this analogy a general framework for packet routing using potential fields is introduced, in which desired communication destinations can be considered as attractive forces and sources of interference as repulsive forces. Furthermore, three examples of how to construct such potential fields are proposed. Two optimal potential field models with respect to delay and energy and one suboptimal, but computationally simple, potential field based on virtual forces are derived. The potential field approach facilitates using physics-based models of propagation in interference modeling. This paper shows that potential field techniques from robotics can provide an attractive solution to routing in a cognitive radio network and that they can achieve good performance in terms of end-to-end delay and energy consumption.

Reinforcement learning-based multiband sensing policy for cognitive radios

Cognitive radios (CR) and dynamic spectrum access (DSA) have been proposed as a way to exploit the underutilized radio spectrum by allowing secondary users to access the licensed frequencies in an opportunistic manner. The constraint set to the secondary use is that it should not interfere the primary users, i.e., the license holder. Hence, the secondary users need to sense the spectrum in order to classify a licensed frequency band as vacant or occupied. However, spectrum sensing can be a demanding task for a single user due to the random nature of the wireless channel, and to mitigate the effects of channel fading cooperative detection algorithms have been proposed. In this paper a multiband spectrum sensing policy for coordinating the cooperative sensing is proposed. It is based on dynamically allocating frequency hopping codes to the secondary users. The proposed policy employs the ∈-greedy reinforcement learning action selection to prioritize the sensing of different subbands and to select the best secondary users to sense them. The results show the proposed policy is able to significantly increase the obtained throughput in the secondary network and to reduce the number of missed detections of the primary signal.

Spatial Interpolation of Cyclostationary Test Statistics in Cognitive Radio Networks: Methods and Field Measurements

The focus of this paper is on evaluating different spatial interpolation methods for the construction of radio environment map using field measurements obtained by cyclostationary-based mobile sensors. Mobile sensing devices employing cyclostationary detectors provide lot of advantages compared to the widely used energy detectors, such as robustness to noise uncertainty and ability to distinguish among different primary user signals. However, mobile sensing results are not available at locations between the sensors making it difficult for a secondary user (possibly without a spectrum sensor) to decide whether to use primary user resources at that location. To overcome this, spatial interpolation of test statistics measured at limited number of locations can be carried out to create a channel occupancy map at unmeasured locations between the sensors. For this purpose, different spatial interpolation techniques for the cyclostationary test statistic have been employed in this paper such as inverse distance weighting, ordinary kriging, and universal kriging. The effectiveness of these methods is demonstrated by applying them on extensive real-world field measurement data obtained by mobile-phone-compliant spectrum sensors. The field measurements were carried out using four mobile spectrum sensors measuring eight digital video broadcasting-terrestrial (DVB-T) channels at more than hundred locations encompassing roughly one-third of the area of the city of Espoo in Finland. The accuracy of the spatial interpolation results based on the field measurements is determined using the cross-validation approach with the widely used root mean square error as the metric. Field measurement results indicate that reliable results with spatial coverage can be achieved using kriging for cyclostationary based test statistics. Comparison of spatial interpolation results of cyclostationary test statistics is also carried out with those of energy values obtained during the measurement campaign in the form of received signal strength indicator. Comparison results clearly show the performance improvement and robustness obtained using cyclostationary based detectors instead of energy detectors.