Deep Reinforcement Learning for Joint Beamwidth and Power Optimization in mmWave Systems
Jiabao Gao, Caijun Zhong, Xiaoming Chen, Hai Lin, Zhaoyang Zhang
aa r X i v : . [ c s . I T ] J un Deep Reinforcement Learning for Joint Beamwidthand Power Optimization in mmWave Systems
Jiabao Gao, Caijun Zhong, Xiaoming Chen, Hai Lin and Zhaoyang Zhang
Abstract — This paper studies the joint beamwidth and trans-mit power optimization problem in millimeter wave communica-tion systems. A deep reinforcement learning based approach isproposed. Specifically, a customized deep Q network is trainedoffline, which is able to make real-time decisions when deployedonline. Simulation results show that the proposed approachsignificantly outperforms conventional approaches in terms ofboth performance and complexity. Besides, strong generalizationability to different system parameters is also demonstrated, whichfurther enhances the practicality of the proposed approach.
Index Terms — Millimeter wave, beamwidth selection, powerallocation, joint optimization, deep reinforcement learning
I. INTRODUCTIONDue to the abundant spectrum at the millimeter frequencyband, millimeter wave (mmWave) communication [1] hasbeen visioned as a key enabling technology to overcome thespectrum shortage challenge in next generation communicationsystems. With high operating frequency, mmWave communi-cations suffer from severe path loss and sensitivity to blockage[2]. Fortunately, highly directional beams can be formed witha large number of antennas, which can effectively alleviateabove issues [3].To fully unleash the potential of mmWave communications,fine alignment of the transmitting and receiving beams is ofcrucial importance, which gives rise to the intrinsic alignmentgain and data transmission time trade-off. In addition, for thescenario with multiple transmitter-receiver pairs, interferencemanagement is essential to further improve the system perfor-mance, which requires sophisticated design of beamwidth andpower control mechanism.Thus far, several works have proposed different approachesto address the above issues. In work [4], two suboptimalalgorithms were proposed, namely underestimation and over-estimation of the interference. The former neglects interferenceand decomposes the joint optimization problem into severalsingle pair subproblems which is convex and can be easilysolved. The latter overestimates interference and only activatespart of pairs without severe mutual interference. Besides, work[5] studied the transmit power control problem in uplinkmmWave cellular networks, while a heuristic algorithm basedon simulated annealing was proposed to jointly optimizethe power and beamwidth in work [6]. Due to the varioussimplifying assumptions adopted in the aforementioned works,the proposed approaches in general yield suboptimal solutions.
J. Gao, C. Zhong, X. Chen, and Z. Zhang are with the College of Infor-mation Science and Electronic Engineering, Zhejiang University, Hangzhou,China, and also with the Zhejiang Provincial Key Laboratory of Informa-tion Processing, Communication and Networking, Hangzhou. (email: [email protected]).H. Lin is with the Department of Electrical and Information Systems, OsakaPrefecture University, Osaka 599-8531, Japan (email: [email protected]).
Responding to this, we aim to tackle the joint beamwidthand power control problem. Specifically, motivated by itsgreat potential of handling complex non-convex problems incommunications [7–9], we pursue a artificial intelligence (AI)based design here. However, the most popular supervisedlearning paradigm is not suitable, mainly due to the pro-hibitive cost of labeling. Therefore, we propose to use deepreinforcement learning (DRL), a mechanism which does notrequire labels naturally. Recently, [10] also proposed to usedeep Q network (DQN) to address decision making problemsin beam optimization. However, the current work differs from[10] in terms of both objective function and action design.Besides, [10] only considers beam selections, while this workjointly optimizes the transmit power and beamwidth to fur-ther enhance system performance. In particular, a customizedDQN is designed to solve the decision making problem. Wecarefully preprocess the channel state information (CSI) andnoise power density to formulate the state tuple, which canenhance effective learning of DRL framework. After offlinetraining, the DQN is subsequently deployed online for real-time decision of the beamwidths and transmit power. Extensivesimulation results are provided to evaluate the performance ofproposed algorithm, which demonstrates the superior perfor-mance of the proposed method over conventional approaches.The rest of this paper is organized as follows. Section IIintroduces the system model and formulates the optimizationproblem. Section III presents the detailed design of the pro-posed DRL based approach, and its superiority is validated byextensive simulation results in Section IV. Finally, the paperis concluded in Section VI.II. S
YSTEM MODEL AND P ROBLEM FORMULATION
We consider a mmWave system consisting of N transmitter-receiver pairs. Each time slot can be divided into two phases,namely beam alignment and data transmission, as shown inFig. 1. In the beam alignment phase , each pair decidesthe optimal transmitting and receiving beam directions thatmaximize the signal-to-noise ratio (SNR) within their sectorsby searching over all possible combinations, as specified inIEEE 802.15.3c [4]. In the data transmission phase, data istransmitted and received using the selected beams.Denote ψ ti and ψ ri as sector-level beamwidths, ϕ ti and ϕ ri as beam-level beamwidths at the transmitter and receiver oflink i respectively. Then, the number of possible combinationsis ⌈ ψ ti ϕ ti ⌉⌈ ψ ri ϕ ri ⌉ . Denote T p as pilot transmission time of each Beam alignment can be further divided into sector-level and beam-levelalignment. In this paper, we assume that sector-level alignment has alreadybeen established as in [11]. (cid:3)(cid:3)(cid:3)(cid:37)(cid:72)(cid:68)(cid:80)(cid:68)(cid:79)(cid:76)(cid:74)(cid:81)(cid:80)(cid:72)(cid:81)(cid:87) (cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:3)(cid:39)(cid:68)(cid:87)(cid:68)(cid:87)(cid:85)(cid:68)(cid:81)(cid:86)(cid:80)(cid:76)(cid:86)(cid:86)(cid:76)(cid:82)(cid:81)(cid:55)(cid:76)(cid:80)(cid:72)(cid:3)(cid:86)(cid:79)(cid:82)(cid:87)(cid:3)(cid:71)(cid:88)(cid:85)(cid:68)(cid:87)(cid:76)(cid:82)(cid:81)(cid:3)(cid:55) (cid:76) (cid:87)
Fig. 1 . Time slot segmentation of link i . T denotes the timeslot duration and τ i denotes beam alignment time.combination, then the total alignment time is τ i ( ϕ ti , ϕ ri ) = ⌈ ψ ti ϕ ti ⌉⌈ ψ ri ϕ ri ⌉ T p . (1)We adopt the antenna model presented in [12], then thetransmission and reception gains at transmitter i and receiver j toward each other can be expressed as g ti,j ( θ ti,j , ϕ ti ) = ( π − (2 π − ϕ ti ) zϕ ti , if | θ ti,j | ≤ ϕ ti z, otherwise , (2) g ri,j ( θ ri,j , ϕ rj ) = ( π − (2 π − ϕ rj ) zϕ rj , if | θ ri,j | ≤ ϕ rj z, otherwise , (3)where θ ti,j and θ ri,j are the angles between the boresightof transmitter i and receiver j and the angle bisectors oftransmitting and receiving beams, z is the side lobe gain, asillustrated in Fig .2. (cid:55) (cid:76) (cid:53) (cid:77) (cid:77)(cid:15)(cid:76)(cid:85) (cid:537) (cid:77)(cid:15)(cid:76)(cid:87) (cid:537) (cid:76)(cid:87) (cid:21)(cid:20) (cid:77) (cid:77)(cid:85) (cid:21)(cid:20) (cid:77) Fig. 2 . The black solid line denotes the boresight betweentransmitter i and receiver j , and the dotted lines denote theangle bisectors of transmitting and receiving beams.Denote g ci,j as the channel gain between transmitter i andreceiver j , p i as the transmission power of transmiter i , N asthe thermal noise power spectral density and W as the systembandwidth. Then, the signal-to-interference-pluse-noise ratio(SINR) of the i -th link can be given asSINR i = p i g ti,i g ri,i g ci,iN P k =1 ,k = i p k g tk,i g rk,i g ck,i + W N . (4)Therefore, the joint beamwidth selection and power alloca-tion optimization problem can be formulated asP1 : max ϕ t , ϕ r , p W N X i =1 (1 − τ i T ) log (1 + SINR i ) (5) s.t. ϕ ti ≤ ψ ti , i = 1 , . . . , N,ϕ ri ≤ ψ ri , i = 1 , . . . , N,ψ ti ψ rj T p /T ≤ ϕ ti ϕ rj , i, j = 1 , . . . , N, ≤ p i ≤ p max , i = 1 , . . . , N, (6)where the first two constraints ensure the beamwidths in beam-level is strictly smaller than that in sector-level, the thirdconstraint means that the beam alignment time can not exceedthe entire time slot duration, and the fourth constraint specifiesthe maximal transmission power limit p max . Since this problemis non-convex, it is challenging to solve by conventionaloptimization approaches.III. D EEP R EINFORCEMENT LEARNING BASED APPROACH
In this Section, we propose a DQN based approach to solvethe above problem, and we start with a brief introduction ofthe DQN algorithm.
A. Brief introduction to DQN
Fig. 3 illustrates the typical agent-environment interactionin a Markov Decision Process (MDP). At time step t , the agenttakes an action a t ∈ A by observing current state s t ∈ S tointeracts with the environment, where S and A are the setsof states and actions, respectively. One time step later, as theconsequence of its action, the agent receives a reward r t +1 and moves into a new state s t +1 . The goal of reinforcementlearning (RL) is to maximize the long-term rewards [13].Specifically, it aims to learn the policy that yields maximalcumulative discounted reward function as follows G t = ∞ X k =0 γ k r t + k +1 , (7)where ≤ γ ≤ is called the discount rate to discountrewards of later time slots. (cid:40)(cid:81)(cid:89)(cid:76)(cid:85)(cid:82)(cid:81)(cid:80)(cid:72)(cid:81)(cid:87)(cid:36)(cid:74)(cid:72)(cid:81)(cid:87) (cid:22)(cid:17) (cid:41)(cid:72)(cid:72)(cid:71)(cid:69)(cid:68)(cid:70)(cid:78)(cid:85)(cid:72)(cid:90)(cid:68)(cid:85)(cid:71)(cid:3)(cid:85) (cid:87)(cid:14)(cid:20) (cid:21)(cid:17) (cid:55)(cid:68)(cid:78)(cid:72)(cid:68)(cid:70)(cid:87)(cid:76)(cid:82)(cid:81)(cid:3)(cid:68) (cid:87) (cid:20)(cid:17) (cid:50)(cid:69)(cid:86)(cid:72)(cid:85)(cid:89)(cid:72)(cid:70)(cid:88)(cid:85)(cid:85)(cid:72)(cid:81)(cid:87)(cid:3)(cid:86)(cid:87)(cid:68)(cid:87)(cid:72)(cid:3)(cid:86) (cid:87) (cid:23)(cid:17) (cid:42)(cid:82)(cid:3)(cid:87)(cid:82)(cid:3)(cid:81)(cid:72)(cid:91)(cid:87)(cid:86)(cid:87)(cid:68)(cid:87)(cid:72)(cid:3)(cid:86) (cid:87)(cid:14)(cid:20) Fig. 3 . The agent-environment interaction in MDP.Q learning, as one of the most popular RL algorithms,maintains a Q table to record Q values of all (state, action)pairs. Under policy π , the Q function of agent with action a in state s is defined as Q π ( s, a ) = E [ G t | s t = s, a t = a ; π ] , (8)where the expectation is taken with respect to the environmentand policy. Through learning from trajectories, the Q table is updatedto approach the real table under optimal policy π ∗ , which isachieved when the following Bellman optimality equation issatisfied [13] Q π ∗ ( s t , a t ) = max a t +1 ( E [ r t +1 + γQ π ∗ ( s t +1 , a t +1 )]) . (9)To further tackle the curse of dimensionality with continu-ous state variables, DQN [14] is proposed to directly predictQ value for any (state, action) pair by a deep neural network.Then, the policy is parameterized by the weights of the Qnetwork θ , which can be updated as θ t +1 = θ t + η ( y t − Q ( s t , a t ; θ t )) ∇ Q ( s t , a t ; θ t ) , (10)where y t = r t +1 + γ max a t +1 Q ( s t +1 , a t +1 ; θ t ) is the optimal Qvalue, η is the learning rate, and ∇ denotes the derivativeoperator. B. Customized DQN design
In this subsection, we present the details about the cus-tomized DQN design for P1, including state, action, reward,network architecture and training strategy.
1) State:
As in [4], we assume that perfect CSI is available.The state is the observation that agent can get from theenvironment. In the current problem, the observation of aparticular link contains the channel gain of the link, N − interfering channel gains from this link’s transmitter to otherlinks’ receivers (ITO), N − interfering channel gains fromother links’ transmitters to this link’s receiver (IFO), and noisepower. Intuitively, one simple way is to directly use the rawdata as state. However, it turns out that the performance isnot satisfactory, which implies that proper preprocessing is ofcritical importance. Motivated by this, we first normalize allelements by the channel gain of the link to better expose thecharacteristics of relative interference and noise level. Then,we choose dB as unit to further reduce the variance of inputand facilitate efficient training. Therefore, the state tuple oflink i at certain time slot contains totally N − elements,which can be expressed as s i = s ITO i ∪ s IFO i ∪ s Noise i , i = 1 , , · · · , N, (11)where s ITO i = { g ci, , . . . , g ci,i − , g ci,i +1 , . . . , g ci,N } /g ci,i , s IFO i = { g c ,i , . . . , g ci − ,i , g ci +1 ,i , . . . , g cN,i } /g ci,i and s Noise i = { W N } /g ci,i .
2) Action:
To achieve the optimal performance, the actionsat all links should be jointly optimized. In such case, theaction number will grow exponentially with the number oflinks, making effective learning impossible. Therefore, toavoid such problem, we let the DQN make decision for everylink separately. For each link, the action is the combination ofbeamwidths for transmitter and receiver and transmit power.Assume transmitter and receiver use the same beamwidth asin [4], i.e., ϕ t = ϕ r = ϕ , then the action of link i at certaintime step can be expressed as a i = [ p i , ϕ i ] .
3) Reward:
Since the objective is to maximize the instan-taneous effective sum rate, the discount rate γ should be set As will be explained later, γ is set to 0, so some critical parts of DQNlike target network is not involved. to 0. Therefore, G t in Equation 7 reduces to r t +1 , where r t +1 is the effective sum rate at time slot t + 1 .
4) Q Network architecture:
The neural network architecturefor Q value estimation is designed by cross validation. Inour experiments, a fully-connected network consisting of twohidden layers with 128 and 64 neurons and an output layerwith the number of neurons equivalent to total action numberworks well.
5) DQN training:
The initial learning rate of neural net-work is 0.001, batch size is 256 and weights are updated bythe Adam optimizer. First, we only generate data for 2000episodes to fill the replay buffer. Then, in order to balanceexploration and exploitation during training, ǫ − greedy policyis adopted [13]. Specifically, the DQN will randomly choosean action from the action set with a small probability ǫ , ratherthan always choose action with the maximal Q value. Theinitial ǫ is set to 0.2, and gradually decreases to 0 in 100000episodes. After that, we continue to train with ǫ = 0 for 10000episodes. Notice that, in practice, a large amount of trainingdata can be generated automatically based on the model.However, mismatch may exist between the assumed channelmodel and the actual propagation environment. In such case,online fine-tuning can be adopted to compensate the modelmismatch, by continuing training with real environmental data.IV. S IMULATION R ESULTS
In this section, extensive simulation results are providedto demonstrate the performance of the proposed DRL basedapproach. The random selection algorithm and underestima-tion of interference algorithm are used as benchmark forperformance comparison. We assume that all the transmitter-receiver pairs are distributed randomly in a square area with aside length of L m. As in [2], the following mmWave channelpathloss model is used: l ( R ) = I ( p ( R )) l LoS ( R ) + I (1 − p ( R )) l NLoS ( R ) , (12)where l LoS ( R ) = C LoS R − α LoS and l NLoS ( R ) = C NLoS R − α NLoS account for the line-of-sight (LoS) and non-line-of-sight(NLoS) loss, respectively. Also, I ( x ) is the indicator functionwhich returns 1 when x = 1 and 0 otherwise, while p ( R ) isa boolean random variable with probability e βR being 1, and R denotes distance between the transmitter and receiver. Thefollowing set of parameters are used in simulation as in [2].Parameter name ValueCarrier frequency 28 GHzSystem bandwidth 1 GHzReference distance 5 mLoS path loss α LoS = 2 , C LoS = − dBNLoS path loss α NLoS = 4 , C NLoS = − dBBlockage model β = 0 . NLoS small-scale fading Nakagami fading of parameter 3Noise power density N = − dBm/HzSector-level beamwidth ψ ti = ψ ri = 90 ◦ for all i Side lobe gain z = 0 . Pilot transmission time T p /T = 0 . TABLE I . Simulation parameters
Next, we investigate the impact of several key factors andparameters, including action discretization, network densityand area size. All the tables and curves about testing per-formance are obtained by averaging over 500 independentexperiments.
A. Impact of action discretization
For DQN training, the actions are discretized depending onhow they affect the system performance. According to equa-tions (1-4), the SINR is a linear function with respect to thetransmit power and the reciprocal of the beamwidth. Therefore,we propose to uniformly discretize the transmit power and thereciprocal of the square of beamwidth in [ P min , P max ] and [1 /ϕ max , /ϕ min ] , respectively, where P min and P max arethe minimal and maximal transmit power, and ϕ min and ϕ max are the minimal and maximal beamwidth. In simulation, weset P min = , P max =
30 dBm , ϕ min = 3 ◦ , ϕ max =30 ◦ . Besides, we use only 8 values for both the transmitpower and beamwidth set to balance the training complexityand performance. To illustrate the performance of the pro-posed approach, the heuristic uniform approach is used asa benchmark, where both the transmit power and bemwidthare uniformly discretized. As can be observed in Fig. 4, theproposed scheme achieves better performance compared withthe uniform scheme. M e a n r e w a r d History
Uniform discretizationProposed discretization
Fig. 4 . Training histories with different discretization patternswhen L = 20 , N = 10 . The vertical coordinate refers to themean reward from the first episode to the current episode, inorder to smooth the curve and highlight the growth trend.To demonstrate the superiority of the proposed approach, itis necessary to see the performance gap when compared withthe performance of exhaustive search (ES) scheme. Due tothe complexity of the ES scheme, it is only feasible to makethe comparison in relatively small scale systems. As shown inTable II, the performance of the DQN approach is close to thatof the ES scheme, and is far superior to that of the randomscheme. B. Impact of network density
Fig. 5 illustrates the impact of network density on thenetwork throughput with L = 20 and different N . It can N ES(G bits/slot) Percentageof DQN (%) Percentageof Random (%)2 19.58 98.72 40.423 28.74 97.56 38.654 35.18 96.09 39.355 45.28 95.02 39.80TABLE II . Network throughput performance of differentapproaches when L = 20 . Transmit power and beamwidthare discretized the same way as above with 4 optional valueseach.be observed that the throughput increases with N , while theslope gradually decreases due to accumulated mutual interfer-ence. Also, the proposed DQN based approach consistentlyoutperforms the underestimation of interference baseline [4]when N ≥ , and the performance gap becomes larger as N increases. The reason is that the proposed DQN methodtakes into account of the interference during the design, hencecan achieve superior performance in crowded scenarios withsevere interference. N e t w o r k t h r o u g h p u t [ G b i t s / s l o t ] DQNUnderestimationRandomDQN trained with N=10
Fig. 5 . The impact of network density on throughput.In practice, the network density may change over time.Therefore, it is of paramount importance to investigate thegeneralization ability of the proposed approach, which is aalso key challenge when implementing AI based wirelesscommunication systems [15]. With a well trained networkwith N , if the actual link number N ∗ is smaller than N ,we simply consider the extra links as virtual links and padeach real link’s state tuple with N ∗ − N zeros. On the otherhand, if N ∗ is greater than N , we sort the interfering channelgains of each link and only keep the largest N ∗ − ones.As illustrated in Fig. 5, where the red curve corresponds tothe case with customized training for different N , while theyellow curve is trained for N = 10 , we observe that the twocurves almost overlap for different N , which demonstrates thesuperior generalization capability of the proposed DQN. C. Impact of area size
Fig. 6 illustrates the impact of area side length on net-work throughput with N = 10 and different L . As can bereadily observed, the throughput decreases with L , which is straightforward due to increased pathloss. Also, the proposedDQN based approach consistently outperforms the underes-timation of interference baseline under various L , while theperformance gap decreases with larger L since the impact ofinterference becomes insignificant. The generalization abilityon L is also investigated. As we can see, the network trainedwith L = 20 works well for a wide range of L with very littleperformance deterioration. N e t w o r k t h r o u g h p u t [ G b i t s / s l o t ] DQNUnderestimationRandomDQN trained with L=20
Fig. 6 . The impact of area side length on throughput.
D. Complexity
When L = 20 , N = 10 , the typical offline trainingtime of the proposed approach on a GeForce GTX 1080 TiGPU is about one hour. However, the online testing phaseonly needs to execute simple forward computation, whichis much faster than the gradient descent process involved inthe underestimation of interference baseline approach. When N = 4 and N = 10 , the average running time of thebaseline approach is 207.40 and 513.04 ms, while the DQNbased approach takes 0.98 and 0.99 ms, respectively, which ishundreds of times faster.V. CONCLUSION AND FUTURE WORK
In this paper, we have proposed a DRL based approach tosolve the joint beamwidth and power allocation problem inmmWave communication systems. A customized DQN is de-signed, and heuristic tricks are used to tackle the generalizationissue. Simulation results show that the proposed approach sig-nificantly outperforms the conventional suboptimal approachesin terms of both performance and complexity. Besides, theproposed DQN has strong generalization ability, which makesit extremely desirable for practical implementation. In thefuture, we will consider the use of advanced DRL algorithmssuch as deep deterministic policy gradient [16] to optimize incontinuous action domain.R
EFERENCES[1] A. M. Al-samman, M. H. Azmi, and T. A. Rahman, “A survey ofmillimeter wave (mm-Wave) communications for 5G: Channel measure-ment below and above 6 GHz,” in
Proc. Int. Conf. Reliable Inf. Commun.Technol. (IRICT) , Berlin, Germany: Springer, 2018, pp. 451-463. [2] T. Bai, V. Desai, and R. W. Heath, “Millimeter wave cellular channelmodels for system evaluation,” in
Proc. IEEE Int. Conf. Netw. Commun.(ICNC) , Honolulu, United States, Feb. 2014, pp. 178-182.[3] S. Singh, R. Mudumbai, and U. Madhow, “Interference analysis forhighly directional 60-GHz mesh networks: The case for rethinkingmedium access control,”
IEEE/ACM Trans. Netw ., vol. 19, no. 5, pp.1513-1527, Oct. 2011.[4] H. Shokri-Ghadikolaei, L. Gkatzikis, and C. Fischione, “Beam-searchingand transmission scheduling in millimeter wave communications,” in
Proc. IEEE Int. Conf. Commun. (ICC) , London, United Kingdom, Jun.2015, pp. 1292-1297.[5] O. Onireti, A. Imran, and M. A. Imran, “Coverage, capacity, and energyefficiency analysis in the uplink of mmWave cellular networks,”
IEEETrans. Veh. Technol. , vol. 67, no. 5, pp. 3982-3997, May 2018.[6] R. Ismayilov, B. Holfeld, R. L. G. Cavalcante, and M. Kaneko, “Powerand beam optimization for uplink millimeter-wave hotspot communica-tion systems,” in
Proc. IEEE Wireless Commun. Netw. Conf. (WCNC) ,Marrakesh, Morocco, Apr. 2019, pp. 1-8.[7] F. Meng, P. Chen, and L. Wu, “Power allocation in multi-user cellularnetworks with deep Q learning approach,” in
Proc. IEEE Int. Conf.Commun. (ICC) , Shanghai, China, May 2019, pp.1-6.[8] J. Gao, C. Zhong, X. Chen, H. Lin, and Z. Zhang, “Unsupervisedlearning for passive beamforming,”
IEEE Commun. Lett. , vol. 24, no. 5,pp. 1052-1056, May 2020.[9] Y. Yang, F. Gao, C. Qian, and G. Liao, “Model-aided deep neuralnetwork for source number detection,”
IEEE Signal Process. Lett. , vol.27, no. 12, pp. 91-95, Dec. 2019.[10] R. Shafin, et al ., “Self-tuning sectorization: Deep reinforcement learningmeets broadcast beam pptimization,”
IEEE Trans. Wireless Commun .,early access.[11] S. Singh, et al ., “Blockage and directivity in 60 GHz wireless personalarea networks: from cross-layer model to multihop MAC design,”
IEEEJ. Sel. Areas Commun. , vol. 27, no. 8, pp. 1400-1413, Oct. 2009.[12] H. Shokri-Ghadikolaei, C. Fischione, G. Fodor, P. Popovski, and M.Zorzi, “Millimeter wave cellular networks: A MAC layer perspective,”
IEEE Trans. Commun. , vol. 63, no. 10, pp. 3437-3458, Oct. 2015.[13] R. Sutton and R. Barto,
Reinforcement learning: An introduction .Cambridge, United Kingdom: MIT press, 2018.[14] V. Mnih et al ., “Human-level control through deep reinforcement learn-ing,”
Nature , vol. 518, pp. 529-533, Feb. 2015.[15] R. Shafin, et al ., “Artificial intelligence-enabled cellular networks: Acritical path to beyond-5G and 6G,”
IEEE Wireless Commun ., vol. 27,no. 2, pp. 212-217, Apr. 2020.[16] T. P. Lillicrap et alet al