Recurrent Neural Networks for Handover Management in Next-Generation Self-Organized Networks
Zoraze Ali, Marco Miozzo, Lorenza Giupponi, Paolo Dini, Stojan Denic, Stavroula Vassaki
aa r X i v : . [ c s . N I] J un Recurrent Neural Networks for HandoverManagement in Next-Generation Self-OrganizedNetworks
Zoraze Ali + , Marco Miozzo + , Lorenza Giupponi + , Paolo Dini + , Stojan Denic ∗ , Stavroula Vassaki ∗ ( + ) Centre Tecnol`ogic de Telecomunicacions de Catalunya (CTTC/CERCA), Barcelona, Spain( ∗ ) Huawei Technologies, Sweden ABemails: { zali, mmiozzo, lgiupponi, pdini } @cttc.es, { stojan.denic, stavroula.vassaki } @huawei.com Abstract —In this paper, we discuss a handover managementscheme for Next Generation Self-Organized Networks. We pro-pose to extract experience from full protocol stack data, to makesmart handover decisions in a multi-cell scenario, where usersmove and are challenged by deep zones of an outage. Traditionalhandover schemes have the drawback of taking into account onlythe signal strength from the serving, and the target cell, beforethe handover. However, we believe that the expected Quality ofExperience (QoE) resulting from the decision of target cell tohandover to, should be the driving principle of the handoverdecision. In particular, we propose two models based on multi-layer many-to-one LSTM architecture, and a multi-layer LSTMAutoEncoder (AE) in conjunction with a MultiLayer Perceptron(MLP) neural network. We show that using experience extractedfrom data, we can improve the number of users finalizing thedownload by 18 %, and we can reduce the time to download,with respect to a standard event-based handover benchmarkscheme. Moreover, for the sake of generalization, we test theLSTM Autoencoder in a different scenario, where it maintains itsperformance improvements with a slight degradation, comparedto the original scenario.
Index Terms —LSTM, RNN, Next Generation Self-organizingnetworks, Deep learning, machine learning, LTE
I. I
NTRODUCTION
It has been almost a decade since when Self-OrganizingNetworks (SON) have been defined and introduced as a featureof Long Term Evolution (LTE), in 3GPP Release 8. However,the vision of an automatic network capable of learning fromexperience and adapting to the environment has still notreached the maturity that operators were initially hoping, inorder to maximize the efficiency of the network, while atthe same time reducing the operational costs. 5G cellularnetworks are characterized by extremely dense and heteroge-neous deployments, in order to increase the network coverageand capacity. In addition, besides traditional sub-6 GHz andlicensed bands, the access can span over a wide range ofbandwidth, including mmWave and unlicensed spectrum. Thehigh diversity of mobile devices and applications, further com-plicates the network architecture and its management. In thiscontext, current and 5G networks generate a massive amountof measurements, control and management information [1][2].This huge amount of information could be efficiently utilizedto address the 5G network management challenges. Recently,the evolution in computational capabilities, has allowed to take advantage of machine learning and novel deep learning solu-tions to tackle multiple problems in different disciplines. In 5Gand its evolution, the possibilities now available for machinelearning and deep learning implementations are infinite andpave the way to an evolved vision of Next Generation SONto be able to address end-to-end solutions.The focus of this work is on the use case of handovermanagement. In standards and literature, handover algorithmsare traditionally based on standard events, e.g., the A3 or A2event, and are mainly focused on the optimization of eventtrigger parameters, e.g., Hysteresis, Time-to-Trigger and Cellindividual Offset [3]. Machine learning solutions have alsobeen proposed in this respect, to adjust online these typicalSON parameters [1]. This approach presents the shortcomingthat it considers the strongest signal for target cell selectionbefore the handover, but not the actual perceived Quality ofExperience (QoE) after the handover. For example, in urbanscenarios where the handover to the strongest neighbour cell issuccessful but, a while after the handover, the transmission isdeeply affected, e.g., by the presence of an obstacle, traditionalhandover approaches fail to provide a satisfactory solution,without taking advantage of available data to gain experienceand make smarter decisions. Those approaches are likely tolead to a severe degradation of QoE, due to the unpredictedcell outage [4].In this paper, first we build a realistic cellular scenariousing a high fidelity, full protocol stack, end-to-end networksimulator, ns-3 , and we extract data from all the layers of theprotocol stack. With this data, we build a wide and completedatabase, which we consider the basis of the experience thata smart network management should be able to construct. Inorder to make smart handover decisions in a scenario, in whichwe artificially generate deep zones of outage using obsta-cles, we use a Long-short-term-memory (LSTM) RecurrentNeural Network (RNN) to take advantage of the temporalcharacteristic of the data extracted from the network. TheLSTM is designed in order to solve a regression problemto estimate the necessary time to download a file transmittedover a Transmission Control Protocol (TCP) transport. This isexpected to capture the QoE of the users. We obtain very goodprediction errors and with these results, we are able to provethat the learning approach outperforms traditional handover Y X -10-8-6-4-2 0 2 4 6 8 10 12 14 16 18 20 S I N R ( d B ) Fig. 1. Simulation scenario solutions. This means that once training is accomplished, thelearning based handover algorithm is able to select a targetcell for handover that could provide a better QoE, in themedium or long term, even if in the short term it offers aweaker signal upon handover decision. Moreover, we use anAutoEncoder (AE) with the purpose to compress the data andreduce its dimensionality. Then, this compressed data is usedas input of an Feedforward Neural Network (FFNN), whichoffers excellent regression results similar to the one obtainedusing LSTM. This means, that the AE successfully reducesthe dimensionality of the data without losing network perfor-mance. Finally, we show that the experience learned by thesemodels in our scenario is also useful for making decisions indifferent deployment scenarios, so that the learned experienceis proven useful to be reused in different geographical areas.The outline of the paper is the following. In Section II, wediscuss the target scenario and the data generation procedure.In Section III we introduce the handover control scheme. InSection IV we discuss the results of the learning scheme incomparison to traditional handover solutions. Finally, SectionV concludes the paper.II. D
ATA GENERATION
A. Simulation Scenario
We implement a realistic simulation scenario through ns-3LENA LTE (Long Term Evolution) - EPC (Evolved PacketCore) simulator [6]. A macro cell outdoor scenario has beenconsidered with a network consisting of three-sectorial eNBs.A cluster of UEs is placed in each sector at a fixed distancefrom the center of a cell, in which the UEs are dropped inrandom positions. Since, in this scenario, we use TCP as thetransport protocol, such deployment of the UEs guaranteesthe establishment of a TCP connection between the remotehost and the UEs. The UEs start moving after receiving thefirst packet, following a predefined mobility pattern. In orderto increase the communication challenges in the scenario,and to generate more random coverage patterns, we introduceobstacles in the scenario, which generate multiple coverage
TABLE IS
IMULATION PARAMETERS
Parameter Value
System bandwidth 5 MHzInter-site distance 500 mHandover algorithm A2-RSRPAdaptive Modulation & CodingScheme Vienna [5]SINR computation for DL CQI Control method [5]eNBs antenna type ParaboliceNBs antenna Beamwidth 70 degreeseNBs antenna max attenuation 20 dBNumber of macro eNBs 21 (7 cells)eNBs Tx Power 46 dBmDistance between the centerpoints of the cluster and the cell 100 mCluster diameter 50 mNumber of UEs in the system 210 (30 per sector)Mobility model RandomWalk2dMobilityModelMode : TimeSpeed : 10 m/sTime : 40 secDistance : 4000 mPath loss model Cost231eNB Antenna height 30 mObstacle height 35 mTraffic TCP Bulk File TransferFile size 1.5 MBMaximum neighbours to han-dover 8Total simulation runs 20Simulation time 40 sec holes, as shown in Fig. 1. Each UE is performing a TCP filetransfer to a remote host in downlink and uplink direction. Thecomplete set of simulation parameters are described in Table I.The simulation consists of 20 runs of a deterministic handoverto a potential neighbour. In this scenario, we observe that themaximum number of neighbours a UE manages to see is 8;therefore, each run is repeated 8 times to measure the QoE ofa UE, i.e., file download time. For every simulation run, a UEpicks a random starting position in the cluster and a randomangle in the range of [0 ◦ to ◦ ] to move away from thesource eNB following a straight line. The data obtained fromthese deterministic handover campaigns for each UE is storedin the form of a dataset, according to the format described inthe next subsection Sec. II-B. B. Dataset creation
We design our handover problem as a regression problem,where we need to estimate the QoE expected from performinghandover to a particular target cell. In general, when workingwith supervised learning, such as in our case, one has tobuild a database to train, test, and evaluate the model. Thisdataset consists of input and output features stored in rows andcolumns. For this purpose, we extracted 84 measurements fromeach layer of the LTE protocol stack Table II. Hereafter, wementioned them as input features. 3GPP already contemplatesthe upload of a part of these measurements, e.g., UE measure-ments, as part of the Minimization of Drive Test (MDT) [7]functionality. We gather measurements from all the layers of
ABLE IIL
IST OF MEASUREMENTS FROM
LTE
PROTOCOL STACK USED TO CREATE THE DATASET
Input featureLayer MeasurementsAPP 1. Throughput UL5. Avg. number of rcvd packets DL 2. Avg. number of rcvd packets UL4. Throughput DL 3. Avg. number of rcvd bytes UL6. Avg. number of rcvd bytes DLRRC 7. Cell ID of serving cell10. Cell ID of neighbour 1 ..
31. Cell ID of neighbour 834. Total number of radio link failures 8. RSRP from serving cell11. RSRP from neighbour 1 ..
32. RSRP from neighbour 835. Total number of handovers 9. RSRQ from serving cell12. RSRQ from neighbour 1 ..
33. RSRQ from neighbour 836. First target cell ID to handoverPDCP 37. Total number of txed PDCP PDUs DL40. Avg. PDCP PDU delay DL43. Min. PDCP PDU size DL46. Total number of rcvd PDCP PDUs UL49. Min. value of the PDCP PDU delay UL52. Max. PDCP PDU size UL 38. Total number of rcvd PDCP PDUs DL41. Min. value of the PDCP PDU delay DL44. Max. PDCP PDU size DL47. Total bytes txed UL50. Max. value of the PDCP PDU delay UL 39. Total bytes txed DL42. Max. value of the PDCP PDU delay DL45. Total number of txed PDCP PDUs UL48. Avg. PDCP PDU delay UL51. Min. PDCP PDU size ULRLC 53. Total number of txed RLC PDUs DL56. Total number of bytes received DL59. Max. value of the RLC PDU delay DL62. Total number of txed RLC PDUs UL65. Total bytes rcvd RLC PDUs UL68. Max. value of the RLC PDU delay UL 54. Total number of rcvd RLC PDUs DL57. Avg. RLC PDU delay DL60. Min. RLC PDU size DL63. Total number of rcvd RLC PDUs UL66. Avg. RLC PDU delay UL69. Minimum RLC PDU size UL 55. Total number of bytes txed DL58. Min. value of the RLC PDU delay DL61. Max. RLC PDU size DL64. Total bytes txed RLC PDUs UL67. Min. value of the RLC PDU delay UL70. Maximum RLC PDU size ULMAC 71. Initial MCS74. Avg. MCS UL77. Avg. RB occupied DL80. UL CQI 72 Avg. TB size UL75. Avg. MCS DL78. DL CQI inband 73. Avg. TB size DL76. Avg. RB occupied UL79. DL CQI widebandPHY 81. Avg. SINR DL84. Avg. number of UL HARQ NACKs 82. AVG. SINR UL 83. Avg. number of DL HARQ NACKsOutput featureAPP 1. File download time [sec] the protocol stack using some logs/traces already available inns-3, e.g., those available for RLC (Radio Link Control) andPDCP (Packet Data Convergence Protocol), and other newcustom trace sources at RRC (Radio Resource Control), MAC(Medium Access Control) and PHY, obtained by leveragingthe tracing system of ns-3 . The input features, for our dataset,are extracted with the periodicity of 200 ms in order to beconsistent with the approximate periodicity with which UEmeasurements are reported from UEs at the RRC level. Thisdataset can be expressed as a matrix X . X = x , x , · · · x ,m x , . . . · · · x ,m ... ... x i,j ... x n, x n, · · · x n,m where the feature vector of size 84 is x i,j ∈ X , ≤ i ≤ n ,and ≤ j ≤ m , with ≤ n ≤ , m = 200 .The total number of samples, i.e., the upper limit of n ,can be computed by multiplying the total number of UEswith the maximum neighbours to handover, and the totalnumber of simulation runs (see Table I). On the other hand, m corresponds to the number of time-steps that the LSTMprocesses to infer the time to download by a UE, which is200 in our case.III. LSTM MODELS FOR HANDOVER MANAGEMENT
As mentioned in Section II, the dataset consists of themeasurements and traces extracted periodically from each layer of LTE protocol stack, forming a time series of mul-tivariate features. We believe that, by exploiting the temporalcharacteristic of this data one could understand the impactof handover decisions. Therefore, in this paper we employRNN with LSTM units [8]. It is a special kind of RNN,which outperforms other machine learning approaches for timeseries analysis [9] [10], and solves the problem of long-termdependency issue found in vanilla RNN [11].In this paper, we present two models to predict the timerequired by a UE to download a file using the dataset. Fig.2,shows the first model, which is based on a multi-layer many-to-one LSTM architecture. This model takes 200 timesteps(i.e., m ), each comprises of 84 features, as input, and process Fig. 2. Proposed model 1 : Many to one LSTM architecture hem in a single lag with multiple batches of size 32, to inferthe time to download. On the other hand, the second modelis based on a multi-layer LSTM AE [12] in conjunction witha MultiLayer Perceptron (MLP) neural network, as shown inFig.3 . An AE is an unsupervised machine learning algorithm,which learns a function to approximate an output identical tothe input. Since it is based on the encoder-decoder paradigm,the input is transformed into a lower-dimensional space, alsoknown as Codeword (CW), to more efficiently model highlynon-linear dependencies in the inputs. The compression op-eration manages to extract more general and useful features,which retain important aspects of a dataset [13]. Our goal isto smartly reduce the data to be used for inferring the time todownload. IV. P
ERFORMANCE E VALUATION
The implementation of the proposed models is done inPython, using Keras and Tensorflow, as backend. In particular,to speedup the training, testing, and evaluation of these modelswe used fast LSTM implementation with Nvidia CUDA DeepNeural Network (CuDNN) library for GPUs [14]. We note that,to select the hyperparameters of the first model, i.e, numberof layers and the number of LSTM units (blue LSTM blocksin Fig. 2) in each layer, we tested nine different combinations.Finally, the hyperparameters resulted in a lowest average MeanSquare Error (MSE) (over 200 epochs) were selected. For thesecond model, we use a similar approach first to select theCW length of the AE, among five different CW lengths. Then,using this selected CW as an input to the MLP neural network,one set of hyperparameters, among 7, was chosen for the MLP.For the readability purpose, Fig. 4, Fig. 5, and Fig. 6 show theloss, i.e., Mean Square Error (MSE) per epoch, only for theselected hyperparameters. In particular, Fig. 4 shows the MSEper epoch of the first model using 3 (i.e.,
L = 1, K = 2 ) layersof LSTM nodes, where the numbers separated by “x” representthe number of hidden LSTM units. We observe that, after 140epochs, this model is able to achieve and maintain very lowtesting loss independently from the number of layers and cells
Fig. 3. Proposed model 2 : Autoencoder + MLP neural network Lo ss Epoch
Training [84x62x42]Validation [84x62x42]
Fig. 4. Mean square error per epoch of LSTM [84x62x42] per layer. Related to the second model, Fig. 5 shows the AEloss using CW length of 100 over 200 epochs. We note that,this loss is the MSE between the decoded version of the inputdata and the original input data fed to the AE. Similarly, Fig. 6shows training and validation loss of the final, two layered (80and 40 neurons in first and second hidden layer, respectively)MLP neural network fed with the input of CW length of 100.The performance evaluation of these models is performedin an offline fashion, i.e., by comparing the real time todownload for each UE, obtained after selecting the targetcell providing the lowest predicted time to download, tothe one achieved by using a benchmark approach, i.e., A2-RSRP based handover algorithm. In particular, to performthis evaluation we consider another dataset generated withtwo extra simulation campaigns using a Run value which wasnot used to build the training dataset (i.e. Run 21). The firstcampaign aims at gathering the file download time using thebenchmark handover algorithm (e.g., A2-RSRP).The secondsimulation campaign is conducted in a similar way as the oneto build the training dataset, i.e., it consists of 8 deterministichandovers. Following this approach, we construct 8 input Lo ss Epoch cw 100
Fig. 5. Mean square error per epoch of AE (CW = 100) Lo ss Epoch
Training [80x40]Validation [80x40]
Fig. 6. Mean square error per epoch of MLP [80x40] strings, for each neighbour of a UE, which consists of 1 rowand 16800 columns (i.e, 84 features x 200 time steps). Toobtain a predicted time to download for the selected LSTMand AE architectures, these strings are used individually astheir input. Finally, for each UE, we select the eNB with theminimum predicted time to download for the handover. Theresults obtained using the benchmark handover algorithm showthat there are 63 UEs out of 210, which are able to finalize thedownload. On the other hand, using the two trained models,77 UEs are able to download the file successfully. This meansthat the machine learning approach manages to increase by 18% the number of UEs, which are able to finalize the downloadduring the simulation time. Moreover, there are 62 commonUEs, which were always able to download the file, irrespectiveof the tested handover solution, i.e., benchmark, LSTM orAE based. Fig.7, plots the Empirical Cumulative DistributionFunction (ECDF) of the difference between the download timeobserved by these UEs using the benchmark, and the twoproposed models. The ECDF trend, on the positive x-axisshows that, using LSTM or AE we are able to reduce the filedownload time for 56 UEs compared to the benchmark case.However, there are 6 UEs which experience marginally higherdownload time compared to the benchmark (see the trend on-ve x axis in Fig.7). We believe that their performance can beimproved by increasing the size of the database used to trainthe models and by further fine tuning their hyperparameters.Moreover, this evaluation shows that the MLP, fed with theAE CW of 100 performs similarly to the LSTM. This provesthat the AE has efficiently transformed the inputs into a lower-dimensional space without losing the meaningful informationof the dataset for the use case of the handover. To furtherinvestigate the reusability of these trained models, we testthem in a simulation scenario with different deployment ofthe obstacles. In this scenario, using the benchmark handoveralgorithm 78 UEs out of 210 are able to download the file. Onthe contrary, the two models perform similar to each other, andoffer an increase of 11.3636 %, (i.e., 88) in the number of UEs,which are able to finish the download. Similarly, the ECDF E C D F LSTM-Run21-84x62x42AE-Run21-Cw100-80x40
Fig. 7. The difference between SOTA and ML DL time for common UEs in Fig.8 shows that using the two models, out of 77 commonUEs, we are able to decrease the file download time for 71.However, due to the presence of new temporal data introducedby the new spatial characteristic of the outages, there are6 UEs, which experience high download time. This can berecovered by extending the available knowledge obtained inthe old scenario, with new incremental data from the newscenario. V. C
ONCLUSIONS
In this paper, we have exploited heterogeneous data, whichis already available inside the network at different layers ofthe LTE protocol stack. This data is used to gain meaningfulexperience to make handover decisions, which are not basedon the signal strength before the handover, but on the expectedQoE after the handover. We have first proposed an RNN thatexploits the temporal characteristic of this data. In particular,an LSTM is designed to model a regression problem thatestimates the expected time to download a file for the different −5.0 −2.5 0.0 2.5 5.0 7.5 10.0Time (sec)0.00.20.40.60.81.0 E C D F LSTM-Run21-84x62x42AE-Run21-Cw100-80x40
Fig. 8. The difference between SOTA and ML DL time for common UEs eighbours of the current serving cell. Our approach outper-forms a traditional event-based benchmark handover schemein terms of the number of successful downloads and time todownload statistics. To reduce the dimensionality of the dataand then facilitate the possibility to transfer the experienceinside the network, we have proposed also a second modelbased on an LSTM-AE to compress the data up to a codewordof 100 and then an MLP neural network that implements theregressor. We tested this model in two different simulationscenarios, and conclude that its performance is equivalent tothe one achieved using the LSTM trained with uncompresseddata. Moreover, this also encourages us to go one step furtherto extend our work in the future, where we could leverage AEbased multitask learning using the same database.A
CKNOWLEDGEMENTS
This work was partially funded by Spanish MINECOgrant TEC2017-88373-R (5G-REFINE) and Generalitat deCatalunya grant 2017 SGR 1195. It was also supported byHuawei Technologies, Sweden AB.R
EFERENCES[1] J. Moysen and L. Giupponi, “From 4G to 5G: Self-organized networkmanagement meets machine learning,”
Computer Communications , vol.129, pp. 248–268, Sep. 2018.[2] N. Baldo, L. Giupponi, and J. Mangues, “Big Data Empowered SelfOrganized Networks,” in
Proceedings of the 20th European WirelessConference , May 2014, pp. 1–8.[3] S. S. Mwanje and A. Mitschele-Thiel, “Distributed cooperative Q-learning for mobility-sensitive handover optimization in LTE SON,”in
Proceedings of the Computers and Communication (ISCC), IEEESymposium on , June 2014, pp. 1–6.[4] Z. Ali, N. Baldo, J. Mangues, and L. Giupponi, “Machine LearningBased Handover Management for Improved QoE in LTE,” in
Proceed-ings of the IEEE/IFIP Network Operations and Management Symposium(NOMS) , Istanbul, Turkey, Apr. 2016.[5] ns 3. 2019, “Network Simulator,” http://code.nsnam.org/ns-3-dev.[6] N. Baldo, M. Miozzo, M. Requena-Esteso, and J. Nin-Guerrero, “Anopen source product-oriented LTE network simulator based on ns-3,” in
Proceedings of the 14th ACM International Conference on Modeling,Analysis and Simulation of Wireless and Mobile Systems , ser. MSWiM’11. New York, NY, USA: ACM, 2011, pp. 293–298.[7] 3GPP TS 36.331, “Radio measurement collection for Minimization ofDrive Tests (MDT); Overall description,” version 10.4.0, Release 10.[8] S. Hochreiter and J. A. Schmidhuber, “Long short-term memory,”
NeuralComputation , vol. 9, no. 8, pp. 1735–1780, 1997.[9] J. Wang, J. Tang, Z. Xu, Y. Wang, G. Xue, X. Zhang, and D. Yang,“Spatiotemporal modeling and prediction in cellular networks: A bigdata enabled deep learning approach,” in
IEEE INFOCOM 2017 - IEEEConference on Computer Communications , May 2017, pp. 1–9.[10] H. D. Trinh, L. Giupponi, and P. Dini, “Mobile traffic predictionfrom raw data using LSTM networks,” in
Proceedings of IEEE 29thAnnual International Symposium on Personal, Indoor, and Mobile RadioCommunication (PIMRC) , Bologna, Italy, Sep. 2018.[11] F. A. Gers, J. A. Schmidhuber, and F. A. Cummins, “Learning to forget:Continual prediction with lstm,”
Neural Comput. , vol. 12, no. 10, pp.2451–2471, Oct. 2000.[12] A. M. Dai and Q. V. Le, “Semi-supervised sequence learning,” in
Advances in Neural Information Processing Systems 28 , C. Cortes, N. D.Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, Eds. CurranAssociates, Inc., 2015, pp. 3079–3087.[13] J. Masci, U. Meier, D. Cires¸an, and J. Schmidhuber, “Stacked convolu-tional auto-encoders for hierarchical feature extraction,” in
Proceedingsof the 21th International Conference on Artificial Neural Networks -Volume Part I , ser. ICANN’11, Espoo, Finland, 2011, pp. 52–59.[14] F. Chollet et al.et al.