Agile Calibration Process of Full-Stack Simulation Frameworks for V2X Communications
Ioannis Mavromatis, Andrea Tassi, Robert J. Piechocki, Andrew Nix
aa r X i v : . [ c s . PF ] O c t Agile Calibration Process of Full-Stack SimulationFrameworks for V2X Communications
Ioannis Mavromatis, Andrea Tassi, Robert J. Piechocki, and Andrew Nix
Department of Electrical and Electronic Engineering, University of Bristol, UKEmails: {Ioan.Mavromatis, A.Tassi, R.J.Piechocki, Andy.Nix}@bristol.ac.uk
Abstract —Computer simulations and real-world car trials areessential to investigate the performance of Vehicle-to-Everything(V2X) networks. However, simulations are imperfect modelsof the physical reality and can be trusted only when theyindicate agreement with the real-world. On the other hand,trials lack reproducibility and are subject to uncertainties anderrors. In this paper, we will illustrate a case study where theinterrelationship between trials, simulation, and the reality-of-interest is presented. Results are then compared in a holisticfashion. Our study will describe the procedure followed tomacroscopically calibrate a full-stack network simulator toconduct high-fidelity full-stack computer simulations.
Keywords —Connected Autonomous Vehicles, V2X, IEEE802.11p/DSRC, Full-Stack Simulator, VEINS, Validation process.
I. I
NTRODUCTION
Connected Autonomous Vehicles (CAVs) are enhanceddaily with new autonomous features, leading gradually tofully autonomous means-of-transports [1]. Being part ofthe ecosystem of Intelligent Transportation Systems (ITSs),CAVs will require an agile interconnecting framework [2],[3], providing a constant service and optimal system be-havior. To optimize and further enhance the performanceof this framework, initial experimental evaluation of real-world trials and simulation results is required.In this work, we aim to establish a connection betweensimulated and trial-based results for a vehicular network.Introducing the procedure followed, we will describe howinconsistencies during the experiments are identified andexcluded from the evaluation. We will later present acalibration framework for the fine-tuning of the imperfectsimulation results to enhance their behavior and achievehigh-fidelity “real-world” results.Simulations are approximated models of the physicalworld. However, they are easily and inexpensively con-ducted using an appropriate network simulator achievingnear-perfect results. For example, the number of vehicleswithin a network can be easily scaled up to increase thenetwork congestion (as in [4]). Furthermore, they offera high degree of flexibility. Using different configurationsand isolating particular parameters, we can examine thebehavior of a system under specific conditions. This is thecase of [5] where the impact of different beacon intervalshas been investigated with respect to the end-to-end delay,for different road networks and city-wide scenarios. On the contrary, real-world trials are based on a “per-fect” model. For example, authors in [6] experimentallyanalyzed the performance of a vehicular network based ondata coming from off-the-shelf IEEE 802.11p devices. Thedisadvantages of this experimental evaluation are the cost,the required time and the inability to reproduce – since itis affected by physical parameters varying over time and itcannot be easily isolated and ignored.A very complex real-world system is harder to modelor might require increased resources to be simulated. Trialresults can aid the design of a system by abstracting variousparameters and introducing them as a priori knowledgein a simulation. For instance, authors in [7], designed asimulator for geometry-based efficient propagation modelsfor Vehicle-to-Vehicle (V2V) communications. Their simu-lator was based on an extensive trial campaign, in orderto identify parameters such as the path loss exponent andthe small-scale signal deviation for different distances andenvironments. Trial-based results though may vary betweendifferent devices, being related with the quality of theequipment used and the software accompanying it.For the above reasons, it is obvious that various approxi-mations, random and systematic errors are introduced dur-ing a system performance evaluation. A direct comparisonbetween the simulated and real-world results will end up ina performance difference. To increase the accuracy of thisscientific evaluation, in this paper we will establish a frame-work where real-world trials and simulations co-exist. Shar-ing knowledge between them, we will fine-tune a full-stacknetwork simulator, enhancing the accuracy of our resultsand giving us the leverage of more precise experimentationlater. For our trials, an open-source testbed will be used,consisting of single-board devices equipped with differentwireless Network Interface Controllers (NICs) designed tobe IEEE 802.11p compliant. The simulated results will beacquired using the VEINS network simulator [8]. This isa vehicular networking framework based on Omnet++ [9]and is compliant with the IEEE 802.11p/Wireless Access inVehicular Environments (WAVE) standards.The rest of the paper is organized as follows. In Sec. II, thehierarchical framework and the interrelationship betweensimulations and trials are described. The different entitiesof this framework, the relationship between them and thepractical issues of the validation process are discussed.In Sec. III, the procedure to fine-tune VEINS is analyzed,tarting from an initial calibration isolating various param-eters and moving towards a full-stack system optimization.Individually analyzing each scenario, their fine-tuned per-formance evaluation is examined in Sec. IV. Finally, Sec. Vconcludes the paper and provides future research avenues.II. H
IERARCHICAL F RAMEWORK : T
RIALS AND S IMULATIONS
Consider an ITS consisting of a number of CAVs and RoadSide Units (RSUs) on a road network. Different kinds of dataare exchanged with respect to the safety- or infotainment-related applications and services running on the ITS. Forexample, WAVE Short Messages (WSMs) [10], are safety-critical messages encapsulating core information aboutCAVs (e.g. position, velocity, size, etc.). These messages areeither broadcast every 100 ms or are triggered to announceroad hazards. They are relatively short (~300-800 B) and ahigh delivery rate and a low one-hop end-to-end delay areregarded as their Quality-of-Service (QoS) constraints.Apart from WSMs, safety-critical applications will bekey in future ITSs. For example, video-assisted over-taking or traffic monitoring applications are tested onCAVs [11]. These applications require the transmission ofvideo streams encapsulated within UDP packets. Increaseddata rate and low jitter are their main QoS requirements,with a more forgiving bit error rate (BER) performance dueto the adoption of Forward Error Correction (FEC) codesand the new generation higher efficiency video encoders.Following the content-related QoS requirements, eachapplication behaves differently under various physical en-vironments. Three different environments can be foundin vehicular networks (urban, suburban, rural) [12]. Forexample, urban environments are affected by blockagesfrom the buildings, significantly attenuating the signal.Urban canyon behaviors can be introduced, under specificcircumstances, waveguiding the signal. On the other hand,foliage is the main form of blockage in rural areas whilevehicles tend to move faster introducing a higher DopplerShift effect. Each scenario should be approached differentlywhen simulated, adapting accordingly the various channelmodel characteristics.
A. Co-operation and Co-existence of Trials and Simulations
Cooperation between trials and simulations is mandatoryto increase the accuracy of a system performance valida-tion. Exchanging information between them can enhancethe outcome, maximize the time utilization and minimizethe cost. Establishing a framework between the reality-of-interest , i.e. the part of the real world (e.g. a city, aneighborhood or a road) that we are interested in inves-tigating, the trials conducted and the simulation models used, will help us better understand the requirements andthe limitations of each one. The interrelationship betweenthese three entities can be found in Fig. 1.The assessment of the simulation accuracy can be di-vided in two phases, the validation and the verification [13].The verification (Fig. 1 – a) is the confirmation that a model
SimulationsModel DevelopmentReality-of-Interest Trials
Experimental Design basedon Engineering ConstraintsMeasurements: Incompletewith uncertainties V e r i fi c a t i o n P r o c e s s E r r o r F i x i n g V a l i d a t i o n P r o c e s s P h y s i c a l I n s i g h t ac d e g ConceptualModels b f C a l i b r a t i o n Fig. 1. The interrelationship between the experiments, the simulationsand the real world as well as the processes that connect them. is correctly implemented and reflects the real world. This isconfirmed by conceptual models – abstractions of a systemthat characterize and standardize a network function, e.g.the OSI model (Fig. 1 – b). Verifying a model, the existingerrors can be determined and fixed to assure that it matchesspecifications and assumptions with respect to the initialconcept (Fig. 1 – c). For this work, we will not discuss theconcept of the model verification for VEINS any further. Formore, we kindly refer the reader to [14].A trial-based evaluation is limited by the engineeringconstraints (Fig. 1 – d). For example, a NIC might supportdecreased transmission power compared to the standardspecifications, limiting the operational range of a device.What is more, trials suffer from uncertainties (e.g. theattenuation of the signal with respect to the weatheris unpredictable, not easily measured and environment-dependent) affecting the reliability and the validity of theexperiment. The replicability of the experiment is also abig concern. All roads are not the same and the deviceshave different specifications, so replicating an experimentis difficult. This leads to incomplete results (Fig. 1 – e) asit is impossible to validate all possible combinations.The validation of a theoretical model assesses the fidelitythat a model reproduces the state and behavior of thereal world (Fig. 1 – f). Verification usually precedes thevalidation of a model. The models can be validated withsimple experiments isolating the external factors affectingthe performance. To achieve a meaningful representationof the real world, a simulator should be fine-tuned at first(Fig. 1 – g) using inputs from measured results (e.g. pathloss exponent) or applying weights at the output to min-imize the divergence error. A direct comparison betweenthe absolute values of the experimental results should beavoided. The validation process should focus on the trendsof performance between the different experiments (e.g.both the simulated and trial-based results have a relativedegradation when one parameter is changed).
B. Hierarchical Validation of a Simulation Model
Differences between trial and simulated results can arisefrom various reasons. Typical examples are measurement er-rors (e.g. calibration errors, noise or data acquisition meth-ods, etc.), formulation errors (e.g. incorrect channel models)or numerical errors (e.g. overflow of integers, subtraction of igh Powertransceiver Low Powertransceiver Road-Side andOn-board UnitHigh-gain AntennaLow-gain Antenna
Fig. 2. Low-latency Linux Kernel implementation of IEEE 802.11p/DSRCunits. The figure shows units designed for RSU and OBU deployment. floating points, etc.). Two different error categories exist.Firstly, the random errors, affecting the relative precision ofa model or a measurement. Secondly, the systematic errorsaffecting the absolute value of a result, being repeatablethough and therefore, easily predicted.In order to achieve the required level of accuracy, thevalidation procedure should be carried out throughout theentire development process following a hierarchical ap-proach [15]. Fragmenting the problem into smaller entitiesand solving them individually, the necessary level of pre-cision can be achieved without increasing the complexity.Generally, the trials should test crucial features of thesimulation models, such as the impact of the consideredassumptions or the simplifications. On the other hand,simulations allow incremental validation towards a “real-world-like” system.III. F
INE -T UNING
VEINS N
ETWORK S IMULATOR
Designing a real-world ITS solution, all the above shouldbe taken into account as well as the initial conditions, theboundaries and the trends of the performance. Isolating thecharacteristics that disruptively affect the performance wecan isolate the systematic and the random errors approach-ing an "ideal-like" system. Using a network simulator, wecan further validate different scenarios based on our initialconfiguration. To do so, the fine-tuning process of VEINSrequires a detailed study on the available hardware, the per-formance metrics used, an insight into how the simulatoroperates and what are the differences of the real-world. Thisstudy will be discussed in the next subsections.
A. Description of the Experimental Setup
For our experimental validation, we prototyped an open-source IEEE 802.11p/DSRC testbed (Fig. 2) meeting thefollowing requirements: • Easily customisable . • Low cost with the potential to be massively deploy later. • Open-source operating system providing enough flexi-bility for future developments. mac80211 ath9kath5k…
Userspace Kernelspace cfg80211_ops ieee80211_ops nl80211 cfg80211Utilities - iw, …
Deadline I/O Scheduler
Frame Transmission
Fig. 3. Linux Kernel Modules that have been modified to implement aIEEE 802.11p/DSRC stack. • Dual-operation as RSUs and On-Board Units (OBUs) . • Weatherproof .To meet these requirements, each device was equippedwith a Mikrotik RB433 single-board computer (CPU300 MHz, 64 MB RAM, 64 MB storage space, x3 Ethernets,x3 MiniPCI slots) [16]. Two IEEE 802.11a NICs were usedfor redundancy in the wireless links. The first one was aMikrotik R52H [17], regarded in this work as a low-power(LP) NIC, with transmission power of up to 25 dBm andconnected to a dipole antenna of 7 dBi gain at 5.9 GHz. Thesecond model was a Mikrotik R5SHPn [18], operating as thehigh-power (HP) transceiver in our experiments (29 dBmmaximum transmission power). This NIC was connected toa 9 dBi-gain dipole antenna.A low-latency OpenWRT Linux distribution was used asthe operating system. The different Atheros chipsets of eachtransceiver (AR5414 for the LP and AR9220 for the HP)required the use of two different Atheros drivers (ath5kfor the LP and ath9k for the HP). Both were modifiedaccordingly to enable IEEE 802.11p compatibility. The Linuxkernel modules that we modified have been summarized inFig. 3. The software modules cfg80211 and nl80211 act as in-terfaces between the user and kernel space, mac80211 is thegeneral driver framework, and iw is the NIC configurationutility. Furthermore, cfg80211_ops and ieee80211_ops definethe operations and the callbacks between the differentblocks. The Outside the Context of a BSS (OCB) mode wasenabled in the MAC layer, allowing the NICs to operatewithout being associated with an access point and the iw utility was modified accordingly to include the newcommands for using OCB mode. Finally, the 5.9 GHz bandwas added in the regulatory domain.Each device can be connected with a GPS dongle via itsUSB interface. Finally, a beaconing interface was developedthat was able to acquire viable vehicle information (e.g.length, position, emissions, etc.) from an Engine ControlUnit (ECU) and broadcast them to surrounding devices. B. Initial V2X Calibration Scenario
Consider a scenario with an ideal channel (no reflections,free-space path loss) between two ideal stationary vehicles(isotropic antennas, zero hardware attenuation). As thisideal system does not exist in the real world, the experi-mental setup was initially demonstrated inside an anechoicchamber (8.5 m × × OpenWRT Barrier Breaker Release no. 14.07 - https://openwrt.org/ wo devices, one acting as a RSU and the other as an OBU.A UDP data stream, transmitted from the OBU to the RSUand generated using iPerf traffic generator , and a periodicbeacon every 100 ms, were used to saturate the channel.As known, the network level performance is affected bythe signal-to-interference-plus-noise ratio (SINR) and thesensitivity levels for each MCS. The SINR degrades based onthe disruptive characteristics of the channel (e.g. distanceattenuation, multipath, antenna misalignment, etc.) and thedevices (e.g. thermal noise, etc.). Using an anechoic cham-ber, we conducted the experiments under near-optimalconditions (SINR greater than the sensitivity level) andtherefore the optimal performance was achieved.The same scenario was designed in VEINS as well. Bypartitioning the design process into smaller problems, wemanaged to achieve the required level of similarity. Firstly,we considered the IEEE 802.11p Physical Layer (PHY) frame,which consists of three fields [19]: (i) The Preamble marksthe beginning of the PHY frame, is responsible for theappropriate antenna selection and corrects the timing andfrequency offsets, (ii) the
Signal field (SIG) specifies theframe rate and length and (iii) The
Data field consistingof the Physical layer Service Data Unit (PSDU) that en-capsulates the MAC frame, the Physical Layer ConvergenceProcedure (PLCP) Service, and a Tail field. The Data fieldcan also be padded with extra bits so its length is a multipleof the coded bits in an OFDM symbol. The above aretransmitted using BPSK / Modulation and Coding Scheme(MCS). The length of each field can be found in Tab. I.In VEINS, the duration of the Preamble and SIG iscontrolled by the parameter preambleDuration , while thebit length of the Data field is set equal to headerBitLength (see Fig. 4). The simulated PHY bitrates are governed bythe wireless interface operational mode – namely, opMode that has been set to “ p ”, in this case. Each MCS shouldbe manually configured for each individual simulation andmatched with the appropriate PHY bitrate using pairs ofthe simulation parameters bitrate and modulation . OtherPHY parameters that should be set within VEINS are thechannel bandwidth , the carrierFrequency , the antennaType ( ConstantAntennaGain in this case) and the gain .The multi-channel operation introduced in the WAVE1609.4 standard [10] was not considered in order to identifythe maximum performance under saturation conditions.Furthermore, the RTS threshold – namely rtsThreshold , wasset to a value greater than the frame size. This ensuredthat the RTS/CTS procedure was disabled. The MAC layerbackoff times are drawn from a Contention Window (CW)starting from CW min ( cwMinData and cwMaxData ). Thevalues chosen for our setup (Tab. I) where proven to beoptimal for vehicular communications [20]. The length ofthe MAC TX queue size is capped by the driver (in the ath5kcase), so the same value was considered in the simulationas well ( maxQueueSize parameter within VEINS). iPerf Traffic Generator - https://iperf.fr TABLE IS
IMULATION AND E XPERIMENTAL P ARAMETERS . Parameter Value
Experiment/Simulation Time 10 sCarrier Frequency 5.9 GHzBandwidth 10 MHzMTU 1500 kBUDP Packet Length 8192 kBBeacon Length 500 BBeacon Interval 100 msPreamble Duration 32 µ sSIG Duration 8 µ sPLCP Service Length 16 bitTail Length (Data Field) 6 bit CW min , CW max [15,1023]TX MAC queue size 50Background Noise N ( − The VEINS parameter sentInterval sets the interval be-tween the generation of two consecutive UDP packets. Inorder to saturate the channel, a very precise interval waschosen for each MCS. Suitable values were found with atrial and error method to fully utilize the channel withouthaving packets discarded from the MAC TX queue. All thesimulation and experimental parameters can be found inTab. I and Sec. III-A.The results for the above calibration scenario are shownin Figs. 5 and 6. The central rectangle is the interquartile re-gion (IQR) between the first and the third quartile, while theline within represents the median. The whiskers constitutethe maximum and the minimum values and the asterisksshow the outliers. A value is regarded as an outlier if itoutside ± σ (99.3% percentage coverage of the normallydistributed samples).Fig. 5 shows the reception throughput measured at thetransport layer. As mentioned in Sec. II-B, a meaningfulcomparison should focus on the trends. Therefore, a trendcan be seen in the performance whereby the simulationresults are slightly better for some MCSs (e.g. QPSK / )while for others (e.g. BPSK / ) they are almost identical. Thedeviation in the mediam is of the order of up to ~0.5 Mbpsfor the HP device and ~1 Mbps for the LP transceiver.Overall, we observed that the LP transceiver has a worsethroughput performance (median values decreased by ~5%)compared to the HP one; following the same trend for allMCSs. This difference is due to the operation of the differentdrivers used.Fig. 6 compares the inter-arrival jitter performance. Thejitter, as defined in RFC 1889 [21], is the statistical varianceof the inter-arrival time between packets. Comparing theabsolute values of the results, we see a huge differencebetween the trials and the simulations. However, comparingthe relative variation in the jitter performance for thedifferent MCSs, it is shown that in both cases, the resultsfollow a similar trend starting with an increased jitter for thelower MCSs, and having a better performance as the bitrate nitial Calibrationwithin the AnechoicChamber Applications
SafetyRelatedUDPStreamsBeaconsPreamble DurationHeader Bit Length
PHY Layer
Operational ModeMCSBitrate MTU lengthBeacon LengthBeacon IntervalTX Queue SizeContention WindowMAC Address: AUTO
MAC Layer
ARP requestsIP Address: AUTO
Link Layer
UDP Packet LengthTransmission Interval
Transport Layer
Propagation ModelPath Loss ModelObstacle Loss ModelSignal Representation
Channel Model
TX PowerRX SensitivitySINR Threshold
Real Device Char.
ThermalNoiseCosmicBackgroundNoiseElectrom.FieldEffectsSystematicErrorsRandomErrors Complex Scenarios
Scenarios
Background Noise
Comparison with Experimental Results
Feedback Line: Step back to fix the errors or apply weights to the outcome.
Initial Calibration Process Real World Representation
Design Phase Testing Phase
SystematicErrorsRandomErrors
Testing PhaseDesign Phase K n o w l e d g e f r o m t h e I n i t i a l C a l i b r a t i o n P ha s e Fig. 4. Hierarchical Validation Process. Each entity is individually fine-tuned to achieve high-fidelity results after the calibration of the entire system.The fine-tuning is a two-phase process starting from an initial calibration and moving to a more complex real-world representation. is increased. To that extent, the jitter values measured withVEINS have been multiplied by 56000 in order to get thesame order of magnitude with the ones obtained by the LPand HP transceivers. The huge difference in the absolutevalues was somehow expected as our devices are built upona single-core CPU, which executes tasks with the samepriority according to the Linux
Deadline I/O Scheduler –thus the CPU cannot fetch/push data streams towards thetransceivers at a constant I/O rate. Since in VEINS thisissue is not present, the simulated and measured jitterperformance may vary significantly.With regards to the framework introduced in Sec. II, wefragmented our system into smaller problems, fine-tuningindividually each of them and managed to build an idealsubsystem able to achieve similar performance with minordeviations between the trial and the simulated results. Inthe next sections, we will introduce more complex entitiessuch as the real device profiles and the channel behaviorapproaching a better representation of the real world.
C. Moving towards a Realistic Representation of the World
The surrounding environment plays a key role in theperformance of vehicular networks. Each environment hasits own characteristics, therefore a theoretical analysis isrequired for the signal degradation due to the differentchannels. The channel behavior within VEINS lies beneaththe physical medium model . This model is further split intodifferent submodules (Fig. 4).At first, the propagation model ( propagationType ) de-scribes the propagation time within the channel. Consid-ering constantSpeed as our propagation model, the prop-agation time is proportional to the distance travelled.To accurately represent the signal and its fluctuation, a
DimensionalAnalogModel was utilized, meaning that thesignal power deviation is represented over both time andfrequency.The long-term signal degradation in the real-world de-pends on the distance, the carrier frequency, the de-vice positioning, etc. In VEINS, the simulation parameter pathLossType describes the path loss model that is respon-sible for computing the power reduction based on thetraveled distance, the velocity factor, the carrier frequencyand the path loss exponent for each environment. Theshort-term signal degradation, affected by the multipathdistortion caused by the surrounding buildings can bedescribed within VEINS using small-scale fading models(e.g. Nakagami, Rician, etc.) and fine-tuning their individ-ual parameters (shape-factor for Nakagami, K-factor forRician, etc.) accordingly. An obstacle loss model can beadded to an existing path loss model. If so, the parameter
ObstacleLossType specifies the material absorption when aray is traveling within an object. The physical obstaclescan be listed using an XML file processed by the physicalenvironment model within VEINS.
D. Integration of Real Device Profiles in VEINS
After configuring the parameters related to the signalpropagation, it is key to consider the different characteris-tics of each simulated NIC. Unfortunately, commercial-off-the-shelf (COTS) devices have frequently to be treated as a“black box” as many of their physical level performancecharacteristics are unknown and not easily measurable.Therefore, we will base some of our simulation parameterson speculations based on the datasheet of each consideredtransceiver.At first, the SINR can fluctuate from random effectssuch as the thermal noise, the cosmic background noise,electromagnetic field effects, etc. These effects are notpredictable and do not particularly come from a specificsource to be isolated. VEINS represents this signal variationwith a background noise model ( backgroundNoiseType ),configured using the backgroundNoise parameter followinga Normal distribution. Also, the cables and the connectorsin a system introduce a systematic attenuation describedas the systemLoss that was measured within our laboratory.According to the manufacturer datasheet, we have accessto the transmission power given at 20 MHz of channel band-width, for each MCS. However, the IEEE 802.11p bandwidthis equal to 10 MHz. From the energy-per-symbol-to-noise ig. 5. Values of throughput obtained from VEINS (initial calibration), the HP and LP transceivers, for different MCSs.Fig. 6. Values of jitter associated to the UDP stream. Obtained from VEINS (initial calibration), the HP and LP transceivers for different MCSs. power spectral density equation it follows that: E s N = CN Bf s (1)where E s is the energy per symbol, N is the noise power, C / N is the carrier-to-noise ratio, B is the channel band-width and finally f s is the symbol rate. In our case, the onlynon-constant variable is B . Therefore, for a 10 MHz channel,the E s / N ratio is expected to be twice as much as thatmeasured using a 20 MHz channel. As N is measured perunit of bandwidth (per MHz), it follows that E s is doubled.Finally, knowing the number of bits per symbol of eachMCS, we can infer the maximum transmission power for a10 MHz channel. These values are summarized in Tab. II.With regards to the sensitivity of the receiver, from theMinimum Operational Sensitivity (MOS) relation, we knowthat: MOS = SI N R thr k T α B ( N F ) G rx (2)where SI N R thr is the minimum SINR needed to process(not just detect) a signal,
N F is the noise figure, k isBoltzmann’s Constant, T α is the effective noise temperaturereferred at the input of the receiver, and G rx is the isotropicantenna gain. Obviously, SI N R thr depends not just on theNIC but also on the MCS in use. Authors in [22] measuredthe
SI N R thr under a V2I scenario for two different antennaheights. Their lower antenna configuration was very similarto ours, so their
SI N R thr results will be utilized for ourscenarios. Knowing the
SI N R thr , the only variable in (2)is B . Therefore, halving B , the MOS will be doubled.Finally, the antenna gain values were taken directly fromthe manufacturer datasheet. All the values are presented inTab. II.IV. P ERFORMANCE E VALUATION AND M ACROSCOPIC V IEW
A. Scenarios and channel analysis
In Sec. III, the full-stack calibration process for VEINS wasdiscussed. As mentioned, the different environments signif-
TABLE IIS
IMULATION P ARAMETERS BASED ON THE M ANUFACTURER D ATASHEET . Modulat. TX power RX sensitivity
SI NR thr [22]
UnitsLP HP LP HP / MCS / MCS
BPSK 27 28 − −
93 10 15 dBmQPSK 26 27 − −
88 10 15 dBm16-QAM 25 26 − −
84 17 17 dBm64-QAM 24 24 − −
80 20 25 dBm icantly change the behavior of the system. To that extent,three different scenarios were designed and evaluated. As inthe initial calibration scenario, we considered one RSU andone OBU devices, stationary during the experiments havingtheir performance being evaluated for different distances,MCSs and for both the HP and LP transceivers. Again a UDPstream and a periodic beaconing transmitted from the OBUto the RSU saturate the channel to evaluate the networkthroughput (as described in Sec. II and III-B). The twodifferent transceiver configurations were simulated withinVEINS using the parameters from Tabs. I and II based onthe analysis preceded in the previous sections.The first scenario (see Fig. 7 – A) is an urban road withbuildings on both sides. The devices are positioned onthe pavement and there is always a Line-of-Sight (LOS).The buildings surrounding the devices cause multipathdistortion. However, since the devices are always in LOSand the experiment was conducted at a relatively shortdistance, a Rician fading model was considered with a K -factor k = α = ig. 7. The three different scenarios were conducted around the city ofBristol, UK. a) the urban, b) the suburban, c) the rural scenario. model with k = α = α = k = α = B. Performance Evaluation
During the experiments carried in scenarios A, B andC, the RSU and OBU were fitted on a tripod at ~1.8 mheight. A consistent setup has been simulated in VEINS.During both the trials and simulations, each performancemetric we measured is the result of an average of multipleexperiments. The ARP probe was disabled by manuallyinserting the addresses of the devices in their respectiveARP tables. Due to limited space, the most meaningfulresults for each distance and MCS will be shown while therest will be described within the text.With regards to the urban scenario, Fig. 8 shows thecommunication throughput that can be sustained by theOBU as a function of distance between RSU and OBU,for each MCSs. For lower modulations (BPSK, QPSK) andall distances, the same trend and performance were seenas in the calibration process (Fig. 5). Again, the differencein the performance between the transceivers, not observedwithin VEINS, is due to the different drivers used. Increasingthe MCS and the distance separating the devices, eventhough the median values remain similar to what shownin Fig. 5, the introduced multipath distortion leads to alarger number of outliers. For 16-QAM / and / , it wasseen that as the distance is increased, the SINR drop startsbeing observed within VEINS for the LP configuration (e.g.110 m) having a different behavior compared to the trials.Finally, for 64-QAM / and / , the signal received from theLP transceiver within VEINS is significantly attenuated. Theperformance degradation is about 1 Mbps compared to thecalibration scenario (50 m) reaching up to 3 Mbps at 110 m.Fig. 9 shows the communication throughput measuredin the case of the suburban scenario, for different MCSs.For distances of 30 m and 60 m, we observe that the resultsfollow the same trend as in the urban case (see Fig. 8).In particular, despite the RSU being hidden after 50 m due to the road slope, the overall communication throughputwas not severely impacted. For greater distances and theHP scenario, VEINS behaves slightly worse compared tothe actual device. However, for the LP scenario, the actualdevice achieved less throughput compared to the simulatedresult. Especially for 64-QAM / and / and a distance of200 m, our LP transceiver achieved zero throughput duringthe trials whereas the VEINS result is around 3 Mbps and1.2 Mbps respectively. This is due to the BER calculationwithin VEINS that is approximated based on a Gaussianerror function not exactly reflecting the reality.Fig. 10 refers to the rural scenario. Again, for lower modu-lation schemes (BPSK, QPSK), the same trend was observedas before. For higher MCS, VEINS again exhibits a sharpperformance degradation when the distance increases. Thisis clearer at 64-QAM / and / where VEINS outperformsthe trial performance at 550 m as expected from the trendobserved. However, it is significantly worse at 700 m.V. C ONCLUSIONS , L
ESSONS L EARNED AND F UTURE W ORK
In this paper, we described the process to follow in orderto calibrate a full-stack network simulator for vehicularcommunications. In our study, we considered VEINS as thenetwork simulator of interest. At first, we introduced aninitial calibration process where an “ideal-like” scenario wasevaluated. Two different transceivers were used to demon-strate the difference emanating from dissimilar hardwareand how it can impact the experimental results. Specifically,we focused on result trends and identified the reasons thatthey exist.To that extent, a trend in the performance was shown forboth the real and the simulated scenario. For some MCSs(e.g. BPSK / , QPSK / , etc.) the network throughput isalmost identical whereas for the others there is a slightdeviation of up to ~1 Mbps (e.g. 64-QAM / , etc.). Thedifferent drivers and hardware introduced a dissimilaritybetween the HP and the LP transceivers, something notobserved within VEINS as the differences in the hardwarecannot be easily simulated. A similar trend was seen in thejitter performance evaluation as well. The huge differencebetween the absolute values due to the Linux schedulingalgorithm serves as a proof of capacity that the direct com-parison between simulated and actual trial results shouldbe avoided.After the initial calibration, VEINS was further fine-tunedto achieve high-fidelity results for more complex scenarios.Three different vehicular environment scenarios were eval-uated with respect to the network throughput. The trendsidentified, showed that for the HP transceiver VEINS hasa sharper performance degradation when the distance isincreased whereas, for the LP one, it behaves smoother. Asdiscussed, some of the simulation parameters are basedon various assumptions and therefore this difference inthe performance is introduced. Finally, we concluded thatthe proposed tuning process allows VEINS to deliver high-fidelity simulation results of IEEE 802.11p/DSRC links, ig. 8. Values of throughput as a function of the distance between RSU and OBU. Results refer to an urban scenario. Each quartet represents theresults for a single position with the order (from left to right): 1) VEINS-HP, 2) Trials-HP, 3) VEINS-LP, 4) Trials-LP.Fig. 9. Values of throughput as a function of the distance between RSU and OBU. Results refer to a suburban scenario. Each quartet follows the sameorder as in Fig. 8 . Fig. 10. Values of throughput as a function of the distance between RSU and OBU. Results refer to a rural scenario. Each boxplot pair is: 1) VEINS-HP,2) Trials-HP. which will be pivotal in city-scale simulation scenarios –scenarios impossible to be calibrated otherwise.R
EFERENCES[1] J. Levinson et al. , “Towards Fully Autonomous Driving: Systems andAlgorithms,” in
IEEE Intel. Veh. Symp. IV , Jun. 2011, pp. 163–168.[2] P. Demestichas et al. , “Intelligent 5G Networks: Managing 5G Wire-less/Mobile Broadband,”
IEEE Veh. Technol. Mag. , vol. 10, no. 3, pp.41–50, Sep. 2015.[3] A. Tassi et al. , “Modeling and Design of Millimeter-Wave Networksfor Highway Vehicular Communication,”
IEEE Trans. Veh. Technol. ,Aug. 2017.[4] C. Han, M. Dianati, and M. Nekovee, “Effective DecentralisedSegmentation-based Scheme for Broadcast in Large-scale Dense
VANETs,” in
IEEE WCNC 2016 , Apr. 2016, pp. 1–6.[5] C. Sommer, O. K. Tonguz, and F. Dressler, “Adaptive Beaconing forDelay-Sensitive and Congestion-aware Traffic Information Systems,”in
Proc. IEEE VNC 2015 , Dec. 2010, pp. 1–8.[6] F. A. Teixeira et al. , “Vehicular Networks Using the IEEE 802.11pStandard: An Experimental Analysis,”
Vehicular Communications ,vol. 1, no. 2, pp. 91 – 96, 2014.[7] M. Boban, J. Barros, and O. Tonguz, “Geometry-Based Vehicle-to-Vehicle Channel Modeling for Large-Scale Simulation,”
IEEE Trans.Vehic. Techn. , vol. 63, no. 9, pp. 4146–4164, Nov. 2014.[8] C. Sommer, R. German, and F. Dressler, “Bidirectionally CoupledNetwork and Road Traffic Simulation for Improved IVC Analysis,”
IEEE Trans. Mobile Comput. , vol. 10, no. 1, pp. 3–15, Jan. 2011.[9] “Omnet++ Discrete Event Simulator.” [Online]. Available: https://omnetpp.org[10] S. A. M. Ahmed, S. H. S. Ariffin, and N. Fisal, “Overview of WirelessAccess in Vehicular Environment (WAVE) Protocols and Standards,”
Indian Journal of Science and Technology , vol. 6, no. 7, 2013.[11] E. Belyaev et al. , “The Use of Automotive Radars in Video-BasedOvertaking Assistance Applications,”
IEEE Trans. Intell. Transp. Syst ,vol. 14, no. 3, pp. 1035–1042, Sep. 2013. [12] S. Zhu et al. , “Probability Distribution of Rician K-Factor in Urban,Suburban and Rural Areas Using Real-World Captured Data,”
IEEETransactions on Antennas and Propagation , vol. 62, no. 7, pp. 3835–3839, Jul. 2014.[13] R. G. Sargent, “An Introductory Tutorial on Verification and Validationof Simulation Models,” in ,Dec. 2015, pp. 1729–1740.[14] F. Klingler, F. Dressler, and C. Sommer, “Ieee 802.11p Unicast Consid-ered Harmful,” in
IEEE VNC 2015 , Dec. 2015, pp. 76–83.[15] P. J. Roache,
Verification and Validation in Computational Science andEngineering . Hermosa Albuquerque, NM, 1998, vol. 895.[16] “Mikrotik RB433 Data Sheet.” [Online]. Available:https://mikrotik.com/product/RB433[17] “Mikrotik R52H Data Sheet.” [Online]. Available:https://mikrotik.com/product/R52H [18] “Mikrotik R5SHPn Data Sheet.” [Online]. Available:https://routerboard.com/R5SHPn[19] A. Abdelgader and W. Lenan, “The Physical Layer of the IEEE 802.11pWAVE Communication Standard: The Specifications and Challenges,”in
Proc. of WCECS 2014 , vol. 2, Oct. 2014.[20] R. Reinders et al. , “Contention Window Analysis for Beaconing inVANETs,” in , Jul. 2011, pp. 1481–1487.[21] H. Schulzrinne et al. , “RTP: A Transport Protocol for Real-TimeApplications,” United States, 2003.[22] A. Paier, D. Faetani, and C. F. MecklenbrÃd’uker, “PerformanceEvaluation of IEEE 802.11p Physical Layer Infrastructure-to-VehicleReal-World Measurements,” in , Nov. 2010, pp. 1–5. [23] O. Renaudin et al. , “Wideband MIMO Car-to-Car Radio ChannelMeasurements at 5.3 GHz,” in2008 IEEE 68th Vehicular TechnologyConference