Assessing virtualization effects in simulations of distributed robotics
aa r X i v : . [ c s . R O ] D ec Assessing virtualization effects in simulations of distributed robotics
Sekou L. Remy
Abstract — In this work, our aim is to identify whether thechoice of virtualization strategy influences the performance ofsimulations in robotics. Performance is quantified in the errorbetween a reference trajectory and the actual trajectory forthe ball moving along the surface of a smooth plate. The two-sample Kolmogorov-Smirnov test is used to assess significanceof variations in performance under the different experimentalsettings. Our results show that the selection of virtualizationtechnology does have a significant effect on simulation, andmoreover this effect can be amplified by the use of someoperating systems. While these results are a strong causefor caution, they also provide reason for optimism for thoseconsidering “repeatable robotics research” using virtualization.
I. INTRODUCTIONRealistic computer based simulation has been an effectivetool – in both academia [1], [2] and industry [3], [4] – toexplore content that is difficult to recreate in the physicalworld (safety, distance), challenging to understand becauseof scale, or even impossible to occur under easily repro-ducible conditions. Simulation environments themselves cansometimes also be difficult to install, configure, and deploy,and if there is variability in any of these phases then it ispossible for the results that are obtained to be influenced ina unexpected way. Simulation technology can benefit fromadvances in computing and information technology, but onlyif these advances can be appropriately incorporated into theworkflow.In this work, we evaluate the impact of widely usedabstractions of computing hardware and subsystems whenrunning simulations of physically based robotics simulationenvironments. This type of simulation is important as itincludes distributed robots and also emulates the networksthat facilitate their control. In this simulation setting, onboardsensors are collecting data on simulated robots which thentransmit this data over an emulated network to another nodewhich calculates the desired control values that the simulatedactuators execute. This entire simulation system is deployedleveraging virtualization . Containers and Virtual Machines(VMs) are the two types of virtualization considered in thiswork. They are widely used to deploy software, and wepresent results which begin to characterize the impact of theiruse in robotics. This work is critical because of the numberof users of the technologies involved. With over 500K pullsof ROS Containers and use of VMs since ROS Fuerte in2012, whether in the cloud or on the desktop, virtualization’spenetration is deep and the impact on experimental resultsmust be fully understood.
Sekou L. Remy is with IBM Research Africa, Nairobi, Kenya [email protected]
We wish to test the null hypothesis H : the observedperformance of a controller is consistent in all deploymentenvironments that a physics-based simulation is performed.Performance is quantified by performing a distributed controltask using an emulated network. This investigation is impor-tant as the research community is embracing virtualization,so we need to ensure that the validity of experimentalfindings generated is not compromised by inconsistenciesthat may arise from the use of virtualization approaches.The main technical contributions of this paper are theimplementation of the robotic simulation environment in anew network emulation framework, the deployment of thesimulation, the experimental assessment of the performancein these simulation, and the statistical evaluation of thisperformance. II. BACKGROUNDHardware virtualization is a software abstraction thatcreates a working representation of a computer system’shardware [5], [6]. Initially developed to correct shortcomingsin computer architectures of the time, one use case for thetechnology was to provide extended “machines” so that userprograms could be executed in the presence of dual statehardware which had a priviledged and non-priviledged mode.Another use case was to permit programs written for specialpurpose hardware to be compiled and executed on generalpurpose ones instead.While a great deal has changed since the 1960’s whenit was first developed, virtualization is still very present inmodern computing. In fact, virtualization is in the midstof a significant resurgence fueled by the architectural ex-tensions which major microprocessor manufacturers haveprovided[7], as well as its integral role in Cloud Computing.Virtual machines (VMs) and containers are two competingvirtualization approaches which have been developed. Virtualmachines differ from containerized virtualization in that eachapplication which runs in a VM is executed on its ownoperating system (i.e. it has its own kernel) and executeson its own virtualized copy of hardware which is providedby a hypervisor. This hypervisor software thus providesisolation for virtual machines running on physical hosts [8].In contrast, containerization does not employ the use of a hy-pervisor, and instead uses a container engine which translatesthe instructions from the guest machine to the host machine’skernel. If there are multiple containers operating on the samehost, they share the same kernel. Isolation between processesis provided through the use of namespaces that the Linuxoperating system provides [9]. These namespaces enable lightweight virtualization, as devices can now effectively behared without collision of the processes and applicationsperforming work at the same time.Virtualization makes computing resources easier to access,and easier to share; And these are two of the challenges thatthose doing research and development in robotics experience.As an example, in the words of the Open Source RoboticsFoundation, curators of the Gazebo simulation environment:With the advancements and standardization of soft-ware containers, roboticists are primed to acquirea host of improved developer tooling for build-ing and shipping software. To help alleviate thegrowing pains and technical challenges of adoptingnew practices, we have focused on providing anofficial resource for using Gazebo with these newtechnologies. The previous quote applies to containerization, howeversince 2012 Gazebo has also been included on preconfiguredVMs, and released with ROS distributions. Moreover, overthe past two years, key robotics conferences like ICRA haveheld workshops and tutorials teaching researchers how to usevirtualization , and papers are frequently shared with deploy-able environments for others to evaluate . Virtualization isclearly addressing a need, and as it is being increasinglydeployed its influence on the work done in the field isgrowing.The goal of this work is to identify whether the choiceof virtualization strategy influences the performance of sim-ulations in robotics, thus influencing the experiments thatare done with virtualization. We are specifically focus onsimulations which incorporate realistic networking settings.The ability to identify the effect of virtualization (if any),would be a precursor to developing models which couldmap research findings from one experimental setting andtransform them to findings from another. The developmentof such models would enable simulations deployed in virtu-alized environments to provide more useful virtual testbedsfor science, instead of being relegated to system integrationframeworks or for use as one off deployment platforms.Many existing multi-robot simulators like Gazebo, We-bots, and MORSE do not provide advanced communicationmodels [10], so researchers have developed extensions tointegrate network simulations and emulations with theirrobot simulations [11], [12]. In our approach we choose mininet based emulation because it readily permits existingrobot control codebases to be used without modification.Extensions to mininet also permit mobility. Mininet is asystem for rapidly prototyping large networks on a singlecomputer which supports running unmodified code on em-ulated machines [13]. Mininet uses containers and allowsprocesses to be placed in namespaces with their own prop-erties like IP addresses. Developed in the Software DefinedNetworking community, mininet makes use of OpenVSwitchfor switching and uses NetEm, a part core Linux subsystem, https://hub.docker.com/ /gazebo/ http://the.swarming.buzz/ICRA2017/virtual-machine-tutorial/ http://d3s.mff.cuni.cz/software/ros sCPS testbed/ TABLE I: Virtualization centric layers for robot simulations
L5 Robot simulationsL4 MininetL3 VirtualizationL2 Host OSL1 Host machine
TABLE II: Scenarios a-i are the 9 possible experimentalconfigurations considered in this work. These scenarios pro-vide different combinations of hardware, operating systems,and virtualization for comparison of the performance of therobotic simulation.
L1 L2 L3Docker Native VirtualBoxi686 Ubuntu a b cIntel i7 OSX d e fARM64 QEMU Ubuntu g h i to emulate the links between virtual hosts (aka the processesrunning in namespaces).Conceptually, there are 5 distinct layers that are essentialto the evaluations in this work. The lowest layer, Layer 1is the hardware. This layer provides the resource that theoperating system (Layer 2) will run upon. It is possible forthis hardware to be provided through virtualization sincewe are focused on the effect of virtualization on simulationwhich is in a higher layer. Layer 2 provides the host operatingsystem that the virtualization environment will be executed.The operating system needs to be able to support virtualiza-tion either by permitting a program to run, or by providingaccess to its subsystem functionality. Layer 3 captures thetwo types of virtualization, VMs and containers. Layers 4and 5 contain mininet and the network simulation codebasethat we will use to characterize the effect of virtualization.Since Layer 3 could be influenced by the layers that enable it,any experiments will have to consider the cumulative effectsof the combined substrate.III. EXPERIMENTAL SETUPThe experimental infrastructure for this work is capturedin the configurations presented in Table II. In this tablenine combinations of base hardware, operating systems, andvirtualization options. These configurations of infrastructurerepresent those that can readily be found in the roboticscommunity.
A. Distributed Robotic Control1) Robot:
The robot, or plant , selected for this work,the ball on plate , is widely studied in control theory andit is an example of a dynamic system with an unstableequilibrium point. In this section we describe the system,first by introducing the ball on beam system, and then wecontinue to introduce the feedback controller we use.For this plant, the position x of the ball is controlled bychanging the angle Y of the beam which causes the ball toig. 1: Control of the ball on plate example. The input to thesystem, R ( s ) , defines the target ball trajectory. E ( s ) capturesthe error from that target and this data is used as input tothe controller. The controller then generates the control inputto the plant, U ( s ) and uses the switch to transmit this datato the robot (See Fig. 2). The output of the plant, Y ( s ) istransmitted back through the switch to the controller. In thisfigure, τ ca is the delay between controller and actuator, and τ sc is the delay between sensor and controller.roll along its surface. By controlling the orientation of thebeam, the position of the ball is thus indirectly changed.To maintain any target ball position, the controller mustactively operate since the ball’s position is affected by gravityat all times. The ball and plate system is modeled as anextension of the ball on beam to two dimensions. Thisextension is rooted in the assumption that the motion ineach dimension of the plane defined by the plate can becontrolled independently. That assumption is valid for smallEuler angles under the linearized dynamics.Consequently, controlling the position of the ball involvesdetermining the input for two actuators that adjust the Eulerangles of the plate’s rotational axes. In this work the stateof the system is partially observable, and only the locationof the ball on the plate, and the roll and pitch angles ofthe plate are known. The reader is referred to [14] for thecomplete derivation of the equations of motion used to modelthe system. In this work, the ball on plate system is modeledas two independent ball on beam systems, so equationspresented for only one dimension of the system. A ( s ) U ( s ) = 1 s ( b s + b ) (1)Using (1) the relationship between the input voltage to themotor ( U ) and the angle of the motor shaft ( A ) can be cal-culated. In this work, the published values of b = 0 . and b = 0 . [15] are applied. These values account forthe physical properties of the system. They incorporate thetotal load inertia, the total load friction, the gear ratio for themotor, and assume negligible armature inductance.A linear mapping (2) is assumed between the Euler angleof the beam ( Y ) and the angle of the motor shaft ( A ).This linear relationship is only valid for small angles (valuesbetween − and degrees), but it results in the simplifiedequation for (3), which is the transfer function for G P ( s ) -the effect of control input on the beam orientation. Y ( s ) A ( s ) = 116 (2) G P ( s ) = Y ( s ) U ( s ) = 116 s ( b s + b ) (3) Fig. 2: An OpenFlow controller manages the transfer of IPpackets flowing through the network switch. This topologyforms layers 4 & 5 of the model presented in Tab. I. This isthe communication substrate that permits the robot controllerto publish data to, and subscribe to data from the robot.The transfer function between the Euler angle and the ballposition is given by (4). This transfer function was obtainedby linearizing the equations of motion for the system aboutan Euler angle and an angular rate of zero. This modelassumes that the ball rolls without slip, and with negligiblefriction along the surface. G P ( s ) = X ( s ) Y ( s ) = − s (4)Ideally, to better simulate the plant, G P ( s ) should update ata much faster rate than G P ( s ) , however in this work theyare updated at the same rate and at the same time step. Assuch, the effective system plant, G P ( s ) = G P ( s ) ∗ G P ( s ) .
2) Robot controllers:
For each axis of the system, theposition of the ball, and the Euler angle of the plate wereeach controlled with one PD controller. This results in fourcontrollers of the form presented in (5). G C ( s ) = K d s + K p ss (5)
3) Networking:
Communication between the controller and the plant was implemented using the Robot OperatingSystem (ROS) [16]. ROS provides a structured communi-cations layer above the host operating systems, and is anopen-source project widely used in the robotics community[17] The controller and the plant were run on hosts sta and sta respectively, while the rosmaster node was run ona third host, h . The ROS publish and subscribe paradigm isused to connect these nodes, and leveraged TCP to providemessage delivery.The three hosts and the network that connected them wereemulated using mininetAPI. As shown in Fig. 2, these hostswere connected to the same switch, and the attributes foreach link were: delay = ms; jitter = ms; loss rate = %;bandwidth = Mb. The reader should note that this resultsin a ms round-trip communication time between each pairof hosts. ) Instrumentation: To characterize performance of thecontrol task, a log is generated by the controller processwhich captures the control action and the subsequent plantstate. Here the state is defined as the position of the ball andthe orientation of the plate. Each of these properties (stateand control) are timestamped by the controller application.Concurrently, the tcpdump utility is used to log all networktraffic on each of the virtual hosts. This utility permitsintrospection of the lower level properties of the networktraffic. Since all the virtual hosts share the same system clockof the underlying OS, their TCP traces are synchronized andthere is complete visibility of the networking traffic.
B. Experiments
The experiments were run in three settings: a ToshibaSatellite computer, an Apple Macbook Pro, and on a cloud-hosted virtual machine (QEMU). To collect data, vitualiza-tion from either a docker container or a VirtualBox VM wasinstalled, and code from a repository was checked out. Thiscode contained python source that configured the mininetAPIand launches the three ROS nodes. There were two referencetrajectories used in the control task, one for the X- andY- axis respectively. Both trajectories were periodic, with T = 100 s . The error between the reference and the actualtrajectory for the ball along each axis is the core performancemetric for this work. This ‘virtual testbed’ was created to runfor four contiguous seconds windows creating data inseparate folders.So the aim will be to uncover whether there is a sta-tistically significant difference in the errors observed whenperforming control under the various conditions. Significancewill be evaluated using the two-sample Kolmogorov-Smirnov(KS) test [18]. In this test, the largest difference between twoconsidered sample distributions are assessed to determineif the two samples could have been drawn from the samedistributions (See Fig. 3). The smaller the difference, thehigher the likelihood that the samples are from the samedistribution.Note: Since Docker does not run on the i chipset, andmininet cannot run on OSX, data cannot be collected forScenarios -a and -e; Data will only be collected for theremaining seven scenarios in Tab. II.IV. RESULTSIf the null hypoothesis is trues, then there is no statisticallysignificant difference in performance observed in the variousexperimental settings considered. This would mean that nomatter what virtualization was used to enable simulation,similar results would be observed, and thus truly repeatablerobotics simulations would be performed.Fig. 4 presents the target and actual trajectory for a con-troller running using Mininet Python API on an Ubuntu hostlaptop (Scenario-b). These figures show that control is beingperformed, however there is error. This error is averaged overthe duration of the period (100 s ) to derive a measure of theperformance over the course of the experiement. In Fig. 5data from Scenario-b are presented as a histogram of the error Fig. 3: Graph depicting an example of the Kolmogorov-Smirnov test. X - po s i t i on ( m ) ActualReference
Fig. 4: One period of the position of the ball when controlledwithout virtualization (Scenario-b from Tab. II).with error data from Scenario-h (with one of the four runsfrom each scenario). Visually, the histograms appear similar,however results from the application of the two-sample KStest will assess this formally. Fig. 6 presents two pairs of thecumulative distribution functions highlighting the KS test inaction.We first consider the consistency in each of the scenarios.Since each scenario was run four times, there are 6 pairsof comparisons for each scenario set. Tab. III shows theresults of the KS test for each of the scenarios from Tab.II. This table shows that there were no statistically signifi-cant differences in performance of controllers when runningnatively on Ubuntu on the Toshiba laptop, nor when runningon the QEMU VM (Scenarios -b and -h respectively). Evenwhen VirtualBox was run on the QEMU VM, there was nostatistically significant difference in performance observed(Scenario-i). In each of the other scenarios at least onepairwise assessment of the null hypothesis was rejected. Asa whole, this data show that there are indeed variationsin performance before considering the relative effect of .00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40Error(m)024681012 C oun t L1 Comparison of error experienced when native simulation running on ARM64 vs QEMU b:i686h:QEMU
Fig. 5: Histograms of presenting the performance errorsobserved in a pair of scenarios considered in this work. Eachof the histograms are asymmetric, and have a non-zero meanTABLE III: Pairwise comparison of the 4 samples of eachscenario. Comparisons are evaluated via the Kolmogorov-Smirnov test and significance assesed at p < . . The datashow that some scenarios have more consistency than others.When mininet is running with either VirtualBox or Dockerthere is at least one comparison where the null hypothesis isrejected. N = 6 L1 L2 L3 Rejectedi686 Ubuntu - 0i686 Ubuntu VirtualBox 3Intel i7 OSX Docker 3Intel i7 OSX VirtualBox 3QEMU Ubuntu Docker 1QEMU Ubuntu - 0QEMU Ubuntu VirtualBox 0 virtualization.Next we consider the pairwise evaluation of data fromdifferent scenarios (Tab. IV). In the experimental controlsamples (where there is no L3 virtualization), the nullhypothesis was not rejected in any of the 16 comparisons.When comparing the errors observed when running on theexperiments on VirtualBox to those observed when runningon Docker however, there were statistically significant differ-ences. On OSX, in more than half of the comparisons the nullhypothesis was rejected. When running on Ubuntu, the nullhypothesis was only rejected in one of the 16 comparisons.This suggests that experiments run in the two virtualizationenvironments on Ubuntu are more consistent with each otherthan experiments run on OSX.Finally, we compare the consistency of the four exper-iments using L3 virtualization with each of the two ex-periments where the technology was not used (the controlscenarios). As presented in Tab. V, in both cases OSX is lessconsistent with the experimental control than Ubuntu. Also,in both cases, Docker is less consistent with the control than TABLE IV: Pairwise comparison of the 4 samples of eachscenario. Comparisons are evaluated via the Kolmogorov-Smirnov test and significance assesed at p < . . The datashow that significant differences between Docker and Virtu-alBox do occur, and suggests that they may be dependent onthe infrastructure used. N = 16 . Scenario-1 Scenario-2 H0 RejectedExp. control i686/Ubuntu ARM64/QEMU 0OSX Docker VirtualBox 9Ubuntu Docker VirtualBox 1
TABLE V: Pairwise comparison of the 4 samples of eachscenario to 4 samples from the experimental controls. Com-parisons are evaluated via the Kolmogorov-Smirnov test andsignificance assesed at p < . . The data show that thereare significant differences that commonly occur based on therun and on the infrastructure used. Also, full virtualizationprovides performance more consistent with hardware thanlightweight virtualization. N = 16 . (a) Control 1: Number of rejectionsOSX UbuntuVirtualBox 0 0Docker 10 2(b) Control 2: Number of rejectionsOSX UbuntuVirtualBox 1 0Docker 7 0 experiments performed with VirtualBox.The purpose of this work was to explore if there are dif-ferences that would be observed in the simulations executedthrough the use of different virtualization strategies, and thedata shows this. Moreover, the data show that the differencesare conditioned on how virtualization is provided. What isnotable about these trends is that they are consistent overdifferent infrastructures since there are different implemen-tations of the virtualization tools for OSX and for Ubuntu.V. CONCLUSIONSThis work shows that there are quantifyable differencesin results for robot simulations when virtualization is usedto deploy them. This means that conclusions drawn fromsuch simulations are dependent on how and where theywere deployed. Our findings were acquired by implementingsimulated robot control over a mininet-emulated network,and then deploying the simulation in VirtualBox, Docker and“natively” in three different operating environments. The datawe present show that the performance of VirtualBox is moreconsistent with non-virtualized environments than Docker.Further, the data show that the variability in performance isdependent on the operating environment itself; e.g. compar-isons on OSX are generally less consistent than Ubuntu. Atpresent, proponents of repeatable robotics simulation must .00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40Error(m)0.00.20.40.60.81.0 CD F L1 Comparison of error experienced when native simulation running on ARM64 vs QEMU b:i686h:QEMU (a) Failed to reject CD F L3 Comparison of error experienced with Docker vs VirtualBox simulations running in OSX/ARM64 d:Dockerf:VirtualBox (b) Rejected
Fig. 6: Cumulative distribution functions for the performance errors observed in two pairs of scenarios considered in thiswork. In one comparison (comparing Docker and VirtualBox), the null hypothesis was rejected, i.e. there was enoughevidence to conclude that the two samples were not drawn from the same distribution. (See Tabs. III-V)be cautiously optimistic about both types of virtualization,and this work highlights the specific areas of concern.The findings in this work are based on experiments requir-ing low bandwidth communication between controller androbot. While the dynamics of the plant still resulted in ameaningful demonstration of control, future work will usehigher bandwidth devices like cameras and depth sensors.We will also extend analysis to include the effects of virtu-alization on the network packets that are used to transmitthe control/sensor data. Such extensions will provide therelevant insight, and data, to develop models that will betterframe simulation results generated with virtualization. Morebroadly, such models will also be key in compensating for theoperation of simulation testbeds in heterogeneous computingenvironments. R
EFERENCES[1] R. Gourdeau, “Object-oriented programming for robotic manipulatorsimulation,”
IEEE Robotics Automation Magazine , vol. 4, no. 3, pp.21–29, Sept 1997.[2] P. Corke,
Robotics, vision and control: fundamental algorithms inMATLAB . Springer, 2011, vol. 73.[3] O. Ma, K. Buhariwala, N. Roger, J. MacLean, and R. Carr, “Mdsfageneric development and simulation facility for flexible, complexrobotic systems,”
Robotica , vol. 15, no. 1, pp. 49–62, 1997.[4] M. A. Lerner, M. Ayalew, W. J. Peine, and C. P. Sundaram, “Doestraining on a virtual reality robotic simulator improve performanceon the da vinci R (cid:13) surgical system?” Journal of Endourology , vol. 24,no. 3, pp. 467–472, 2010.[5] R. P. Goldberg, “Survey of virtual machine research,”
Computer ,vol. 7, no. 6, pp. 34–45, 1974.[6] R. J. Creasy, “The origin of the vm/370 time-sharing system,”
IBMJournal of Research and Development , vol. 25, no. 5, pp. 483–490,1981.[7] K. Adams and O. Agesen, “A comparison of software and hardwaretechniques for x86 virtualization,”
ACM SIGOPS Operating SystemsReview , vol. 40, no. 5, pp. 2–13, 2006. [8] A. M. Joy, “Performance comparison between linux containers andvirtual machines,” in , March 2015, pp. 342–346.[9] E. W. Biederman and L. Networx, “Multiple instances of the globallinux namespaces,” in
Proceedings of the Linux Symposium , vol. 1,2006, pp. 101–112.[10] M. Kudelski, L. M. Gambardella, and G. A. Di Caro, “Robonetsim:An integrated framework for multi-robot and network simulation,”
Robotics and Autonomous Systems , vol. 61, no. 5, pp. 483–496, 2013.[11] W. Ye, R. T. Vaughan, G. S. Sukhatme, J. Heidemann, D. Estrin, andM. J. Mataric, “Evaluating control strategies for wireless-networkedrobots using an integrated robot and network simulation,” in
Roboticsand Automation, 2001. Proceedings 2001 ICRA. IEEE InternationalConference on , vol. 3. IEEE, 2001, pp. 2941–2947.[12] V. Matena, T. Bures, I. Gerostathopoulos, and P. Hnetynka, “Modelproblem and testbed for experiments with adaptation in smart cyber-physical systems,” in
Software Engineering for Adaptive and Self-Managing Systems (SEAMS), 2016 IEEE/ACM 11th InternationalSymposium on . IEEE, 2016, pp. 82–88.[13] B. Lantz, B. Heller, and N. McKeown, “A network in a laptop: rapidprototyping for software-defined networks,” in
Proceedings of the 9thACM SIGCOMM Workshop on Hot Topics in Networks . ACM, 2010,p. 19.[14] S. Awtar, C. Bernard, N. Boklund, A. Master, D. Ueda, and K. Craig,“Mechatronic design of a ball-on-plate balancing system,”
Mechatron-ics , vol. 12, no. 2, pp. 217–228, 2002.[15] R. Randolph, K. Choi, C. Lim, and M. Hunt, “Design and imple-mentation of a software simulation, animation, and real-time controlenvironment for a ball-and-beam hardware system,” 1998.[16] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs,R. Wheeler, and A. Y. Ng, “Ros: an open-source robot operatingsystem,” in
ICRA workshop on open source software , vol. 3, no. 3.2.Kobe, 2009, p. 5.[17] W. Curran, T. Thornton, B. Arvey, and W. D. Smart, “Evaluatingimpact in the ros ecosystem,” in
Robotics and Automation (ICRA),2015 IEEE International Conference on . IEEE, 2015, pp. 6213–6219.[18] S. A. Metchev and J. E. Grindlay, “A two-dimensional kolmogorov–smirnov test for crowded field source detection: Rosat sources in ngc6397,”