Reproducibility Report for the Paper: Modeling of Request Cloning in Cloud Server Systems using Processor Sharing
RReproducibility Report for the Paper:
Modeling of Request Cloning in Cloud ServerSystems using Processor Sharing
Alessandro [email protected] 12, 2020
Abstract
The authors have uploaded their artifact on Zenodo, which ensures a long-term retention of the artifact.The code is suitably documented, and some examples are given. A minimalistic overall description of theengine is provided. The artifact allows to setup the environment quite quickly, and the dependencies arewell documented. The process to regenerate data for the figures in the paper completes, and all results arereproducible.This paper can thus receive the
Artifacts Available badge and the
Artifacts Evaluated—Functional . Giventhe high quality of the artifact, also the
Artifacts Evaluated—Reusable badge can be assigned.
The paper
Modeling of Request Cloning in Cloud Server Systems using Processor Sharing by Tommi Nylander,Johan Ruuskanen, Karl-Erik Årzén, and Martina Maggio [1] presents theoretical results related to theperformance of client/server applications (in the context of cloud-based deploys) when request cloning isexploited to reduce the time to serve the requests.The authors present a G/G/1 model which allows to represent a system of servers using cloning toserve the requests as if it were a single server, basing their approach on the idea that the service must besynchronized.The authors validate their theoretical results by means of a discrete event simulation model, writtenin python, which is the focus of this report. The authors have also implemented the sequential runtimecore for the model, thus making the artifact self-contained, which could be a benefit for some end users,provided they don’t have strong performance requirements.
In this section we provide Figures taken from the original paper. The authors have given permission touse their original results in this report.
The authors have provided a link to a permanent repository on Zenodo ( http://doi.org/10.5281/zenodo.3611398 ), making it permanently available (
Artifacts Available badge ). This reviewer encour-ages the authors to place this link also in the main paper, to increase the dissemination of their results.The amount of dependencies required to run the artifacts is extremely reduced. Simulations can berun by relying solely on python. There is a subsequent post-processing phase, required to analyze theresults of the simulations to prepare the datasets to replot the figures which requires matlab. This laststep, anyhow, is only related to the reproducibility phase, while the simulation model is self-contained ina reduced number of scripts.The authors have provided a docker environment to run the simulations, which installs all the requiredpython dependencies, which has been gratefully appreciated. a r X i v : . [ c s . PF ] F e b .2 Quality of the artifact The artifact is relevant to the associated paper, as it allows to re-generate all the data and all the figures, itis well documented (from the point of view of reproducibility), and it is easy to use thanks to the providedsupporting environment. No component to re-run the simulations and to post-process the data is lacking(
Artifacts Evaluated—Functional ).The authors have also provided the results of their simulation runs in the artifact. They propose to relyon this data to regenerate the plots, in case no sufficient computational power is available. I can confirmthat by relying on this data, the post-processing and plotting modules do work, and the figures are 1:1 withrespect to those in the paper. In the artifact, the authors also provide the post-processed data, thus making itpossible to only replot the figures. Matlab scripts are written so that if these post-processed files are found,they are not overwritten. For this reproducibility assessment, this reviewer has rerun all simulations fromscratch, and has cleared out all post-processed results before launching the post-processing activity.The authors have provided a minimalistic documentation to the artifact, consisting in an html file witha high-level description of the simulation model. The authors have also provided a description of thetimeline of a cloned request in their software. This documentation does not really allow to repurpose theartifact. Nevertheless, there are some examples which allow to get an insight on how to repurpose theartifact. The model/kernel documentation is good, making it easier to understand. Overall, this reviewerthinks that the quality of the artifact is good (
Artifacts Evaluated—Reusable ). Replication of the experiments has been carried out on multiple environments. Python simulations havebeen run on a cluster of 4 parallel high-end machines, all running Debian 9, in the provided docker con-tainer. Post processing and plotting have been run on an Arch Linux machine, using matlab R2019b rev 3.The artifact is easy to run, and all data required to support the reproducibility is provided (
ArtifactsEvaluated—Functional ). It took a more than one week to run all the simulations using ∼ cores.Initially, two of the post-processing matlab scripts were failing. This was preventing to generate the datawhich are later required by the tikz scripts in the main latex file to generate the plots.The authors have been informed via email on Feb 02, 2020 of this issue, receiving all the data from sim-ulations which were re-generated by this reviewer. The authors have replied on Feb 04, 2020, by acknowl-edging the issues and uploading a revised version of the artifact which has fixed all the aforementionedissues.In this section, I provide a short description of my reproducibility results, and how they were obtained.There is a total of 11 Figures and 2 Tables in the original paper. Tab. 1 in the paper shows probabilities from a distribution, which does not require to be replicated.Similarly,
Tab. 2 presents data coming from a theoretical analysis which does not require to be replicated.
Fig. 1 in the paper is a drawing and needs not to be reproduced.
Fig. 2 in the paper is a graphicalrepresentation of multiple equations, thus needs not to be reproduced.
Fig. 3 in the paper corresponds to Figure 1 in this report. The data for this figure is generated by runningthe sim_gg1_3dist.py wrapper script, which spawns a couple of simulation instances. Data is postprocessed by the analyze_gg1_example.m matlab script, which completes successfully. The results arevisually and substantially comparable.(a) Original. . . . . Response time (s) E m p i r i c a l C D F Three Servers with CloningEquivalent Single Server (b) Reproduced.
Figure 1: Empirical response time CDFs for the Example with Heterogeneous Servers. Data retrieved through20 repeated simulations of requests each. The 95% confidence intervals lie within the lines. Fig. 4 in the paper corresponds is a visual representation of a theoretical analysis, which thus does notrequire reproducibility.
Fig. 5 in the paper is a drawing, which does not require reproducibility. ig. 6 in the paper corresponds to Figure 2 in this report. The data for this figure is generated byrunning the sim_optimal_clone-ps.py wrapper script, which spawns 3110 simulation instances. Datais post processed by the analyze_opt_clone.m matlab script, which completes successfully. The resultsare slightly different from the original ones in the paper, but the trends are comparable and do not divergetoo much. I emphasize that this difference is also related to the reduced number of simulations which havebeen run for this particular figure, due to a clerical error in the first version of the artifact.(a) Original. . . . E [ T ] ( s ) Arrival rate / server (1/s) C l o n i n g f a c t o r c f E [ T ] opt (theory) E [ T ] opt (sim) c opt f (theory) c opt f (sim) (b) Reproduced. Figure 2: Clone-to-All: Comparison of theoretical values with 95% confidence intervals for the simulationresults.
Fig. 7 in the paper corresponds to Figure 3 in this report. The data for this figure is generated byrunning the sim_codesigns_icpe.py wrapper script, which spawns 880 simulation instances. Data ispost processed by the analyze_codesigns.m matlab script, which completes successfully. The resultsare visually and substantially comparable.(a) Original. . .
38 0 .
52 0 .
62 0 . C l o n e f a c t o r c f . .
38 0 .
52 0 .
62 0 . Arrival rate λ / server (1/s) E [ T ] ( s ) c-JSQ-d (sim) theoryc-R-d (sim) theory (b) Reproduced. Figure 3: Co-designs: Comparison of theoretical values with 95% confidence intervals for the simulationresults. The legend applies to both figures.
Fig. 8(a) in the paper corresponds to Figure 4 in this report. The data for this figure is generatedby running the sim_randomized_arrival_delays.py wrapper script, which spawns 1000 simulationinstances. Data is post processed by the analyze_randomized_arrival_delays.m matlab script, whichcompletes successfully. The results are visually and substantially comparable, except for the point at x =0 . , but the deviation can be deemed as negligible. Fig. 8(b) in the paper corresponds to Figure 5 in this report. The data for this figure is generated byrunning the sim_randomized_cancellation_delays.py wrapper script, which spawns 1000 simula-tion instances. Data is post processed by the analyze_randomized_cancellation_delays.m matlabscript, which completes successfully. The results are visually and substantially comparable, except for the a) Original. .
01 0 .
02 0 .
05 0 . . . E [ a ] E [ X ] N o r m a l i z e d E [ T ] Upper boundDelaysNo delays (b) Reproduced.
Figure 4: Arrival delays only. point at x = 0 . , but the deviation can be deemed as negligible.(a) Original. .
01 0 .
02 0 .
05 0 . . . . . E [ c ] E [ X ] N o r m a l i z e d E [ T ] (b) Reproduced. Figure 5: Cancellation delays only.
Fig. 8(c) in the paper corresponds to Figure 6 in this report. The data for this figure is generated byrunning the sim_randomized_combined_delays.py wrapper script, which spawns 1000 simulation in-stances. Data is post processed by the analyze_randomized_combined_delays.m matlab script, whichcompletes successfully, except for the error bar at x = 0 . , but the deviation can be deemed as negligibleand the trend might look overall better than the original plot in the paper. Fig. 9 in the paper corresponds to Figure 7 in this report. The data for this figure is generated by run-ning the sim_randomized_sync_vs_nonsync_icpe.py wrapper script, which spawns 4000 simulationinstances. Data is post processed by the analyze_randomized_scenarios.m matlab script, which com-pletes successfully, except for the points at x = 0 . , but the deviation can be deemed as negligible and thetrend might look overall better than the original plot in the paper.Overall, the plots which have been generated are consistent with the original publication. References [1] Tommi Nylander, Johan Ruuskanen, Karl-Erik Årzén, and Martina Maggio, “Modeling of RequestCloning in Cloud Server Systems using Processor Sharing," In
Proceedings of the 11 th ACM/SPEC Inter-national Conference on Performance Engineering , April 2020. a) Original. .
01 0 .
02 0 .
05 0 . . . . . E [ a + c ] E [ X ] N o r m a l i z e d E [ T ] (b) Reproduced. Figure 6: Both delays present. (a) Original. . . . . . . . . . . . . E [ (cid:15) ] a-JSQ-da-R-d . . . . . . . . . . . . . . . Utilization ρ N o r m a l i z e d E [ T ] a-JSQ-da-R-dc- (cid:96) -d (b) Reproduced. Figure 7: Simulations comparing a- (cid:96) -d to c- (cid:96) -d for random and JSQ. The normalization of E [ T ] is performedsuch that each value is divided by the value for the c- (cid:96)(cid:96)