Gregory Ewing

Coverage of confidence intervals in sequential steady-state simulation☆

Abstract Stochastic discrete-event simulation has become one of the most-used tools for performance evaluation in science and engineering. But no innovation can replace the responsibility of simulators for obtaining credible results from their simulation experiments. In this paper we address the problem of the statistical correctness of simulation output data analysis, in the context of sequential steady-state stochastic simulation, conducted for studying long run behavior of stable systems. Such simulations are stopped as soon as the relative precision of estimates, defined as the relative half-width of confidence intervals at a specified confidence level, reaches the required level. We formulate basic rules for the proper experimental analysis of the coverage of steady-state interval estimators. Our main argument is that such an analysis should be done sequentially. The numerical results of our coverage analysis of the method of non-overlapping batch means and spectral analysis are presented, and compared with those obtained by the traditional, non-sequential approach. Two scenarios for stochastic simulation are considered: traditional sequential simulation on a single processor, and fast concurrent simulation based on multiple replications in parallel (MRIP), with multiple processors cooperating in the production of output data.

Experimental evaluation of confidence interval procedures in sequential steady-state simulation

Sequential analysis of simulation output is generally accepted as the most efficient way for securing representativeness of samples of collected observations. In this scenario a simulation experiment is stopped when the relative precision of estimates, defined as the relative width of confidence intervals at an assumed confidence level, reaches the required level. This paper deals with the statistical correctness of the methods proposed for estimating confidence intervals for mean values in sequential steady-state stochastic simulation. We formulate basic rules that should be followed in proper experimental analysis of coverage of different steady state interval estimators. Our main argument is that such analysis should be done sequentially. The numerical results of our preliminary coverage analysis of the method of Spectral Analysis (SA/HW) and Non overlapping Batch Means are presented, and compared with those obtained by traditional, non-sequential approaches.

AKAROA2: A Controller Of Discrete-Event Simulation Which Exploits The Distributed Computing Resources Of Networks.

This paper describes and summarises our research on enhancing the methodology of automated discrete-event simulation and its implementation in Akaroa2, a controller of such simulation studies. Akaroa2 addresses two major practical issues in the application of stochastic simulation in performance evaluation studies of complex dynamic systems: (i) accuracy of the final results; and (ii) the length of time required to achieve these results. (i) is addressed by running simulations sequentially, with on-line analysis of statistical errors until these reach an acceptably low level. For (ii), Akaroa2 launches multiple copies of a simulation program on networked processors, applying the Multiple Replications in Parallel (MRIP) scenario. In MRIP the processors run independent replications, generating statistically equivalent streams of simulation output data. These data are fed to a global data analyser responsible for analysis of the results and for stopping the simulation. We outline main design issues of Akaroa2, and detail some of the improvements and extensions to this tool over the last 10 years.

Transient Deletion And The Quality Of Sequential Steady-State Simulation

In discrete event steady-state simulation, deletion of data from the initial transient phase of the simulation is usually recommended in order to reduce the bias of the final estimates. Various heuristics and tests have been proposed to aid with this. The plummeting cost of simulation, combined with uncertainties about the overall reliability of the estimation of the transient period, suggest revisiting the notion that deletion is essential, especially for longer simulations. We consider this in a sequential simulation framework.

World-wide distributed multiple replications in parallel for quantitative sequential simulation

With the recent deployment of global experimental networking facilities, dozens of computer networks with large numbers of computers have become available for scientific studies. Multiple Replications in Parallel (MRIP) is a distributed scenario of sequential quantitative stochastic simulation which offers significant speedup of simulation if it is executed on multiple computers of a local area network. We report results of running MRIP simulations on PlanetLab, a global overlay network which can currently access more than a thousand computers in forty different countries round the globe. Our simulations were run using Akaroa2, a universal controller of quantitative discrete event simulation designed for automatic launching of MRIP-based experiments. Our experimental results provide strong evidence that global experimental networks, such as PlanetLab, can efficiently be used for quantitative simulation, without compromising speed and efficiency.

Some effects of transient deletion on sequential steady-state simulation

Abstract In discrete event steady-state simulation, deleting the initial transient phase of the simulation is usually recommended in order to reduce bias in the results. Various heuristics and tests have been proposed to determine how many observations to delete. The plummeting cost of simulation, combined with uncertainties about the overall reliability of transient methods, suggests revisiting the notion that deletion is essential. We consider this in a framework of sequential simulation, where the simulation is run until a pre-specified accuracy of the results is reached. Our results show that for run lengths required for commonly used levels of accuracy, there is no substantial difference in point or interval estimates of means due to deleting the initial transient for the models we consider. However, in sequential simulation, deleting the initial transient turns out to have considerable value in reducing the risk that the simulation stops too early, thus ensuring that the accuracy of the final results is closer to that specified by the decision-maker.