David Goldsman

Simple Procedures for Selecting the Best Simulated System When the Number of Alternatives is Large

In this paper, we address the problem of finding the simulated system with the best (maximum or minimum) expected performance when the number of alternatives is finite, but large enough that ranking-and-selection (R&S) procedures may require too much computation to be practical. Our approach is to use the data provided by the first stage of sampling in an R&S procedure to screen out alternatives that are not competitive, and thereby avoid the (typically much larger) second-stage sample for these systems. Our procedures represent a compromise between standard R&S procedures--which are easy to implement, but can be computationally inefficient--and fully sequential procedures--which can be statistically efficient, but are more difficult to implement and depend on more restrictive assumptions. We present a general theory for constructing combined screening and indifference-zone selection procedures, several specific procedures and a portion of an extensive empirical evaluation.

A Stochastic Model of Airline Operations

We present a stochastic model of the daily operations of an airline. Its primary purpose is to evaluate plans, such as crew schedules, as well as recovery policies in a random environment. We describe the structure of the stochastic model, sources of disruptions, recovery policies, and performance measures. Then, we describe SimAir--our simulation implementation of the stochastic model, and we give computational results. Finally, we give future directions for the study of airline recovery policies and planning under uncertainty.

Ranking, selection and multiple comparisons in computer simulations

We present a state-of-the-art review of ranking, selection and multiple-comparison procedures that are used to compare system designs via computer simulation. We describe methods for four classes of problems: screening a large number of system designs, selecting the best system, comparing all systems to a standard and comparing alternatives to a default. Rather than give a comprehensive review, we present the methods we would be likely to use in practice and emphasize recent results. Where possible, we unify the ranking-and-selection and multiple-comparison perspectives.

Ranking and Selection for Steady-State Simulation: Procedures and Perspectives

We present and evaluate three ranking-and-selection procedures for use in steady-state simulation experiments when the goal is to find which among a finite number of alternative systems has the largest or smallest long-run average performance. All three procedures extend existing methods for independent and identically normally distributed observations to general stationary output processes, and all procedures are sequential. We also provide our thoughts about the evaluation of simulation design and analysis procedures, and illustrate these concepts in our evaluation of the new procedures.

Finding the non-dominated Pareto set for multi-objective simulation models

This article considers a multi-objective Ranking and Selection (R+S) problem, where the system designs are evaluated in terms of more than one performance measure. The concept of Pareto optimality is incorporated into the R+S scheme, and attempts are made to find all of the non-dominated designs rather than a single “best” one. In addition to a performance index to measure how non-dominated a design is, two types of errors are defined to measure the probabilities that designs in the true Pareto/non-Pareto sets are dominated/non-dominated based on observed performance. Asymptotic allocation rules are derived for simulation replications based on a Lagrangian relaxation method, under the assumption that an arbitrarily large simulation budget is available. Finally, a simple sequential procedure is proposed to allocate the simulation replications based on the asymptotic allocation rules. Computational results show that the proposed solution framework is efficient when compared to several other algorithms in terms of its capability of identifying the Pareto set.

ASAP3: a batch means procedure for steady-state simulation analysis

We introduce ASAP3, a refinement of the batch means algorithms ASAP and ASAP2, that delivers point and confidence-interval estimators for the expected response of a steady-state simulation. ASAP3 is a sequential procedure designed to produce a confidence-interval estimator that satisfies user-specified requirements on absolute or relative precision as well as coverage probability. ASAP3 operates as follows: the batch size is progressively increased until the batch means pass the Shapiro-Wilk test for multivariate normality; and then ASAP3 fits a first-order autoregressive (AR(1)) time series model to the batch means. If necessary, the batch size is further increased until the autoregressive parameter in the AR(1) model does not significantly exceed 0.8. Next, ASAP3 computes the terms of an inverse Cornish-Fisher expansion for the classical batch means t-ratio based on the AR(1) parameter estimates; and finally ASAP3 delivers a correlation-adjusted confidence interval based on this expansion. Regarding not only conformance to the precision and coverage-probability requirements but also the mean and variance of the half-length of the delivered confidence interval, ASAP3 compared favorably to other batch means procedures (namely, ABATCH, ASAP, ASAP2, and LBATCH) in an extensive experimental performance evaluation.

To batch or not to batch

When designing steady-state computer simulation experiments, one may be faced with the choice of batching observations in one long run or replicating a number of smaller runs. Both methods are potentially useful in the course of undertaking simulation output analysis. The tradeoffs between the two alternatives are well known: batching ameliorates the effects of initialization bias, but produces batch means that might be correlated; replication yields independent sample means, but may suffer from initialization bias at the beginning of each of the runs. We present several new results and specific examples to lend insight as to when one method might be preferred over the other. In steady-state, batching and replication perform similarly in terms of estimating the mean and variance parameter, but replication tends to do better than batching with regard to the performance of confidence intervals for the mean. Such a victory for replication may be hollow, for in the presence of an initial transient, batching often performs better than replication when it comes to point and confidence-interval estimation of the steady-state mean. We conclude---like other classic references---that in the context of estimation of the steady-state mean, batching is typically the wiser approach.

A Review of Healthcare, Public Health, and Syndromic Surveillance

ABSTRACT Due to the ongoing desire for healthcare performance improvement, the latest outbreaks of the avian influenza, and the continuing bioterrorism threat, there is an urgent need for research in healthcare and disease surveillance. In this article we present an overview and review of the general issues involved in healthcare, public health, and syndromic surveillance. In particular, we review existing data collection and surveillance systems, popular surveillance methods, and appropriate performance measures in healthcare and disease surveillance. We also discuss specific challenges and future research in temporal and spatiotemporal surveillance.

An investigation of finite-sample behavior of confidence interval estimators

We investigate the small-sample behavior and convergence properties of confidence interval estimators (CIEs) for the mean of a stationary discrete process. We consider CIEs arising from nonoverlapping batch means, overlapping batch means, and standardized time series, all of which are commonly used in discrete-event simulation. The performance measures of interest are the coverage probability, and the expected value and variance of the half-length. We use empirical and analytical methods to make detailed comparisons regarding the behavior of the CIEs for a variety of stochastic processes. All the CIEs under study are asymptotically valid; however, they are usually invalid for small sample sizes. We find that for small samples, the bias of the variance parameter estimator figures significantly in CIE coverage performance—the less bias the better. A secondary role is played by the marginal distribution of the stationary process. We also point out that some CIEs require fewer observations before manifesting ...

Statistical screening, selection, and multiple comparison procedures in computer simulation

We present a state-of-the-art review of screening, selection, and multiple comparison procedures that are used to compare system designs via computer simulation. We describe methods for three broad classes of problems: screening a large number of system designs, selecting the best system, and comparing all systems to a standard (either known or unknown). We concentrate primarily on recent methods that we would be likely to use in practice. Where possible, we unify the screening, selection, and multiple comparison perspectives.