Robert G. Sargent

Validation and verification of simulation models

In this paper we discuss validation and verification of simulation models. Four different approaches to deciding model validity are described; two different paradigms that relate validation and verification to the model development process are presented; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are discussed; a way to document results is given; a recommended procedure for model validation is presented; and accreditation is briefly discussed.

A methodology for fitting and validating metamodels in simulation

This expository paper discusses the relationships among metamodels, simulation models, and problem entities. A metamodel or response surface is an approximation of the input/output function implied by the underlying simulation model. There are several types of metamodel: linear regression, splines, neural networks, etc. This paper distinguishes between fitting and validating a metamodel. Metamodels may have different goals: (i) understanding, (ii) prediction, (iii) optimization, and (iv) verification and validation. For this metamodeling, a process with thirteen steps is proposed. Classic design of experiments (DOE) is summarized, including standard measures of fit such as the R-square coefficient and cross-validation measures. This DOE is extended to sequential or stagewise DOE. Several validation criteria, measures, and estimators are discussed. Metamodels in general are covered, along with a procedure for developing linear regression (including polynomial) metamodels.

A Unifying View of Hybrid Simulation/Analytic Models and Modeling

In this paper we give unifying definitions for both hybrid simulation/analytic models and modeling. We present four classes of hybrid simulation/analytic models and give examples of each class, including numerical results for two of the examples Four usages of hybrid simulation/analytic modeling are also presented with examples.

A methodology for cost-risk analysis in the statistical validation of simulation models

A methodology is presented for constructing the relationships among model users risk, model builders risk, acceptable validity range, sample sizes, and cost of data collection when statistical hypothesis testing is used for validating a simulation model of a real, observable system. The use of the methodology is illustrated for the use of Hotellings two-sample T 2 test in testing the validity of a multivariate stationary response stimulation model.

Verification, validation and accreditation of simulation models

The paper discusses verification, validation, and accreditation of simulation models. The different approaches to deciding model validity are presented; how model verification and validation relate to the model development process are discussed; various validation techniques are defined; conceptual model validity, model verification, operational validity, and data validity are described; ways to document results are given; a recommended procedure is presented; and accreditation is briefly discussed.

Perspectives on the Evolution of Simulation

Simulation is introduced in terms of its different forms and uses, but the focus on discrete event modeling for systems analysis is dominant as it has been during the evolution of the technique within operations research and the management sciences. This evolutionary trace of over almost fifty years notes the importance of bidirectional influences with computer science, probability and statistics, and mathematics. No area within the scope of operations research and the management sciences has been affected more by advances in computing technology than simulation. This assertion is affirmed in the review of progress in those technical areas that collectively define the art and science of simulation. A holistic description of the field must include the roles of professional societies, conferences and symposia, and publications. The closing citation of a scientific value judgment from over 30 years in the past hopefully provides a stimulus for contemplating what lies ahead in the next 50 years.

Verifying and validating simulation models

In this paper verification and validation of simulation models are discussed. Different approaches to deciding model validity are described and a graphical paradigm that relates verification and validation to the model development process is presented and explained. Conceptual model validity, model verification, operational validity, and data validity are discussed and a recommended procedure for model validation is presented.

Analysis of future event set algorithms for discrete event simulation

New analytical and empirical results for the performance of future event set algorithms in discrete event simulation are presented. These results provide a clear insight to the factors affecting algorithm performance, permit evaluation of the hold model, and determine the best algorithm(s) to use. The analytical results include a classification of distributions for efficient insertion scanning of a linear structure. In addition, it is shown that when more than one distribution is present, there is generally an increase in the probability that new insertions will have smaller times than those in the future event set. Twelve algorithms, including most of those recently proposed, were empirically evaluated using primarily simulation models. Of the twelve tested, four performed well, three performed fairly, and five performed poorly.

Some approaches and paradigms for verifying and validating simulation models

In this paper we discuss verification and validation of simulation models. The different approaches to deciding model validity are described, two different paradigms that relate verification and validation to the model development process are presented, the use of graphical data statistical references for operational validity is discussed, and a recommended procedure for model validation is given.

An investigation of finite-sample behavior of confidence interval estimators

We investigate the small-sample behavior and convergence properties of confidence interval estimators (CIEs) for the mean of a stationary discrete process. We consider CIEs arising from nonoverlapping batch means, overlapping batch means, and standardized time series, all of which are commonly used in discrete-event simulation. The performance measures of interest are the coverage probability, and the expected value and variance of the half-length. We use empirical and analytical methods to make detailed comparisons regarding the behavior of the CIEs for a variety of stochastic processes. All the CIEs under study are asymptotically valid; however, they are usually invalid for small sample sizes. We find that for small samples, the bias of the variance parameter estimator figures significantly in CIE coverage performance—the less bias the better. A secondary role is played by the marginal distribution of the stationary process. We also point out that some CIEs require fewer observations before manifesting ...