Jack P. C. Kleijnen

Kriging metamodeling in simulation: A review

This article reviews Kriging (also called spatial correlation modeling). It presents the basic Kriging assumptions and formulas contrasting Kriging and classic linear regression metamodels. Furthermore, it extends Kriging to random simulation, and discusses bootstrapping to estimate the variance of the Kriging predictor. Besides classic one-shot statistical designs such as Latin Hypercube Sampling, it reviews sequentialized and customized designs. It ends with topics for future research.

A methodology for fitting and validating metamodels in simulation

This expository paper discusses the relationships among metamodels, simulation models, and problem entities. A metamodel or response surface is an approximation of the input/output function implied by the underlying simulation model. There are several types of metamodel: linear regression, splines, neural networks, etc. This paper distinguishes between fitting and validating a metamodel. Metamodels may have different goals: (i) understanding, (ii) prediction, (iii) optimization, and (iv) verification and validation. For this metamodeling, a process with thirteen steps is proposed. Classic design of experiments (DOE) is summarized, including standard measures of fit such as the R-square coefficient and cross-validation measures. This DOE is extended to sequential or stagewise DOE. Several validation criteria, measures, and estimators are discussed. Metamodels in general are covered, along with a procedure for developing linear regression (including polynomial) metamodels.

Performance Metrics in Supply Chain Management

This survey paper starts with a critical analysis of various performance metrics for supply chain management (SCM), used by a specific manufacturing company. Then it summarizes how economic theory treats multiple performance metrics. Actually, the paper proposes to deal with multiple metrics in SCM via the balanced scorecard — which measures customers, internal processes, innovations, and finance. To forecast how the values of these metrics will change — once a supply chain is redesigned — simulation may be used. This paper distinguishes four simulation types for SCM: (i) spreadsheet simulation, (ii) system dynamics, (iii) discrete-event simulation, and (iv) business games. These simulation types may explain the bullwhip effect, predict fill rate values, and educate and train users. Validation of simulation models requires sensitivity analysis; a statistical methodology is proposed. The paper concludes with suggestions for a possible research agenda in SCM. A list with 50 references for further study is included.

State-of-the-Art Review: A User's Guide to the Brave New World of Designing Simulation Experiments

Many simulation practitioners can get more from their analyses by using the statistical theory on design of experiments (DOE) developed specifically for exploring computer models.In this paper, we discuss a toolkit of designs for simulationists with limited DOE expertise who want to select a design and an appropriate analysis for their computational experiments.Furthermore, we provide a research agenda listing problems in the design of simulation experiments -as opposed to real world experiments- that require more investigation.We consider three types of practical problems: (1) developing a basic understanding of a particular simulation model or system; (2) finding robust decisions or policies; and (3) comparing the merits of various decisions or policies.Our discussion emphasizes aspects that are typical for simulation, such as sequential data collection.Because the same problem type may be addressed through different design types, we discuss quality attributes of designs.Furthermore, the selection of the design type depends on the metamodel (response surface) that the analysts tentatively assume; for example, more complicated metamodels require more simulation runs.For the validation of the metamodel estimated from a specific design, we present several procedures.

An Overview of the Design and Analysis of Simulation Experiments for Sensitivity Analysis

Sensitivity analysis may serve validation, optimization, and risk analysis of simulation models.This review surveys classic and modern designs for experiments with simulation models.Classic designs were developed for real, non-simulated systems in agriculture, engineering, etc.These designs assume a few factors (no more than ten factors) with only a few values per factor (no more than five values).These designs are mostly incomplete factorials (e.g., fractionals).The resulting input/output (I/O) data are analyzed through polynomial metamodels, which are a type of linear regression models.Modern designs were developed for simulated systems in engineering, management science, etc.These designs allow many factors (more than 100), each with either a few or many (more than 100) values.These designs include group screening, Latin Hypercube Sampling (LHS), and other space filling designs.Their I/O data are analyzed through second-order polynomials for group screening, and through Kriging models for LHS.

Application-driven sequential designs for simulation experiments: Kriging metamodelling

This paper proposes a novel method to select an experimental design for interpolation in simulation. Although the paper focuses on Kriging in deterministic simulation, the method also applies to other types of metamodels (besides Kriging), and to stochastic simulation. The paper focuses on simulations that require much computer time, so it is important to select a design with a small number of observations. The proposed method is therefore sequential. The novelty of the method is that it accounts for the specific input/output function of the particular simulation model at hand; that is, the method is application-driven or customized. This customization is achieved through cross-validation and jackknifing. The new method is tested through two academic applications, which demonstrate that the method indeed gives better results than either sequential designs based on an approximate Kriging prediction variance formula or designs with prefixed sample sizes.

Simulation: a statistical perspective

Random numbers sampling from non-uniform distributions economic and corporate models operations research models simulation software statistical applications regression metamodels design of experiments tactical aspects verification and validation epilogue.

Supply chain simulation tools and techniques: a survey

The main contribution of this paper is twofold: • it surveys different types of simulation for supply chain management • it discusses several methodological issues. These different types of simulation are spreadsheet simulation, system dynamics, discrete-event simulation and business games. Which simulation type should be applied, depends on the type of managerial question to be answered by the model. The methodological issues concern validation and verification, sensitivity, optimisation, and robustness analyses. This sensitivity analysis yields a shortlist of the truly important factors in large simulation models with (say) a hundred factors. The robustness analysis optimises the important factors controllable by management, while accounting for the noise created by the important non-controllable, environmental factors. The various methodological issues are illustrated by a case study involving the simulation of a supply chain in the mobile communications industry in Sweden. In general, simulation is important because it may support the quantification of the benefits resulting from supply chain management.

Searching for important factors in simulation models with many factors: Sequential bifurcation

Abstract This paper deals with the problem of ‘screening’; that is, how to find the important factors in simulation models that have many (for example, 300) ‘factors’ (also called simulation parameters or input variables). Screening assumes that only a few factors are really important (parsimony principle). This paper solves the screening problem by a novel technique called ‘sequential bifurcation’. This technique is both effective and efficient; that is, it does find all important factors, yet it requires relatively few simulation runs. The technique is demonstrated through a realistic case study, concerning a complicated simulation model, called ‘IMAGE’. This simulation models the greenhouse phenomenon (the worldwide increase of temperatures). This case study gives surprising results: the technique identifies some factors as being important that the ecological experts initially thought to be unimportant. Sequential bifurcation assumes that the input/output behavior of the simulation model may be approximated by a first-order polynomial (main effects), possibly augmented with interactions between factors. The technique is sequential; that is, it specifies and analyzes simulation runs, one after the other.

Kriging for interpolation in random simulation

Whenever simulation requires much computer time, interpolation is needed. Simulationists use different interpolation techniques (eg linear regression), but this paper focuses on Kriging. This technique was originally developed in geostatistics by DG Krige, and has recently been widely applied in deterministic simulation. This paper, however, focuses on random or stochastic simulation. Essentially, Kriging gives more weight to ‘neighbouring’ observations. There are several types of Kriging; this paper discusses—besides Ordinary Kriging—a novel type, which ‘detrends’ data through the use of linear regression. Results are presented for two examples of input/output behaviour of the underlying random simulation model: Ordinary and Detrended Kriging give quite acceptable predictions; traditional linear regression gives the worst results.