Russell R. Barton

Computationally Inexpensive Metamodel Assessment Strategies

In many scientific and engineering domains, it is common to analyze and simulate complex physical systems using mathematical models. Although computing resources continue to increase in power and speed, computer simulation and analysis codes continue to grow in complexity and remain computationally expensive, limiting their use in design and optimization. Consequently, many researchers have developed different metamodeling strategies to create inexpensive approximations of computationally expensive computer simulations. These approximations introduce a new element of uncertainty during design optimization, and there is a need to develop efficient methods to assess metamodel validity. We investigate computationally inexpensive assessment methods for metamodel validation based on leave-k-out cross validation and develop guidelines for selecting k for different types of metamodels. Based on the results from two sets of test problems, k = 1 is recommended for leave-k-out cross validation of low-order polynomial and radial basis function metamodels, whereas k=0.1N or N is recommended for kriging metamodels, where N is the number of sample points used to construct the metamodel.

A review on design, modeling and applications of computer experiments

In this paper, we provide a review of statistical methods that are useful in conducting computer experiments. Our focus is on the task of metamodeling, which is driven by the goal of optimizing a complex system via a deterministic simulation model. However, we also mention the case of a stochastic simulation, and examples of both cases are discussed. The organization of our review first presents several engineering applications, it then describes approaches for the two primary tasks of metamodeling: (i) selecting an experimental design; and (ii) fitting a statistical model. Seven statistical modeling methods are included. Both classical and newer experimental designs are discussed. Finally, our own computational study tests the various metamodeling options on two two-dimensional response surfaces and one ten-dimensional surface.

Metamodels for simulation input-output relations

The simulation community has used metarnodels to study the behavior of computer simulations for over twenty-five years. The most popular teebniques have been based on parametric polynomial response surface approximations. In this state of the art review, we present recent developments in this area. We also discuss seven alternative modeling strategies that are active topics in the current literature.

Chapter 18 Metamodel-Based Simulation Optimization

Abstract Simulation models allow the user to understand system performance and assist in behavior prediction, to support system diagnostics and design. Iterative optimization methods are often used in conjunction with engineering simulation models to search for designs with desired properties. These optimization methods can be difficult to employ with a discrete-event simulation, due to the stochastic nature of the response(s) and the potentially extensive run times. A metamodel, or model of the simulation model, simplifies the simulation optimization in two ways: the metamodel response is deterministic rather than stochastic, and the run times are generally much shorter than the original simulation. Metamodels based on first- or second-order polynomials generally provide good fit only locally, and so a series of metamodels are fit as the optimization progresses. Other classes of metamodels can provide good global fit; in these cases one can fit a (global) metamodel once, at the start of the optimization, and use it to find a design that will meet the optimality criteria. Both approaches are discussed in this chapter and illustrated with an example.

Metamodeling: a state of the art review

The simulation community has used metamodels to study the behavior of computer simulations for over twenty-five years. The most popular techniques have been based on parametric polynomial response surface approximations. In this state of the art review, we present recent developments in this area, with a particular focus on new developments in the experimental designs employed.

Metamodeling of Combined Discrete/Continuous Responses

Metamodels are effective for providing fast-running surrogate approximations of product or system performance. Because these approximations are generally based on continuous functions, they can provide poor fits of discontinuous response functions. Many engineering models produce functions that are only piecewise continuous, due to changes in modes of behavior or other state variables. The use of a state-selecting metamodeling approach that provides an accurate approximation for piecewise continuous responses is investigated. The proposed approach is applied to a desk lamp performance model. Three types of metamodels, quadratic polynomials, spatial correlation (kriging) models, and radial basis functions, and five types of experimental designs, full factorial designs, D-best Latin hypercube designs, fractional Latin hypercubes, Hammersley sampling sequences, and uniform designs, are compared based on three error metrics computed over the design space. The state-selecting metamodeling approach outperforms a combined metamodeling approach in this example, and radial basis functions perform well for metamodel construction.

Uniform and bootstrap resampling of empirical distributions

Stochastic simulation models are used to predict the behavior of real systems whose components have random variation. The simulation model generates artificial random quantities based on the nature of the random variation in the real system. Very often, the probability distributions occurring in the real system are unknown, and must be estimated using finite samples. This paper presents two ways to estimate simulation model output errors due to the errors in the empirical distributions used to drive the simulation. These approaches are applied to simulations of the M/M/1 queue with an empirically sampled interarrival time. They capture components of variance in the estimate of mean time in the system that are ignored when the empirical distribution is treated as the true distribution.

Resampling methods for input modeling

Stochastic simulation models are used to predict the behavior of real systems whose components have random variation. The simulation model generates artificial random quantities based on the nature of the random variation in the real system. Very often, the probability distributions occurring in the real system are unknown, and must be estimated using finite samples. This paper shows three methods for incorporating the error due to input distributions that are based on finite samples, when calculating confidence intervals for output parameters.

Experimental design for simulation

This tutorial introduces some of the ideas, issues, challenges, solutions, and opportunities in deciding how to experiment with simulation models to learn about their behavior. Careful planning, or designing, of simulation experiments is generally a great help, saving time and effort by providing efficient ways to estimate the effects of changes in the models inputs on its outputs. Traditional experimental-design methods are discussed in the context of simulation experiments, as are the broader questions pertaining to planning computer-simulation experiments.

Ch. 7. A review of design and modeling in computer experiments

In this chapter, we provide a review of statistical methods that are useful in conducting computer experiments. Our focus is primarily on the task of metamodeling, which is driven by the goal of optimizing a complex system via a deterministic simulation model. However, we also mention the case of a stochastic simulation, and examples of both cases are discussed. The organization of our review separates the two primary tasks for metamodeling: (1) select an experimental design; (2) fit a statistical model. We provide an overview of the general strategy and discuss applications in electrical engineering, chemical engineering, mechanical engineering, and dynamic programming. Then, we dedicate a section to statistical modeling methods followed by a section on experimental designs. Designs are discussed in two paradigms, model-dependent and model-independent, to emphasize their different objectives. Both classical and modern methods are discussed.