Ehsan Mehdad

Multivariate versus Univariate Kriging Metamodels for Multi-Response Simulation Models

To analyze the input/output behavior of simulation models with multiple responses, we may apply either univariate or multivariate Kriging (Gaussian process) metamodels. In multivariate Kriging we face a major problem: the covariance matrix of all responses should remain positive-definite; we therefore use the recently proposed “nonseparable dependence” model. To evaluate the performance of univariate and multivariate Kriging, we perform several Monte Carlo experiments that simulate Gaussian processes. These Monte Carlo results suggest that the simpler univariate Kriging gives smaller mean square error.

Classic Kriging versus Kriging with Bootstrapping or Conditional Simulation: Classic Kriging's Robust Confidence Intervals and Optimization

Kriging is a popular method for estimating the global optimum of a simulated system. Kriging approximates the input/output function of the simulation model. Kriging also estimates the variances of the predictions of outputs for input combinations not yet simulated. These predictions and their variances are used by ‘efficient global optimization’ (EGO), to balance local and global search. This article focuses on two related questions: (1) How to select the next combination to be simulated when searching for the global optimum? (2) How to derive confidence intervals for outputs of input combinations not yet simulated? Classic Kriging simply plugs the estimated Kriging parameters into the formula for the predictor variance, so theoretically this variance is biased. This article concludes that practitioners may ignore this bias, because classic Kriging gives acceptable confidence intervals and estimates of the optimal input combination. This conclusion is based on bootstrapping and conditional simulation.

Conditional simulation for efficient global optimization

A classic Kriging or Gaussian process (GP) metamodel estimates the variance of its predictor by plugging-in the estimated GP (hyper)parameters; namely, the mean, variance, and covariances. The problem is that this predictor variance is biased. To solve this problem for deterministic simulations, we propose “conditional simulation” (CS), which gives predictions at an old point that in all bootstrap samples equal the observed value. CS accounts for the randomness of the estimated GP parameters. We use the CS predictor variance in the “expected improvement” criterion of “efficient global optimization” (EGO). To quantify the resulting small-sample performance, we experiment with multi-modal test functions. Our main conclusion is that EGO with classic Kriging seems quite robust; EGO with CS only tends to perform better in expensive simulation with small samples.

Efficient Global Optimization for Black-Box Simulation Via Sequential Intrinsic Kriging

Efficient Global Optimization (EGO) is a popular method that searches sequentially for the global optimum of a simulated system. EGO treats the simulation model as a black-box, and balances local and global searches. In deterministic simulation, EGO uses ordinary Kriging (OK), which is a special case of universal Kriging (UK). In our EGO variant we use intrinsic Kriging (IK), which eliminates the need to estimate the parameters that quantify the trend in UK. In random simulation, EGO uses stochastic Kriging (SK), but we use stochastic IK (SIK). Moreover, in random simulation, EGO needs to select the number of replications per simulated input combination, accounting for the heteroscedastic variances of the simulation outputs. A popular selection method uses optimal computer budget allocation (OCBA), which allocates the available total number of replications over simulated combinations. We derive a new allocation algorithm. We perform several numerical experiments with deterministic simulations and random simulations. These experiments suggest that (1) in deterministic simulations, EGO with IK outperforms classic EGO; (2) in random simulations, EGO with SIK and our allocation rule does not differ significantly from EGO with SK combined with the OCBA allocation rule.

Stochastic intrinsic Kriging for simulation metamodeling: Stochastic Intrinsic Kriging for Simulation Metamodeling

Kriging (or a Gaussian process) provides metamodels for deterministic and random simulation models. Actually, there are several types of Kriging; the classic type is the so‐called universal Kriging, which includes ordinary Kriging. These classic types require estimation of the trend in the input‐output data of the underlying simulation model; this estimation weakens the Kriging metamodel. We therefore consider the so‐called intrinsic Kriging (IK), which originated in geostatistics, and derive IK types for deterministic simulations and random simulations, respectively. Moreover, for random simulations, we derive experimental designs that specify the number of replications that varies with the input combination of the simulation model. To compare the performance of IK and classic Kriging, we use several numerical experiments with deterministic simulations and random simulations, respectively. These experiments show that IK gives better metamodels, in most experiments.

Estimating the Variance of the Predictor in Stochastic Kriging

We study the correct estimation of the true variance of the predictor in stochastic Kriging (SK). First, we obtain macroreplications for a SK metamodel that approximates a single-server simulation model; these macroreplications give independently and identically distributed predictions. This simulation may use common random numbers (CRN). From these macroreplications we conclude that the usual plug-in estimator of the variance significantly underestimates the true variance. Because macroreplications of practical simulation models are computationally expensive, we next formulate two bootstrap methods that use a single macroreplication: (i) a distribution-free method that resamples simulation replications (within the single macroreplication), and (ii) a parametric method that assumes a Gaussian distribution for the SK predictor, and estimates the (hyper)parameters of that distribution from the single macroreplication. Altogether we recommend distribution-free bootstrapping for the estimation of the SK predictor variance in practical simulation experiments.