Bert Bettonvil

Searching for important factors in simulation models with many factors: Sequential bifurcation

Abstract This paper deals with the problem of ‘screening’; that is, how to find the important factors in simulation models that have many (for example, 300) ‘factors’ (also called simulation parameters or input variables). Screening assumes that only a few factors are really important (parsimony principle). This paper solves the screening problem by a novel technique called ‘sequential bifurcation’. This technique is both effective and efficient; that is, it does find all important factors, yet it requires relatively few simulation runs. The technique is demonstrated through a realistic case study, concerning a complicated simulation model, called ‘IMAGE’. This simulation models the greenhouse phenomenon (the worldwide increase of temperatures). This case study gives surprising results: the technique identifies some factors as being important that the ecological experts initially thought to be unimportant. Sequential bifurcation assumes that the input/output behavior of the simulation model may be approximated by a first-order polynomial (main effects), possibly augmented with interactions between factors. The technique is sequential; that is, it specifies and analyzes simulation runs, one after the other.

Finding the Important Factors in Large Discrete-Event Simulation: Sequential Bifurcation and its Applications

This contribution discusses experiments with many factors: the case study includes a simulation model with 92 factors.The experiments are guided by sequential bifurcation.This method is most efficient and effective if the true input/output behavior of the simulation model can be approximated through a first-order polynomial possibly augmented with two-factor interactions.The method is explained and illustrated through three related discrete-event simulation models.These models represent three supply chain configurations, studied for an Ericsson factory in Sweden.After simulating 21 scenarios (factor combinations) each replicated five times to account for noise a shortlist with the 11 most important factors is identified for the biggest of the three simulation models.

Statistical testing of optimality conditions in multiresponse simulation-based optimization

This paper derives a novel procedure for testing the Karush-Kuhn-Tucker (KKT) first-order optimality conditions in models with multiple random responses.Such models arise in simulation-based optimization with multivariate outputs. This paper focuses on expensive simulations, which have small sample sizes. The paper estimates the gradients (in the KKT conditions) through low-order polynomials, fitted locally. These polynomials are estimated using Ordinary Least Squares (OLS), which also enables estimation of the variability of the estimated gradients. Using these OLS results, the paper applies the bootstrap (resampling) method to test the KKT conditions. Furthermore, it applies the classic Student t test to check whether the simulation outputs are feasible, and whether any constraints are binding. The paper applies the new procedure to both a synthetic example and an inventory simulation; the empirical results are encouraging.

Factor screening by sequential bifurcation

The purpose of screening is the reduction of a large set of explanatory variables or factors to the set of important variables, assuming that there are only a few really important explanatory variables. The technique for screening that we propose is a modification of Jacoby and Harrisons (1962) sequential bifurcation, and resembles binary search. It assumes that the direction of the effect of potential variables is already known.This paper starts with deterministic linear response surfaces, for ease of description and easy comparison with alternative techniques. A large scale application illustrates the method. Next the technique is extended with normally distributed random errors with a known common standard deviation. Simulation results include the number of (correctly) found important variables, the number of (incorrectly) found unimportant variables, and the number of observations needed. Further extensions and limitations are briefly indicated.

Validation of trace-driven simulation models: regression analysis revisited

For the validation of trace-driven simulation models this paper recommends a simple statistical test that uses elementary regression analysis in a novel way. This test concerns a (joint) null-hypothesis: the outputs of the simulated and the real systems have the same means and the same variances. Technically, the differences between simulated and real outputs are regressed on their sums, and the resulting slope and intercept are tested to be zero. This paper further proves that it is wrong to use a naive test that regresses the simulation outputs on the real outcomes, and hypothesizes that the resulting regression line gives a 45 /spl deg/ line through the origin. The new and the old tests are investigated in Monte Carlo experiments with inventory systems. The conclusion is that the new test has the correct type I error probability, whereas the old test (falsely) rejects a valid simulation model substantially more often than the nominal alpha level. The power of the new test increases, as the simulation model deviates more from the real system.

IMAGE - Modeling the Greenhouse Effect: Identifying the Important Factors in Simulation Models with Many Factors (FEW 498)

In practice, simulation models usually have a great many parameters and input variables. This paper presents a screening technique which identifies the really important factors. The technique treats the simulation model as a black box and uses a regression metamodel to approximate the input/output behaviour of that black box. The metamodel can account for fitting errors with unknown variance and for interactions among factors. The technique requires relatively few simulation runs and applies to both random and deterministic simulations. The technique is demon strated through a case study of a complicated ecological model.