Thomas W. Lucas

State-of-the-Art Review: A User's Guide to the Brave New World of Designing Simulation Experiments

Many simulation practitioners can get more from their analyses by using the statistical theory on design of experiments (DOE) developed specifically for exploring computer models.In this paper, we discuss a toolkit of designs for simulationists with limited DOE expertise who want to select a design and an appropriate analysis for their computational experiments.Furthermore, we provide a research agenda listing problems in the design of simulation experiments -as opposed to real world experiments- that require more investigation.We consider three types of practical problems: (1) developing a basic understanding of a particular simulation model or system; (2) finding robust decisions or policies; and (3) comparing the merits of various decisions or policies.Our discussion emphasizes aspects that are typical for simulation, such as sequential data collection.Because the same problem type may be addressed through different design types, we discuss quality attributes of designs.Furthermore, the selection of the design type depends on the metamodel (response surface) that the analysts tentatively assume; for example, more complicated metamodels require more simulation runs.For the validation of the metamodel estimated from a specific design, we present several procedures.

Efficient Nearly Orthogonal and Space-Filling Latin Hypercubes.

This article presents an algorithm for constructing orthogonal Latin hypercubes, given a fixed sample size, in more dimensions than previous approaches. In addition, we detail a method that dramatically improves the space-filling properties of the resultant Latin hypercubes at the expense of inducing small correlations between the columns in the design matrix. Although the designs are applicable to many situations, they were developed to provide Department of Defense analysts flexibility in fitting models when exploring high-dimensional computer simulations where there is considerable a priori uncertainty about the forms of the response surfaces.

Exploring the world of agent-based simulations: simple models, complex analyses

Agent-based simulations are models where multiple entities sense and stochastically respond to conditions in their local environments, mimicking complex large-scale system behavior. We provide an overview of some important issues in the modeling and analysis of agent-based systems. Examples are drawn from a range of fields: biological modeling, sociological modeling, and industrial applications, though we focus on recent results for a variety of military applications. Based on our experiences with various agent-based models, we describe issues that simulation analysts should be aware of when embarking on agent-based model development. We also describe a number of tools (both graphical and analytical) that we have found particularly useful for analyzing these types of simulation models. We conclude with a discussion of areas in need of further investigation.

Military applications of agent-based simulations

There continues to be increasing interest from a broad range of disciplines in agent-based and artificial life simulations. This includes the Department of Defense - which uses simulations heavily in its decision making process. Indeed, military conflicts can have many attributes that are consistent with complex adaptive systems - such as many entities interacting with some degree of autonomy, each of which is continually making decisions to satisfy a variety of sometimes conflicting objectives. In this paper, we present three applications of agent-based simulations used to analyze military problems. The first uses the MANA model to explore the ability of the U.S. Armys network-based Future Force to perform with degraded communications. The second studies how unmanned surface vehicles can be used in force protection missions with the Pythagoras model. The last example examines the standard Army squad size with an integrated effort using MANA, Pythagoras, and the high-resolution simulation JANUS.

A two-phase screening procedure for simulation experiments

Analysts examining complex simulation models often conduct screening experiments to identify the most important factors. Controlled sequential bifurcation (CSB) is a screening procedure, developed specifically for simulation experiments, that uses a sequence of hypothesis tests to classify the factors as either important or unimportant. CSB controls the probability of Type I error for each factor, and the power at each bifurcation step, under heterogeneous variance conditions. CSB does, however, require the user to correctly state the directions of the effects prior to running the experiments. Experience indicates that this can be problematic with complex simulations. We propose a hybrid two-phase approach, FF-CSB, to relax this requirement. Phase 1 uses an efficient fractional factorial experiment to estimate the signs and magnitudes of the effects. Phase 2 uses these results in controlled sequential bifurcation. We describe this procedure and provide an empirical evaluation of its performance.

Defense and homeland security applications of multi-agent simulations

Department of defense and homeland security analysts are increasingly using multi-agent simulation (MAS) to examine national security issues. This paper summarizes three MAS national security studies conducted at the Naval Postgraduate School. The first example explores equipment and employment options for protecting critical infrastructure. The second case considers non-lethal weapons within the spectrum of force-protection options in a maritime environment. The final application investigates emergency (police, fire, and medical) responses to an urban terrorist attack. There are many potentially influential factors and many sources of uncertainty associated with each of these simulated scenarios. Thus, efficient experimental designs and computing clusters are used to enable us to explore many thousands of computational experiments, while simultaneously varying many factors. The results illustrate how MAS experiments can provide valuable insights into defense and homeland security operations.

Constructing nearly orthogonal latin hypercubes for any nonsaturated run-variable combination

We present a new method for constructing nearly orthogonal Latin hypercubes that greatly expands their availability to experimenters. Latin hypercube designs have proven useful for exploring complex, high-dimensional computational models, but can be plagued with unacceptable correlations among input variables. To improve upon their effectiveness, many researchers have developed algorithms that generate orthogonal and nearly orthogonal Latin hypercubes. Unfortunately, these methodologies can have strict limitations on the feasible number of experimental runs and variables. To overcome these restrictions, we develop a mixed integer programming algorithm that generates Latin hypercubes with little or no correlation among their columns for most any determinate run-variable combination—including fully saturated designs. Moreover, many designs can be constructed for a specified number of runs and factors—thereby providing experimenters with a choice of several designs. In addition, our algorithm can be used to quickly adapt to changing experimental conditions by augmenting existing designs by adding new variables or generating new designs to accommodate a change in runs.

Efficient experimental design tools for exploring large simulation models

Simulation experiments are typically faster, cheaper and more flexible than physical experiments. They are especially useful for pilot studies of complicated systems where little prior knowledge of the system behavior exists. One key characteristic of simulation experiments is the large number of factors and interactions between factors that impact decision makers. Traditional simulation approaches offer little help in analyzing large numbers of factors and interactions, which makes interpretation and application of results very difficult and often incorrect. In this paper we implement and demonstrate efficient design of experiments techniques to analyze large, complex simulation models. Looking specifically within the domain of organizational performance, we illustrate how our approach can be used to analyze even immense results spaces, driven by myriad factors with sometimes unknown interactions, and pursue optimal settings for different performance measures. This allows analysts to rapidly identify the most important, results-influencing factors within simulation models, employ an experimental design to fully explore the simulation space efficiently, and enhance system design through simulation. This dramatically increases the breadth and depth of insights available through analysis of simulation data, reduces the time required to analyze simulation-driven studies, and extends the state of the art in computational and mathematical organization theory.

Two-phase screening procedure for simulation experiments

Analysts examining complex simulation models often conduct screening experiments to identify the most important factors. Controlled sequential bifurcation (CSB) is a screening procedure, developed specifically for simulation experiments, that uses a sequence of hypothesis tests to classify the factors as either important or unimportant. CSB controls the probability of type I error for each factor, and the power at each bifurcation step, under heterogeneous variance conditions. CSB does, however, require the user to correctly state the directions of the effects prior to running the experiments. Experience indicates that this can be problematic with complex simulations. We propose a hybrid two-phase approach, FF-CSB, to relax this requirement. Phase 1 uses an efficient fractional factorial experiment to estimate the signs and magnitudes of the effects. Phase 2 uses these results in controlled sequential bifurcation. We describe this procedure and provide an empirical evaluation of its performance.

Selecting random latin hypercube dimensions and designs through estimation of maximum absolute pairwise correlation

Latin hypercubes are the most widely used class of design for high-dimensional computer experiments. However, the high correlations that can occur in developing these designs can complicate subsequent analyses. Efforts to reduce or eliminate correlations can be complex and computationally expensive. Consequently, researchers often use uncorrected Latin hypercube designs in their experiments and accept any resulting multicollinearity issues. In this paper, we establish guidelines for selecting the number of runs and/or the number of variables for random Latin hypercube designs that are likely to yield an acceptable degree of correlation. Applying our policies and tools, analysts can generate satisfactory random Latin hypercube designs without the need for complex algorithms.