Susan M. Sanchez

State-of-the-Art Review: A User's Guide to the Brave New World of Designing Simulation Experiments

Many simulation practitioners can get more from their analyses by using the statistical theory on design of experiments (DOE) developed specifically for exploring computer models.In this paper, we discuss a toolkit of designs for simulationists with limited DOE expertise who want to select a design and an appropriate analysis for their computational experiments.Furthermore, we provide a research agenda listing problems in the design of simulation experiments -as opposed to real world experiments- that require more investigation.We consider three types of practical problems: (1) developing a basic understanding of a particular simulation model or system; (2) finding robust decisions or policies; and (3) comparing the merits of various decisions or policies.Our discussion emphasizes aspects that are typical for simulation, such as sequential data collection.Because the same problem type may be addressed through different design types, we discuss quality attributes of designs.Furthermore, the selection of the design type depends on the metamodel (response surface) that the analysts tentatively assume; for example, more complicated metamodels require more simulation runs.For the validation of the metamodel estimated from a specific design, we present several procedures.

Exploring the world of agent-based simulations: simple models, complex analyses

Agent-based simulations are models where multiple entities sense and stochastically respond to conditions in their local environments, mimicking complex large-scale system behavior. We provide an overview of some important issues in the modeling and analysis of agent-based systems. Examples are drawn from a range of fields: biological modeling, sociological modeling, and industrial applications, though we focus on recent results for a variety of military applications. Based on our experiences with various agent-based models, we describe issues that simulation analysts should be aware of when embarking on agent-based model development. We also describe a number of tools (both graphical and analytical) that we have found particularly useful for analyzing these types of simulation models. We conclude with a discussion of areas in need of further investigation.

Work smarter, not harder: guidelines for designing simulation experiments

We present the basic concepts of experimental design, the types of goals it can address, and why it is such an important and useful tool for simulation. A well-designed experiment allows the analyst to examine many more factors than would otherwise be possible, while providing insights that cannot be gleaned from trial-and-error approaches or by sampling factors one at a time. We focus on experiments that can cut down the sampling requirements of some classic designs by orders of magnitude, yet make it possible and practical to develop a better understanding of a complex simulation model. Designs we have found particularly useful for simulation experiments are illustrated using simple simulation models, and we provide links to other resources for those wishing to learn more. Ideally, this tutorial will leave you excited about experimental designs - and prepared to use them - in your upcoming simulation studies.

Military applications of agent-based simulations

There continues to be increasing interest from a broad range of disciplines in agent-based and artificial life simulations. This includes the Department of Defense - which uses simulations heavily in its decision making process. Indeed, military conflicts can have many attributes that are consistent with complex adaptive systems - such as many entities interacting with some degree of autonomy, each of which is continually making decisions to satisfy a variety of sometimes conflicting objectives. In this paper, we present three applications of agent-based simulations used to analyze military problems. The first uses the MANA model to explore the ability of the U.S. Armys network-based Future Force to perform with degraded communications. The second studies how unmanned surface vehicles can be used in force protection missions with the Pythagoras model. The last example examines the standard Army squad size with an integrated effort using MANA, Pythagoras, and the high-resolution simulation JANUS.

Robust design: seeking the best of all possible worlds

We describe a framework for analyzing simulation output in order to find solutions that will work well after implementation. We show how the use of a loss function that incorporates both system mean and system variability can be used to efficiently and effectively carry out system optimization and improvement efforts. For models whose behavior depends on quantitative factors, we illustrate how robust design can be accomplished by using simple experimental designs in conjunction with response-surface metamodels. The results can yield new insights into system behavior, and may lead to recommended system configurations that differ substantially from those selected by analysis solely on the basis of mean response.

Very large fractional factorial and central composite designs

We present a concise representation of fractional factorials and an algorithm to quickly generate resolution V designs. The description is based on properties of a complete, orthogonal discrete-valued basis set called Walsh functions. We tabulate two-level resolution V fractional factorial designs, as well as central composite designs allowing estimation of full second-order models, for experiments involving up to 120 factors. The simple algorithm provided can be used to characterize even larger designs, and a fast Walsh transform method quickly generates design matrices from our representation.

Designing simulation experiments: Taguchi methods and response surface metamodels

G. Taguchi (1987) has made an innovative contribution to quality planning activities through the integrated use of loss functions and orthogonal arrays. The authors focus on the improvement and implementation of some of these techniques in the simulation arena. The orthogonal arrays advocated by Taguchi are related to classical experimental designs, which have played important tactical roles in the exploration of mathematical metamodels for the simulation response surface. However, the loss function and the associated robust design philosophy provide fresh insights into the process of optimizing or improving the simulations performance. The authors use examples to illustrate concepts such as the simultaneous treatment of variability and mean of performance measures, strategies for achieving system robustness, and implementation of noise through factorial designs. Relationships to other issues in designing and analyzing simulation experiments, such as response surface metamodels and variance reduction, are discussed.<<ETX>>

Emerging issues in healthcare simulation

Despite the size and importance of the health care industry, simulation is less prevalent in health care than in other fields such as manufacturing, logistics, and military applications. Yet simulation clearly has the potential to play a role in health care decision-making at many levels. The purpose of this panel is to discuss some of the issues that practioners must be aware of in order to tap the potential of simulation in the health care arena. The panelists have extensive experience in health care and the use of simulation in that environment. They have provided statements outlining several key issues for achieving success in current and future health care simulation projects. These will serve as the starting point for discussion at the conference.

A ROBUST DESIGN METHODOLOGY FOR KANBAN SYSTEM DESIGN

Many companies are interested in implementing just-in-time (JIT) manufacturing philosophies in response to increased competitive pressures on manufacturing. At the shop floor level, one application of JIT is through the introduction of Kanbans (or cards) so as to control in-process inventory. Traditionally, it has been argued that Kanban systems work well when the shop floor environment is fairly stable. In this paper, we propose and illustrate a methodology, based on the robust design concept of Taguchi, to implement Kanban systems in uncertain environments. We show how this procedure can be used to determine appropriate settings for the decision factors based on the inherent variations on the shop floor. From a managerial perspective, this procedure can be used not only for Kanban system design but also to identify shop floor factors which can be the targets of improvement efforts.

Effective Engineering Design through Simulation

Abstract This paper presents a framework for designing, analyzing and improving systems and processes via discrete event simulation. The framework incorporates a robust design philosophy into a response surface metamodeling approach, and the simulation setting provides the analyst with an increased level of control relative to industrial experimentation. System optimization and improvement efforts can be carried out efficiently and effectively, providing insights into system behavior and suggesting optimal system configurations which may yield substantial improvements over those selected using more traditional approaches. One noteworthy benefit of the simulation framework is that robust design methodologies can be applied prospectively — at the inception and conceptualization phases of an engineering design project. We illustrate the method by considering the design of a small job shop.