Osman Balci

Validation, verification, and testing techniques throughout the life cycle of a simulation study

Life cycle validation, verification, and testing (VV&T) is extremely important for the success of a simulation study. This paper surveys current software VV&T techniques and current simulation model VV&T techniques and describes how they can all be applied throughout the life cycle of a simulation study. The processes and credibility assessment stages of the life cycle are described and the applicability of the VV&T techniques for each stage is stated. A glossary is provided to explicitly define important terms and VV&T techniques.

Verification validation and accreditation of simulation models

This paper presents guidelines for conducting verification, validation and accreditation (VV&A) of simulation models. Fifteen guiding principles are introduced to help the researchers, practitioners and managers better comprehend what VV&A is all about. The VV&A activities are described in the modeling and simulation life cycle. A taxonomy of more than 77 V&V techniques is provided to assist simulationists in selecting proper approaches for conventional simulation model V&V. Another taxonomy of 38 V&V techniques is presented for object-oriented simulation models. 1. INTRODUCTION Assuring total quality in a modeling and simulation (M&S) effort involves the measurement and assessment of a variety of quality characteristics such as accuracy, execution efficiency, maintainability, portability, reusability, and usability (human-computer interface). This paper is concerned only with the accuracy quality characteristic. Verification, validation, testing, accreditation, certification and credibility assessment activities primarily deal with the measurement and assessment of accuracy of models and simulations (M&S). Model VeriJication is substantiating that the model is transformed from one form into another, as intended, with sufficient accuracy. Model verification deals with building the model right. The accuracy of transforming a problem formulation into a model specification or the accuracy of converting a model representation from a micro flowchart form into an executable computer program is evaluated in model verification. Model Validation is substantiating that the model, within its domain of applicability, behaves with satisfactory accuracy consistent with the M&S objectives. Model validation deals with building the right model. An activity of accuracy assessment can be labeled as verification or validation based on an answer to the following question: In assessing the accuracy, is the model behavior compared with respect to the corresponding system behavior through mental or computer execution? If

Modeling and analyzing a physician clinic environment using discrete-event (visual) simulation

Abstract This paper examines the design and development of a discrete-event (visual) simulation model of a physician clinic environment within a physician network. Biological & Popular Culture, Inc. (Biopop) sought to partner with healthcare professionals to provide high-quality, cost-effective medical care within a physician network setting. Towards this end, a discrete-event (visual) simulation model that captures both the operations of a family practice healthcare clinic and a centralized information center is presented. The research presented in this article focuses on the family practice healthcare clinic. This simulation model is built in an object-oriented, visual manner utilizing the visual simulation environment (VSE). Application of the object-oriented paradigm (OOP) allows simulation objects in the model to be easily reused. The simulation model provides a tool for risk-free evaluation of operating policies in the clinical environment. Results of a fractional factorial design to determine those input factors which significantly affect overall clinic effectiveness are reported. Scope and purpose With increased pressures from governmental and insurance agencies, todays physician devotes less time to patient care and more time to administration. To alleviate this problem, Biological & Popular Culture, Inc. (Biopop) proposed the building of partnerships with healthcare professionals to provide high-quality, cost-effective medical care in a physician network setting. The proposed physician network is to be composed of clinics (i.e., outpatient medical facilities where patient care is delivered) located throughout the United States. Member clinics would benefit from centrally managed non-medical operations and business consultation from Biopop. Biopops overall goal is to increase patient flow and customer satisfaction in the clinical environment. To assist Biopop in evaluating potential operating procedures, a discrete-event simulation model has been constructed. The model is built in an object-oriented, visual manner utilizing Orca Computer, Inc.s visual simulation environment (VSE). The model examines both internal Biopop operations and external clinic operations. The research presented herein describes how the simulation model was designed and built, including the verification, validation, and testing procedures applied. Results of simulation experiments run on the clinical environment using the simulation model are also reported. The results of these computer runs suggest ways in which healthcare clinics can be redesigned, without sacrificing service levels or profitability of the facility.

Verification, validation, and accreditation

This paper presents guidelines for conducting verification, validation, and accreditation (VV&A) of M&S applications. Fifteen guiding principles are introduced to help the researchers, practitioners and managers better comprehend what VV&A is all about. The VV&A activities are described in two M&S life cycles. Applicability of 77 V&V techniques is shown for the major stages of the two M&S life cycles. A methodology for accreditation of M&S applications is briefly introduced.

A methodology for cost-risk analysis in the statistical validation of simulation models

A methodology is presented for constructing the relationships among model users risk, model builders risk, acceptable validity range, sample sizes, and cost of data collection when statistical hypothesis testing is used for validating a simulation model of a real, observable system. The use of the methodology is illustrated for the use of Hotellings two-sample T 2 test in testing the validity of a multivariate stationary response stimulation model.

Principles of Simulation Model Validation, Verification, and Testing

Sufficient experience has been gained over the last decade in simulation model validation, verification, and testing (VV&T) to establish basic principles and its characteristics. This paper presents 15 principles of simulation model VV&T. These principles help the researchers, practitioners, and manaagers better understand what model VV&T is all about. They serve to provide the underpinnings for the VV&T techniques that can be used throughout the life cycle of a simulation study. Understanding and applying these principles is crucially important for the success of a simulation study.

Guidelines for successful simulation studies

Guidelines are given for conducting a simulation study. The guidelines are provided throughout the entire life cycle of a simulation study that is composed of ten processes, ten phases, and 13 credibility assessment stages. An overall evaluation scheme for assessing the acceptability and credibility of simulation results is proposed.<<ETX>>

A methodology for certification of modeling and simulation applications

Certification of modeling and simulation (M&S) applications poses significant technical challenges for M&S program managers, engineers, and practitioners. Certification is becoming increasingly more important as M&S applications are used more and more for military training, complex system design evaluation, M&S-based acquisition, problem solving, and critical decision making. Certification, a very complex process, involves the measurement and evaluation of hundreds of qualitative and quantitative elements, mandates subject matter expert evaluation, and requires the integration of different evaluations. Planning and managing such measurements and evaluations requires a unifying methodology and should not be performed in an ad hoc manner. This paper presents such a methodology. The methodology consists of the following body of methods, rules, and postulates: (a) employment of subject matter experts, (b) construction of a hierarchy of indicators, (c) relative criticality weighting of indicators using the analytic hierarchy process, (d) using a rule-based expert knowledge base with an object-oriented specification language, (e) assignment of crisp, fuzzy, and nominal scores for the indicators, (f) aggregation of indicator scores, (g) graphical representation of the indicator scores and weights, (h) hypertext certification report, and (i) interpretation of the results. The methodology can be used for certification of any kind of M&S application either throughout the M&S development life cycle or after the development is completed.

The implementation of four conceptual frameworks for simulation modeling in high-level languages

This is a tutorial paper on how to implement a simulation model in a high-level programming language (e.g., C, Pascal, FORTRAN) by using the following conceptual frameworks (also called world views, simulation strategies, and formalisms): (1) event scheduling, (2) activity scanning, (3) three-phase approach, and (4) process interaction. Implementation details under each conceptual framework are covered in a high level without being concerned about execution efficiency. The purpose is to reveal the characteristics of the four conceptual frameworks so that the programmer can select and implement one to achieve certain model quality characteristics such as maintainability, reusability, and execution efficiency. A problem is defined for use as an example for illustrating the concepts throughout the paper.

Requirements for model development environments

Abstract This paper deals with the initial phase of our ongoing research project on the Definition of a Discrete Event Simulation MDE which started on 1 June 1983. The first phase of the rapid prototyping approach we are using in designing the MDE involves the requirements specification. A literature review revealed eleven current problems in modeling. To address these problems, a MDE was identified as composed of four layers: (1) hardware and operating system, (2) kernel MDE, (3) minimal MDE and (4) MDEs. Requirements were then perceived for each layer and are reported in this paper. The feasibility of the requirements have been assessed throughout our prototyping efforts. This paper has provided significant guidance to our research group in designing the MDE and its associated tools. We believe that the designers and implementers of other types of MDEs can benefit from the research described herein.