Kathryn Hoad

Automating warm-up length estimation

AbstractThere are two key issues in assuring the accuracy of estimates of performance obtained from a simulation model. The first is the removal of any initialisation bias, the second is ensuring that enough output data is produced to obtain an accurate estimate of performance. This paper is concerned with the first issue, and more specifically warm-up estimation. Our aim is to produce an automated procedure, for inclusion into commercial simulation software, for estimating the length of warm-up and hence removing initialisation bias from simulation output data. This paper describes the extensive literature search that was carried out in order to find and assess the various existing warm-up methods, the process of short-listing and testing of candidate methods. In particular it details the extensive testing of the warm-up MSER-5 method.

Automated selection of the number of replications for a discrete-event simulation

Selecting an appropriate number of replications to run with a simulation model is important in assuring that the desired accuracy and precision of the results are attained with minimal effort. If too few are selected then accuracy and precision are lost; if too many are selected then valuable time is wasted. Given that simulation is often used by non-specialists, it seems important to provide guidance on the number of replications required with a model. In this paper an algorithm for automatically selecting the number of replications is described. Test results show the algorithm to be effective in obtaining coverage of the expected mean at a given level of precision and in reducing the bias in the estimate of the mean of the simulation output. The algorithm is consistent in selecting the expected number of replications required for a model output.

Automating DES output analysis: how many replications to run

This paper describes the selection and automation of a method for estimating how many replications should be run to achieve a required accuracy in the output. The motivation is to provide an easy to use method that can be incorporated into existing simulation software that enables practitioners to obtain results of a specified accuracy. The processes and decisions involved in selecting and setting up a method for automation are explained. The extensive test results are outlined, including results from applying the algorithm to a collection of artificial and real models.

AutoSimOA: a framework for automated analysis of simulation output

There are two key issues in assuring the accuracy of estimates of performance obtained from a simulation model. The first is the removal of any initialisation bias; the second is ensuring that enough output data are produced to obtain an accurate estimate of performance. Our aim is to produce an automated procedure for inclusion into commercial simulation software to address both of these issues. This paper describes the results of a 3-year project to produce such an analyser. Our Automated Simulation Output Analyser identifies the warm-up period, estimates the number of replications, and/or analyses output from a single run, with the aim of providing the user with accurate and precise measures of their chosen output statistics.

Are we there yet? Simulation modellers on what needs to be done to involve agent-based simulation in practical decision making

The most recent meeting of the Society Simulation Special Interest Group (SSIG) (24 March 2011) was in collaboration with the Centre for Research in Social Simulation (CRESS), based in the Department of Sociology at the University of Surrey. CRESS is described as ‘a multidisciplinary centre bringing together the social sciences, software engineering and agent-based computing to promote and support the use of social simulation in research in the human sciences’. Hosted by CRESS as part of its ESRC-funded project, SIMIAN, the SSIG meeting explored the application of agent-based simulation modelling from a practical decision-making point of view. The speakers and delegates were gathered from both the social sciences and operational research/management areas, in order to provoke discussion and encourage cross-disciplinary learning and collaboration. The speakers included: Professor Nigel Gilbert, Director, CRESS, University of Surrey; Professor Stewart Robinson, Professor of O.R. at the University of Warwick; Professor Scott Moss, of Scott Moss Associates and Visiting Professor at the University of KoblenzLandau; Dr Peer-Olaf Siebers, Senior Research Fellow in Computer Science at the University of Nottingham; David Buxton, dseConsulting Ltd; Mark Temple, Consultant in Public Health Medicine (NHS) working in the Communicable Disease Surveillance Centre for Wales, and a SIMIAN user fellow; and Jeremy Franklin, Home Office, also a SIMIAN user fellow. Delegates were entertained and informed by seven halfhour talks around the subject of using agent-based simulation (ABS) in policy decision making. Kicking the day off, Professor Nigel Gilbert welcomed us all to CRESS and proceeded to outline what he considers to be the main ingredients of an agent-based simulation model: agents, which represent any type of social actor, possessing properties such as perception, policies, performance and memory; and the environment within and upon which the agents act, which provides resources and a communication medium. He outlined agent-based modelling (ABM) methodology as first starting with a theory that could then be explored by ABS. The next steps were described as proposing some agent actions and the context in which they occur (‘micro foundations’), then creating a model to represent those agents and their interactions within an environment. The modeller could then observe the ‘macro behaviour’ of this model and compare the output with observations from the social world. Professor Stewart Robinson, in his talk entitled ‘Are ABS and OR commensurable paradigms’, pointed out that this first step of finding or developing a theory upon which the model would be designed is one of the key features that sets apart ABS social science simulation from operational research (OR) simulation. He argued that there were several key differences between OR and ABS paradigms, namely that OR has an empirical basis, whereas ABS has a theoretical basis; OR has an emphasis on improving the real world, whereas ABS has an emphasis on thinking about the real world; OR puts great emphasis on the collection and analysis of data, whereas ABS concentrates more on dynamic hypotheses; OR expects validation of its models, whereas ABS seeks plausibility; and OR puts an emphasis on the implementation of findings, whereas ABS strives for learning or understanding. He stressed that despite these differences, ABS and OR can still sometimes be used interchangeably and therefore there is a small area of overlap between the two methodologies or scientific theories. He argued that this overlap can be made larger by pushing OR further into the theoretical by embracing theoretical ABS, and expanding ABS more into the empirical by greater emphasis on data collection. *Correspondence: K Hoad, ORMS Group, Warwick Business School, University of Warwick, University Road, Coventry, CV4 &AL, UK. Journal of Simulation (2012) 6, 67–70 r 2012 Operational Research Society Ltd. All rights reserved. 1747-7778/12

A note on the use of multiple comparison scenario techniques in education and practice

Our main aim in this paper is to highlight current practice and education in multiple scenario comparison within DES experimentation and to illustrate the possible benefits of employing false discovery rate (FDR) control as opposed to strict family-wise error rate (FWER) control when comparing large numbers of scenarios in an exploratory manner. We present the results of a small survey into the current practice of scenario analysis by simulation practitioners and academics. The results indicated that the range of scenarios used in DES studies may prohibit the use of FWER control methods such as the Bonferroni Correction referred to in DES textbooks. Furthermore, 80% of our sample were not familiar with any of the multiple comparison control procedures presented to them. We provide a practical example of the FDR in action and argue that it is preferable to employ FDR instead of no multiple comparison control in exploratory style studies.

The use of search experimentation in discrete-event simulation practice

The practice of DES experimentation has not been rigorously assessed in over a decade. Past studies of DES practice report little transfer of experimentation theory into real-world application. We conducted an international survey of over 300 modellers to investigate the extent to which simulation optimisation, meta-modelling and design of experiments are used in practice. Over the last decade there has been substantial growth in the use of optimisation and to a lesser extent design of experiments to tackle practical problems. However, users rarely make use of optimisers bundled with commercial software, opting instead for custom or third-party solutions. Outside of academia, the use of methods is hampered by a lack of application knowledge and a persisting view that such techniques are not necessary. It is clear that academics must not become complacent regarding the dissemination of theory into common practice and continue to reach out to industry users.

Implementing MSER-5 in commercial simulation software and its wider implications

Starting a model from an unrealistic state can lead to initialization bias in the simulation output. This, in turn, can produce bias in the results and lead to incorrect conclusions. One method for dealing with this problem is to run the model for a warm-up period until steady state is reached and remove the initialization bias by deleting the data within that warm-up period. Our previous research identified the MSER-5 algorithm as the best candidate warm-up method for implementation into an automated output analysis system, and for inclusion into existing DES software products. However, during an attempt to implement an automatable sequential version of the MSER-5 procedure into existing discrete-event simulation software several issues arose. This paper describes the framework and associated adaption of MSER-5 in order to automate it. It then discusses in detail the implementation issues that arose and some potential solutions.

Modelling local and global effects on the risk of contracting Tuberculosis using stochastic Markov-chain models.

For some diseases, the transmission of infection can cause spatial clustering of disease cases. This clustering has an impact on how one estimates the rate of the spread of the disease and on the design of control strategies. It is, however, difficult to assess such clustering, (local effects on transmission), using traditional statistical methods. A stochastic Markov-chain model that takes into account possible local or more dispersed global effects on the risk of contracting disease is introduced in the context of the transmission dynamics of tuberculosis. The model is used to analyse TB notifications collected in the Asembo and Gem Divisions of Nyanza Province in western Kenya by the Kenya Ministry of Health/National Leprosy and Tuberculosis Program and the Centers for Disease Control and Prevention. The model shows evidence of a pronounced local effect that is significantly greater than the global effect. We discuss a number of variations of the model which identify how this local effect depends on factors such as age and gender. Zoning/clustering of villages is used to identify the influence that zone size has on the models ability to distinguish local and global effects. An important possible use of the model is in the design of a community randomised trial where geographical clusters of people are divided into two groups and the effectiveness of an intervention policy is assessed by applying it to one group but not the other. Here the model can be used to take the effect of case clustering into consideration in calculating the minimum difference in an outcome variable (e.g. disease prevalence) that can be detected with statistical significance. It thereby gauges the potential effectiveness of such a trial. Such a possible application is illustrated with the given time/spatial TB data set.

Teaching system dynamics and discrete event simulation together : a case study

System dynamics (SD) and discrete event simulation (DES) follow two quite different modeling philosophies and can bring very different but, nevertheless, complimentary insights in understanding the same ‘real world’ problem. Thus, learning SD and DES approaches requires students to absorb different modeling philosophies usually through specific and distinct courses. We run a course where we teach model conceptualization for SD and DES in parallel and, then, the technical training on SD and DES software in sequential order. The ability of students to assimilate, and then put into practice both modeling approaches, was evaluated using simulation-based problems. While we found evidence that students can master both simulation techniques, we observed that they were better able to develop skills at representing the tangible characteristics of systems, the realm of DES, rather than conceptualizing the intangible properties of systems such as feedback processes, the realm of SD. Suggestions and reflections on teaching both simulation methods together are proposed.