Geertje Bekebrede

The research and evaluation of serious games: Toward a comprehensive methodology

The authors present the methodological background to and underlying research design of an ongoing research project on the scientific evaluation of serious games and/or computer-based simulation games (SGs) for advanced learning. The main research questions are: (1) what are the requirements and design principles for a comprehensive social scientific methodology for the evaluation of SGs?; (2) to what extent do SGs contribute to advanced learning?; (3) what factors contribute to or determine this learning?; and (4) to what extent and under what conditions can SG-based learning be transferred to the real world? In the Netherlands between 2005 and 2012, several hundred SG sessions with 12 SGs were evaluated systematically, uniformly and quantitatively to create a dataset, which comprises data on 2488 respondents in higher education or work organizations. The authors present the research model, the quasi-experimental design and the evaluation instruments. This focus in this paper is on the methodology and dataset, which form a sound foundation for forthcoming publications on the empirical results. [ABSTRACT FROM AUTHOR]

Understanding Complexity

Complexity is increasingly tied into systems of societies, technologies and organizations. As these systems are characterized by non-linear interactions, it becomes more and more difficult to manage, educate, understand and predict their processes and outcomes. This symposium presents cases of a variety of simulation games that facilitate in these different types of systems; it also examines the theoretical challenges of the design of simulation games for such complex systems. The examples show how both computer-based and traditional board games can be used in various domains to analyze and design complex processes such as asset management, manufacturing procedures, railway infrastructure innovations, and land use development. The case studies presented in this symposium introduce simulation games as alternative solutions to complex organizational and even societal problems.

Learning in a game-based virtual environment: a comparative evaluation in higher education

The authors define the requirements and a conceptual model for comparative evaluation research of simulation games and serious games (SGs) in a learning context. A first operationalisation of the model was used to comparatively evaluate a suite of 14 SGs on varying topics played between 2004 and 2009 in 13 institutes of higher education in the Netherlands. The questions in this research were: what is the perceived learning effectiveness of the games and what factors explain it? How can we comparatively evaluate games for learning? Data were gathered through pre- and post-game questionnaires among 1000 students, leading to 500 useful datasets and 230 complete datasets for analysis (factor analysis, scaling, t-test and correlation analysis) to give an explorative, structural model. The findings are discussed and a number of propositions for further research are formulated. The conclusion of the analysis is that the students’ motivation and attitudes towards game-based learning before the game, their actual enjoyment, their efforts during the game and the quality of the facilitator/teacher are most strongly correlated with their learning satisfaction. The degree to which the experiences during the game were translated back into the underlying theories significantly determines the students’ learning satisfaction. The quality of the virtual game environment did not matter so much. The authors reflect upon the general methodology used and offer suggestions for further research and development.

Critical infrastructures: a review from a complex adaptive systems perspective

Critical infrastructures are complex systems that may be studied in large interdisciplinary teams. Interaction between researchers in these interdisciplinary teams can suffer from the lack of a shared view on the object of study or the meaning of words. To facilitate the understanding of complex infrastructure systems, we reviewed literature on complex systems from different fields. We created a three-layered framework that clarifies the relationships between the properties and the different levels of complex systems. We illustrated our framework with an example of a transportation system that shows how this complex system may be understood in terms of the framework.

How to use a systems diagram to analyse and structure complex problems for policy issue papers

Many policy problems are complex in the sense that natural, technological, social and human elements interact. Problem exploration and structuring are essential as a basis for deliberate and focused approaches towards problem resolution. The results of problem exploration efforts can be laid down in the form of a policy issue paper. We have developed a systemic, stepwise approach, which has been elaborated and taught for over a decade to hundreds of students. This seven-step approach centers on the construction of a system diagram as a means to provide structure to the conceptualisation of a complex problem situation. The approach is based on a conscious combination of existing relatively straightforward analytical methods including objectives hierarchy, means-ends analysis, causal diagramming, stakeholder analysis, and contextual scenarios. The obtained insights are then summed up in a policy issue paper, which is the basis for further planning and research.

Learning in Single-Versus Multiplayer Games

From the observations of successful entertainment games, it is hypothesized that implementing a single-player option may require a different approach from that in a multiplayer option, in terms of game design. To find out whether this could be true and to understand what the specific approaches could entail for educational games, three separate yet related investigations were conducted to examine single- versus multiplayer games: an investigation of two educational games designed and evaluated by the authors, a theoretical investigation from a game and learning perspective, and an empirical investigation of 23 case studies. From these three investigations, it turned out that a “single-player approach” is data intensive, has formal rules, and uses direct transfer and individual learning. On the other hand, a “multiplayer approach” is less straightforward. From a game perspective, it can be characterized as process intensive and having social rules. When related to learning, however, it could be positioned on any dimension. This exploration shows that the approaches differ to a large extent and that designers have to find a fit between what option they choose and approach they take.

Understanding Complex Systems Through Mental Models and Shared Experiences

Background. Our world today is made up of complex, socio-technical systems like large infrastructure systems, global logistic chains and information networks. Simulation games represent one way to enable participation and governance of such systems. Aim. The current study explores the development of mental models and shared experiences of professionals and students through a simulation game. Method. The simulation game SIMPORT-MV2 focused on the development of a new port area in the Port of Rotterdam, The Netherlands. Insights in the perceived development of mental models were measured through questionnaires and through observations in the debriefing. Likewise, observations in the debriefing were obtained for measurements of shared experiences. Results. Results showed that participants perceived positive learning on their declarative, procedural and strategic knowledge (mental model) about the port project. A significant difference was found for both the overall perceived knowledge and the strategic knowledge between students and professionals. Furthermore, qualitative findings showed that players are enabled to probe designs of, to gain insights into, and to develop an understanding of complex systems in a safe environment through shared experiences. Conclusions. This study shows that it is possible to design a simulation game that represents such a complex socio-technical system based on a layered approach of agent, network and system behavior.

A Scientific Foundation of Simulation Games for the Analysis and Design of Complex Systems

Background. The use of simulation games for complex systems analysis and design has been acknowledged about 50 years ago. However, articles do not combine all salient factors for successful simulation games, and often stem from a clear view of one particular field of science only. With combining multiple disciplines, connect analysis and design as well as research and practice, we provide deep insights in design and use of simulation games. Aim. This article analyzes the design and evaluation process of a variety of game-based projects and activities, using existing scientific concepts and approaches, in order to establish games as a valid research tool. Our focus lies on the approach towards the use of games as design instrument; using them as an intervention in a larger, complex context, in order to design this context. With our contribution, we aim at providing insights and recommendations on the design and use of games as valid research tools, the limitations of this use, possible pitfalls, but also best practices. Method. We carried out a literature review of related work to identify the most important scientific concepts related to our approach of game design. Further use of combined quantitative and qualitative case study analyses highlights the design process and results of our own game studies. Results. The analyses yielded a consolidated conceptualization of simulation games as research instruments in complex systems analysis and design. The results also include methods for the evaluation of simulation games, additional evaluation methods, and limitations to use simulation games as research instruments. Conclusions. We propose guidelines for using simulation games as research instruments that may be of value to practitioners and scientists alike. Recommendation. We recommend practitioners and scientists to apply the guidelines presented here in their efforts to analyze and design complex systems.

Understanding complex infrastructure systems: the case of SimPort-MV2

Planning and design of infrastructures is a complicated process, which is caused by the complexity of the subject matter. The theory of Complex Adaptive Systems offers insights that can be of use in the design process. However, translating theory into practice is difficult. Gaming is proposed as a tool to bridge this gap by simulating complex adaptive systems. The objective of the paper is to analyze whether gaming can simulate the behavior of a complex adaptive system. We use the game SimPort-MV2 as a case study. Based on this example, we conclude that games can simulate complex adaptive systems. However, we raise specific concerns about the validity of the outcome and use of the game.

Design Considerations for Building a Scalable Digital Version of a Multi-player Educational Board Game for a MOOC in Logistics and Transportation

With more flexible and large-scale learning environments, new design requirements for games emerge. Massive Open Online Courses (MOOCs) are one of the most important innovations in the learning field. Still, it is a challenge to motivate learners and to keep them motivated in such huge learning environments. To address this challenge, we redesigned a board game targeting at an integrated view on disruption and communication management in an intermodal transportation situation. From the redesign, we have learned that an online game works better with fewer roles, requires immediate feedback, and an engaging way of challenge to keep players motivated. Our findings can inform the design of games for large groups of players in an online environment.