Harald Warmelink

The research and evaluation of serious games: Toward a comprehensive methodology

The authors present the methodological background to and underlying research design of an ongoing research project on the scientific evaluation of serious games and/or computer-based simulation games (SGs) for advanced learning. The main research questions are: (1) what are the requirements and design principles for a comprehensive social scientific methodology for the evaluation of SGs?; (2) to what extent do SGs contribute to advanced learning?; (3) what factors contribute to or determine this learning?; and (4) to what extent and under what conditions can SG-based learning be transferred to the real world? In the Netherlands between 2005 and 2012, several hundred SG sessions with 12 SGs were evaluated systematically, uniformly and quantitatively to create a dataset, which comprises data on 2488 respondents in higher education or work organizations. The authors present the research model, the quasi-experimental design and the evaluation instruments. This focus in this paper is on the methodology and dataset, which form a sound foundation for forthcoming publications on the empirical results. [ABSTRACT FROM AUTHOR]

Learning in a game-based virtual environment: a comparative evaluation in higher education

The authors define the requirements and a conceptual model for comparative evaluation research of simulation games and serious games (SGs) in a learning context. A first operationalisation of the model was used to comparatively evaluate a suite of 14 SGs on varying topics played between 2004 and 2009 in 13 institutes of higher education in the Netherlands. The questions in this research were: what is the perceived learning effectiveness of the games and what factors explain it? How can we comparatively evaluate games for learning? Data were gathered through pre- and post-game questionnaires among 1000 students, leading to 500 useful datasets and 230 complete datasets for analysis (factor analysis, scaling, t-test and correlation analysis) to give an explorative, structural model. The findings are discussed and a number of propositions for further research are formulated. The conclusion of the analysis is that the students’ motivation and attitudes towards game-based learning before the game, their actual enjoyment, their efforts during the game and the quality of the facilitator/teacher are most strongly correlated with their learning satisfaction. The degree to which the experiences during the game were translated back into the underlying theories significantly determines the students’ learning satisfaction. The quality of the virtual game environment did not matter so much. The authors reflect upon the general methodology used and offer suggestions for further research and development.

A frame‐reflective discourse analysis of serious games

The authors explore how framing theory and the method of frame-reflective discourse analysis provide foundations for the emerging discipline of serious games (SGs) research. Starting with Wittgensteins language game and Berger and Luckmanns social constructivist view on science, the authors demonstrate why a definitional or taxonomic approach to SGs is problematic and unfruitful. Using Goffmans frame analysis as an alternative, they construct four frames, with sample illustrations, demonstrating the different ways in which the utility of games for society, business and politics is considered. These are SGs as: (1) tool (therapy, drug), (2) innovation (economic utility), (3) persuasion (idea, belief) and (4) self-organization (complexity). The frames are based upon different values and perceive different impacts of games in society, business and politics. The authors discuss the implications of their approach.

Oasistan: An Intercultural Role-Playing Simulation Game to Recognize Cultural Dimensions

Aim. Although cultural dimensions theory is a topical strand of quantitative cultural research, few intercultural simulation games use it. We present the design and review of the application of OAS...Aim. Although cultural dimensions theory is a topical strand of quantitative cultural research, few intercultural simulation games use it. We present the design and review of the application of OASISTAN, an intercultural role-playing simulation game that is specifically based on cultural dimensions theory. Method. OASISTAN was first designed in 1999 for use in Master’s courses on cross-cultural management at Delft University of Technology in the Netherlands, attracting 20-23 year old students with a Bachelor degree in engineering and from various cultural backgrounds. Since its first design the game has been played approximately 45 times at Delft University of Technology in the Netherlands and three times at Harbin Institute of Technology in China in the years 2006-2008. We reviewed their experiences designing and facilitating OASISTAN since 1999. Results. The game has a no-tech role-play design and revolves around the geopolitically complex region of the Caspian Sea, specifically the fictional country of ‘Oasistan’. The game consists of students forming small teams of Oasistani, Western and non-Western public/private actors collaborating with each other to try and reach the common goal of oil exploration and production in this country. In total 15-30 students were involved. We found that OASISTAN allowed its players not only to intensely experience the difficulty and awkwardness of being confronted with cultural differences, but also to interpret and understand these differences through cultural dimensions. Students who played OASISTAN identified ten out of the 12 dimensions by Maleki and De Jong. The two dimensions that students were not able to identify are uncertainty avoidance and collaborativeness. Conclusion. OASISTAN shows how a game design field (i.e., intercultural simulation gaming) can be reinvigorated in light of new or updated scientific theories pertaining to the field’s subject matter (i.e., cultural dimensions). Several opportunities for future research are identified.

AMELIO: Evaluating the Team-building Potential of a Mixed Reality Escape Room Game

The authors investigate the potential of Mixed Reality (MR) games for team building and assessment. The AMELIO game was designed for a highly immersive MR lab. The game is a multi-player team challenge based on the concept of an escape room, staged in a space colony emergency situation. An explorative empirical pre-post measurement study was carried out to establish whether playing AMELIO influences team cohesiveness. Ten teams of three played AMELIO and filled out pre- and post-game questionnaires with validated measurements of team cohesiveness and mediating factors related to team composition, game experience and team dynamics. The findings show a positive and significant increase in team cohesiveness, with stronger effects for teams with lower pre-game familiarity. In terms of game experience and team dynamics, audio aesthetics and empathy proved to be significant mediating factors. This aids game validation and improvement, and understanding and guiding the team building process.

The Game Jam as a Format for Formal Applied Game Design and Development Education

This paper introduces the design and results of an applied game jam integrated in a game design and development curriculum, which took place February 2016 at HKU University of the Arts Utrecht in the Netherlands. The game jam followed a four-phased structure over the course of four days. Forty-five participants shared their demographics, baseline competency, attitude towards game jams, and learning expectations in a pre-jam questionnaire. In a post-jam questionnaire they assessed their collaboration and learning outcomes. Results are generally positive, although some measures of collaboration constructs were unreliable. Nevertheless, a game jam is in principal a format worthy of introduction into formal game design and development education. Several implications and next steps are discussed.

Using the Revised Bloom Taxonomy to Analyze Psychotherapeutic Games

Most of the scientific literature on computer games aimed at offering or aiding in psychotherapy provides little information on the relationship between the game’s design and the player’s cognitive processes. This article investigates the use of Bloom’s taxonomy in describing a psychotherapeutic game in terms of knowledge level and cognitive processing. It introduces the Revised Bloom Taxonomy and applies this to five psychotherapeutic games (Personal Investigator, Treasure Hunt, Ricky and the Spider, Moodbot, and SuperBetter) in a two-round procedure. In the first round consensus was reached on the Player Actions with Learning Objectives (PALOs) in each game. The second round sought to determine what level of knowledge and cognitive processing can be attributed to the PALOs by placing them in the taxonomy. Our low intercoder reliability in the second round indicates that Bloom’s Revised Taxonomy is not suitable to compare and contrast content between games.

Maritime Spatial Planning – A Board Game for Stakeholder Involvement

The Maritime Spatial Planning (MSP) Challenge game: Short Sea Shipping (SSS) Edition is a table-top strategy board game, designed for policy-makers and stakeholders involved in MSP, short-sea shipping and the Blue Economy. It is a ‘serious game’, allowing the development of a better understanding of the issues involved in MSP through creative and imaginative role playing, taking into account the relevant professional and personal experience of the players. The authors present and discuss the use of the MSP Challenge board game to test how, and to what extent, the concept can help stakeholders understand Maritime Spatial Planning.

Looking for Metacognition

Most of scientific literature on computer games aimed at offering or aiding in psychotherapy has little information on how the game exactly relates to the relatively recent development of the ‘third wave’ of behavioural psychotherapy, which includes metacognition. This paper first introduces metacognition and subsequently studies five cases of psychotherapeutic games (Personal Investigator, Treasure Hunt, Ricky and the Spider, Moodbot and SuperBetter) by looking at them through the lens of Blooms’ Revised Taxonomy of Knowledge. The paper offers design recommendations for future (metacognitive) psychotherapeutic games.

Get It Right! Introducing a Framework for Integrating Validation in Applied Game Design

Validation of applied games is generally a lengthy and costly process, usually done after the game has almost been completed. This article argues that validation throughout the design process is vital for creating games that achieve their desired objective. It proposes a framework to integrate validation throughout the applied game design process. Iterative design and design fidelity levels form the foundation of the framework. In the framework four types of design fidelity specifications, concepts, prototypes, and integrations are linked to five types of validity content, face, construct, concurrent and predictive validity, which lead to eight applicable validation tools and techniques. The framework is a starting point for further research into applied game validation.