Anissa All

Assessing the effectiveness of digital game-based learning

In recent years, research into the effectiveness of digital game-based learning (DGBL) has increased. However, a large heterogeneity in methods for assessing the effectiveness of DGBL exist, leading to questions regarding reliability and validity of certain methods. This has resulted in the need for a scientific basis to conduct this type of research, providing procedures, frameworks and methods that can be validated. The present study is part of a larger systematic process towards the development of a standardized procedure for conducting DGBL effectiveness studies. In a first phase, the variety in methods that are used for sampling, implementation of the interventions, measures and data analysis were mapped in a systematic literature review using Cochrane guidelines. The present paper reflects the second stage, where this variety in elements are presented to experts in psychology and pedagogy by means of semi-structured interviews, in order to define preferred methods for conducting DGBL effectiveness studies. The interview was structured according to five dimensions that were used in the literature review: 1) participants (e.g., characteristics of the sample involved) 2) intervention (e.g., contents, format, timings and treatment lengths, intervention(s) in control group(s)) 3) methods (sampling, assignment of participants to conditions, number of testing moments) 4) outcome measures (e.g., instruments used to measure a certain outcome) and 5) data-analysis. The interviews were transcribed and analyzed using qualitative software package nVivo. Our results show that areas for improvement involve the intervention dimension and the methods dimension. The proposed improvements relate to implementation of the interventions in both the experimental and control group, determining which elements are preferably omitted during the intervention (such as guidance by the instructor, extra elements that consist of substantive information) and which elements would be aloud (e.g., procedural help, training session). Also, variables on which similarity between experimental and control condition should be attained were determined (e.g., time exposed to intervention, instructor, day of the week). With regard to the methods dimension, proposed improvements relate to assignment of participants to conditions (e.g., variables to take into account when using blocked randomized design), general design (e.g. necessity of a pre-test and control group) test development (e.g., develop and pilot parallel tests) and testing moments (e.g., follow up after minimum 2 weeks). In sum, the present paper provides best practices that cover all aspects of the study design and consist of game specific elements. While several suggestions have been previously made regarding research design of DGBL effectiveness studies, these often do not cover all aspects of the research design. Hence, the results of this study can be seen as a base for a more systematic approach, which can be validated in the future in order to develop a standardized procedure for assessing the effectiveness of DGBL that can be applied flexibly across different contexts. Questions regarding current assessment practices of DGBL.Best practices for effectiveness assessment of educational games are defined.Interviews with experts in pedagogy/psychology were conducted.Areas for improvement and suggestions for improvement are defined.More systematic approach for effectiveness assessment in this field is recommended.

Towards a conceptual framework for assessing the effectiveness of digital game-based learning

In recent years, interest has grown in the systematic assessment of the effectiveness of digital game-based learning (DGBL). A conceptual framework describing what effectiveness means in the context of DGBL and which are its subcomponents has hitherto been lacking however. Hence, the goal of this paper is to propose a conceptualization and operationalization of effectiveness rooted in social-cognitive theory. In order to identify desired outcomes and be able to operationalize effectiveness, focus groups were organized with three stakeholder groups following a user requirements analysis methodology.Results indicate that three categories of desired outcomes can be distinguished: learning, motivational and efficiency outcomes. For the different outcomes, different subcomponents can be extracted which can be organized hierarchically. Learning outcomes that are seen as relevant to the effectiveness of DGBL are 1) increased interest in the subject matter, 2) improvement in objective performance (e.g., in a test), and 3) transfer, referring to the players ability to apply acquired knowledge or skills to real-world situations. Relevant motivational outcomes concern 1) enjoyment, the extent to which playing the game evoked an enjoyable experience, and 2) increased motivation to learn using DGBL. Efficiency outcomes relevant to DGBL effectiveness, finally, are related to 1) time management and 2) cost-effectiveness. Overall, it can be stated that a DGBL intervention is effective when it achieves similar or higher scores compared to other instructional methods in relation to any of the above mentioned outcomes without significantly (in the common, not the statistical sense) diminishing any of the others. An operationalization of digital game-based learning effectiveness is provided.The operationalization is based on desired outcomes of digital game-based learning.A user requirements analysis among relevant stakeholders groups was conducted.Learning, motivational and efficiency outcomes can be distinguished.

Cognitive abilities, digital games and arithmetic performance enhancement

Besides entertainment, games have shown to have the potential to impact a broader variety of cognitive abilities. Research has consistently shown that several aspects in cognition such as visual short-memory, multitasking and spatial skills can be enhanced by game play. In a previous study, it was found that playing Monkey Tales, a game aimed at training arithmetic skills, helped second grade pupils to increase their accuracy in mental calculation as compared to paper exercises. In this follow up study we explore whether traditional methods and game training differ in terms of the cognitive processes that both are able to impact. We incorporated standardized measures of working memory and visuo-motor skills. Additionally, the mathematics game was modified and its contents extracted to allow precise comparison between the gaming and paper exercises condition. Thus each single math exercise, type of question (e.g., multiple choice), quantity and order was perfectly matched in the game training and the traditional training conditions. Gains in arithmetical performance, and self-reported measures of enjoyment were also investigated. We found some evidence suggesting that arithmetic performance enhancement induced by game play and paper exercises differ not only in terms of enjoyment but also of working memory capacity improvements. We explore differences between traditional methods and game training.In both training conditions all math exercises were perfectly matched.We used standardized measures of working memory and visuo-motor skills.Differences in terms of enjoyment and working memory capacity were observed.

An evaluation of the added value of co-design in the development of an educational game for road safety

This study explores the added value of co-design in addition to other innovation research methods in the process of developing a serious game design document for a road safety game. The sessions aimed at exploring 4 aspects of a location-based game experience: themes, game mechanics, mobile phone applications and locations for mini-games. In total, 72 adolescents between 15 and 18 years participated in five co-design sessions lead by a researcher and a professional game designer. The sessions provided useful input regarding the aspects the authors wished to explore. The sessions were especially useful in gathering input on scoring systems, ways to give instructions about next tasks and organizing level systems. In sum, their study indicates that co-design can be a source of additional ideas on top of other research methods such as state of the art analysis and expert consultation and thus lead to more effective interactive content creation.

Evaluating User Experience in Smart Home Contexts: A Methodological Framework

Similar to the concept of ‘smart city’, the phrase ‘smart home’ is being adopted by many businesses and stakeholders as a priority which recognizes the growing importance of digital technologies in the home context. However, few targeted methodologies exist that take into account the dynamic and interactive aspects of this environment when studying user experience. To date, the multi-disciplinary field of user experience studies, which investigates individuals perception about using a particular product, system or service, lacks a methodological and conceptual framework to study these smart homes that are connected to the internet and to a rapidly increasing amount of both sensors and actuators.

Human-Computer Interaction to Human-Computer-Context Interaction: Towards a Conceptual Framework for Conducting User Studies for Shifting Interfaces

Computer interfaces have been diversifying: from mobile and wearable technologies to the human body as an interface. Moreover, new sensing possibilities have allowed input to interfaces to go beyond the traditional mouse- and keyboard. This has resulted in a shift from manifest to latent interactions, where interactions between the human and the computer are becoming less visible. Currently, there is no framework available that fully captures the complexity of the multidimensional, multimodal, often latent interactions with these constantly shifting interfaces. In this manuscript, the Human-Computer-Context Interaction (HCCI) framework is proposed. This framework defines 5 relevant interaction levels to be considered during user research in all stages of the new product development process in order to optimize user experience. More specifically, the interaction context is defined in terms of user-object, user-user, user-content, user-platform and user-context interactions. The HCCI framework serves as a concrete tool to use in a new product development process by HCI researchers, designers, and developers and aims to be technology independent and future-proof. This framework is a preliminary suggestion to be matched against other innovation development projects and needs to be further validated.

Do Not Disturb: Psychophysiological Correlates of Boredom, Flow and Frustration During VR Gaming

Since the technology behind virtual reality (VR) is evolving rapidly and the number of VR applications is growing every year, research on the user’s experience of being in the virtual environment itself and the methodologies to measure these experiences becomes highly important. In this study, we apply the methodology of measuring attentional allocation by means of a dual-task paradigm to the topic of VR gaming. The idea is to ask participants to react to oddball sounds (secondary task), pulling attention away from the primary task (the game). The behavioral (reaction time and accuracy) and neural response (P300 component) to these oddball sounds then tells us something about indirect attentional allocation to the game and possibly the experience of flow. In order to check the latter, we created experiences of boredom, flow and frustration by manipulating the mechanics of the game. In addition, we were interested in other psychophysiological correlates like brain oscillations and average heart rate and whether these differed between gaming with a regular and a VR set-up. Although we were not able to accurately induce feelings of boredom, flow and frustration and could not replicate previous studies showing increased reaction times for oddball sounds during flow, we did find a decreased P300 and more high-frequency brain oscillations in VR compared to regular gaming (indicating more attention to the game). Together, this suggests that psychophysiological measures are promising tools to quantify attentional allocation in VR, but more research is needed to clarify whether and how this translates to flow.