Julia C. Lo

Using gaming simulation experiments to test railway innovations: implications for validity

Gaming simulation in the railway sector often uses the same conceptual model as in computer simulation, and enables operators to interact with this model during a simulation run. Therefore, gaming simulation validation poses different challenges. This paper aims to answer the question to what extent gaming simulation can be used as an experimental research setting, due to its loosely demarcated experimental features. Focusing on validity issues, we study five cases in which the Dutch railway sector used gaming simulation to test innovations in a controlled environment. The results show that in addition to traditional external validity issues, human game players inherently open up this controlled environment, bringing in many confounding variables. By signaling what the specific validity threats are, this paper strives to improve gaming simulation for testing innovations that tackle social and technical elements of a system.

Debriefing in gaming simulation for research: opening the black box of the non-trivial machine to assess validity and reliability

Gaming simulation allows for experiments with sociotechnical systems and has as such been employed in the railway sector to study the effects of innovations on robustness and punctuality. Systems work as non-trivial machines and the effect of an innovation on a dependent variable is potentially context, time and history dependent. However, several constraints inhibit the use of validity increasing measures such as repeated runs and increasing sample size. Based on a debriefing framework, insights from qualitative process research and six games with Dutch and UK railway traffic operators, we provide a guide on how to assess and increase reliability and validity. The key is for game players, observers and facilitators to open up the black box and thereby assessing how the innovation brought about any changes, if these changes are insensitive to changes in parameters and if the conclusions hold outside the game.

Explicit or Implicit Situation Awareness? Situation Awareness Measurements of Train Traffic Controllers in a Monitoring Mode

Railway traffic control faces the challenge of ensuring a high infrastructure capacity to maintain a constant train traffic flow. The current study assesses the situation awareness SA, as a predictor of decision-making, of train traffic controllers to gain novel insights in their cognition. This study puts emphasis on levels of implicit and explicit situation awareness in a monitoring mode, through measures of SAGAT, MARS and performance. A human-in-the-loop simulator, called the PRL game is used to simulate the workspace of train traffic controllers. Initial findings indicate rather low levels of explicit SA, on the contrary to higher subjective SA scores through MARS and observer ratings, and a high performance on the punctuality and unplanned stops of trains.

Gaming Simulation Design for Individual and Team Situation Awareness

Situation awareness is a key concept in understanding operator behaviour. Shortly, it can be described as knowing what is going on. For the past decades, human-in-the-loop simulators have been the traditional type of gaming simulations for studying or training situation awareness. The overall characteristic of gaming simulations is that they are a simulation of a system using gaming methods in which humans take part. Depending on a range of design choices, these gaming simulations take upon different visualizations and approaches to simulate aspects of the real world. Thus, a fundamental question is: what are the minimal requirements of a game to ensure natural levels of (team) situation awareness? This paper aims to capture and define the boundaries and limitations of gaming simulation design, in which the situation awareness of individuals and teams can be simulated and measured.

Measuring group situation awareness in a multiactor gaming simulation : A pilot study of railway and passenger traffic operators

This paper provides initial results for the gaming simulation design and measurement of group situation awareness (SA) through a low-tech multi-actor board gaming simulation for the Dutch railway operation. Group situation awareness is measured in this study, as railway operations consist of many dyadic teams and predominantly unique roles. Gaming simulations are herein defined as a simulation of a system using gaming methods, in which humans take part. This particular type of gaming simulation provides a relative fast and low-cost alternative to measure situation awareness in a multi-actor environment compared to the traditional human-in-the-loop-like simulator environment for SA measurements. However, due to variations in their abstraction level, exploration is needed on the validity for measurements of situation awareness in these environments. Thus, the main aim in this study is to determine whether, and if so, how, group SA can be measured in gaming simulations up to a quality that provides significant data for research. The results show potential for SA measurements in low-tech board gaming simulations, although improvement is needed with regards to the different validity types for gaming simulation. This may be achieved through the explicit use of gaming simulation design principles for SA. Future work should focus on further validation and research on the theoretical implications of group situation awareness.

A Trust-Based Situation Awareness Model

Trust is a social phenomenon that impacts the situation awareness of individuals and indirectly their decision-making. However, most of the existing computational models of situation awareness do not take interpersonal trust into account. Contrary to those models, this study introduces a computational, agent-based situation awareness model incorporating trust to enable building more human-like decision making tools. To illustrate the proposed model, a simulation case study has been conducted in the airline operation control domain. According to the results of this study, the trustworthiness of information sources had a significant effect on airline operation controller’s situation awareness.

Individual Markers of Resilience in Train Traffic Control: The Role of Operators’ Goals and Strategic Mental Models and Implications for Variation, Expertise, and Performance

Objective: The aim of this study was to examine individual markers of resilience and obtain quantitative insights into the understanding and the implications of variation and expertise levels in train traffic operators’ goals and strategic mental models and their impact on performance. Background: The Dutch railways are one of the world’s most heavy utilized railway networks and have been identified to be weak in system and organizational resilience. Method: Twenty-two train traffic controllers enacted two scenarios in a human-in-the-loop simulator. Their experience, goals, strategic mental models, and performance were assessed through questionnaires and simulator logs. Goals were operationalized through performance indicators and strategic mental models through train completion strategies. Results: A variation was found between operators for both self-reported primary performance indicators and completion strategies. Further, the primary goal of only 14% of the operators reflected the primary organizational goal (i.e., arrival punctuality). An incongruence was also found between train traffic controllers’ self-reported performance indicators and objective performance in a more disrupted condition. The level of experience tends to affect performance differently. Conclusion: There is a gap between primary organizational goals and preferred individual goals. Further, the relative strong diversity in primary operator goals and strategic mental models indicates weak resilience at the individual level. Application: With recent and upcoming large-scale changes throughout the sociotechnical space of the railway infrastructure organization, the findings are useful to facilitate future railway traffic control and the development of a resilient system.

Modeling Network Controller Decisions Based Upon Situation Awareness through Agent-Based Negotiation

The Dutch railway traffic control is in an urgent need for innovation and therefore turns to gaming simulation as a platform to test and train future configurations of the system. The presence of relevant participants is necessary to keep the fidelity of the gaming simulation high. Network controllers are often needed in such games, but are expensive, scarce, and often have limited action, thus making their involvement less than desirable. To overcome this, the current paper introduces the use of intelligent software agents to replace some roles. The cognitive construct of situation awareness is required to model the evaluation of an offer in a negotiation setting, in which a situation awareness model (SAM) is introduced for evaluating offers in complex and dynamic systems.

Measuring Mental Workload With Low-Cost and Wearable Sensors: Insights Into the Accuracy, Obtrusiveness, and Research Usability of Three Instruments:

The affordability of wearable psychophysiological sensors has led to opportunities to measure the mental workload of operators in complex sociotechnical systems in ways that are more objective and less obtrusive. This study primarily focuses on the sensors themselves by investigating low-cost and wearable sensors in terms of their accuracy, obtrusiveness, and usability for research purposes. Two sensors were assessed on their accuracy as tools to measure mental workload through heart rate variability (HRV): the E3 from Empatica and the emWave Pro from HeartMath. The BioPatch from Zephyr Technology, which is an U.S. Food and Drug Administration–approved device, was used as a gold standard to compare the data obtained from the other 2 devices regarding their accuracy on HRV. Linear dependencies for 6 of 10 HRV parameters were found between the emWave and BioPatch data and for 1 of 10 for the E3 sensor. In terms of research usability, both the E3 and the BioPatch had difficulty acquiring either sufficiently high data recording confidence values or normal distributions. However, the BioPatch output files do not require postprocessing, which reduces costs and effort in the analysis stage. None of the sensors was perceived as obtrusive by the participants.