Sanne Jensen

Simulation studies for the evaluation of health information technologies: experiences and results

It is essential for new health information technologies (IT) to undergo rigorous evaluations to ensure they are effective and safe for use in real-world situations. However, evaluation of new health IT is challenging, as field studies are often not feasible when the technology being evaluated is not sufficiently mature. Laboratory-based evaluations have also been shown to have insufficient external validity. Simulation studies seem to be a way to bridge this gap. The aim of this study was to evaluate, using a simulation methodology, the impact of a new prototype of an electronic medication management system on the appropriateness of prescriptions and drug-related activities, including laboratory test ordering or medication changes. This article presents the results of a controlled simulation study with 50 simulation runs, including ten doctors and five simulation patients, and discusses experiences and lessons learnt while conducting the study. Although the new electronic medication management system showed tendencies to improve medication safety when compared with the standard system, this tendency was not significant. Altogether, five distinct situations were identified where the new medication management system did help to improve medication safety. This simulation study provided a good compromise between internal validity and external validity. However, several challenges need to be addressed when undertaking simulation evaluations including: preparation of adequate test cases; training of participants before using unfamiliar applications; consideration of time, effort and costs of conducting the simulation; technical maturity of the evaluated system; and allowing adequate preparation of simulation scenarios and simulation setting. Simulation studies are an interesting but time-consuming approach, which can be used to evaluate newly developed health IT systems, particularly those systems that are not yet sufficiently mature to undergo field evaluation studies.

Clinical simulation

Use of clinical simulation in the design and evaluation of eHealth systems and applications has increased during the last decade. This paper describes a methodological approach for using clinical simulations in the design and evaluation of clinical information systems. The method is based on experiences from more than 20 clinical simulation studies conducted at the ITX-lab in the Capital Region of Denmark during the last 5 years. A ten-step approach to conducting simulations is presented in this paper. To illustrate the approach, a clinical simulation study concerning implementation of Digital Clinical Practice Guidelines in a prototype planning and coordination module is presented. In the case study potential benefits were assessed in a full-scale simulation test including 18 health care professionals. The results showed that health care professionals can benefit from such a module. Unintended consequences concerning terminology and changes in the division of responsibility amongst healthcare professionals were also identified, and questions were raised concerning future workflow across sector borders. Furthermore unexpected new possible benefits concerning improved communication, content of information in discharge letters and quality management emerged during the testing. In addition new potential groups of users were identified. The case study is used to demonstrate the potential of using the clinical simulation approach described in the paper.

The Role of Simulation in Clinical Information Systems Development

This paper describes the role of simulation involving end-users in Health Informatics. Simulation has long been established as a widely accepted method in clinical skills training. During the last decade simulation has also gained a place in the development and evaluation of clinical information systems. Simulation is especially well suited for the evaluation of human factors and organizational aspects in relation to application of information systems. In full-scale simulation tests it is possible to evaluate socio-technical interaction. A near to real life experience can be achieved by creating high fidelity environments. The paper discusses how simulation may be used during the lifecycle of clinical information systems, and the requirements on simulation fidelity in various situations. We recommend that simulation should get a more prominent role in the design and evaluation of clinical information systems.

Clinical simulation as a boundary object in design of health IT-systems

Healthcare organizations are very complex, holding numerous stakeholders with various approaches and goals towards the design of health IT-systems. Some of these differences may be approached by applying the concept of boundary objects in a participatory IT-design process. Traditionally clinical simulation provides the opportunity to evaluate the design and the usage of clinical IT-systems without endangering the patients and interrupting clinical work. In this paper we present how clinical simulation additionally holds the potential to function as a boundary object in the design process. The case points out that clinical simulation provides an opportunity for discussions and mutual learning among the various stakeholders involved in design of standardized electronic clinical documentation templates. The paper presents and discusses the use of clinical simulation in the translation, transfer and transformation of knowledge between various stakeholders in a large healthcare organization.

Fidelity in clinical simulation: How low can you go?

Clinical simulation may be used to identify user needs for context sensitive functionalities in e-Health. The objective with this paper is to describe how user requirements and use cases in a large EHR-platform procurement may be validated by clinical simulation using a very low-fidelity prototype without any existing test data. Instead of using test scenarios and use cases, the healthcare professionals who are participating in the clinical simulation are generating both scenario and patient data themselves. We found that this approach allows for an imaginative discussion, not restricted by known functionalities and limitations, of the ideal EHR-platform. Subsequently, we discuss benefits and challenges of using an extremely low fidelity environment and discuss the degree of fidelity necessary for conducting clinical simulation.

Boundary objects in clinical simulation and design of eHealth

Development and implementation of eHealth is challenging due to the complexity of clinical work practices and organizations. Standardizing work processes and documentation procedures is one way of coping with these challenges, and acceptance of these initiatives and acceptance of the clinical information system are vital for success. Clinical simulation may be used as “boundary objects” and help transferring of knowledge between groups of stakeholders and help to better understand needs and requirements in other parts of the organization. This article presents a case study about design of electronic documentation templates for nurses’ initial patient assessment, where clinical simulation was used as a boundary object and thereby achieved mutual clinical agreement on the content. Results showed that meetings prior to and in between workshops allowed all communities of practice an opportunity to voice their point of view and affect the final result. Implications of considering clinical simulations as boundary objects are discussed.

Benefits of a Clinical Planning and Coordination Module: A Simulation Study

Digital Clinical Practice Guidelines are commonly used in Danish health care. Planning and decision support are particularly important to patients with chronic diseases, who often are in contact with General Practitioners, Community Nurses and hospitals. In the Capital Region of Denmark the potential benefits of a planning and coordination module has been assessed in a full-scale simulation test including 18 health care professionals. The results showed that health care professionals can benefit from such a module. Furthermore unexpected new possible benefits concerning communication and quality management emerged during the test and potential new groups of users were identified.

Clinical Simulation: A Protocol for Evaluation of Mobile Technology

For mobile technology to be accepted at point of care in healthcare environments there is a need to demonstrate benefits whilst ameliorating the risks and challenges. To provide a standardised approach to evaluation of mobile technology a simulation protocol was developed to provide guidance for its use in healthcare environments. Simulated conditions provide the opportunity to assess intended and unintended consequences and identify potential workarounds when using technology. The protocol can also be used to demonstrate the importance of the development of digital professionalism by end-users prior to students entering the clinical practice setting. The mobile technology protocol was adapted from a health information systems protocol developed and used at the ITX Lab, Denmark for use in other simulation laboratories. Use case scenarios were developed to enable evaluation of mobile technology for mobile learning of nurses, nurse supervisors, students and patients. The scenarios can be used in a range of simulated environments including hospital bedside, outpatient clinic or community settings. A case study exemplar of a nurse and patient is included to demonstrate how the mobile technology protocol can be applied.