Andre W. Kushniruk

Cognitive and usability engineering methods for the evaluation of clinical information systems

Increasingly healthcare policy and decision makers are demanding evidence to justify investments in health information systems. This demand requires an adequate evaluation of these systems. A wide variety of approaches and methodologies have been applied in assessing the impact of information systems in health care, ranging from controlled clinical trials to use of questionnaires and interviews with users. In this paper we describe methodological approaches which we have applied and refined for the past 10 years for the evaluation of health information systems. The approaches are strongly rooted in theories and methods from cognitive science and the emerging field of usability engineering. The focus is on assessing human computer interaction and in particular, the usability of computer systems in both laboratory and naturalistic settings. The methods described can be a part of the formative evaluation of systems during their iterative development, and can also complement more traditional assessment methods used in summative system evaluation of completed systems. The paper provides a review of the general area of systems evaluation with the motivation and rationale for methodological approaches underlying usability engineering and cognitive task analysis as applied to health information systems. This is followed by a detailed description of the methods we have applied in a variety of settings in conducting usability testing and usability inspection of systems such as computer-based patient records. Emerging trends in the evaluation of complex information systems are discussed.

Comparing Approaches to Measuring the Adoption and Usability of Electronic Health Records: Lessons Learned from Canada, Denmark and Finland

Internationally, the adoption of health information technology is increasing. However, a number of issues have complicated the adoption of electronic health records (EHRs). In addition to adoption issues, it is becoming increasingly recognized that healthcare providers face a variety of usability issues. In this paper, we consider approaches that have been taken to assess both adoption and usability of EHRs in Canada, Denmark and Finland. Although all three countries deploy surveys to assess adoption, the approach and focus of the surveys differs across the countries. In Denmark and Finland, these surveys are dedicated to assessing information technology (IT) usage; while in Canada, questions about IT usage are part of a larger physician survey. Regarding usability, approaches vary considerably. In Finland, the approach includes a national survey about EHR usability. In Canada, ratings of system usability are reported regionally on web sites; while in Denmark, regional study results are reported based on evaluation of commercial products. This paper highlights the need to consider different evaluation approaches internationally.

Evaluation in the design of health information systems: application of approaches emerging from usability engineering

This paper examines the role of evaluation in the design of health care information systems. A framework is presented for considering evaluation in the context of software development processes, in particular, the systems development life cycle (SDLC). Variations on standard design methodologies are then discussed, including methods based on rapid development and continual evaluation of prototype systems. Usability testing is presented as a key method for conducting evaluations during iterative system development. The emergence of design methodologies, where evaluation is viewed as a central part of the development cycle is also discussed. Evaluation methodologies are then considered along a continuum, ranging from studies involving a high degree of experimental control to observational approaches. A full cycle approach to evaluation of health care systems is argued for, involving deployment of new methods across the SDLC. Implications for future work exploring the integration of design and evaluation processes in health informatics are discussed.

Unpacking the potential of educational gaming: A new tool for gaming research

The article begins by reviewing the theoretical bases for the contention that advanced computer-based educational gaming can provide powerful learning experiences, and overviews the limited research on the use of such games. Although studies to date have generally supported their value, most of the published investigations have methodological limitations. Critical process data are typically not collected, and unreliable student and teacher self-reports are heavily relied on in evaluating the educational efficacy of many games. To address these and other limitations, the authors have developed research software that can remotely and unobtrusively record screen activity during game play in classroom settings together with synchronized audio of player discussion. A field trial of this data collection system in which 42 college students were studied as they played a coursework-related Web-based learning game is described, and the article discusses how the trial outcomes concretely demonstrate the methodological advantages the tool offers researchers.

Empowering patients through social media: The benefits and challenges:

This article explores the range of social media platforms used by patients and examines the benefits and challenges of using these tools from a patient perspective. A literature review was performed to investigate the use of social media technology by patients. The MEDLINE database was searched using the terms “social media” and “patient.” The search was conducted in September 2012 and yielded 765 abstracts. Initially, 63 abstracts were selected. All articles dating from 2004 through 2012 were included. Only 12 articles were found to be relevant for the purposes of the review. The results of this research found that there appears to be an increase in the use of social media by patients across the healthcare spectrum. The research indicates a promising future for the use of social media by patients; however, evidence related to the efficacy and effectiveness of social media is currently limited. Various challenges have also been identified relating to privacy and security concerns, usability, the manipulation of identity, and misinformation. The use of social media technology is an emerging trend for patients who are seeking health information. Conclusions are that such technology holds promise for improving patient engagement and empowerment and community building. Social media has a future in healthcare, especially with regard to patient engagement and empowerment; however, there are several challenges to overcome before the technology can achieve its potential.

PERSIVAL, a system for personalized search and summarization over multimedia healthcare information

In healthcare settings, patients need access to online information tha t can help them understand their medical situation. Physicians need information that is clinically relevant to an individual patient. In this paper, we present our progress on developing a system, PERSIVAL, that is designed to provide personalized access to a distributed patient care digital library. Using the secure, online patient records at New York Presbyterian Hospital as a user model, PERSIVALs components tailor search, presentation and summarization of online multimedia information to both patients and healthcare providers.

National efforts to improve health information system safety in Canada, the United States of America and England

OBJECTIVE In this paper we review progress as well as challenges encountered in Canada, the United States and England with regard to ensuring safety of health information technology. METHOD A review of major programs and initiatives for ensuring safety of health information technology in the three countries was conducted. Published literature and Web resources from national programs were reviewed for relevant information. RESULTS It was found that in all three countries the issue of technology-induced error has been recognized as being of critical importance. The three countries have developed approaches for dealing with the issue that have some commonalities; however, they are at varying different stages of maturity, with England having the longest standing and most well developed safety programs, while Canada and the United States are at earlier stages. The types of approaches employed have included work on developing standards related to usability and interface design, certifications, directives from regulatory bodies, educational initiatives in health information technology (HIT) safety as well as research into safer HIT design and implementation methods. CONCLUSIONS HIT promises to lead to improved patient safety. However, it has become recognized that if not designed and deployed appropriately, such systems can lead to new types of errors. Based on this recognition, a variety of initiatives are being undertaken in Canada, the United States and England to promote the safe design, procurement and deployment of HIT. It is concluded that improved approaches to system design, testing, regulation, error reporting, safety education and cross-country collaboration will be needed to further promote safer HIT.

A framework for evaluating usability of clinical monitoring technology

Technology design is a complex task, and acceptability is enhanced when usability is central to its design. Evaluating usability is a challenge for purchasers and developers of technology. We have developed a framework for testing the usability of clinical monitoring technology through literature review and experience designing clinical monitors. The framework can help designers meet key international usability norms. The framework includes these direct testing methods: thinking aloud, question asking, co-discovery, performance and psychophysiological measurement. Indirect testing methods include: questionnaires and interviews, observation and ethnographic studies, and self-reporting logs. Inspection, a third usability testing method, is also included. The use of these methods is described and practical examples of how they would be used in the development of an innovative monitor are given throughout. This framework is built on a range of methods to ensure harmony between users and new clinical monitoring technology, and have been selected to be practical to use.

Enhancing the Effectiveness of Consumer-Focused Health Information Technology Systems Through eHealth Literacy: A Framework for Understanding Users' Needs

Background eHealth systems and applications are increasingly focused on supporting consumers to directly engage with and use health care services. Involving end users in the design of these systems is critical to ensure a generation of usable and effective eHealth products and systems. Often the end users engaged for these participatory design processes are not actual representatives of the general population, and developers may have limited understanding about how well they might represent the full range of intended users of the eHealth products. As a consequence, resulting information technology (IT) designs may not accommodate the needs, skills, cognitive capacities, and/or contexts of use of the intended broader population of health consumers. This may result in challenges for consumers who use the health IT systems, and could lead to limitations in adoption if the diversity of user attributes has not been adequately considered by health IT designers. Objective The objective of this paper is to propose how users’ needs and competences can be taken into account when designing new information and communications technology solutions in health care by expanding the user-task-context matrix model with the domains of a new concept of eHealth literacy. Methods This approach expands an existing method for supporting health IT system development, which advocates use of a three-dimensional user-task-context matrix to comprehensively identify the users of health IT systems, and what their needs and requirements are under differing contexts of use. The extension of this model involved including knowledge about users’ competences within the seven domains of eHealth literacy, which had been identified based on systematic engagement with computer scientists, academics, health professionals, and patients recruited from various patient organizations and primary care. A concept map was constructed based on a structured brainstorm procedure, card sorting, and computational analysis. Results The new eHealth literacy concept (based on 7 domains) was incorporated as a key factor in expanding the user-task-context matrix to describe and qualify user requirements and understanding related to eHealth literacy. This resulted in an expanded framework and a five-step process, which can support health IT designers in understanding and more accurately addressing end-users’ needs, capabilities, and contexts to improve effectiveness and broader applicability of consumer-focused health IT systems. It is anticipated that the framework will also be useful for policy makers involved in the planning, procuring, and funding of eHealth infrastructure, applications, and services. Conclusions Developing effective eHealth products requires complete understanding of the end-users’ needs from multiple perspectives. In this paper, we have proposed and detailed a framework for modeling users’ needs for designing eHealth systems that merges prior work in development of a user-task-context matrix with the emerging area of eHealth literacy. This framework is intended to be used to guide design of eHealth technologies and to make requirements explicitly related to eHealth literacy, enabling a generation of well-targeted, fit-for-purpose, equitable, and effective products and systems.

The role of organizational context and individual nurse characteristics in explaining variation in use of information technologies in evidence based practice

BackgroundThere is growing awareness of the role of information technology in evidence-based practice. The purpose of this study was to investigate the role of organizational context and nurse characteristics in explaining variation in nurses’ use of personal digital assistants (PDAs) and mobile Tablet PCs for accessing evidence-based information. The Promoting Action on Research Implementation in Health Services (PARIHS) model provided the framework for studying the impact of providing nurses with PDA-supported, evidence-based practice resources, and for studying the organizational, technological, and human resource variables that impact nurses’ use patterns.MethodsA survey design was used, involving baseline and follow-up questionnaires. The setting included 24 organizations representing three sectors: hospitals, long-term care (LTC) facilities, and community organizations (home care and public health). The sample consisted of 710 participants (response rate 58%) at Time 1, and 469 for whom both Time 1 and Time 2 follow-up data were obtained (response rate 66%). A hierarchical regression model (HLM) was used to evaluate the effect of predictors from all levels simultaneously.ResultsThe Chi square result indicated PDA users reported using their device more frequently than Tablet PC users (p = 0.001). Frequency of device use was explained by ‘breadth of device functions’ and PDA versus Tablet PC. Frequency of Best Practice Guideline use was explained by ‘willingness to implement research,’ ‘structural and electronic resources,’ ‘organizational slack time,’ ‘breadth of device functions’ (positive effects), and ‘slack staff’ (negative effect). Frequency of Nursing Plus database use was explained by ‘culture,’ ‘structural and electronic resources,’ and ‘breadth of device functions’ (positive effects), and ‘slack staff’ (negative). ‘Organizational culture’ (positive), ‘breadth of device functions’ (positive), and ‘slack staff ‘(negative) were associated with frequency of Lexi/PEPID drug dictionary use.ConclusionAccess to PDAs and Tablet PCs supported nurses’ self-reported use of information resources. Several of the organizational context variables and one individual nurse variable explained variation in the frequency of information resource use.