Laura G. Militello

Characterizing a Naturalistic Decision-Making Phenomenon Loss of System Resilience Associated With Implementation of New Technology

We describe a phenomenon viewed through the conceptual lens of a naturalistic decision-making perspective: a loss of system resilience, due to increased difficulty in performing macrocognition functions, associated with the implementation of new information technology. Examples of the phenomenon collected in a targeted literature review are characterized by stakeholder groups, technology, typical changes in workflow before and after implementation, and potential impacts on macrocognition and patient outcomes for four clinical care environments. The loss of system resilience is due to increased difficulty in performing macrocognition functions: (a) sensemaking due to less effective cognitive warm-up and collaborative framing strategies, (b) detecting events due to missing trends in data and changes to orders, and (c) coordinating due to less clinical knowledge during scheduling and updating information and less effective cross-checks. Potential impacts to patient safety include an increase in unnecessary care, missed care, delays in diagnoses and treatment, redundant care, inaccurate diagnoses, medication errors, and adverse events. We recommended future conceptually driven research in other complex, sociotechnical settings in order to develop useful metrics and reduce the risk of incurring undesirable and unnecessary impacts on cognitive work associated with new technology.

THE FORGOTTEN HISTORY OF COGNITIVE TASK ANALYSIS

Many reports of recent research on topics in cognitive systems engineering describe their methods by distinguishing cognitive task analysis from “traditional” or “behavioral” task analysis. A hallmark of modern cognitive task analysis (CTA) methods is that they place primary focus on understanding the cognitive demands of a task and the knowledge and strategies that underlie performance. While cognitive task analysis may have seemed to be a revolutionary approach introduced in recent years, a review of the history of task analysis reveals many things that have been lost in modern treatments of the history of human factors. This presentation will review some highlights of this forgotten history, including the ideas and methods of the Psychotechnicians, the earliest industrial psychologists, the Taylorists, and others who contributed to modern CTA. Task analysis never lacked cognitive categories. Even microscale “time and motion” studies involved the analysis of the work of domain experts. Basic ideas of human -machine systems and of complexity also appear in some of the earliest literatures — industrial psychology of the first decades of the 20th century.

Design concepts to support management of outpatient consultations in the Veterans Health Administration

This poster describes a project to improve understanding of the challenges associated with managing consultations in the Veterans Health Administration (VHA). We conducted interviews and observations with primary care providers and specialists at two VHA facilities. Using qualitative analysis, we identified cognitive requirements, challenges associated with each, and design seeds. During the poster session, we will present design concepts exploring interventions to support management of consultations.

The Divide between Applied Research and Operations in Health Information Technology Advancement Reducing the Gap

This practice-oriented presentation describes several factors that contribute to the divide between applied research and operations for health information technology (HIT). Factors and corresponding recommendations were drawn from experiences working in both research and operations positions as human factors professionals within a large healthcare system, the Veterans Health Administration (VHA). We offer specific recommendations for addressing each of these factors to help reduce the gap. Although our findings and recommendations were derived from our VHA experience, we expect many of the lessons learned are relevant to other major healthcare organizations. Indeed, we strongly suspect that the factors presented represent fundamental differentiators between researchers and operational-based professionals. Understanding these differences and findings ways to bridge them is essential for advancing HIT.

Discussion Panel How to Recognize a “Good” Cognitive Task Analysis?

Cognitive Task Analysis (CTA) has become part of the standard tool set of cognitive engineering. CTAs are routinely used to understand the cognitive and collaborative demands that contribute to performance problems, the basis of expertise, as well as the opportunities to improve performance through new forms of training, user interfaces, or decision aids. While the need to conduct CTAs has become well established, there is little in the way of available guidance with respect to ‘best practice’ for how to conduct a CTA or how to evaluate the quality of a CTA that has been conducted by others. This is an important gap as the range of consumers of CTAs is expanding to include program managers and regulators who may need to make decisions based on CTA findings. This panel brings together some of the leaders in development and application of CTA methods to address the question: Given the variety of methods available, and the lack of rigid guidance on how to perform a CTA, how does one judge the quality of a CTA?” The goal of the panel is to explore points of consensus with respect to ‘best practice’ in conducting and evaluating a CTA, in spite of differences in particular CTA methods, as well as to draw insights from unique and provocative perspectives.

Clinical Reminders: Explaining Variability in Adoption by Nurses and Physicians at Four Outpatient Clinics

Computerized clinical reminders (CRs) can improve compliance with evidence-based practices in preventive care and chronic disease management. However, observational research is needed to explain the known variability in the use of CRs. We conducted ethnographic observations of nurses and providers using CRs in outpatient primary care clinics for two days in each of four geographically distributed Veterans Administration (VA) medical centers. We found that use of CRs was impeded by (1) lack of coordination between nurses and providers, (2) using the reminders while not with the patient, impairing data acquisition and/or implementation of recommended actions, (3) workload, (4) lack of CR flexibility, and (5) poor interface usability. We discuss implications of these findings for CR system design and provide recommendations for redesign to facilitate the effective use of CRs.

A Systematic Method for Using Critical Decision Method Interviews to Generate a Garden Path Scenario for Human-in-the-Loop Simulations for Training and Evaluation of New Technologies

Strategies for assessing macrocognitive skill must be tailored to work domains and specific tasks. This paper describes one approach for leveraging the critical decision method, a structured interview technique, to develop an assessment method for new technologies. The approach uses garden path scenarios, where initial cues suggest a different (false prime) diagnosis than later cues and thus requires changing the leading diagnosis over time, to assess sensemaking and re-planning skills in the context of tactical combat casualty care. Critical decision method interviews with emergency medicine physicians revealed critical cues specific to particular injuries and common across different injuries, and also provided cases that were used as the foundation for garden path scenarios. Evaluations using this approach with garden path scenarios enables having an objective measure to compare performance with and without a new technology on a continuous variable of the time until landmark events, such as recognition of a critical cue, committing to a likely diagnosis, or ruling out an inaccurate diagnosis. Additional measures include whether or not particular cues are noticed based upon gaze tracking data and think aloud statements, and exams that assess knowledge of anatomy and treatment priorities. Re-planning measures will focus on comparing performance to an expert model such as for tourniquet application, whether or not tasks on a checklist are conducted in the expected order including for prioritizing where to look for patient assessment, action priorities, and the trainee’s ability to link diagnosis to appropriate treatment.

Augmented Reality: The Future of Medic Training

Augmented reality (AR) has been used successfully in the marketing and entertainment industries to bring products and simulated experiences to life (Kipper & Rampolla, 2013). AR technology is a powerful way to provide the experience of first-hand exposure to realistic perceptual cues (e.g., visual, haptic, and auditory stimuli) in situations that might ordinarily be very difficult to replicate. This makes AR an increasingly viable tool to deliver occupational training in complex environments. While virtual reality (VR) immerses a person in a virtual environment, AR allows the users to see the real world with virtual objects superimposed (Zhu et al., 2014). The relatively low cost and extreme flexibility of AR applications makes them practical and portable.

Human Factors Research and Design Trends for the Modern Exam and Hospital Room

Although much research has been conducted on the impact of the electronic health record (EHR) and health information technology (HIT) on provider-patient interaction, challenges persist on how to effectively integrate these tools into healthcare environments, while maintaining patient-centered care. Panelists will present and discuss their experience relating to: the introduction of new technology and how it changes the nature of work; the positive impact of greater focus on the patient; the nature of interruptions and strategies to reduce their negative impact; enhancing the workflow of medication ordering; and designing collaborative spaces considering all relevant stakeholders (patient care team, patients, family members). These topics represent some of the latest human factors research and design trends with a common goal: advancing the design of the modern healthcare environment, with consideration of the additional complexity introduced by evolutionary changes in clinical information systems and HIT.

Employing decision-centered design to develop decision support for colorectal cancer screening (extended abstract)

This paper describes a project employing decision-centered design to develop a decision support application called the Screening & Surveillance App (SSA) to aid primary care providers in tracking and managing colorectal cancer (CRC) screening for their patients. This paper details the path from cognitive task analysis (CTA) to a decision support application. We conducted ethnographic observations and CTA interviews to identify common elements of clinical workflow across health systems and articulate decision requirements. We describe three design iterations that emerged over the course of CTA and software development. Findings from a beta test evaluating the SSA suggest that it saves time, increases accuracy in responding to CRC-related patient-specific questions, and is perceived to reduce workflow as compared to participants’ current electronic health records (EHR). Usability and usefulness ratings for the SSA as measured by the Health ITUES were above 4 on a 5-point scale where 5 was most positive.