Michael M. Wagner

Accuracy of Data in Computer-based Patient Records

Data in computer-based patient records (CPRs) have many uses beyond their primary role in patient care, including research and health-system management. Although the accuracy of CPR data directly affects these applications, there has been only sporadic interest in, and no previous review of, data accuracy in CPRs. This paper reviews the published studies of data accuracy in CPRs. These studies report highly variable levels of accuracy. This variability stems from differences in study design, in types of data studied, and in the CPRs themselves. These differences confound interpretation of this literature. We conclude that our knowledge of data accuracy in CPRs is not commensurate with its importance and further studies are needed. We propose methodological guidelines for studying accuracy that address shortcomings of the current literature. As CPR data are used increasingly for research, methods used in research databases to continuously monitor and improve accuracy should be applied to CPRs.

Implementing Syndromic Surveillance: A Practical Guide Informed by the Early Experience

Syndromic surveillance refers to methods relying on detection of individual and population health indicators that are discernible before confirmed diagnoses are made. In particular, prior to the laboratory confirmation of an infectious disease, ill persons may exhibit behavioral patterns, symptoms, signs, or laboratory findings that can be tracked through a variety of data sources. Syndromic surveillance systems are being developed locally, regionally, and nationally. The efforts have been largely directed at facilitating the early detection of a covert bioterrorist attack, but the technology may also be useful for general public health, clinical medicine, quality improvement, patient safety, and research. This report, authored by developers and methodologists involved in the design and deployment of the first wave of syndromic surveillance systems, is intended to serve as a guide for informaticians, public health managers, and practitioners who are currently planning deployment of such systems in their regions.

Technical Description of RODS: A Real-time Public Health Surveillance System

Abstract This report describes the design and implementation of the Real-time Outbreak and Disease Surveillance (RODS) system, a computer-based public health surveillance system for early detection of disease outbreaks. Hospitals send RODS data from clinical encounters over virtual private networks and leased lines using the Health Level 7 (HL7) message protocol. The data are sent in real time. RODS automatically classifies the registration chief complaint from the visit into one of seven syndrome categories using Bayesian classifiers. It stores the data in a relational database, aggregates the data for analysis using data warehousing techniques, applies univariate and multivariate statistical detection algorithms to the data, and alerts users of when the algorithms identify anomalous patterns in the syndrome counts. RODS also has a Web-based user interface that supports temporal and spatial analyses. RODS processes sales of over-the-counter health care products in a similar manner but receives such data in batch mode on a daily basis. RODS was used during the 2002 Winter Olympics and currently operates in two states—Pennsylvania and Utah. It has been and continues to be a resource for implementing, evaluating, and applying new methods of public health surveillance.

The emerging science of very early detection of disease outbreaks.

A surge of development of new public health surveillance systems designed to provide more timely detection of outbreaks suggests that public health has a new requirement: extreme timeliness of detection. The authors review previous work relevant to measuring timeliness and to defining timeliness requirements. Using signal detection theory and decision theory, the authors identify strategies to improve timeliness of detection and position ongoing system development within that framework.

Classifying free-text triage chief complaints into syndromic categories with natural languages processing

OBJECTIVE Develop and evaluate a natural language processing application for classifying chief complaints into syndromic categories for syndromic surveillance. INTRODUCTION Much of the input data for artificial intelligence applications in the medical field are free-text patient medical records, including dictated medical reports and triage chief complaints. To be useful for automated systems, the free-text must be translated into encoded form. METHODS We implemented a biosurveillance detection system from Pennsylvania to monitor the 2002 Winter Olympic Games. Because input data was in free-text format, we used a natural language processing text classifier to automatically classify free-text triage chief complaints into syndromic categories used by the biosurveillance system. The classifier was trained on 4700 chief complaints from Pennsylvania. We evaluated the ability of the classifier to classify free-text chief complaints into syndromic categories with a test set of 800 chief complaints from Utah. RESULTS The classifier produced the following areas under the ROC curve: Constitutional = 0.95; Gastrointestinal = 0.97; Hemorrhagic = 0.99; Neurological = 0.96; Rash = 1.0; Respiratory = 0.99; Other = 0.96. Using information stored in the systems semantic model, we extracted from the Respiratory classifications lower respiratory complaints and lower respiratory complaints with fever with a precision of 0.97 and 0.96, respectively. CONCLUSION Results suggest that a trainable natural language processing text classifier can accurately extract data from free-text chief complaints for biosurveillance.

Roundtable on bioterrorism detection: information system-based surveillance.

During the 2001 AMIA Annual Symposium, the Anesthesia, Critical Care, and Emergency Medicine Working Group hosted the Roundtable on Bioterrorism Detection. Sixty-four people attended the roundtable discussion, during which several researchers discussed public health surveillance systems designed to enhance early detection of bioterrorism events. These systems make secondary use of existing clinical, laboratory, paramedical, and pharmacy data or facilitate electronic case reporting by clinicians. This paper combines case reports of six existing systems with discussion of some common techniques and approaches. The purpose of the roundtable discussion was to foster communication among researchers and promote progress by 1) sharing information about systems, including origins, current capabilities, stages of deployment, and architectures; 2) sharing lessons learned during the development and implementation of systems; and 3) exploring cooperation projects, including the sharing of software and data. A mailing list server for these ongoing efforts may be found at http://bt.cirg.washington.edu.

The Accuracy of Medication Data in an Outpatient Electronic Medical Record

Objective: To measure the accuracy-of medication records stored in the electronic medical record (EMR) of an outpatient geriatric center. The authors analyzed accuracy from the perspective of a clinician using the data and the perspective of a computer-based medical decision-support system (MDSS). Design: Prospective cohort study. Methods: The EMR at the geriatric center captures medication data both directly from clinicians and indirectly using encounter forms and data-entry clerks. During a scheduled office visit for medical care, the treating clinician determined whether the medication records for the patient were an accurate representation of the medications that the patient was actually taking. Using the available sources of information (the patient, the patient’s vials, any caregivers, and the medical chart), the clinician determined whether the recorded data were correct, whether any data were missing, and the type and cause for each discrepancy found. Results: At the geriatric center, 83% of medication records represented correctly the compound, dose, and schedule of a current medication; 91% represented correctly the compound. 0.37 current medications were missing per patient. The principal cause of errors was the patient (36.1% of errors), who misreported a medication at a previous visit or changed (stopped, started, or dose-adjusted) a medication between visits. The second most frequent cause of errors was failure to capture changes to medications made by outside clinicians, accounting for 25.9% of errors. Transcription errors were a relatively ucommon cause (8.2% of errors). When the accuracy of records from the center was analyzed from the perspective of a MDSS, 90% were correct for compound identity and 1.38 medications were missing or uncoded per patient. The cause of the additional errors of omission was a free-text “comments” field-which it is assumed would be unreadable by current MDSS applications-that was used by clinicians in 18% of records to record the identity of the medication. Conclusions: Medication records in an outpatient EMR may have significant levels of data error. Based on an analysis of correctable causes of error, the authors conclude that the most effective extension to the EMR studied would be to expand its scope to include all clinicians who can potentially change medications. Even with EMR extensions, however, ineradicable error due to patients and data entry will remain. Several implications of ineradicable error for MDSSs are discussed. The provision of a free-text “comments” field increased the accuracy of medication lists for clinician users at the expense of accuracy for a MDSS.

Automatic Electronic Laboratory-Based Reporting of Notifiable Infectious Diseases

Electronic laboratory-based reporting, developed by the University of Pittsburgh Medical Center (UPMC) Health System, was evaluated to determine if it could be integrated into the conventional paper-based reporting system. We reviewed reports of 10 infectious diseases from 8 UPMC hospitals that reported to the Allegheny County Health Department in southwestern Pennsylvania during January 1–November 26, 2000. Electronic reports were received a median of 4 days earlier than conventional reports. The completeness of reporting was 74% (95% confidence interval [CI] 66% to 81%) for the electronic laboratory-based reporting and 65% (95% CI 57% to 73%) for the conventional paper-based reporting system (p>0.05). Most reports (88%) missed by electronic laboratory-based reporting were caused by using free text. Automatic reporting was more rapid and as complete as conventional reporting. Using standardized coding and minimizing free text usage will increase the completeness of electronic laboratory-based reporting.

Automated Syndromic Surveillance for the 2002 Winter Olympics

The 2002 Olympic Winter Games were held in Utah from February 8 to March 16, 2002. Following the terrorist attacks on September 11, 2001, and the anthrax release in October 2001, the need for bioterrorism surveillance during the Games was paramount. A team of informaticists and public health specialists from Utah and Pittsburgh implemented the Real-time Outbreak and Disease Surveillance (RODS) system in Utah for the Games in just seven weeks. The strategies and challenges of implementing such a system in such a short time are discussed. The motivation and cooperation inspired by the 2002 Olympic Winter Games were a powerful driver in overcoming the organizational issues. Over 114,000 acute care encounters were monitored between February 8 and March 31, 2002. No outbreaks of public health significance were detected. The system was implemented successfully and operational for the 2002 Olympic Winter Games and remains operational today.

The Informatics Response in Disaster, Terrorism, and War

The United States currently faces several new, concurrent large-scale health crises as a result of terrorist activity. In particular, three major health issues have risen sharply in urgency and public consciousness--bioterrorism, the threat of widespread delivery of agents of illness; mass disasters, local events that produce large numbers of casualties and overwhelm the usual capacity of health care delivery systems; and the delivery of optimal health care to remote military field sites. Each of these health issues carries large demands for the collection, analysis, coordination, and distribution of health information. The authors present overviews of these areas and discuss ongoing work efforts of experts in each.