Leonard W. D'Avolio

Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutions

This issue of JAMIA focuses on natural language processing (NLP) techniques for clinical-text information extraction. Several articles are offshoots of the yearly ‘Informatics for Integrating Biology and the Bedside’ (i2b2) (http://www.i2b2.org) NLP shared-task challenge, introduced by Uzuner et al ( see page 552 )1 and co-sponsored by the Veterans Administration for the last 2 years. This shared task follows long-running challenge evaluations in other fields, such as the Message Understanding Conference (MUC) for information extraction,2 TREC3 for text information retrieval, and CASP4 for protein structure prediction. Shared tasks in the clinical domain are recent and include annual i2b2 Challenges that began in 2006, a challenge for multi-label classification of radiology reports sponsored by Cincinnati Childrens Hospital in 2007,5 a 2011 Cincinnati Childrens Hospital challenge on suicide notes,6 and the 2011 TREC information retrieval shared task involving retrieval of clinical cases from narrative records.7 Although NLP research in the clinical domain has been active since the 1960s, progress in the development of NLP applications for clinical text has been slow and lags behind progress in the general NLP domain. There are several barriers to NLP development in the clinical domain, and shared tasks like the i2b2/VA Challenge address some of these barriers. Nevertheless, many barriers remain and unless the community takes a more active role in developing novel approaches for addressing the barriers, advancement and innovation will continue to be slow. Historically, there have been substantial barriers to NLP development in the clinical domain. These barriers are not unique to the clinical domain: they also occur in the fields of software engineering and general NLP. ### Lack of access to shared data Because of concerns regarding patient privacy and worry about revealing unfavorable institutional practices, hospitals and clinics have been extremely reluctant to allow access to clinical data for researchers from outside … Correspondence to Dr Wendy W Chapman, Department of Biomedical Informatics, University of California San Diego, 9500 Gilman Dr, Bldg 2 #0728, La Jolla, California, USA; wwchapman{at}ucsd.edu

Statins and Prostate Cancer Diagnosis and Grade in a Veterans Population

BACKGROUND Although prostate cancer is commonly diagnosed, few risk factors for high-grade prostate cancer are known and few prevention strategies exist. Statins have been proposed as a possible treatment to prevent prostate cancer. METHODS Using electronic and administrative files from the Veterans Affairs New England Healthcare System, we identified 55,875 men taking either a statin or antihypertensive medication. We used age- and multivariable-adjusted Cox proportional hazard models to calculate hazard ratios (HRs) and 95% confidence intervals (CIs) for prostate cancer incidence among patients taking statins (n = 41,078) compared with patients taking antihypertensive medications (n = 14,797). We performed similar analyses for all lipid parameters including total cholesterol examining each lipid parameter as a continuous variable and by quartiles. All statistical tests were two-sided. RESULTS Compared with men taking an antihypertensive medication, statin users were 31% less likely (HR = 0.69, 95% CI = 0.52 to 0.90) to be diagnosed with prostate cancer. Furthermore, statin users were 14% less likely (HR = 0.86, 95% CI = 0.62 to 1.20) to be diagnosed with low-grade prostate cancer and 60% less likely (HR = 0.40, 95% CI = 0.24 to 0.65) to be diagnosed with high-grade prostate cancer compared with antihypertensive medication users. Increased levels of total cholesterol were also associated with both total (HR = 1.02, 95% CI = 1.00 to 1.05) and high-grade (HR = 1.06, 95% CI = 1.02 to 1.10) prostate cancer incidence but not with low-grade prostate cancer incidence (HR = 1.01, 95% CI = 0.98 to 1.04). CONCLUSIONS Statin use is associated with statistically significantly reduced risk for total and high-grade prostate cancer, and increased levels of serum cholesterol are associated with higher risk for total and high-grade prostate cancer. These findings indicate that clinical trials of statins for prostate cancer prevention are warranted.

Implementation of the Department of Veterans Affairs' first point-of-care clinical trial

OBJECTIVES The Massachusetts Veterans Epidemiology Research and Information Center in collaboration with the Stanford Center for Innovative Study Design set out to test the feasibility of a new method of evidence generation. The first pilot of a point-of-care clinical trial (POCCT), adding randomization and other study processes to an electronic medical record (EMR) system, was launched to compare the effectiveness of two insulin regimens. MATERIALS AND METHODS Existing functionalities of the Veterans Affairs (VA) computerized patient record system (CPRS)/veterans health information systems and technology architecture (VISTA) were modified to support the activities of a randomized controlled trial including enrolment, randomization, and longitudinal data collection. RESULTS The VAs CPRS/VISTA was successfully adapted to support the processes of a clinical trial and longitudinal study data are being collected from the medical record automatically. As of 30 June 2011, 55 of the 67 eligible patients approached received a randomized intervention. DISCUSSION The design of CPRS/VISTA made integration of study workflows and data collection possible. Institutions and investigators considering similar designs must carefully map clinical workflows and clinical trial workflows to EMR capabilities. POCCT study teams are necessarily interdisciplinary and interdepartmental. As a result, executive sponsorship is critical. CONCLUSION POCCT represent a promising new method for conducting clinical science. Much work is needed to understand better the optimal uses and designs for this new approach. Next steps include focus groups to measure patient and clinician perceptions, multisite deployment of the current pilot, and implementation of additional studies.

Comparative Effectiveness Research and Medical Informatics

As is the case for environmental, ecological, astronomical, and other sciences, medical practice and research finds itself in a tsunami of data. This data deluge, due primarily to the introduction of digitalization in routine medical care and medical research, affords the opportunity for improved patient care and scientific discovery. Medical informatics is the subdiscipline of medicine created to make greater use of information in order to improve healthcare. The 4 areas of medical informatics research (information access, structure, analysis, and interaction) are used as a framework to discuss the overlap in information needs of comparative effectiveness research and potential contributions of medical informatics. Examples of progress from the medical informatics literature and the Veterans Affairs Healthcare System are provided.

Automated classification of psychotherapy note text: implications for quality assessment in PTSD care.

In recent years, studies have attempted to use various methods to characterize the quality of care for post-traumatic stress disorder (PTSD) delivered in United States Veterans Administration (VA) outpatient clinics. Dieperink et al. used manual chart review to characterize care for 150 veterans at three VA medical centres during the 2001 fiscal year [1]. They found wide variation in the types of social services, psychotherapy and pharmacotherapy received in the 6 months following entry into specialized PTSD programmes. For example, clinics in Minneapolis and Memphis tended to provide pharmacotherapy while clinics in Boston tended to provide psychotherapy. As a result, there was wide variation in the amount of care received; veterans in Minneapolis received an average of seven mental health contacts per year (visits with a psychiatrist, psychologist, social worker or other mental health practitioner) while veterans in Boston received an average of 16 mental health contacts per year. Although it was not clear which approach was superior, it was clear that PTSD care was not standardized among the three VA facilities. In order to include more sites and be able to generalize across sites, further studies on the quality of care for PTSD in the VA have attempted to use national administrative data rather than chart review to capture information about the process of care. This work has relied on a combination of Current Procedural Technology (CPT) codes, International Classification of Diseases, Ninth Revision (ICD-9) codes, and pharmacy data. Cully et al. used administrative data to examine the receipt of psychotherapy in the VA nationally for the 12 months following initial diagnoses of PTSD, anxiety and depression in the 2004 fiscal year [2]. While the 77 743 veterans with new diagnoses of PTSD had a higher chance of receiving psychotherapy than veterans with anxiety or depression, the amount of psychotherapy was still very low; only 10% received an adequate number of sessions (defined as eight in this study), and the median wait to start psychotherapy was 50 days. Two studies using similar methods were published in 2010. Using stricter inclusion criteria, Spoont et al. examined care for 20 284 veterans with a new diagnosis of PTSD from the 2004 mid-fiscal year through the 2005 mid-fiscal year [3]. They evaluated whether veterans received a large enough medication supply that they could have gotten an adequate trial of pharmacotherapy or whether they received enough psychotherapy visits that they could have received an adequate trial of psychotherapy (defined, again, as eight visits). Based on this resource utilization, they concluded that, at most 33%, of veterans could have received an adequate trial of evidence-based treatment for PTSD. Seal et al. examined all mental health visits over the year following a new PTSD diagnosis in veterans returning from Iraq and Afghanistan [4]. The study included 49 425 veterans enrolling in VA care from the 2002 mid-fiscal year through the 2008 mid-fiscal year. They asserted that delivery of evidence-based psychotherapies endorsed in the VA Mental Health Uniformed Services Package [5] (prolonged exposure [6] and cognitive processing therapy [7]) required at least nine sessions over 15 weeks and found that only 9.5% received this level of service. A key limitation in these three studies is that they tell us only the best possible scenario about the amount of psychotherapy that could have been delivered based on the number of visits – we do not know whether veterans actually received psychotherapy during these visits. Therefore, it is possible that these studies overestimate the amount of psychotherapy actually delivered to veterans with PTSD. Researchers and policy makers wishing to understand care delivery for PTSD are left with a dilemma. Manual medical record review can generate detailed information about clinical processes, including psychotherapists’ reports of the specific techniques they used in a session. However, the method is time-consuming and difficult to apply on a large scale. Administrative review techniques are applicable on a large scale, but are limited in the granularity of the information they provide. We learn how much of a given service practitioners report providing, but because we do not read the notes, we have little information about the content of those services. Automated text-based information retrieval technologies, such as natural language processing (NLP) have the potential to bridge this gap by extracting detailed information found in a medical record review on the larger scale permitted by administrative review. NLP is an effort to have computers draw specific information from free text. The application of NLP has traditionally been limited by the need to customize programming for each new application. However, modern NLP applications can use a technique known as machine learning to ‘teach’ a computer to recognize patterns in documents [8,9]. Through the recognition of language patterns within a document the NLP application can help users make inferences about the content of the text. We sought to understand whether using administrative data to determine the number of psychotherapy sessions veterans receive is equivalent to manual medical records review. We thought it was possible that psychotherapy billing codes might sometimes be misapplied to other services delivered by psychotherapy-oriented practitioners, such as psychologists and social workers. These might include intakes, psychological testing and case management. Alternatively, administrative data review might be accurate, making manual or automated review of note text an unnecessary method. Our primary hypothesis was that administrative data overestimates the number of psychotherapy sessions delivered to veterans when compared to manual chart review (as some sessions administratively coded as psychotherapy are actually used for other purposes). Our secondary hypothesis was that if administrative data review was inaccurate, our manual medical record review could be approximated using an automated NLP programme, creating the potential for a more accurate method to be efficiently applied to large-scale treatment studies.

Using the Electronic Medical Record to Identify Community-Acquired Pneumonia: Toward a Replicable Automated Strategy

Background Timely information about disease severity can be central to the detection and management of outbreaks of acute respiratory infections (ARI), including influenza. We asked if two resources: 1) free text, and 2) structured data from an electronic medical record (EMR) could complement each other to identify patients with pneumonia, an ARI severity landmark. Methods A manual EMR review of 2747 outpatient ARI visits with associated chest imaging identified x-ray reports that could support the diagnosis of pneumonia (kappa score  = 0.88 (95% CI 0.82∶0.93)), along with attendant cases with Possible Pneumonia (adds either cough, sputum, fever/chills/night sweats, dyspnea or pleuritic chest pain) or with Pneumonia-in-Plan (adds pneumonia stated as a likely diagnosis by the provider). The x-ray reports served as a reference to develop a text classifier using machine-learning software that did not require custom coding. To identify pneumonia cases, the classifier was combined with EMR-based structured data and with text analyses aimed at ARI symptoms in clinical notes. Results 370 reference cases with Possible Pneumonia and 250 with Pneumonia-in-Plan were identified. The x-ray report text classifier increased the positive predictive value of otherwise identical EMR-based case-detection algorithms by 20–70%, while retaining sensitivities of 58–75%. These performance gains were independent of the case definitions and of whether patients were admitted to the hospital or sent home. Text analyses seeking ARI symptoms in clinical notes did not add further value. Conclusion Specialized software development is not required for automated text analyses to help identify pneumonia patients. These results begin to map an efficient, replicable strategy through which EMR data can be used to stratify ARI severity.

The Clinical Outcomes Assessment Toolkit: A Framework to Support Automated Clinical Records- based Outcomes Assessment and Performance Measurement Research

The Clinical Outcomes Assessment Toolkit (COAT) was created through a collaboration between the University of California, Los Angeles and Brigham and Womens Hospital to address the challenge of gathering, formatting, and abstracting data for clinical outcomes and performance measurement research. COAT provides a framework for the development of information pipelines to transform clinical data from its original structured, semi-structured, and unstructured forms to a standardized format amenable to statistical analysis. This system includes a collection of clinical data structures, reusable utilities for information analysis and transformation, and a graphical user interface through which pipelines can be controlled and their results audited by nontechnical users. The COAT architecture is presented, as well as two case studies of current implementations in the domain of prostate cancer outcomes assessment.

Natural language processing to identify foley catheter-days.

APR-DRG weights were available for 1,303 patients (80%). The mean APR-DRG weights were 2.3 for those who never had a VC, 3.6 for those who average 0-1 VC, and 6.9 for those who averaged more than 1 VC. Pairwise comparisons of APR-DRG means were significant, with P values of .001, <.0001, and <.0001, respectively (Figure 1). We undertook an observational pilot study to better understand the intensity of VC use in our ICUs. Our findings demonstrate that device utilization, as denned by the NHSN, underestimates the actual number of VCs employed. By counting the actual number of catheters, our DUR doubled. Our work also demonstrated an association between increased intensity of VC use and measures of SOI. All of the 18 CLABSIs identified during the study period occurred in patients who averaged more than 1 VC. The risk of BSI with different types of VCs, including arterial lines, has been reported previously. However, the incremental risk as more devices are required is unknown. Although Tejedor et al recently demonstrated that a large percentage of CVCdays in a non-ICU population were potentially unnecessary, our study suggests that, in the ICU, more line-days are associated with increased SOI and not with increased opportunity for line removal. Our findings also indicate that not all ICU patients are at similar risk for CLABSI and support total VC-days as a denominator for defining a population for CLABSI rate calculation. Tokars et al have demonstrated that central-line-days, as opposed to patient-days, was an appropriate risk adjustment for comparing interfacility rates, because percentile error increased as ICU DUR decreased. Our findings suggest that, for ICUs caring for patients with a high SOI, total VC-days should similarly be considered as an appropriate risk adjustment, because a hospitals CLABSI rate is increasingly used to publicly compare the quality of care provided.

From Prototype to Deployable System: Framing the Adoption of Digital Library Services

The Alexandria Digital Earth Prototype Project (ADEPT) is a 5-year (1999-2004) effort, with a goal of developing effective models for implementing digital libraries in undergraduate instruction. The ADEPT team has created a digital learning environment (DLE) that adds educational value to a digital library by offering a suite of services for teaching. Encouraged by the results of implementations in undergraduate geography classrooms, the team now shifts its focus from experimental prototype to deployable system. Everett Rogers’ Diffusion of Innovations theories are used as frameworks for analyzing this complex transition. Recommendations for lowering the barriers to adoption related to complexity, trialability, and observability include the prioritization of development efforts focused on stabilizing the system, the creation of documentation and an online demonstration, and anonymous logins to the system. To increase perceived relative advantage, existing technical and copyright issues in integrating the Alexandria Digital Library must be overcome. To increase compatibility, the speed at which pedagogical change is achieved must be rethought. Finally, recruitment efforts should focus on innovators and early adopters before moving on to early majority, late majority, or laggard adopters.

The automated retrieval console (ARC): open source software for streamlining the process of natural language processing

Open source natural language processing (NLP) frameworks have made it easier for NLP developers and researchers to develop more reusable and modular components and to capitalize on the work of others. With the Automated Retrieval Console (ARC) we attempt to build upon this foundation by streamlining the many processes surrounding the development, evaluation, and deployment of natural language processing technologies. Toward this end, ARC offers graphical user interfaces to facilitate corpus import, reference set creation, annotation, and inter-annotator agreement calculation. To speed task-specific information extraction development, ARC combines NLP-generated features from UIMA pipelines with machine learning classifiers and calculates performance statistics against a reference set. We also use ARC to explore automated algorithm creation for specific information extraction tasks in an effort to reduce the need for custom code and rules development. We present a detailed description of the ideas implemented in this proof-of-concept and a brief overview of two empirical evaluations.