Jianwei Leng

Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text

The Health Insurance Portability and Accountability Act (HIPAA) Safe Harbor method requires removal of 18 types of protected health information (PHI) from clinical documents to be considered “de-identified” prior to use for research purposes. Human review of PHI elements from a large corpus of clinical documents can be tedious and error-prone. Indeed, multiple annotators may be required to consistently redact information that represents each PHI class. Automated de-identification has the potential to improve annotation quality and reduce annotation time. For instance, using machine-assisted annotation by combining de-identification system outputs used as pre-annotations and an interactive annotation interface to provide annotators with PHI annotations for “curation” rather than manual annotation from “scratch” on raw clinical documents. In order to assess whether machine-assisted annotation improves the reliability and accuracy of the reference standard quality and reduces annotation effort, we conducted an annotation experiment. In this annotation study, we assessed the generalizability of the VA Consortium for Healthcare Informatics Research (CHIR) annotation schema and guidelines applied to a corpus of publicly available clinical documents called MTSamples. Specifically, our goals were to (1) characterize a heterogeneous corpus of clinical documents manually annotated for risk-ranked PHI and other annotation types (clinical eponyms and person relations), (2) evaluate how well annotators apply the CHIR schema to the heterogeneous corpus, (3) compare whether machine-assisted annotation (experiment) improves annotation quality and reduces annotation time compared to manual annotation (control), and (4) assess the change in quality of reference standard coverage with each added annotator’s annotations.

Persistence With Conventional Triple Therapy Versus a Tumor Necrosis Factor Inhibitor and Methotrexate in US Veterans With Rheumatoid Arthritis

To compare persistence and adherence to triple therapy with the nonbiologic disease‐modifying antirheumatic drugs (DMARDs) methotrexate (MTX), hydroxychloroquine, and sulfasalazine, versus a tumor necrosis factor inhibitor (TNFi) plus MTX in patients with rheumatoid arthritis (RA).

Performance of a Natural Language Processing (NLP) Tool to Extract Pulmonary Function Test (PFT) Reports from Structured and Semistructured Veteran Affairs (VA) Data.

Introduction/Objective: Pulmonary function tests (PFTs) are objective estimates of lung function, but are not reliably stored within the Veteran Health Affairs data systems as structured data. The aim of this study was to validate the natural language processing (NLP) tool we developed—which extracts spirometric values and responses to bronchodilator administration—against expert review, and to estimate the number of additional spirometric tests identified beyond the structured data. Methods: All patients at seven Veteran Affairs Medical Centers with a diagnostic code for asthma Jan 1, 2006–Dec 31, 2012 were included. Evidence of spirometry with a bronchodilator challenge (BDC) was extracted from structured data as well as clinical documents. NLP’s performance was compared against a human reference standard using a random sample of 1,001 documents. Results: In the validation set NLP demonstrated a precision of 98.9 percent [d1] (95 percent confidence intervals (CI): 93.9 percent, 99.7 percent), recall of 97.8 percent (95 percent CI: 92.2 percent, 99.7 percent), and an F-measure of 98.3 percent for the forced vital capacity preand post pairs and precision of 100 percent (95 percent CI: 96.6 percent, 100 percent), recall of 100 percent (95 percent CI: 96.6 percent, 100 percent), and an F-measure of 100 percent for the forced expiratory volume in one second preand post pairs for bronchodilator administration. Application of the NLP increased the proportion identified with complete bronchodilator challenge by 25 percent. Discussion/Conclusion: This technology can improve identification of PFTs for epidemiologic research. Caution must be taken in assuming that a single domain of clinical data can completely capture the scope of a disease, treatment, or clinical test.Introduction/Objective: Pulmonary function tests (PFTs) are objective estimates of lung function, but are not reliably stored within the Veteran Health Affairs data systems as structured data. The aim of this study was to validate the natural language processing (NLP) tool we developed—which extracts spirometric values and responses to bronchodilator administration—against expert review, and to estimate the number of additional spirometric tests identified beyond the structured data. Methods: All patients at seven Veteran Affairs Medical Centers with a diagnostic code for asthma Jan 1, 2006–Dec 31, 2012 were included. Evidence of spirometry with a bronchodilator challenge (BDC) was extracted from structured data as well as clinical documents. NLP’s performance was compared against a human reference standard using a random sample of 1,001 documents. Results: In the validation set NLP demonstrated a precision of 98.9 percent (95 percent confidence intervals (CI): 93.9 percent, 99.7 percent), recall of 97.8 percent (95 percent CI: 92.2 percent, 99.7 percent), and an F-measure of 98.3 percent for the forced vital capacity pre- and post pairs and precision of 100 percent (95 percent CI: 96.6 percent, 100 percent), recall of 100 percent (95 percent CI: 96.6 percent, 100 percent), and an F-measure of 100 percent for the forced expiratory volume in one second pre- and post pairs for bronchodilator administration. Application of the NLP increased the proportion identified with complete bronchodilator challenge by 25 percent. Discussion/Conclusion: This technology can improve identification of PFTs for epidemiologic research. Caution must be taken in assuming that a single domain of clinical data can completely capture the scope of a disease, treatment, or clinical test.

Treatment patterns and annual biologic costs in US veterans with rheumatic conditions or psoriasis

Abstract Objective: To determine annual biologic drug and administration costs to the US Veterans Health Administration (VHA) per treated patient with rheumatoid arthritis (RA), psoriasis (PsO), psoriatic arthritis (PsA), or ankylosing spondylitis (AS) who received abatacept, adalimumab, certolizumab pegol, etanercept, golimumab, infliximab, rituximab, tocilizumab, or ustekinumab. Methods: Adults with at least one biologic claim between January 1, 2008 and December 31, 2011 were included. Evidence of enrollment in the VHA was required from 365 days before (pre-index) to 360 days after (post-index) the date of the first biologic claim (index date). Included patients had pre-index diagnoses of RA, PsO, PsA, and/or AS. Drug costs were from Federal Supply Schedule or ‘Big Four’ in November 2014. Administration costs were VHA fixed costs for infused (

Predicting Psychiatric Hospitalizations among Elderly Veterans with a History of Mental Health Disease

169) and subcutaneous (

Cohort identification of axial spondyloarthritis in a large healthcare dataset: current and future methods

25) biologics. Results: Of the 20,465 patients in the analysis, 10,711 received etanercept, 7838 received adalimumab, and 1196 received infliximab as the index biologic. In these patients, across all uses studied, the VHA incurred greater annual cost per treated patient for infliximab (

Identifying Axial Spondyloarthritis in Electronic Medical Records of US Veterans.

18,066) compared with adalimumab (

The use of natural language processing on narrative medication schedules to compute average weekly dose.

16,523) and etanercept (

FRI0179 Comparative Effectiveness of Biologic Disease Modifying Anti-Rheumatic Drug Therapy in US Veterans with Rheumatoid Arthritis

16,526). In the first year post-index, ∼80% of patients were either persistent on these index biologics or re-started these index biologics after a ≥45–day treatment gap. Other biologics comprised <5% of the study population, with sample sizes ranging from 3–374 patients each. Cost by indication for biologics used by >20 patients ranged from

A Prototype Tool Set to Support Machine-Assisted Annotation

15,056 (etanercept) to