Elias Brandt

Testing process errors and their harms and consequences reported from family medicine practices: a study of the American Academy of Family Physicians National Research Network

Context: Little is known about the types and outcomes of testing process errors that occur in primary care. Objective: To describe types, predictors and outcomes of testing errors reported by family physicians and office staff. Design: Events were reported anonymously. Each office completed a survey describing their testing processes prior to event reporting. Setting and participants: 243 clinicians and office staff of eight family medicine offices. Main outcome measures: Distribution of error types, associations with potential predictors; predictors of harm and consequences of the errors. Results: Participants submitted 590 event reports with 966 testing process errors. Errors occurred in ordering tests (12.9%), implementing tests (17.9%), reporting results to clinicians (24.6%), clinicians responding to results (6.6%), notifying patient of results (6.8%), general administration (17.6%), communication (5.7%) and other categories (7.8%). Charting or filing errors accounted for 14.5% of errors. Significant associations (p<0.05) existed between error types and type of reporter (clinician or staff), number of labs used by the practice, absence of a results follow-up system and patients’ race/ethnicity. Adverse consequences included time lost and financial consequences (22%), delays in care (24%), pain/suffering (11%) and adverse clinical consequence (2%). Patients were unharmed in 54% of events; 18% resulted in some harm, and harm status was unknown for 28%. Using multilevel logistic regression analyses, adverse consequences or harm were more common in events that were clinician-reported, involved patients aged 45–64 years and involved test implementation errors. Minority patients were more likely than white, non-Hispanic patients to suffer adverse consequences or harm. Conclusions: Errors occur throughout the testing process, most commonly involving test implementation and reporting results to clinicians. While significant physical harm was rare, adverse consequences for patients were common. The higher prevalence of harm and adverse consequences for minority patients is a troubling disparity needing further investigation.

An Electronic Practice-Based Network for Observational Comparative Effectiveness Research

Key Summary Points The Distributed Ambulatory Research in Therapeutics Network (DARTNet) is a federated network linking health data from 8 organizations representing more than 500 clinicians and more than 400 000 patients. Electronic health record and other clinical data are aggregated, standardized, and stored within each organization, then deidentified and made available for secure queries through the Web. A full set of patient data never leaves the individual clinical sites. DARTNet can be used for observational comparative effectiveness research, which may provide important information about safe and effective health care. DARTNet tools can prompt clinicians to obtain specific information during a patient encounter. Key clinical information traditionally missing in comparative effectiveness research is now available through DARTNet. The Distributed Ambulatory Research in Therapeutics Network (DARTNet) is a federated network of electronic health data that was created to facilitate advanced observational comparative effectiveness research and examine outcomes associated with prescription medications and devices. A federated network links geographically and organizationally separate databases so that a single database query can return results from multiple databases while maintaining the privacy and confidentiality of patient data. DARTNet currently links patient-level clinical data from 8 organizations representing more than 500 clinicians and more than 400000 patients. This article presents an overview of DARTNet, explains the contributions of DARTNet to the field of comparative effectiveness research, and summarizes the initial research study and lessons learned. Overview of DARTNet The Agency for Healthcare Research and Quality funded the creation of DARTNet through its Developing Evidence to Inform Decisions about Effectiveness Network. DARTNet was created to examine outcomes associated with prescription medications, devices, and various approaches to medical care. Within each member organization, DARTNet assembles patient-level information (such as vital signs, social history, family history, and physical examination findings) from electronic health records, laboratory tests, imaging results, pharmacy utilization databases, and billing systems. All of this information is put into 1 database that is deidentified and made available for secure access through the Web. Each organizations aggregated, deidentified clinical database is linked to similar databases in other DARTNet member organizations; the relationship among these databases is the federated network. The DARTNet research team can submit a precise request for information through a database query. The query runs simultaneously on each federated database, and the results provide highly detailed information about care and outcomes for hundreds of thousands of patients across the United States. The DARTNet technology currently works with 5 brands of ambulatory electronic health records, but it is capable of working with essentially any ambulatory electronic health record. In addition to facilitating queries among the standardized and federated databases, the DARTNet system can prompt clinicians to obtain specific information during a patient encounter. This capability allows the research team to collect additional data beyond what would normally be recorded in electronic records. Thus, DARTNet is designed to combine the essential elements of observational research with elements of a practical clinical trial (1). DARTNet is also designed to help support a learning community. By comparing clinical care provided across the network, high-performing practices and systems can be identified. One expectation of DARTNet membership is that high-performing sites will share their approaches to care with other members. The DARTNet staff is working to develop methods and systems to support this process. The learning community process is one of the main reasons that clinicians and practices wish to join DARTNet. Advancing the Science of Clinical Effectiveness Research By allowing examination of routine care from many clinicians, DARTNet provides insight that would not otherwise be available. Much of the current evidence base for health care depends on the results of randomized trials; however, those trials do not adequately account for the variability seen in actual care (1). Furthermore, randomized trials are not a practical way to answer important questions about the many possible variations in chronic disease treatment (1). Patients, physicians, payers, purchasers, health care administrators, and public health policymakers need better information to compare the effectiveness of treatments and thus make sound health care decisions from the individual to the national level (1). Traditionally, observational comparative effectiveness researchers examine data sets created for other purposes, such as insurance claims data. Cohorts of patients created from large populations are then compared. Statistical matching algorithms are applied to try to account for underlying clinical differences among the groups (2, 3), and the resulting analysis provides comparisons of outcomes of treatment. Although this methodology is powerful, it has been criticized because it cannot account for important clinical information not available in claims databases (4). Two brief examples demonstrate how the findings of a comparative effectiveness study might be skewed. First, a typical observational comparative effectiveness study of oral hypoglycemic medications cannot account for body mass index differences between cohorts. In addition, a comparative effectiveness examination of liver toxicity related to a particular medication would be much more robust if it could account for the use of alcohol or acetaminophen among cohort patients. DARTNet provides key missing clinical information of this type, which will advance the field of comparative effectiveness research. Research Using DARTNet To demonstrate the capabilities of DARTNet, we conducted a retrospective cohort study of patterns of use and comparative effectiveness and safety of oral diabetes medications for adults with type 2 diabetes. In phase 1 of the study, a limited set of data elements from a commercially available, integrated medical claims database (the Ingenix National Managed Care Impact database [Ingenix, Eden Prairie, Minnesota]) was used to examine the comparative effectiveness and safety of oral diabetes medications. This is a standard approach to observational comparative effectiveness research. A secondary aim was to identify limitations in claims databases for which DARTNet could supply useful and otherwise unavailable information. Phase 2 was designed to replicate phase 1 of the study, this time using DARTNet data. For phase 2, we studied a smaller patient sample but examined a broader range of data elements. We looked at electronic health record data and tested point-of-care data collection. Phase 2 findings show that DARTNet identified similarly sized panels of diabetic patients and patients receiving various oral diabetes medications, permitting analyses of similar power to the claims-based study in phase 1. In additional, DARTNet added useful clinical data (such as body weight, height, self-reported alcohol intake, and self-reported hypoglycemic events) to the analysis of comparative effectiveness of oral diabetes medications. Lessons Learned and Next Steps Our experience in developing DARTNet indicates that the system has broad utility and power. This power derives from the systems ability to access and standardize an array of data (approximately 150 data elements at this time) from various electronic health records and other clinical databases. Nonetheless, we encountered difficulty locating particular types of data in the electronic health records. It can be challenging to standardize data from separate practices, even those using the same electronic health record (5). We also found that many data fields in electronic health records lack reasonable range checks. For instance, most electronic health records will accept a systolic blood pressure in the thousands of mm Hg. Correction of these errors, although infrequent, will require work from the developers of electronic health records. DARTNet currently relies on a third-party clinical decision support system to extract and aggregate clinical data at each organization (6). The ongoing use of extracted data for clinical purposes serves as a continuous quality control feedback process: Clinicians who rely on the clinical decision support quickly recognize errors in the data and therefore routinely correct problems. Even with the use of powerful clinical decision support tools, we encountered difficulty in our attempts to view a discrete episode of care that spans multiple encounters. Many essential features linked to a given visit, including diagnoses, medication refills, and procedures, can be lost when the data are extracted for clinical decision support purposes. To improve its ability to examine episodes of care, DARTNet will include billing data to better distinguish encounters and procedures as well as focused point-of-care data collection. Even with effective decision support tools at their disposal, practices varied greatly in clinical outcomes and performance. This could be construed as a data quality issue, but it also highlights the potential for the system to add value to members through their participation in a learning community that identifies top-performing members, disseminates best practices, and provides facilitation to enhance clinical care at the practice level. The DARTNet prototype was connected to both small offices and large group practices specifically to demonstrate that it can include a wide array of organizations with varying informatics configurations, support levels, and ability to manipulate their information management environment.

Barriers and Motivators for Making Error Reports from Family Medicine Offices: A Report from the American Academy of Family Physicians National Research Network (AAFP NRN)

Context: Reporting of medical errors is a widely recognized mechanism for initiating patient safety improvement, yet we know little about the feasibility of error reporting in physician offices, where the majority of medical care in the United States is rendered. Objective: To identify barriers and motivators for error reporting by family physicians and their office staff based on the experiences of those participating in a testing process error reporting study. Design: Qualitative focus group study, analyzed using the editing method. Setting: Eight volunteer practices of the American Academy of Family Physicians National Research Network. Participants: 139 physicians, nurse practitioners, physician assistants, nurses, and staff who took part in 18 focus groups. Instrument: Interview questions asked about making reports, what prevents more reports from being made, and decisions about when to make reports. Results: Four factors were seen as central to making error reports: the burden of effort to report, clarity regarding the information requested in an error report, the perceived benefit to the reporter, and properties of the error (eg, severity, responsibility). The most commonly mentioned barriers were related to the high burden of effort to report and lack of clarity regarding the requested information. The most commonly mentioned motivator was perceived benefit. Conclusion: Successful error reporting systems for physicians’ offices will need to have low reporting burden, have great clarity regarding the information requested, provide direct benefit through feedback useful to reporters, and take into account error severity and personal responsibility.

Enhancing Electronic Health Record Measurement of Depression Severity and Suicide Ideation: A Distributed Ambulatory Research in Therapeutics Network (DARTNet) Study

Background: Depression is a leading cause of morbidity worldwide. The majority of treatment for depression occurs in primary care, but effective care remains elusive. Clinical decision making and comparative studies of real-world antidepressant effectiveness are limited by the absence of clinical measures of severity of illness and suicidality. Methods: The Distributed Ambulatory Research in Therapeutics Network (DARTNet) was engaged to systematically collect data using the 9-item Patient Health Questionnaire (PHQ-9) at the point of care. We used electronic health records (EHRs) and the PHQ-9 to capture, describe, and compare data on both baseline severity of illness and suicidality and response and suicidality after diagnosis for depressed patients in participating DARTNet practices. Results: EHR data were obtained for 81,028 episodes of depression (61,464 patients) from 14 clinical organizations. Over 9 months, data for 4900 PHQ-9s were collected from 2969 patients in DARTNet practices (this included 1892 PHQ-9s for 1019 adults and adolescents who had at least one depression diagnosis). Only 8.3% of episodes identified in our depression cohort had severity of illness information available in the EHR. For these episodes, considerable variation existed in both severity of illness (32.05% with no depression, 26.89% with minimal, 19.54% with mild, 12.04% with moderate, and 9.47% with severe depression) and suicidality (69.43% with a score of 0, 22.58% with a score of 1, 4.97% with a score of 2, and 3.02% with a score of 3 on item 9 of the PHQ-9). Patients with an EHR diagnosis of depression and a PHQ-9 (n = 1019) had similar severity but slightly higher suicidality levels compared with all patients for which PHQ-9 data were available. The PHQ-9 showed higher sensitivity for identifying depression response and emergent (after diagnosis) severity and suicidality; 25% to 30% of subjects had some degree of suicidal thought at some point in time according to the PHQ-9. Conclusions: This study demonstrated the value of adding PHQ-9 data and prescription fulfillment data to EHRs to improve diagnosis and management of depression in primary care and to enable more robust comparative effectiveness research on antidepressants.

Comparative Effectiveness Research in DARTNet Primary Care Practices Point of Care Data Collection on Hypoglycemia and Over-the- Counter and Herbal Use Among Patients Diagnosed With Diabetes

Background:The Distributed Ambulatory Research in Therapeutics Network (DARTNet) is a federated network of electronic health record (EHR) data, designed as a platform for next-generation comparative effectiveness research in real-world settings. DARTNet links information from nonintegrated primary care clinics that use EHRs to deliver ambulatory care to overcome limitations with traditional observational research. Objective:Test the ability to conduct a remote, electronic point of care study in DARTNet practices by prompting clinic staff to obtain specific information during a patient encounter. Research Design:Prospective survey of patients identified through queries of clinical data repositories in federated network organizations. On patient visit, survey is triggered and data are relinked to the EHR, de-identified, and copied for evaluation. Subjects:Adult patients diagnosed with diabetes mellitus that scheduled a clinic visit for any reason in a 2-week period in DARTNet primary care practices. Measures:Survey on hypoglycemic events (past month) and over-the-counter and herbal supplement use. Results:DARTNet facilitated point of care data collection triggered by an electronic prompt for additional information at a patient visit. More than one-third of respondents (33% response rate) reported either mild (45%) or severe hypoglycemic events (5%) in the month before the survey; only 3 of those were also coded using the ICD-9 (a significant difference in detection rates 37% vs. 1%). Nearly one-quarter of patients reported taking an OTC/herbal, 4% specifically for the treatment of symptoms of diabetes. Conclusions:Prospective data collection is feasible in DARTNet and can enable comparative effectiveness and safety research.

Practice-based Research Network Studies and Institutional Review Boards: Two New Issues

Background: Practice-based research network (PBRN) study investigators must interface with multiple Institutional Review Boards (IRBs), many of which are unfamiliar with PBRN research. Objective: To present 2 IRB-related issues that have not appeared in the literature but occurred during the course of a large 5-year PBRN study involving 32 sites dispersed around the United States. Results: Our study required IRB approval from a total of 19 local, hospital, academic center, and professional organization-based IRBs that reviewed a protocol of postpartum depression screening and follow-up completed in English or Spanish. Initial approval of the protocol and consent forms proceeded with only the usual barriers of submitting 19 different forms, and no protocol amendments were required. However, 2 unanticipated IRB issues provided significant additional work for the study team and the local practice sites. First, several IRBs required staff to repeat human subjects training every 1 to 2 years, resulting in 92 practicing physicians, residents, and members of the nursing staff spending time completing the exact same human subjects’ training at least twice during the course of this study. Second, 17 of the 19 IRBs required that the patient be given consent forms that were newly stamped and dated each year, requiring the central site to reprint and replace consent forms yearly. Because not all IRBs returned the newly stamped and dated forms in a timely fashion, study enrollment with valid consent forms was interrupted in 4 sites for periods of 2 to 13 weeks. Conclusions: IRB requirements not directly responsive to federal regulations can add significant costs, frustrations, and burdens to PBRN studies. Non–federally mandated IRB requirements should be based on an identified need with evidence to support the solution.

Scalable Architecture for Federated Translational Inquiries Network (SAFTINet) Technology Infrastructure for a Distributed Data Network

Introduction: Distributed Data Networks (DDNs) offer infrastructure solutions for sharing electronic health data from across disparate data sources to support comparative effectiveness research. Data sharing mechanisms must address technical and governance concerns stemming from network security and data disclosure laws and best practices, such as HIPAA. Methods: The Scalable Architecture for Federated Translational Inquiries Network (SAFTINet) deploys TRIAD grid technology, a common data model, detailed technical documentation, and custom software for data harmonization to facilitate data sharing in collaboration with stakeholders in the care of safety net populations. Data sharing partners host TRIAD grid nodes containing harmonized clinical data within their internal or hosted network environments. Authorized users can use a central web-based query system to request analytic data sets. Discussion: SAFTINet DDN infrastructure achieved a number of data sharing objectives, including scalable and sustainable systems for ensuring harmonized data structures and terminologies and secure distributed queries. Initial implementation challenges were resolved through iterative discussions, development and implementation of technical documentation, governance, and technology solutions.

Improving the Management of Skin and Soft Tissue Infections in Primary Care: A Report From State Networks of Colorado Ambulatory Practices and Partners (SNOCAP-USA) and the Distributed Ambulatory Research in Therapeutics Network (DARTNet)

Background: Purulent skin and soft tissue infections (SSTIs) requiring medical attention are often managed in primary care. The prevalence of SSTIs caused by community-acquired Methicillin-resistant Staphylococcus aureus (CA-MRSA) has been increasing rapidly, including in otherwise healthy individuals. The Centers for Disease Control and Prevention (CDC) issued guidelines to improve the management of SSTIs in primary care. Purpose: In primary care settings, to assess the prevalence of CA-MRSA using an electronic chart audit and then evaluate SSTI management strategies consistent with CDC guidelines. Methods: A practical intervention that compared a historical cohort to an intervention cohort of patients seen for SSTI in 16 primary care practices in two health care systems. The intervention included a ready-made kit for I & D procedures, MRSA information for clinicians, a patient information handout, provider education, and patient follow-up. Results: A total of 3112 SSTI cases (cellulitis or purulent) were observed during the preintervention period and 1406 cases during the intervention. For purulent infections in the intervention period (n = 148), univariate and multivariate analyses showed no significant improvement in the rate of I & D procedures or cultures obtained but showed increased use of antibiotics overall and agents that typically cover MRSA strains (OR, 2.183; 95% CI, 1.443 to 3.303 and 2.624; 95% CI, 1.500 to 4.604, respectively). For infections that were cellulitis with or without purulence (n = 1258), overall rates in the use of antibiotics and those that cover MRSA increased significantly, but secular trends could not be ruled out as an explanation for this increase. Conclusion: In SSTIs, this intervention resulted in increased use of antibiotics, including antibiotics that typically cover MRSA strains, but did not demonstrate increased rates of recommended drainage procedures. It is replicable and portable, and may improve antibiotic selection in other settings.

Impact of family history assessment on communication with family members and health care providers: A report from the Family Healthware™ Impact Trial (FHITr)

OBJECTIVE This study examines the impact of Family Healthware™ on communication behaviors; specifically, communication with family members and health care providers about family health history. METHODS A total of 3786 participants were enrolled in the Family Healthware™ Impact Trial (FHITr) in the United States from 2005-7. The trial employed a two-arm cluster-randomized design, with primary care practices serving as the unit of randomization. Using generalized estimating equations (GEE), analyses focused on communication behaviors at 6month follow-up, adjusting for age, site and practice clustering. RESULTS A significant interaction was observed between study arm and baseline communication status for the family communication outcomes (ps<.01), indicating that intervention had effects of different magnitude between those already communicating at baseline and those who were not. Among participants who were not communicating at baseline, intervention participants had higher odds of communicating with family members about family history risk (OR=1.24, p=0.042) and actively collecting family history information at follow-up (OR=2.67, p=0.026). Family Healthware™ did not have a significant effect on family communication among those already communicating at baseline, or on provider communication, regardless of baseline communication status. Greater communication was observed among those at increased familial risk for a greater number of diseases. CONCLUSION Family Healthware™ prompted more communication about family history with family members, among those who were not previously communicating. Efforts are needed to identify approaches to encourage greater sharing of family history information, particularly with health care providers.

Mitigation of patient harm from testing errors in family medicine offices: a report from the American Academy of Family Physicians National Research Network

Objectives: Little research has focused on preventing harm from errors that occur in primary care. We studied mitigation of patient harm by analysing error reports from family physicians’ offices. Methods: The data for this analysis come from reports of testing process errors identified by family physicians and their office staff in eight practices in the American Academy of Family Physicians National Research Network. We determined how often reported error events were mitigated, described factors related to mitigation and assessed the effect of mitigation on the outcome of error events. Results: We identified mitigation in 123 (21%) of 597 testing process event reports. Of the identified mitigators, 79% were persons from inside the practice, and 7% were patients or patient’s family. Older age was the only patient demographic attribute associated with increased likelihood of mitigation occurring (unadjusted OR 18–44 years compared with 65 years of age or older = 0.27; p = 0.007). Events that included testing implementation errors (11% of the events) had lower odds of mitigation (unadjusted OR = 0.40; p = 0.001), and events containing reporting errors (26% of the events) had higher odds of mitigation (unadjusted OR = 1.63; p = 0.021). As the number of errors reported in an event increased, the odds of that event being mitigated decreased (unadjusted OR = 0.58; p = 0.001). Multivariate logistic regression showed that an event had higher odds of being mitigated if it included an ordering error or if the patient was 65 years of age or older, and lower odds of being mitigated if the patient was between age 18 and 44, or if the event included an implementation error or involved more than one error. Mitigated events had lower odds of patient harm (unadjusted OR = 0.16; p<0.0001) and negative consequences (unadjusted OR = 0.28; p<0.0001). Mitigated events resulted in less severe and fewer detrimental outcomes compared with non-mitigated events. Conclusion: Nearly a quarter of testing process errors reported by family physicians and their staff had evidence of mitigation, and mitigated errors resulted in less frequent and less serious harm to patients. Vigilance throughout the testing process is likely to detect and correct errors, thereby preventing or reducing harm.