Howard Balshem

AHRQ Series Paper 2: Principles for developing guidance: AHRQ and the Effective Health-Care Program

OBJECTIVE This article describes some of the fundamental principles that have been developed to guide the work of producing comparative effectiveness reviews (CERs). STUDY DESIGN AND SETTING We briefly describe the role stakeholders play in providing important insights that inform the evidence-gathering process, and discuss the critical role of analytic frameworks in illuminating the relationship between surrogate measures and health outcomes, providing an understanding of the context in which clinical decisions are made and the uncertainties that underlie clinical controversies. RESULTS We describe the Effective Health Care program conceptual model for considering different types of evidence that emphasizes minimizing the risk of bias, but places high-quality, highly applicable evidence about effectiveness at the top of the hierarchy. Finally, we briefly describe areas of future methodological research. CONCLUSION CERs have become a foundation for decision-making in clinical practice and health policy. To be useful, CERs must approach the evidence from a patient-centered perspective; explore the clinical logic underlying the rationale for a service; cast a broad net with respect to types of evidence, placing a high value on effectiveness and applicability, in addition to internal validity; and, present benefits and harms for treatments and tests in a consistent way.

Updating comparative effectiveness reviews: Current efforts in AHRQ's Effective Health Care Program

OBJECTIVES To review the current knowledge and efforts on updating systematic reviews (SRs) as applied to comparative effectiveness reviews (CERs). STUDY DESIGN AND SETTING This article outlines considerations for updating CERs by including a definition of the updating process, describing issues around assessing whether to update, and providing general guidelines for the update process. Key points to consider include (1) identifying when to update CERs, (2) how to update CERs, and (3) how to present, report, and interpret updated results in CERs. RESULTS Currently, there is little information about what proportion of SRs needs updating. Similarly, there is no consensus on when to initiate updating and how best to carry it out. CONCLUSION CERs need to be regularly updated as new evidence is produced. Lack of attention to updating may lead to outdated and sometimes misleading conclusions that compromise health care and policy decisions. The article outlines several specific goals for future research, one of them being the development of efficient guideline for updating CERs applicable across evidence-based practice centers.

Finding evidence for comparing medical interventions: AHRQ and the Effective Health Care Program.

OBJECTIVE This article discusses search methodology to identify evidence for comparative effectiveness reviews (CERs) as practiced by the Effective Health Care program. STUDY DESIGN AND SETTING Review. RESULTS Search methods described attempt to overcome the bias inherent in the publication and distribution of clinical evidence. Bibliographic databases and search strategies are discussed with special emphasis on searching for observational studies and harms data. Other techniques described include the use of key articles, citation tracking, hand searching, and personal communications. Strategies for locating gray literature, such as clinical trial protocols and regulatory information, are described. Search reporting and other practical matters are also discussed. CONCLUSION Better reporting and further research on search strategies is needed to develop additional evidence-based recommendations.

US Food and Drug Administration documents can provide unpublished evidence relevant to systematic reviews

OBJECTIVES A key systematic review (SR) methodology is comprehensive searching. The Drug Effectiveness Review Project (DERP) SRs search US Food and Drug Administration (FDA) documents to identify unpublished evidence. This study evaluates the success of those efforts. STUDY DESIGN AND SETTING We examined DERP reports published since 2003 for the use of FDA preapproval and postmarketing documents. We categorized evidence as (1) unique unpublished studies, (2) supplemental unpublished data, or (3) FDA postmarketing data analysis. Three reviewers independently assigned predetermined impact categories (e.g., qualitative or quantitative usage, fills gaps, confirms findings, and alters conclusions), resolving disagreements through consensus. RESULTS Among 114 DERP reports, 19% included unpublished studies and/or supplemental data and 10% included postmarketing analyses. From 175 preapproval documents, 14% provided eligible unpublished studies and 4.0% supplemental unpublished data that helped confirm previous findings, identify important harms, and fill gaps in knowledge about understudied subpopulations, outcomes, and comparisons. Report conclusion statements changed in 9 of 33 instances of premarketing documents compared with 4 of 12 postmarketing analyses. CONCLUSIONS The FDA documents can provide important unpublished evidence for SRs, although in a small proportion of cases. Future research should identify attributes that predict which reviews may benefit most from review of FDA documents.