James Reston

AHRQ Series Paper 5: Grading the strength of a body of evidence when comparing medical interventions—Agency for Healthcare Research and Quality and the Effective Health-Care Program

OBJECTIVE To establish guidance on grading strength of evidence for the Evidence-based Practice Center (EPC) program of the U.S. Agency for Healthcare Research and Quality. STUDY DESIGN AND SETTING Authors reviewed authoritative systems for grading strength of evidence, identified domains and methods that should be considered when grading bodies of evidence in systematic reviews, considered public comments on an earlier draft, and discussed the approach with representatives of the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) working group. RESULTS The EPC approach is conceptually similar to the GRADE system of evidence rating; it requires assessment of four domains: risk of bias, consistency, directness, and precision. Additional domains to be used when appropriate include dose-response association, presence of confounders that would diminish an observed effect, strength of association, and publication bias. Strength of evidence receives a single grade: high, moderate, low, or insufficient. We give definitions, examples, mechanisms for scoring domains, and an approach for assigning strength of evidence. CONCLUSION EPCs should grade strength of evidence separately for each major outcome and, for comparative effectiveness reviews, all major comparisons. We will collaborate with the GRADE group to address ongoing challenges in assessing the strength of evidence.

Meta-analysis of short-term and mid-term outcomes following off-pump coronary artery bypass grafting

BACKGROUND Uncertainty continues to surround the relative benefits and harms of conventional coronary artery bypass grafting (CABG) and off-pump coronary artery bypass grafting (OPCABG). Possible reasons are that high-quality studies have not comprehensively examined relevant patient outcomes and have enrolled a limited range of patients. Some studies may have been too small to detect clinically important differences in patient outcomes. The present study addresses these issues using meta-analysis. METHODS We comprehensively retrieved randomized and nonrandomized controlled studies according to predetermined criteria. We performed meta-analyses for each outcome and empirically determined whether potential biases that might result from differences in study design or patient characteristics actually biased a studys results. We also conducted sensitivity analyses and tested for publication bias. RESULTS Rates of perioperative myocardial infarction, stroke, reoperation for bleeding, renal failure, and mortality were lower after OPCABG than after CABG. Reductions in length of hospital stay, atrial fibrillation, and wound infection were also associated with OPCABG, but statistically significant differences among study results for these outcomes could not be explained by available information. Midterm (3 to 25 months) angina recurrence did not appear to differ between treatments; a trend was noticed toward lower reintervention rates with CABG, and a trend toward lower overall mortality with OPCABG, at least when performed at experienced centers. These midterm outcome results require confirmation. CONCLUSIONS Off-pump coronary artery bypass grafting appears to reduce length of hospital stay, operative morbidity, and operative mortality relative to on-pump CABG. More studies are required before firm conclusions can be drawn concerning the effect of OPCABG on midterm mortality, angina recurrence, and repeat intervention.

The Top Patient Safety Strategies That Can Be Encouraged for Adoption Now

Over the past 12 years, since the publication of the Institute of Medicines report, “To Err is Human: Building a Safer Health System,” improving patient safety has been the focus of considerable public and professional interest. Although such efforts required changes in policies; education; workforce; and health care financing, organization, and delivery, the most important gap has arguably been in research. Specifically, to improve patient safety we needed to identify hazards, determine how to measure them accurately, and identify solutions that work to reduce patient harm. A 2001 report commissioned by the Agency for Healthcare Research and Quality, “Making Health Care Safer: A Critical Analysis of Patient Safety Practices” (1), helped identify some early evidence-based safety practices, but it also highlighted an enormous gap between what was known and what needed to be known.

Autologous hematopoietic cell transplantation for multiple sclerosis: a systematic review

Background and objectives: The purpose of this systematic review was to evaluate the safety and efficacy of autologous hematopoietic cell transplantation in patients with progressive multiple sclerosis (MS) refractory to conventional medical treatment. Methods: Eight case series met our a priori inclusion criteria for the primary outcome of progression-free survival. Individual study quality was rated using an 11-item scale for case series. The strength of the overall body of evidence for each outcome was rated using a system developed by the ECRI Institute. Data from different studies were statistically combined using meta-analysis. An additional six studies were included for a summary of mortality and morbidity. Results: For secondary progressive MS, immunoablative therapy with autologous bone marrow/peripheral blood stem cell transplantation was associated with higher progression-free survival (up to 3 years following treatment) when using intermediate-intensity conditioning regimens compared with high-intensity conditioning regimens. The evidence was insufficient to determine whether the treatment was effective in patients with other types of MS. Treatment-related mortality was about 2.7%. Conclusions: Patients with secondary progressive MS refractory to conventional medical treatment have longer progression-free survival following autologous stem cell transplantation with intermediate-intensity conditioning regimens than with high-intensity conditioning regimens.

In-Facility Delirium Prevention Programs as a Patient Safety Strategy: A Systematic Review

Delirium, an acute decline in attention and cognition, occurs among hospitalized patients at rates estimated to range from 14% to 56% and increases the risk for morbidity and mortality. The purpose of this systematic review was to evaluate the effectiveness and safety of in-facility multicomponent delirium prevention programs. A search of 6 databases (including MEDLINE, EMBASE, and CINAHL) was conducted through September 2012. Randomized, controlled trials; controlled clinical trials; interrupted time series; and controlled before-after studies with a prospective postintervention portion were eligible for inclusion. The evidence from 19 studies that met the inclusion criteria suggests that most multicomponent interventions are effective in preventing onset of delirium in at-risk patients in a hospital setting. Evidence was insufficient to determine the benefit of such programs in other care settings. Future comparative effectiveness studies with standardized protocols are needed to identify which components in multicomponent interventions are most effective for delirium prevention.

The predictive validity of quality of evidence grades for the stability of effect estimates was low: a meta-epidemiological study

OBJECTIVE To determine the predictive validity of the U.S. Evidence-based Practice Center (EPC) approach to GRADE (Grading of Recommendations Assessment, Development and Evaluation). STUDY DESIGN AND SETTING Based on Cochrane reports with outcomes graded as high quality of evidence (QOE), we prepared 160 documents which represented different levels of QOE. Professional systematic reviewers dually graded the QOE. For each document, we determined whether estimates were concordant with high QOE estimates of the Cochrane reports. We compared the observed proportion of concordant estimates with the expected proportion from an international survey. To determine the predictive validity, we used the Hosmer-Lemeshow test to assess calibration and the C (concordance) index to assess discrimination. RESULTS The predictive validity of the EPC approach to GRADE was limited. Estimates graded as high QOE were less likely, estimates graded as low or insufficient QOE more likely to remain stable than expected. The EPC approach to GRADE could not reliably predict the likelihood that individual bodies of evidence remain stable as new evidence becomes available. C-indices ranged between 0.56 (95% CI, 0.47 to 0.66) and 0.58 (95% CI, 0.50 to 0.67) indicating a low discriminatory ability. CONCLUSION The limited predictive validity of the EPC approach to GRADE seems to reflect a mismatch between expected and observed changes in treatment effects as bodies of evidence advance from insufficient to high QOE.

Ongoing Clinical Trials

This section provides information for researchers and health care professionals who are interested in enrolling patients into currently recruiting clinical trials. Studies may be funded by the NIH, other federal agencies, nonprofit organizations, or industry. Listed are phase II/III trials currently recruiting in Europe that are registered in the ClinicalTrials.gov database. In this issue of Oncology Research and Treatment trials on hematological malignancies are listed.

Time to Intervention During Cardiac Interventions. Are We Forgetting a Confounder

As we evolve in the field of contemporary cardiothoracic surgery and witness modern applications of new techniques and technology, we need to be careful of how statistical methods are executed. Publications with hidden mediators that are not adequately addressed can lead to biased conclusions, especially when meta-analyzed. Public health policies need to be sure that their statements are as unbiased as possible for correct inference, leading to optimal patient safety and well-being. Careful analysis of hidden mediators is important in studies comparing the effectiveness of procedures and devices. Such analysis is critical in identifying mediators such as waiting time that should be considered when constructing interventions to be evaluated in the next RCT. In particular, RCTs of devices and procedures should always conduct (and report) ITT analysis, capturing all events from the time of randomization forward to control for differential waiting time. Similarly, observational registries and databases should count time zero as the time when patients are first referred for therapy, rather than when they enter a hospital to receive treatment; this would ensure that events during the waiting period are captured.