Kerry Dwan

Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias

Background The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention. Methodology/Principal Findings We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. Conclusions Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.

The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews

Objective To examine the prevalence of outcome reporting bias—the selection for publication of a subset of the original recorded outcome variables on the basis of the results—and its impact on Cochrane reviews. Design A nine point classification system for missing outcome data in randomised trials was developed and applied to the trials assessed in a large, unselected cohort of Cochrane systematic reviews. Researchers who conducted the trials were contacted and the reason sought for the non-reporting of data. A sensitivity analysis was undertaken to assess the impact of outcome reporting bias on reviews that included a single meta-analysis of the review primary outcome. Results More than half (157/283 (55%)) the reviews did not include full data for the review primary outcome of interest from all eligible trials. The median amount of review outcome data missing for any reason was 10%, whereas 50% or more of the potential data were missing in 70 (25%) reviews. It was clear from the publications for 155 (6%) of the 2486 assessable trials that the researchers had measured and analysed the review primary outcome but did not report or only partially reported the results. For reports that did not mention the review primary outcome, our classification regarding the presence of outcome reporting bias was shown to have a sensitivity of 88% (95% CI 65% to 100%) and specificity of 80% (95% CI 69% to 90%) on the basis of responses from 62 trialists. A third of Cochrane reviews (96/283 (34%)) contained at least one trial with high suspicion of outcome reporting bias for the review primary outcome. In a sensitivity analysis undertaken for 81 reviews with a single meta-analysis of the primary outcome of interest, the treatment effect estimate was reduced by 20% or more in 19 (23%). Of the 42 meta-analyses with a statistically significant result only, eight (19%) became non-significant after adjustment for outcome reporting bias and 11 (26%) would have overestimated the treatment effect by 20% or more. Conclusions Outcome reporting bias is an under-recognised problem that affects the conclusions in a substantial proportion of Cochrane reviews. Individuals conducting systematic reviews need to address explicitly the issue of missing outcome data for their review to be considered a reliable source of evidence. Extra care is required during data extraction, reviewers should identify when a trial reports that an outcome was measured but no results were reported or events observed, and contact with trialists should be encouraged.

Exercise for depression

Depression is a common and important cause of morbidity and mortality worldwide. It is commonly treated with anti depressants and/or psychological therapy, but some people prefer alternative approaches such as exercise. There are a number of theoretical reasons why exercise may improve depression.

The clinical effectiveness of central venous catheters treated with anti-infective agents in preventing catheter-related bloodstream infections : A systematic review

Objectives:To assess the clinical effectiveness of central venous catheters (CVCs) treated with anti-infective agents (AI-CVCs) in preventing catheter-related bloodstream infections (CRBSI). Data Sources:MEDLINE (OVID), EMBASE, SCI//Web of Science, SCI/ISI Proceedings, and the Cochrane Library. Study Selection:A systematic review of the literature was conducted using internationally recognized methodology. All included articles were reports of randomized controlled trials comparing the clinical effectiveness of CVCs treated with AI-CVCs with either standard CVCs or another anti-infective treated catheter. Articles requiring in-house preparation of catheters or that only reported interim data were excluded. Data Extraction:Data extraction was carried out independently and crosschecked by two reviewers using a pretested data extraction form. Data Synthesis:Meta-analyses were conducted to assess the effectiveness of AI-CVCs in preventing CRBSI, compared with standard CVCs. Results are presented in forest plots with 95% confidence intervals. Results:Thirty-eight randomized controlled trials met the inclusion criteria. Methodologic quality was generally poor. Meta-analyses of data from 27 trials assessing CRBSI showed a strong treatment effect in favor of AI-CVCs (odds ratio 0.49 (95% confidence interval 0.37–0.64) fixed effects, test for heterogeneity, chi-square = 28.78, df = 26, p = 0.321, I2 = 9.7). Results subgrouped by the different types of anti-infective treatments generally demonstrated treatment effects favoring the treated catheters. Sensitivity analyses investigating the effects of methodologic differences showed no differences to the overall conclusions of the primary analysis. Conclusion:AI-CVCs appear to be effective in reducing CRBSI compared with standard CVCs. However, it is important to establish whether this effect remains in settings where infection-prevention bundles of care are established as routine practice. This review does not address this question and further research is required.

Exercise for Depression

CLINICAL QUESTION Is exercise an effective treatment for depression? BOTTOM LINE Exercise is associated with a greater reduction in depression symptoms compared with no treatment, placebo, or active control interventions, such as relaxation or meditation. However, analysis of high-quality studies alone suggests only small benefits.

Assessing the potential for outcome reporting bias in a review: a tutorial

BackgroundOutcome reporting bias (ORB) occurs when variables are selected for publication based on their results. This can impact upon the results of a meta-analysis, biasing the pooled treatment effect estimate.The aim of this paper is to show how to assess a systematic review and corresponding trial reports for ORB using an example review of intravenous and nebulised magnesium in the treatment of asthma.MethodsThe review was assessed for ORB by 1) checking the reasons, when available, for excluding studies to ensure that no studies were excluded because they did not report the outcomes of interest in the review; 2) assessing the eligible studies as to whether the review outcomes of interest were reported. Each study was classified using a system developed in the ORBIT (Outcome Reporting Bias In Trials) project to indicate whether ORB was suspected and a reason for the suspicion. Authors of trials that did not report the outcomes of interest were contacted for information. A sensitivity analysis was performed to assess the robustness of the conclusions of the review to this potential source of bias.ResultsTwenty-four studies were included in the review; two studies had been excluded for not reporting either of the two outcomes of interest. Six included studies did not report hospital admission and two did not report pulmonary function. There was high suspicion of outcome reporting bias in four studies. Results from the sensitivity analysis indicate that review conclusions were not overturned.ConclusionThis paper demonstrates, with the example of the magnesium review, how to assess a review for outcome reporting bias. A review should not exclude studies if they have not reported the outcomes of interest and should consider the potential for outcome reporting bias in all included studies.

Impregnated central venous catheters for prevention of bloodstream infection in children (the CATCH trial): a randomised controlled trial

BACKGROUND Impregnated central venous catheters are recommended for adults to reduce bloodstream infections but not for children because there is not enough evidence to prove they are effective. We aimed to assess the effectiveness of any type of impregnation (antibiotic or heparin) compared with standard central venous catheters to prevent bloodstream infections in children needing intensive care. METHODS We did a randomised controlled trial of children admitted to 14 English paediatric intensive care units. Children younger than 16 years were eligible if they were admitted or being prepared for admission to a participating paediatric intensive care unit and were expected to need a central venous catheter for 3 or more days. Children were randomly assigned (1:1:1) to receive a central venous catheter impregnated with antibiotics, a central venous catheter impregnated with heparin, or a standard central venous catheter with computer generated randomisation in blocks of three and six, stratified by method of consent, site, and envelope storage location within the site. The clinician responsible for inserting the central venous catheter was not masked to allocation, but allocation was concealed from patients, their parents, and the paediatric intensive care unit personnel responsible for their care. The primary outcome was time to first bloodstream infection between 48 h after randomisation and 48 h after central venous catheter removal with impregnated (antibiotic or heparin) versus standard central venous catheters, assessed in the intention-to-treat population. Safety analyses compared central venous catheter-related adverse events in the subset of children for whom central venous catheter insertion was attempted (per-protocol population). This trial is registered with ISRCTN number, ISRCTN34884569. FINDINGS Between Nov 25, 2010, and Nov 30, 2012, 1485 children were recruited to this study. We randomly assigned 502 children to receive standard central venous catheters, 486 to receive antibiotic-impregnated catheters, and 497 to receive heparin-impregnated catheters. Bloodstream infection occurred in 18 (4%) of those in the standard catheters group, 7 (1%) in the antibiotic-impregnated group, and 17 (3%) assigned to heparin-impregnated catheters. Primary analyses showed no effect of impregnated (antibiotic or heparin) catheters compared with standard central venous catheters (hazard ratio [HR] for time to first bloodstream infection 0.71, 95% CI 0.37-1.34). Secondary analyses showed that antibiotic central venous catheters were better than standard central venous catheters (HR 0.43, 0.20-0.96) and heparin central venous catheters (HR 0.42, 0.19-0.93), but heparin did not differ from standard central venous catheters (HR 1.04, 0.53-2.03). Clinically important and statistically significant absolute risk differences were identified only for antibiotic-impregnated catheters versus standard catheters (-2.15%, 95% CI -4.09 to -0.20; number needed to treat [NNT] 47, 95% CI 25-500) and antibiotic-impregnated catheters versus heparin-impregnated catheters (-1.98%, -3.90 to -0.06, NNT 51, 26-1667). Nine children (2%) in the standard central venous catheter group, 14 (3%) in the antibiotic-impregnated group, and 8 (2%) in the heparin-impregnated group had catheter-related adverse events. 45 (8%) in the standard group, 35 (8%) antibiotic-impregnated group, and 29 (6%) in the heparin-impregnated group died during the study. INTERPRETATION Antibiotic-impregnated central venous catheters significantly reduced the risk of bloodstream infections compared with standard and heparin central venous catheters. Widespread use of antibiotic-impregnated central venous catheters could help prevent bloodstream infections in paediatric intensive care units. FUNDING National Institute for Health Research, UK.

Comparison of randomized controlled trial registry entries and content of reports in surgery journals.

Objective:To evaluate discrepancies between trial registry entries and final reports of randomized controlled trials (RCTs) published in major general surgical journals. Background:Health care decisions are based on published results in peer-reviewed journals. Mandatory trial registration was introduced to increase transparency and reduce publication and outcome reporting bias. Methods:The discrepancy rate between trial registry entries and final reports of all RCTs published during 2010 in the Annals of Surgery, Archives of Surgery, and British Journal of Surgery was evaluated. Results:Of 596 identified studies, 545 were excluded because they were not RCTs or interim reports/secondary analysis of RCTs or because of missing trial registry information.In the remaining 51 RCTs, prospective registration was found in 9.8% (n = 5), registration during trial conduct in 33.3% (n = 17), and retrospective registration in 56.9% (n = 29), respectively.For the primary and secondary outcomes, there was no discrepancy in 54.9% and 33.3%, complete omission in 7.8% and 31.3%, new introduction in 7.8% and 39.2%, a change in definition in 9.8% and 5.8%, downgrading from primary to secondary in 21.6%, and upgrading from secondary to primary in 13.7%. There were few discrepancies in randomization, blinding, and intervention and some in targeted sample size and inclusion/exclusion criteria. Conclusions:When interpreting the results of surgical RCTs, the possibility of selective reporting, and thus outcome reporting bias, has to be kept in mind. For future trials, prospective registration should be strictly respected with the ultimate goal to increase transparency and contribute to high-level evidence reports for optimal patient care in surgery.

Reporting of clinical trials: a review of research funders' guidelines.

BackgroundRandomised controlled trials (RCTs) represent the gold standard methodological design to evaluate the effectiveness of an intervention in humans but they are subject to bias, including study publication bias and outcome reporting bias. National and international organisations and charities give recommendations for good research practice in relation to RCTs but to date no review of these guidelines has been undertaken with respect to reporting bias.MethodsNational and international organisations and UK based charities listed on the Association for Medical Research Charities website were contacted in 2007; they were considered eligible for this review if they funded RCTs. Guidelines were obtained and assessed in relation to what was written about trial registration, protocol adherence and trial publication. It was also noted whether any monitoring against these guidelines was undertaken. This information was necessary to discover how much guidance researchers are given on the publication of results, in order to prevent study publication bias and outcome reporting bias.ResultsSeventeen organisations and 56 charities were eligible of 140 surveyed for this review, although there was no response from 12. Trial registration, protocol adherence, trial publication and monitoring against the guidelines were often explicitly discussed or implicitly referred too. However, only eleven of these organisations or charities mentioned the publication of negative as well as positive outcomes and just three of the organisations specifically stated that the statistical analysis plan should be strictly adhered to and all changes should be reported.ConclusionOur review indicates that there is a need to provide more detailed guidance for those conducting and reporting clinical trials to help prevent the selective reporting of results. Statements found in the guidelines generally refer to publication bias rather than outcome reporting bias. Current guidelines need to be updated and include the statement that all primary and secondary outcomes prespecified in the protocol should be fully reported and should not be selected for inclusion in the final report based on their results.

The clinical effectiveness and cost-effectiveness of the PROGENSA® prostate cancer antigen 3 assay and the Prostate Health Index in the diagnosis of prostate cancer: a systematic review and economic evaluation

BACKGROUND There is no single definitive test to identify prostate cancer in men. Biopsies are commonly used to obtain samples of prostate tissue for histopathological examination. However, this approach frequently misses cases of cancer, meaning that repeat biopsies may be necessary to obtain a diagnosis. The PROGENSA(®) prostate cancer antigen 3 (PCA3) assay (Hologic Gen-Probe, Marlborough, MA, USA) and the Prostate Health Index (phi; Beckman Coulter Inc., Brea, CA, USA) are two new tests (a urine test and a blood test, respectively) that are designed to be used to help clinicians decide whether or not to recommend a repeat biopsy. OBJECTIVE To evaluate the clinical effectiveness and cost-effectiveness of the PCA3 assay and the phi in the diagnosis of prostate cancer. DATA SOURCES Multiple publication databases and trial registers were searched in May 2014 (from 2000 to May 2014), including MEDLINE, EMBASE, The Cochrane Library, ISI Web of Science, Medion, Aggressive Research Intelligence Facility database, ClinicalTrials.gov, International Standard Randomised Controlled Trial Number Register and World Health Organization International Clinical Trials Registry Platform. REVIEW METHODS The assessment of clinical effectiveness involved three separate systematic reviews, namely reviews of the analytical validity, the clinical validity of these tests and the clinical utility of these tests. The assessment of cost-effectiveness comprised a systematic review of full economic evaluations and the development of a de novo economic model. SETTING The perspective of the evaluation was the NHS in England and Wales. PARTICIPANTS Men suspected of having prostate cancer for whom the results of an initial prostate biopsy were negative or equivocal. INTERVENTIONS The use of the PCA3 score or phi in combination with existing tests (including histopathology results, prostate-specific antigen level and digital rectal examination), multiparametric magnetic resonance imaging and clinical judgement. RESULTS In addition to documents published by the manufacturers, six studies were identified for inclusion in the analytical validity review. The review identified issues concerning the precision of the PCA3 assay measurements. It also highlighted issues relating to the storage requirements and stability of samples intended for analysis using the phi assay. Fifteen studies met the inclusion criteria for the clinical validity review. These studies reported results for 10 different clinical comparisons. There was insufficient evidence to enable the identification of appropriate test threshold values for use in a clinical setting. In addition, the implications of adding either the PCA3 assay or the phi to clinical assessment were not clear. Furthermore, the addition of the PCA3 assay or the phi to clinical assessment plus magnetic resonance imaging was not found to improve discrimination. No published papers met the inclusion criteria for either the clinical utility review or the cost-effectiveness review. The results from the cost-effectiveness analyses indicated that using either the PCA3 assay or the phi in the NHS was not cost-effective. LIMITATIONS The main limitations of the systematic review of clinical validity are that the review conclusions are over-reliant on findings from one study, the descriptions of clinical assessment vary widely within reviewed studies and many of the reported results for the clinical validity outcomes do not include either standard errors or confidence intervals. CONCLUSIONS The clinical benefit of using the PCA3 assay or the phi in combination with existing tests, scans and clinical judgement has not yet been confirmed. The results from the cost-effectiveness analyses indicate that the use of these tests in the NHS would not be cost-effective. STUDY REGISTRATION This study is registered as PROSPERO CRD42014009595. FUNDING The National Institute for Health Research Health Technology Assessment programme.