Carrol Gamble

Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias

Background The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias has been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making. Until recently, outcome reporting bias has received less attention. Methodology/Principal Findings We review and summarise the evidence from a series of cohort studies that have assessed study publication bias and outcome reporting bias in randomised controlled trials. Sixteen studies were eligible of which only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Eleven of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40–62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies. Conclusions Recent work provides direct empirical evidence for the existence of study publication bias and outcome reporting bias. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.

The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews

Objective To examine the prevalence of outcome reporting bias—the selection for publication of a subset of the original recorded outcome variables on the basis of the results—and its impact on Cochrane reviews. Design A nine point classification system for missing outcome data in randomised trials was developed and applied to the trials assessed in a large, unselected cohort of Cochrane systematic reviews. Researchers who conducted the trials were contacted and the reason sought for the non-reporting of data. A sensitivity analysis was undertaken to assess the impact of outcome reporting bias on reviews that included a single meta-analysis of the review primary outcome. Results More than half (157/283 (55%)) the reviews did not include full data for the review primary outcome of interest from all eligible trials. The median amount of review outcome data missing for any reason was 10%, whereas 50% or more of the potential data were missing in 70 (25%) reviews. It was clear from the publications for 155 (6%) of the 2486 assessable trials that the researchers had measured and analysed the review primary outcome but did not report or only partially reported the results. For reports that did not mention the review primary outcome, our classification regarding the presence of outcome reporting bias was shown to have a sensitivity of 88% (95% CI 65% to 100%) and specificity of 80% (95% CI 69% to 90%) on the basis of responses from 62 trialists. A third of Cochrane reviews (96/283 (34%)) contained at least one trial with high suspicion of outcome reporting bias for the review primary outcome. In a sensitivity analysis undertaken for 81 reviews with a single meta-analysis of the primary outcome of interest, the treatment effect estimate was reduced by 20% or more in 19 (23%). Of the 42 meta-analyses with a statistically significant result only, eight (19%) became non-significant after adjustment for outcome reporting bias and 11 (26%) would have overestimated the treatment effect by 20% or more. Conclusions Outcome reporting bias is an under-recognised problem that affects the conclusions in a substantial proportion of Cochrane reviews. Individuals conducting systematic reviews need to address explicitly the issue of missing outcome data for their review to be considered a reliable source of evidence. Extra care is required during data extraction, reviewers should identify when a trial reports that an outcome was measured but no results were reported or events observed, and contact with trialists should be encouraged.

The SANAD study of effectiveness of carbamazepine, gabapentin, lamotrigine, oxcarbazepine, or topiramate for treatment of partial epilepsy: an unblinded randomised controlled trial

BACKGROUND Carbamazepine is widely accepted as a drug of first choice for patients with partial onset seizures. Several newer drugs possess efficacy against these seizure types but previous randomised controlled trials have failed to inform a choice between these drugs. We aimed to assess efficacy with regards to longer-term outcomes, quality of life, and health economic outcomes. METHODS SANAD was an unblinded randomised controlled trial in hospital-based outpatient clinics in the UK. Arm A recruited 1721 patients for whom carbamazepine was deemed to be standard treatment, and they were randomly assigned to receive carbamazepine, gabapentin, lamotrigine, oxcarbazepine, or topiramate. Primary outcomes were time to treatment failure, and time to 12-months remission, and assessment was by both intention to treat and per protocol. This study is registered as an International Standard Randomised Controlled Trial, number ISRCTN38354748. FINDINGS For time to treatment failure, lamotrigine was significantly better than carbamazepine (hazard ratio [HR] 0.78 [95% CI 0.63-0.97]), gabapentin (0.65 [0.52-0.80]), and topiramate (0.64 [0.52-0.79]), and had a non-significant advantage compared with oxcarbazepine (1.15 [0.86-1.54]). For time to 12-month remission carbamazepine was significantly better than gabapentin (0.75 [0.63-0.90]), and estimates suggest a non-significant advantage for carbamazepine against lamotrigine (0.91 [0.77-1.09]), topiramate (0.86 [0.72-1.03]), and oxcarbazepine (0.92 [0.73-1.18]). In a per-protocol analysis, at 2 and 4 years the difference (95% CI) in the proportion achieving a 12-month remission (lamotrigine-carbamazepine) is 0 (-8 to 7) and 5 (-3 to 12), suggesting non-inferiority of lamotrigine compared with carbamazepine. INTERPRETATION Lamotrigine is clinically better than carbamazepine, the standard drug treatment, for time to treatment failure outcomes and is therefore a cost-effective alternative for patients diagnosed with partial onset seizures.

The SANAD study of effectiveness of valproate, lamotrigine, or topiramate for generalised and unclassifiable epilepsy: an unblinded randomised controlled trial.

BACKGROUND Valproate is widely accepted as a drug of first choice for patients with generalised onset seizures, and its broad spectrum of efficacy means it is recommended for patients with seizures that are difficult to classify. Lamotrigine and topiramate are also thought to possess broad spectrum activity. The SANAD study aimed to compare the longer-term effects of these drugs in patients with generalised onset seizures or seizures that are difficult to classify. METHODS SANAD was an unblinded randomised controlled trial in hospital-based outpatient clinics in the UK. Arm B of the study recruited 716 patients for whom valproate was considered to be standard treatment. Patients were randomly assigned to valproate, lamotrigine, or topiramate between Jan 12, 1999, and Aug 31, 2004, and follow-up data were obtained up to Jan 13, 2006. Primary outcomes were time to treatment failure, and time to 1-year remission, and analysis was by both intention to treat and per protocol. This study is registered as an International Standard Randomised Controlled Trial, number ISRCTN38354748. FINDINGS For time to treatment failure, valproate was significantly better than topiramate (hazard ratio 1.57 [95% CI 1.19-2.08]), but there was no significant difference between valproate and lamotrigine (1.25 [0.94-1.68]). For patients with an idiopathic generalised epilepsy, valproate was significantly better than both lamotrigine (1.55 [1.07-2.24] and topiramate (1.89 [1.32-2.70]). For time to 12-month remission valproate was significantly better than lamotrigine overall (0.76 [0.62-0.94]), and for the subgroup with an idiopathic generalised epilepsy 0.68 (0.53-0.89). But there was no significant difference between valproate and topiramate in either the analysis overall or for the subgroup with an idiopathic generalised epilepsy. INTERPRETATION Valproate is better tolerated than topiramate and more efficacious than lamotrigine, and should remain the drug of first choice for many patients with generalised and unclassified epilepsies. However, because of known potential adverse effects of valproate during pregnancy, the benefits for seizure control in women of childbearing years should be considered.

Immediate versus deferred antiepileptic drug treatment for early epilepsy and single seizures: a randomised controlled trial

BACKGROUND The relative risks and benefits of starting or withholding antiepileptic drug treatment in patients with few or infrequent seizures are unclear. We sought to compare policies of immediate versus deferred treatment in such patients and to assess the effects of these policies on short-term recurrence and long-term outcomes. METHODS We undertook an unmasked, multicentre, randomised study of immediate and deferred antiepileptic drug treatment in 1847 patients with single seizures and early epilepsy. Outcomes comprised time to first, second, and fifth seizures; time to 2-year remission; no seizures between years 1 and 3 and between years 3 and 5 after randomisation; and quality of life. Analysis was by intention to treat. FINDINGS 404 patients invited to join the trial did not consent to randomisation; 722 were subsequently assigned immediate treatment with antiepileptic drugs and 721 were assigned deferred treatment. Immediate treatment increased time to first seizure (hazard ratio 1.4 [95% CI 1.2 to 1.7]), second seizure (1.3 [1.1 to 1.6]), and first tonic-clonic seizure (1.5 [1.2 to 1.8]). It also reduced the time to achieve 2-year remission of seizures (p=0.023). At 5-years follow-up, 76% of patients in the immediate treatment group and 77% of those in the deferred treatment group were seizure free between 3 and 5 years after randomisation (difference -0.2% [95% CI -5.8% to 5.5%]). The two policies did not differ with respect to quality of life outcomes or serious complications. INTERPRETATION Immediate antiepileptic drug treatment reduces the occurrence of seizures in the next 1-2 years, but does not affect long-term remission in individuals with single or infrequent seizures.

Public knowledge, private grief: a study of public attitudes to epilepsy in the United Kingdom and implications for stigma.

Summary:  Purpose: For many people with epilepsy, the continuing social reality of their condition is as a stigma, thus representing a source of much private grief. To understand fully the nature of epilepsy stigma, it is important to examine attitudes and beliefs of not just the “targets” but also of the “perceivers” of stigma. Perceivers may hold erroneous beliefs and stereotypes that lead them to have negative expectations of people affected by epilepsy. This study examined levels of knowledge and attitudes among perceivers of epilepsy stigma in the U.K.

Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialists

Objectives To provide information on the frequency and reasons for outcome reporting bias in clinical trials. Design Trial protocols were compared with subsequent publication(s) to identify any discrepancies in the outcomes reported, and telephone interviews were conducted with the respective trialists to investigate more extensively the reporting of the research and the issue of unreported outcomes. Participants Chief investigators, or lead or coauthors of trials, were identified from two sources: trials published since 2002 covered in Cochrane systematic reviews where at least one trial analysed was suspected of being at risk of outcome reporting bias (issue 4, 2006; issue 1, 2007, and issue 2, 2007 of the Cochrane library); and a random sample of trial reports indexed on PubMed between August 2007 and July 2008. Setting Australia, Canada, Germany, the Netherlands, New Zealand, the United Kingdom, and the United States. Main outcome measures Frequency of incomplete outcome reporting—signified by outcomes that were specified in a trial’s protocol but not fully reported in subsequent publications—and trialists’ reasons for incomplete reporting of outcomes. Results 268 trials were identified for inclusion (183 from the cohort of Cochrane systematic reviews and 85 from PubMed). Initially, 161 respective investigators responded to our requests for interview, 130 (81%) of whom agreed to be interviewed. However, failure to achieve subsequent contact, obtain a copy of the study protocol, or both meant that final interviews were conducted with 59 (37%) of the 161 trialists. Sixteen trial investigators failed to report analysed outcomes at the time of the primary publication, 17 trialists collected outcome data that were subsequently not analysed, and five trialists did not measure a prespecified outcome over the course of the trial. In almost all trials in which prespecified outcomes had been analysed but not reported (15/16, 94%), this under-reporting resulted in bias. In nearly a quarter of trials in which prespecified outcomes had been measured but not analysed (4/17, 24%), the “direction” of the main findings influenced the investigators’ decision not to analyse the remaining data collected. In 14 (67%) of the 21 randomly selected PubMed trials, there was at least one unreported efficacy or harm outcome. More than a quarter (6/21, 29%) of these trials were found to have displayed outcome reporting bias. Conclusion The prevalence of incomplete outcome reporting is high. Trialists seemed generally unaware of the implications for the evidence base of not reporting all outcomes and protocol changes. A general lack of consensus regarding the choice of outcomes in particular clinical settings was evident and affects trial design, conduct, analysis, and reporting.

Outcome selection bias in meta-analysis.

Publication bias has been previously identified as a threat to the validity of a meta-analysis. Recently, new evidence has documented an additional threat to validity, the selective reporting of trial outcomes within published studies. Several diseases have several possible measures of outcome. Some articles might report only a selection of those outcomes, perhaps those with statistically significant results. In this article, we review this problem while addressing the questions: what is within-study selective reporting? how common is it? why is it done? how can it mislead? how can it be detected?, and finally, what is the solution? We recommend that both publication bias and selective reporting should be routinely investigated in systematic reviews.

Interventions to improve recruitment and retention in clinical trials: a survey and workshop to assess current practice and future priorities

BackgroundDespite significant investment in infrastructure many trials continue to face challenges in recruitment and retention. We argue that insufficient focus has been placed on the development and testing of recruitment and retention interventions.MethodsIn this current paper, we summarize existing reviews about interventions to improve recruitment and retention. We report survey data from Clinical Trials Units in the United Kingdom to indicate the range of interventions used by these units to encourage recruitment and retention. We present the views of participants in a recent workshop and a priority list of recruitment interventions for evaluation (determined by voting among workshop participants). We also discuss wider issues concerning the testing of recruitment interventions.ResultsMethods used to encourage recruitment and retention were categorized as: patient contact, patient convenience, support for recruiters, monitoring and systems, incentives, design, resources, and human factors. Interventions felt to merit investigation by respondents fell into three categories: training site staff, communication with patients, and incentives.ConclusionsSignificant resources continue to be invested into clinical trials and other high quality studies, but recruitment remains a significant challenge. Adoption of innovative methods to develop, test, and implement recruitment interventions are required.

Indirect Comparisons: A Review of Reporting and Methodological Quality

Background The indirect comparison of two interventions can be valuable in many situations. However, the quality of an indirect comparison will depend on several factors including the chosen methodology and validity of underlying assumptions. Published indirect comparisons are increasingly more common in the medical literature, but as yet, there are no published recommendations of how they should be reported. Our aim is to systematically review the quality of published indirect comparisons to add to existing empirical data suggesting that improvements can be made when reporting and applying indirect comparisons. Methodology/Findings Reviews applying statistical methods to indirectly compare the clinical effectiveness of two interventions using randomised controlled trials were eligible. We searched (1966–2008) Database of Abstracts and Reviews of Effects, The Cochrane library, and Medline. Full review publications were assessed for eligibility. Specific criteria to assess quality were developed and applied. Forty-three reviews were included. Adequate methodology was used to calculate the indirect comparison in 41 reviews. Nineteen reviews assessed the similarity assumption using sensitivity analysis, subgroup analysis, or meta-regression. Eleven reviews compared trial-level characteristics. Twenty-four reviews assessed statistical homogeneity. Twelve reviews investigated causes of heterogeneity. Seventeen reviews included direct and indirect evidence for the same comparison; six reviews assessed consistency. One review combined both evidence types. Twenty-five reviews urged caution in interpretation of results, and 24 reviews indicated when results were from indirect evidence by stating this term with the result. Conclusions This review shows that the underlying assumptions are not routinely explored or reported when undertaking indirect comparisons. We recommend, therefore, that the quality of indirect comparisons should be improved, in particular, by assessing assumptions and reporting the assessment methods applied. We propose that the quality criteria applied in this article may provide a basis to help review authors carry out indirect comparisons and to aid appropriate interpretation.