Yoon K. Loke

Dissemination and publication of research findings: an updated review of related biases.

OBJECTIVES To identify and appraise empirical studies on publication and related biases published since 1998; to assess methods to deal with publication and related biases; and to examine, in a random sample of published systematic reviews, measures taken to prevent, reduce and detect dissemination bias. DATA SOURCES The main literature search, in August 2008, covered the Cochrane Methodology Register Database, MEDLINE, EMBASE, AMED and CINAHL. In May 2009, PubMed, PsycINFO and OpenSIGLE were also searched. Reference lists of retrieved studies were also examined. REVIEW METHODS In Part I, studies were classified as evidence or method studies and data were extracted according to types of dissemination bias or methods for dealing with it. Evidence from empirical studies was summarised narratively. In Part II, 300 systematic reviews were randomly selected from MEDLINE and the methods used to deal with publication and related biases were assessed. RESULTS Studies with significant or positive results were more likely to be published than those with non-significant or negative results, thereby confirming findings from a previous HTA report. There was convincing evidence that outcome reporting bias exists and has an impact on the pooled summary in systematic reviews. Studies with significant results tended to be published earlier than studies with non-significant results, and empirical evidence suggests that published studies tended to report a greater treatment effect than those from the grey literature. Exclusion of non-English-language studies appeared to result in a high risk of bias in some areas of research such as complementary and alternative medicine. In a few cases, publication and related biases had a potentially detrimental impact on patients or resource use. Publication bias can be prevented before a literature review (e.g. by prospective registration of trials), or detected during a literature review (e.g. by locating unpublished studies, funnel plot and related tests, sensitivity analysis modelling), or its impact can be minimised after a literature review (e.g. by confirmatory large-scale trials, updating the systematic review). The interpretation of funnel plot and related statistical tests, often used to assess publication bias, was often too simplistic and likely misleading. More sophisticated modelling methods have not been widely used. Compared with systematic reviews published in 1996, recent reviews of health-care interventions were more likely to locate and include non-English-language studies and grey literature or unpublished studies, and to test for publication bias. CONCLUSIONS Dissemination of research findings is likely to be a biased process, although the actual impact of such bias depends on specific circumstances. The prospective registration of clinical trials and the endorsement of reporting guidelines may reduce research dissemination bias in clinical research. In systematic reviews, measures can be taken to minimise the impact of dissemination bias by systematically searching for and including relevant studies that are difficult to access. Statistical methods can be useful for sensitivity analyses. Further research is needed to develop methods for qualitatively assessing the risk of publication bias in systematic reviews, and to evaluate the effect of prospective registration of studies, open access policy and improved publication guidelines.

ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions

Non-randomised studies of the effects of interventions are critical to many areas of healthcare evaluation, but their results may be biased. It is therefore important to understand and appraise their strengths and weaknesses. We developed ROBINS-I (“Risk Of Bias In Non-randomised Studies - of Interventions”), a new tool for evaluating risk of bias in estimates of the comparative effectiveness (harm or benefit) of interventions from studies that did not use randomisation to allocate units (individuals or clusters of individuals) to comparison groups. The tool will be particularly useful to those undertaking systematic reviews that include non-randomised studies.

Long-term use of thiazolidinediones and fractures in type 2 diabetes: a meta-analysis

Background: Rosiglitazone and pioglitazone may increase the incidence of fractures. We aimed to determine systematically the risk of fractures associated with thiazolidinedione therapy and to evaluate the effect of the therapy on bone density. Methods: We searched MEDLINE, EMBASE, the Cochrane Central Register of Controlled Trials (CENTRAL), other trial registries and product information sheets through June 2008. We selected long-term (≥ 1 year) randomized controlled trials involving patients with type 2 diabetes and controlled observational studies that described the risk of fractures or changes in bone density with thiazolidinediones. We calculated pooled odds ratios (ORs) for fractures and the weighted mean difference in bone density. Results: We analyzed data from 10 randomized controlled trials involving 13 715 participants and from 2 observational studies involving 31 679 participants. Rosiglitazone and pioglitazone were associated with a significantly increased risk of fractures overall in the 10 randomized controlled trials (OR 1.45, 95% confidence interval [CI] 1.18–1.79; p < 0.001). Five randomized controlled trials showed a significantly increased risk of fractures among women (OR 2.23, 95% CI 1.65–3.01; p < 0.001) but not among men (OR 1.00, 95% CI 0.73–1.39; p = 0.98). The 2 observational studies demonstrated an increased risk of fractures associated with rosiglitazone and pioglitazone. Bone mineral density in women exposed to thiazolidinediones was significantly reduced at the lumbar spine (weighted mean difference –1.11%, 95% CI –2.08% to –0.14%; p = 0.02) and hip (weighted mean difference –1.24%, 95%CI –2.34% to –0.67%; p < 0.001) in 2 randomized controlled trials. Interpretation: Long-term thiazolidinedione use doubles the risk of fractures among women with type 2 diabetes, without a significant increase in risk of fractures among men with type 2 diabetes.

Risk of Clostridium difficile infection with acid suppressing drugs and antibiotics: meta-analysis.

OBJECTIVES:Several studies have raised concern regarding the possible association between proton-pump inhibitors (PPIs) and Clostridium difficile infection (CDI). We aimed to perform a systematic review of incident and recurrent CDI in PPI users, and to evaluate the relative impact of concurrent antibiotic use, or switching acid suppression to histamine-2-receptor antagonists (H2RAs).METHODS:We searched MEDLINE and EMBASE from inception to December 2011 for controlled observational studies that reported on the risk of CDI with and without PPI use. We performed random effects meta-analysis and assessed statistical heterogeneity using the I2 statistic.RESULTS:We included 42 observational studies (30 case-control, 12 cohort) totalling 313,000 participants overall. Pooled analysis of 39 studies showed a statistically significant association between PPI use and risk of developing CDI, odds ratio (OR) 1.74 (95% confidence interval (CI) 1.47–2.85, P<0.001, I2=85%) compared with non-users. A pooled analysis of three studies showed a significant associated risk of recurrent CDI associated with PPIs, OR 2.51 (95% CI 1.16–5.44, P=0.005, I2=78%). Subgroup analysis failed to fully clarify the source of the substantial statistical heterogeneity. Adjusted indirect comparison demonstrated that use of H2RAs as an alternative carried a lower-risk OR 0.71 (95% CI 0.53–0.97) compared with PPIs. Conversely, concomitant use of PPI and antibiotics conferred a greater-risk OR 1.96 (95% CI 1.03–3.70) above that of PPIs alone. For PPI and antibiotics, the Rothmans synergy index was 1.36 and attributable proportion of risk from interaction 0.19, indicating an increased risk from interaction beyond the effects of each drug alone.CONCLUSIONS:Despite the substantial statistical and clinical heterogeneity, our findings indicate a probable association between PPI use and incident and recurrent CDI. This risk is further increased by concomitant use of antibiotics and PPI, whereas H2RAs may be less harmful.

Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews

Objective To investigate basic assumptions and other methodological problems in the application of indirect comparison in systematic reviews of competing healthcare interventions. Design Survey of published systematic reviews. Inclusion criteria Systematic reviews published between 2000 and 2007 in which an indirect approach had been explicitly used. Data extraction Identified reviews were assessed for comprehensiveness of the literature search, method for indirect comparison, and whether assumptions about similarity and consistency were explicitly mentioned. Results The survey included 88 review reports. In 13 reviews, indirect comparison was informal. Results from different trials were naively compared without using a common control in six reviews. Adjusted indirect comparison was usually done using classic frequentist methods (n=49) or more complex methods (n=18). The key assumption of trial similarity was explicitly mentioned in only 40 of the 88 reviews. The consistency assumption was not explicit in most cases where direct and indirect evidence were compared or combined (18/30). Evidence from head to head comparison trials was not systematically searched for or not included in nine cases. Conclusions Identified methodological problems were an unclear understanding of underlying assumptions, inappropriate search and selection of relevant trials, use of inappropriate or flawed methods, lack of objective and validated methods to assess or improve trial similarity, and inadequate comparison or inappropriate combination of direct and indirect evidence. Adequate understanding of basic assumptions underlying indirect and mixed treatment comparison is crucial to resolve these methodological problems. Appendix 1 PubMed search strategy Appendix 2 Characteristics of identified reports Appendix 3 Identified studies References of included studies

Long-term Use of Inhaled Corticosteroids and the Risk of Pneumonia in Chronic Obstructive Pulmonary Disease: A Meta-analysis

BACKGROUND Recent studies have suggested a possible association between pneumonia and the use of inhaled corticosteroids. We aimed to ascertain the risk of pneumonia with long-term inhaled corticosteroid use among patients with chronic obstructive pulmonary disease (COPD). METHODS We performed systematic searches with no date restrictions through June 30, 2008, of MEDLINE, EMBASE, the Cochrane Database of Systematic Reviews, regulatory documents, and trial registries. We included randomized controlled trials of any inhaled corticosteroid vs a control treatment for COPD, with at least 24 weeks of follow-up and reporting of pneumonia as an adverse event. Outcomes evaluated included any pneumonia, serious pneumonia, pneumonia-related mortality, and overall mortality. RESULTS Eighteen randomized controlled trials (n = 16 996) with 24 to 156 weeks of follow-up were included after a detailed screening of 97 articles. Inhaled corticosteroids were associated with a significantly increased risk of any pneumonia (relative risk [RR], 1.60; 95% confidence interval [CI], 1.33-1.92 [P < .001]; I(2) = 16%) and serious pneumonia (1.71; 1.46-1.99 [P < .001]; I(2) = 0%) but without a significantly increased risk of pneumonia-related mortality (1.27; 0.80-2.03 [P = .31]; I(2) = 0%) or overall mortality (0.96; 0.86-1.08 [P = .51]; I(2) = 0%). Inhaled corticosteroids were associated with a significantly increased risk of serious pneumonia when compared with placebo (RR, 1.81; 95% CI, 1.44-2.29 [P < .001]) or when the combination of inhaled corticosteroids and long-acting beta-agonists was compared with long-acting beta-agonists (1.68; 1.20-2.34 [P = .002]). CONCLUSION Among patients with COPD, inhaled corticosteroid use for at least 24 weeks is associated with a significantly increased risk of serious pneumonia, without a significantly increased risk of death.

Risk of serious adverse cardiovascular events associated with varenicline: a systematic review and meta-analysis

Background: There have been postmarketing reports of adverse cardiovascular events associated with the use of varenicline, a widely used smoking cessation drug. We conducted a systematic review and meta-analysis of randomized controlled trials to ascertain the serious adverse cardiovascular effects of varenicline compared with placebo among tobacco users. Methods: We searched MEDLINE, EMBASE, the Cochrane Database of Systematic Reviews, websites of regulatory authorities and registries of clinical trials, with no date or language restrictions, through September 2010 (updated March 2011) for published and unpublished studies. We selected double-blind randomized controlled trials of at least one week’s duration involving smokers or people who used smokeless tobacco that reported on cardiovascular events (ischemia, arrhythmia, congestive heart failure, sudden death or cardiovascular-related death) as serious adverse events asociated with the use of varenicline. Results: We analyzed data from 14 double-blind randomized controlled trials involving 8216 participants. The trials ranged in duration from 7 to 52 weeks. Varenicline was associated with a significantly increased risk of serious adverse cardiovascular events compared with placebo (1.06% [52/4908] in varenicline group v. 0.82% [27/3308] in placebo group; Peto odds ratio [OR] 1.72, 95% confidence interval [CI] 1.09–2.71; I2 = 0%). The results of various sensitivity analyses were consistent with those of the main analysis, and a funnel plot showed no publication bias. There were too few deaths to allow meaningful comparisons of mortality. Interpretation: Our meta-analysis raises safety concerns about the potential for an increased risk of serious adverse cardiovascular events associated with the use of varenicline among tobacco users.

Mortality associated with tiotropium mist inhaler in patients with chronic obstructive pulmonary disease: systematic review and meta-analysis of randomised controlled trials

Objective To systematically review the risk of mortality associated with long term use of tiotropium delivered using a mist inhaler for symptomatic improvement in chronic obstructive pulmonary disease. Data sources Medline, Embase, the pharmaceutical company clinical trials register, the US Food and Drug Administration website, and ClinicalTrials.gov for randomised controlled trials from inception to July 2010. Study selection Trials were selected for inclusion if they were parallel group randomised controlled trials of tiotropium solution using a mist inhaler (Respimat Soft Mist Inhaler, Boehringer Ingelheim) versus placebo for chronic obstructive pulmonary disease; the treatment duration was more than 30 days, and they reported data on mortality. Relative risks of all cause mortality were estimated using a fixed effect meta-analysis, and heterogeneity was assessed with the I2 statistic. Results Five randomised controlled trials were eligible for inclusion. Tiotropium mist inhaler was associated with a significantly increased risk of mortality (90/3686 v 47/2836; relative risk 1.52, 95% confidence interval, 1.06 to 2.16; P=0.02; I2=0%). Both 10 µg (2.15, 1.03 to 4.51; P=0.04; I2=9%) and 5 µg (1.46, 1.01 to 2.10; P=0.04; I2=0%) doses of tiotropium mist inhaler were associated with an increased risk of mortality. The overall estimates were not substantially changed by sensitivity analysis of the fixed effect analysis of the five trials combined using the random effects model (1.45, 1.02 to 2.07; P=0.04), limiting the analysis to three trials of one year’s duration each (1.50, 1.05 to 2.15), or the inclusion of additional data on tiotropium mist inhaler from another investigational drug programme (1.42, 1.01 to 2.00). The number needed to treat for a year with the 5 µg dose to see one additional death was estimated to be 124 (95% confidence interval 52 to 5682) based on the average control event rate from the long term trials. Conclusions This meta-analysis explains safety concerns by regulatory agencies and indicates a 52% increased risk of mortality associated with tiotropium mist inhaler in patients with chronic obstructive pulmonary disease.

Comparative cardiovascular effects of thiazolidinediones: systematic review and meta-analysis of observational studies

Objective To determine the comparative effects of the thiazolidinediones (rosiglitazone and pioglitazone) on myocardial infarction, congestive heart failure, and mortality in patients with type 2 diabetes. Design Systematic review and meta-analysis of observational studies. Data sources Searches of Medline and Embase in September 2010. Study selection Observational studies that directly compared the risk of cardiovascular outcomes for rosiglitazone and pioglitazone among patients with type 2 diabetes mellitus were included. Data extraction Random effects meta-analysis (inverse variance method) was used to calculate the odds ratios for cardiovascular outcomes with thiazolidinedione use. The I2 statistic was used to assess statistical heterogeneity. Results Cardiovascular outcomes from 16 observational studies (4 case-control studies and 12 retrospective cohort studies), including 810 000 thiazolidinedione users, were evaluated after a detailed review of 189 citations. Compared with pioglitazone, use of rosiglitazone was associated with a statistically significant increase in the odds of myocardial infarction (n=15 studies; odds ratio 1.16, 95% confidence interval 1.07 to 1.24; P<0.001; I2=46%), congestive heart failure (n=8; 1.22, 1.14 to 1.31; P<0.001; I2=37%), and death (n=8; 1.14, 1.09 to 1.20; P<0.001; I2=0%). Numbers needed to treat to harm (NNH), depending on the population at risk, suggest 170 excess myocardial infarctions, 649 excess cases of heart failure, and 431 excess deaths for every 100 000 patients who receive rosiglitazone rather than pioglitazone. Conclusion Among patients with type 2 diabetes, use of rosiglitazone is associated with significantly higher odds of congestive heart failure, myocardial infarction, and death relative to pioglitazone in real world settings.

Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview

Su Golder and colleagues carry out an overview of meta-analyses to assess whether estimates of the risk of harm outcomes differ between randomized trials and observational studies. They find that, on average, there is no difference in the estimates of risk between overviews of observational studies and overviews of randomized trials.