Jonathan A C Sterne

The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials

Flaws in the design, conduct, analysis, and reporting of randomised trials can cause the effect of an intervention to be underestimated or overestimated. The Cochrane Collaboration’s tool for assessing risk of bias aims to make the process clearer and more accurate

QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.

In 2003, the QUADAS tool for systematic reviews of diagnostic accuracy studies was developed. Experience, anecdotal reports, and feedback suggested areas for improvement; therefore, QUADAS-2 was developed. This tool comprises 4 domains: patient selection, index test, reference standard, and flow and timing. Each domain is assessed in terms of risk of bias, and the first 3 domains are also assessed in terms of concerns regarding applicability. Signalling questions are included to help judge risk of bias. The QUADAS-2 tool is applied in 4 phases: summarize the review question, tailor the tool and produce review-specific guidance, construct a flow diagram for the primary study, and judge bias and applicability. This tool will allow for more transparent rating of bias and applicability of primary diagnostic accuracy studies.

Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls

Most studies have some missing data. Jonathan Sterne and colleagues describe the appropriate use and reporting of the multiple imputation approach to dealing with them

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study

Objective To examine whether the association of inadequate or unclear allocation concealment and lack of blinding with biased estimates of intervention effects varies with the nature of the intervention or outcome. Design Combined analysis of data from three meta-epidemiological studies based on collections of meta-analyses. Data sources 146 meta-analyses including 1346 trials examining a wide range of interventions and outcomes. Main outcome measures Ratios of odds ratios quantifying the degree of bias associated with inadequate or unclear allocation concealment, and lack of blinding, for trials with different types of intervention and outcome. A ratio of odds ratios <1 implies that inadequately concealed or non-blinded trials exaggerate intervention effect estimates. Results In trials with subjective outcomes effect estimates were exaggerated when there was inadequate or unclear allocation concealment (ratio of odds ratios 0.69 (95% CI 0.59 to 0.82)) or lack of blinding (0.75 (0.61 to 0.93)). In contrast, there was little evidence of bias in trials with objective outcomes: ratios of odds ratios 0.91 (0.80 to 1.03) for inadequate or unclear allocation concealment and 1.01 (0.92 to 1.10) for lack of blinding. There was little evidence for a difference between trials of drug and non-drug interventions. Except for trials with all cause mortality as the outcome, the magnitude of bias varied between meta-analyses. Conclusions The average bias associated with defects in the conduct of randomised trials varies with the type of outcome. Systematic reviewers should routinely assess the risk of bias in the results of trials, and should report meta-analyses restricted to trials at low risk of bias either as the primary analysis or in conjunction with less restrictive analyses.

Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials.

Funnel plots, and tests for funnel plot asymmetry, have been widely used to examine bias in the results of meta-analyses. Funnel plot asymmetry should not be equated with publication bias, because it has a number of other possible causes. This article describes how to interpret funnel plot asymmetry, recommends appropriate tests, and explains the implications for choice of meta-analysis model

Prognosis of HIV-1-infected patients starting highly active antiretroviral therapy: a collaborative analysis of prospective studies.

BACKGROUND Insufficient data are available from single cohort studies to allow estimation of the prognosis of HIV-1 infected, treatment-naive patients who start highly active antiretroviral therapy (HAART). The ART Cohort Collaboration, which includes 13 cohort studies from Europe and North America, was established to fill this knowledge gap. METHODS We analysed data on 12,574 adult patients starting HAART with a combination of at least three drugs. Data were analysed by intention-to-continue-treatment, ignoring treatment changes and interruptions. We considered progression to a combined endpoint of a new AIDS-defining disease or death, and to death alone. The prognostic model that generalised best was a Weibull model, stratified by baseline CD4 cell count and transmission group. FINDINGS During 24,310 person-years of follow up, 1094 patients developed AIDS or died and 344 patients died. Baseline CD4 cell count was strongly associated with the probability of progression to AIDS or death: compared with patients starting HAART with less than 50 CD4 cells/microL, adjusted hazard ratios were 0.74 (95% CI 0.62-0.89) for 50-99 cells/microL, 0.52 (0.44-0.63) for 100-199 cells/microL, 0.24 (0.20-0.30) for 200-349 cells/microL, and 0.18 (0.14-0.22) for 350 or more CD4 cells/microL. Baseline HIV-1 viral load was associated with a higher probability of progression only if 100,000 copies/microL or above. Other independent predictors of poorer outcome were advanced age, infection through injection-drug use, and a previous diagnosis of AIDS. The probability of progression to AIDS or death at 3 years ranged from 3.4% (2.8-4.1) in patients in the lowest-risk stratum for each prognostic variable, to 50% (43-58) in patients in the highest-risk strata. INTERPRETATION The CD4 cell count at initiation was the dominant prognostic factor in patients starting HAART. Our findings have important implications for clinical management and should be taken into account in future treatment guidelines.

Funnel plots for detecting bias in meta-analysis: Guidelines on choice of axis

Asymmetry in funnel plots may indicate publication bias in meta-analysis, but the shape of the plot in the absence of bias depends on the choice of axes. We evaluated standard error, precision (inverse of standard error), variance, inverse of variance, sample size and log sample size (vertical axis) and log odds ratio, log risk ratio and risk difference (horizontal axis). Standard error is likely to be the best choice for the vertical axis: the expected shape in the absence of bias corresponds to a symmetrical funnel, straight lines to indicate 95% confidence intervals can be included and the plot emphasises smaller studies which are more prone to bias. Precision or inverse of variance is useful when comparing meta-analyses of small trials with subsequent large trials. The use of sample size or log sample size is problematic because the expected shape of the plot in the absence of bias is unpredictable. We found similar evidence for asymmetry and between trial variation in a sample of 78 published meta-analyses whether odds ratios or risk ratios were used on the horizontal axis. Different conclusions were reached for risk differences and this was related to increased between-trial variation. We conclude that funnel plots of meta-analyses should generally use standard error as the measure of study size and ratio measures of treatment effect.

Systematic reviews in health care: Investigating and dealing with publication and other biases in meta-analysis.

This is the second in a series of four articles Studies that show a significant effect of treatment are more likely to be published, be published in English, be cited by other authors, and produce multiple publications than other studies.1–8 Such studies are therefore also more likely to be identified and included in systematic reviews, which may introduce bias.9 Low methodological quality of studies included in a systematic review is another important source of bias.10 All these biases are more likely to affect small studies than large ones. The smaller a study the larger the treatment effect necessary for the results to be significant. The greater investment of time and money in larger studies means that they are more likely to be of high methodological quality and published even if their results are negative. Bias in a systematic review may therefore become evident through an association between the size of the treatment effect and study size—such associations may be examined both graphically and statistically. #### Summary points Asymmetrical funnel plots may indicate publication bias or be due to exaggeration of treatment effects in small studies of low quality Bias is not the only explanation for funnel plot asymmetry; funnel plots should be seen as a means of examining “small study effects” (the tendency for the smaller studies in a meta-analysis to show larger treatment effects) rather than a tool for diagnosing specific types of bias Statistical methods may be used to examine the evidence for bias and to examine the robustness of the conclusions of the meta-analysis in sensitivity analyses “Correction” of treatment effect estimates for bias should be avoided as such corrections may depend heavily on the assumptions made Multivariable models may be used, with caution, to examine the relative importance of different types of bias ### Funnel plots Funnel …

Mortality of HIV-1-infected patients in the first year of antiretroviral therapy: comparison between low-income and high-income countries

BACKGROUND Highly active antiretroviral therapy (HAART) is being scaled up in developing countries. We compared baseline characteristics and outcomes during the first year of HAART between HIV-1-infected patients in low-income and high-income settings. METHODS 18 HAART programmes in Africa, Asia, and South America (low-income settings) and 12 HIV cohort studies from Europe and North America (high-income settings) provided data for 4810 and 22,217, respectively, treatment-naïve adult patients starting HAART. All patients from high-income settings and 2725 (57%) patients from low-income settings were actively followed-up and included in survival analyses. FINDINGS Compared with high-income countries, patients starting HAART in low-income settings had lower CD4 cell counts (median 108 cells per muL vs 234 cells per muL), were more likely to be female (51%vs 25%), and more likely to start treatment with a non-nucleoside reverse transcriptase inhibitor (NNRTI) (70%vs 23%). At 6 months, the median number of CD4 cells gained (106 cells per muL vs 103 cells per muL) and the percentage of patients reaching HIV-1 RNA levels lower than 500 copies/mL (76%vs 77%) were similar. Mortality was higher in low-income settings (124 deaths during 2236 person-years of follow-up) than in high-income settings (414 deaths during 20,532 person-years). The adjusted hazard ratio (HR) of mortality comparing low-income with high-income settings fell from 4.3 (95% CI 1.6-11.8) during the first month to 1.5 (0.7-3.0) during months 7-12. The provision of treatment free of charge in low-income settings was associated with lower mortality (adjusted HR 0.23; 95% CI 0.08-0.61). INTERPRETATION Patients starting HAART in resource-poor settings have increased mortality rates in the first months on therapy, compared with those in developed countries. Timely diagnosis and assessment of treatment eligibility, coupled with free provision of HAART, might reduce this excess mortality.

Sifting the evidence—what's wrong with significance tests?Another comment on the role of statistical methods

The findings of medical research are often met with considerable scepticism, even when they have apparently come from studies with sound methodologies that have been subjected to appropriate statistical analysis. This is perhaps particularly the case with respect to epidemiological findings that suggest that some aspect of everyday life is bad for people. Indeed, one recent popular history, the medical journalist James Le Fanus The Rise and Fall of Modern Medicine , went so far as to suggest that the solution to medicines ills would be the closure of all departments of epidemiology.1 One contributory factor is that the medical literature shows a strong tendency to accentuate the positive; positive outcomes are more likely to be reported than null results.2–4 By this means alone a host of purely chance findings will be published, as by conventional reasoning examining 20 associations will produce one result that is “significant at P=0.05” by chance alone. If only positive findings are published then they may be mistakenly considered to be of importance rather than being the necessary chance results produced by the application of criteria for meaningfulness based on statistical significance. As many studies contain long questionnaires collecting information on hundreds of variables, and measure a wide range of potential outcomes, several false positive findings are virtually guaranteed. The high volume and often contradictory nature5 of medical research findings, however, is not only because of publication bias. A more fundamental problem is the widespread misunderstanding of the nature of statistical significance. #### Summary points P values, or significance levels, measure the strength of the evidence against the null hypothesis; the smaller the P value, the stronger the evidence against the null hypothesis An arbitrary division of results, into “significant” or “non-significant” according to the P value, was not the intention of the …