Alex J. Sutton

Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials.

Funnel plots, and tests for funnel plot asymmetry, have been widely used to examine bias in the results of meta-analyses. Funnel plot asymmetry should not be equated with publication bias, because it has a number of other possible causes. This article describes how to interpret funnel plot asymmetry, recommends appropriate tests, and explains the implications for choice of meta-analysis model

Publication bias in meta-analysis : prevention, assessment and adjustments

Preface. Acknowledgements. Notes on Contributors. Chapter 1: Publication Bias in Meta-Analysis (Hannah R. Rothstein, Alexander J. Sutton and Michael Borenstein). Part A: Publication bias in context. Chapter 2: Publication Bias: Recognizing the Problem, Understanding Its Origins and Scope, and Preventing Harm (Kay Dickersin). Chapter 3: Preventing Publication Bias: Registries and Prospective Meta-Analysis (Jesse A. Berlin and Davina Ghersi). Chapter 4: Grey Literature and Systematic Reviews (Sally Hopewell, Mike Clarke and Sue Mallett). Part B: Statistical methods for assessing publication bias. Chapter 5: The Funnel Plot (Jonathan A.C. Sterne, Betsy Jane Becker and Matthias Egger). Chapter 6: Regression Methods to Detect Publication and Other Bias in Meta-Analysis (Jonathan A.C. Sterne and Matthias Egger). Chapter 7: Failsafe N or File-Drawer Number (Betsy Jane Becker). Chapter 8: The Trim and Fill Method (Sue Duval). Chapter 9: Selection Method Approaches (Larry V. Hedges and Jack Vevea). Chapter 10: Evidence Concerning the Consequences of Publication and Related Biases (Alexander J. Sutton). Chapter 11: Software for Publication Bias (Michael Borenstein). Part C: Advanced and emerging approaches. Chapter 12: Bias in Meta-Analysis Induced by Incompletely Reported Studies (Alexander J. Sutton and Therese D. Pigott). Chapter 13: Assessing the Evolution of Effect Sizes over Time (Thomas A. Trikalinos and John P.A. Ioannidis). Chapter 14: Do Systematic Reviews Based on Individual Patient Data Offer a Means of Circumventing Biases Associated with Trial Publications? (Lesley Stewart, Jayne Tierney and Sarah Burdett). Chapter 15: Differentiating Biases from Genuine Heterogeneity: Distinguishing Artifactual from Substantive Effects (John P.A. Ioannidis). Chapter 16: Beyond Conventional Publication Bias: Other Determinants of Data Suppression (Scott D. Halpern and Jesse A. Berlin). Appendices. Appendix A: Data Sets. Appendix B: Annotated Bibliography (Hannah R. Rothstein and Ashley Busing). Glossary. Index.

Synthesising qualitative and quantitative evidence: a review of possible methods

Background The limitations of traditional forms of systematic review in making optimal use of all forms of evidence are increasingly evident, especially for policy-makers and practitioners. There is an urgent need for robust ways of incorporating qualitative evidence into systematic reviews. Objectives In this paper we provide a brief overview and critique of a selection of strategies for synthesising qualitative and quantitative evidence, ranging from techniques that are largely qualitative and interpretive through to techniques that are largely quantitative and integrative. Results A range of methods is available for synthesising diverse forms of evidence. These include narrative summary, thematic analysis, grounded theory, meta-ethnography, meta-study, realist synthesis, Miles and Hubermans data analysis techniques, content analysis, case survey, qualitative comparative analysis and Bayesian meta-analysis. Methods vary in their strengths and weaknesses, ability to deal with qualitative and quantitative forms of evidence, and type of question for which they are most suitable. Conclusions We identify a number of procedural, conceptual and theoretical issues that need to be addressed in moving forward with this area, and emphasise the need for existing techniques to be evaluated and modified, rather than inventing new approaches.

Pharmacological and lifestyle interventions to prevent or delay type 2 diabetes in people with impaired glucose tolerance: systematic review and meta-analysis

Objective To quantify the effectiveness of pharmacological and lifestyle interventions to prevent or delay type 2 diabetes in people with impaired glucose tolerance. Data sources Medline, Embase, and the Cochrane library searched up to July 2006. Expert opinions sought and reference lists of identified studies and any relevant published reviews checked. Study selection Randomised controlled trials that evaluated interventions to delay or prevent type 2 diabetes in individuals with impaired glucose tolerance. Results 21 trials met the inclusion criteria, of which 17, with 8084 participants with impaired glucose tolerance, reported results in enough detail for inclusion in the meta-analyses. From the meta-analyses the pooled hazard ratios were 0.51 (95% confidence interval 0.44 to 0.60) for lifestyle interventions v standard advice, 0.70 (0.62 to 0.79) for oral diabetes drugs v control, 0.44 (0.28 to 0.69) for orlistat v control, and 0.32 (0.03 to 3.07) for the herbal remedy jiangtang bushen recipe v standard diabetes advice. These correspond to numbers needed to treat for benefit (NNTB) and harm (NNTH) of 6.4 for lifestyle (95% credible interval, NNTB 5.0 to NNTB 8.4), 10.8 for oral diabetes drugs (NNTB 8.1 to NNTB 15.0), 5.4 for orlistat (NNTB 4.1 to NNTB 7.6), and 4.0 for jiangtang bushen (NNTH 16.9 to NNTB 24.8). Conclusions Lifestyle and pharmacological interventions reduce the rate of progression to type 2 diabetes in people with impaired glucose tolerance. Lifestyle interventions seem to be at least as effective as drug treatment.

Empirical assessment of effect of publication bias on meta-analyses

Abstract Objective: To assess the effect of publication bias on the results and conclusions of systematic reviews and meta-analyses. Design: Analysis of published meta-analyses by trim and fill method. Studies: 48 reviews in Cochrane Database of Systematic Reviews that considered a binary endpoint and contained 10 or more individual studies. Main outcome measures: Number of reviews with missing studies and effect on conclusions of meta-analyses. Results: The trim and fill fixed effects analysis method estimated that 26 (54%) of reviews had missing studies and in 10 the number missing was significant. The corresponding figures with a random effects model were 23 (48%) and eight. In four cases, statistical inferences regarding the effect of the intervention were changed after the overall estimate for publication bias was adjusted for. Conclusions: Publication or related biases were common within the sample of meta-analyses assessed. In most cases these biases did not affect the conclusions. Nevertheless, researchers should check routinely whether conclusions of systematic reviews are robust to possible non-random selection mechanisms.

Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups

BackgroundConventional systematic review techniques have limitations when the aim of a review is to construct a critical analysis of a complex body of literature. This article offers a reflexive account of an attempt to conduct an interpretive review of the literature on access to healthcare by vulnerable groups in the UKMethodsThis project involved the development and use of the method of Critical Interpretive Synthesis (CIS). This approach is sensitised to the processes of conventional systematic review methodology and draws on recent advances in methods for interpretive synthesis.ResultsMany analyses of equity of access have rested on measures of utilisation of health services, but these are problematic both methodologically and conceptually. A more useful means of understanding access is offered by the synthetic construct of candidacy. Candidacy describes how peoples eligibility for healthcare is determined between themselves and health services. It is a continually negotiated property of individuals, subject to multiple influences arising both from people and their social contexts and from macro-level influences on allocation of resources and configuration of services. Health services are continually constituting and seeking to define the appropriate objects of medical attention and intervention, while at the same time people are engaged in constituting and defining what they understand to be the appropriate objects of medical attention and intervention. Access represents a dynamic interplay between these simultaneous, iterative and mutually reinforcing processes. By attending to how vulnerabilities arise in relation to candidacy, the phenomenon of access can be better understood, and more appropriate recommendations made for policy, practice and future research.DiscussionBy innovating with existing methods for interpretive synthesis, it was possible to produce not only new methods for conducting what we have termed critical interpretive synthesis, but also a new theoretical conceptualisation of access to healthcare. This theoretical account of access is distinct from models already extant in the literature, and is the result of combining diverse constructs and evidence into a coherent whole. Both the method and the model should be evaluated in other contexts.

Dissemination and publication of research findings: an updated review of related biases.

OBJECTIVES To identify and appraise empirical studies on publication and related biases published since 1998; to assess methods to deal with publication and related biases; and to examine, in a random sample of published systematic reviews, measures taken to prevent, reduce and detect dissemination bias. DATA SOURCES The main literature search, in August 2008, covered the Cochrane Methodology Register Database, MEDLINE, EMBASE, AMED and CINAHL. In May 2009, PubMed, PsycINFO and OpenSIGLE were also searched. Reference lists of retrieved studies were also examined. REVIEW METHODS In Part I, studies were classified as evidence or method studies and data were extracted according to types of dissemination bias or methods for dealing with it. Evidence from empirical studies was summarised narratively. In Part II, 300 systematic reviews were randomly selected from MEDLINE and the methods used to deal with publication and related biases were assessed. RESULTS Studies with significant or positive results were more likely to be published than those with non-significant or negative results, thereby confirming findings from a previous HTA report. There was convincing evidence that outcome reporting bias exists and has an impact on the pooled summary in systematic reviews. Studies with significant results tended to be published earlier than studies with non-significant results, and empirical evidence suggests that published studies tended to report a greater treatment effect than those from the grey literature. Exclusion of non-English-language studies appeared to result in a high risk of bias in some areas of research such as complementary and alternative medicine. In a few cases, publication and related biases had a potentially detrimental impact on patients or resource use. Publication bias can be prevented before a literature review (e.g. by prospective registration of trials), or detected during a literature review (e.g. by locating unpublished studies, funnel plot and related tests, sensitivity analysis modelling), or its impact can be minimised after a literature review (e.g. by confirmatory large-scale trials, updating the systematic review). The interpretation of funnel plot and related statistical tests, often used to assess publication bias, was often too simplistic and likely misleading. More sophisticated modelling methods have not been widely used. Compared with systematic reviews published in 1996, recent reviews of health-care interventions were more likely to locate and include non-English-language studies and grey literature or unpublished studies, and to test for publication bias. CONCLUSIONS Dissemination of research findings is likely to be a biased process, although the actual impact of such bias depends on specific circumstances. The prospective registration of clinical trials and the endorsement of reporting guidelines may reduce research dissemination bias in clinical research. In systematic reviews, measures can be taken to minimise the impact of dissemination bias by systematically searching for and including relevant studies that are difficult to access. Statistical methods can be useful for sensitivity analyses. Further research is needed to develop methods for qualitatively assessing the risk of publication bias in systematic reviews, and to evaluate the effect of prospective registration of studies, open access policy and improved publication guidelines.

Synthesising qualitative and quantitative evidence: A review of possible methods

BACKGROUND The limitations of traditional forms of systematic review in making optimal use of all forms of evidence are increasingly evident, especially for policy-makers and practitioners. There is an urgent need for robust ways of incorporating qualitative evidence into systematic reviews. OBJECTIVES In this paper we provide a brief overview and critique of a selection of strategies for synthesising qualitative and quantitative evidence, ranging from techniques that are largely qualitative and interpretive through to techniques that are largely quantitative and integrative. RESULTS A range of methods is available for synthesising diverse forms of evidence. These include narrative summary, thematic analysis, grounded theory, meta-ethnography, meta-study, realist synthesis, Miles and Hubermans data analysis techniques, content analysis, case survey, qualitative comparative analysis and Bayesian meta-analysis. Methods vary in their strengths and weaknesses, ability to deal with qualitative and quantitative forms of evidence, and type of question for which they are most suitable. CONCLUSIONS We identify a number of procedural, conceptual and theoretical issues that need to be addressed in moving forward with this area, and emphasise the need for existing techniques to be evaluated and modified, rather than inventing new approaches.

Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry

OBJECTIVES To present the contour-enhanced funnel plot as an aid to differentiating asymmetry due to publication bias from that due to other factors. STUDY DESIGN AND SETTING An enhancement to the usual funnel plot is proposed that allows the statistical significance of study estimates to be considered. Contour lines indicating conventional milestones in levels of statistical significance (e.g., <0.01, <0.05, <0.1) are added to funnel plots. RESULTS This contour overlay aids the interpretation of the funnel plot. For example, if studies appear to be missing in areas of statistical nonsignificance, then this adds credence to the possibility that the asymmetry is due to publication bias. Conversely, if the supposed missing studies are in areas of higher statistical significance, this would suggest the cause of the asymmetry may be more likely to be due to factors other than publication bias, such as variable study quality. CONCLUSIONS We believe this enhancement to funnel plots (i) is simple to implement, (ii) is widely applicable, (iii) greatly improves interpretability, and (iv) should be used routinely.

Severe hypoglycaemia and glycaemic control in Type 1 diabetes: meta‐analysis of multiple daily insulin injections compared with continuous subcutaneous insulin infusion

Aims  Continuous subcutaneous insulin infusion (CSII) is a recommended treatment for reducing severe hypoglycaemia in Type 1 diabetes, but the change in hypoglycaemia compared with multiple daily insulin injections (MDI) is unclear. We therefore conducted a meta‐analysis comparing severe hypoglycaemia and glycaemic control during CSII and MDI.