Hugh Waddington

ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions

Non-randomised studies of the effects of interventions are critical to many areas of healthcare evaluation, but their results may be biased. It is therefore important to understand and appraise their strengths and weaknesses. We developed ROBINS-I (“Risk Of Bias In Non-randomised Studies - of Interventions”), a new tool for evaluating risk of bias in estimates of the comparative effectiveness (harm or benefit) of interventions from studies that did not use randomisation to allocate units (individuals or clusters of individuals) to comparison groups. The tool will be particularly useful to those undertaking systematic reviews that include non-randomised studies.

Effectiveness and sustainability of water, sanitation, and hygiene interventions in combating diarrhoea

This paper presents a synthetic review of impact evaluations examining effectiveness of water, sanitation, and hygiene interventions in reducing diarrhoea among children. The evaluations were conducted in 35 low- and middle-income countries during the past three decades. The paper challenges the existing consensus that water treatment at point-of-use and hygiene interventions are necessarily the most effective and sustainable interventions for promoting reduction of diarrhoea. The analysis suggests that sanitation ‘hardware’ interventions are highly effective in reducing diarrhoea morbidity. Moreover, while there is a wealth of trials documenting the effectiveness of water treatment interventions, studies conducted over longer periods tend to show smaller effectiveness and evidence suggests compliance rates and therefore impact may fall markedly over time.

How to do a good systematic review of effects in international development: a tool kit

We provide a ‘how to’ guide to undertake systematic reviews of effects in international development, by which we mean, synthesis of literature relating to the effectiveness of particular development interventions. Our remit includes determining the reviews questions and scope, literature search, critical appraisal, methods of synthesis including meta-analysis, and assessing the extent to which generalisable conclusions can be drawn using a theory-based approach. Our work draws on the experiences of the International Initiative for Impact Evaluations (3ies) systematic reviews programme.

Why do we care about evidence synthesis? An introduction to the special issue on systematic reviews

Systematic reviews are currently in high demand in international development. At least 100 new reviews are ongoing or already completed on a range of topics across the board in international development, many of which were commissioned by policy-making agencies. These new reviews need to be based on answerable questions, using methods of analysis and reporting which are appropriate for social and economic development programmes and relevant to users. This introductory paper lays out why we believe systematic reviews should be an important component of evidence-informed development policy and practice. It concludes by introducing the papers collected in this issue, which aim to demonstrate how reviews can be made to live up to the promises generated around them.

Assessing ‘what works’ in international development: meta-analysis for sophisticated dummies

Many studies of development interventions are individually unable to provide convincing conclusions because of low statistical significance, small size, limited geographical purview and so forth. Systematic reviews and meta-analysis are forms of research synthesis that combine studies of adequate methodological quality to produce more convincing conclusions. In the social sciences, study designs, types of analysis and methodological quality vary tremendously. Combining these studies for meta-analysis entails more demanding risk of bias assessments to ensure that only studies with largely appropriate methodological characteristics are included, and sensitivity analysis should be performed. In this article, we discuss assessing risk of bias and meta-analysis using such diverse studies.

Advancing the field of health systems research synthesis

Those planning, managing and working in health systems worldwide routinely need to make decisions regarding strategies to improve health care and promote equity. Systematic reviews of different kinds can be of great help to these decision-makers, providing actionable evidence at every step in the decision-making process. Although there is growing recognition of the importance of systematic reviews to inform both policy decisions and produce guidance for health systems, a number of important methodological and evidence uptake challenges remain and better coordination of existing initiatives is needed. The Alliance for Health Policy and Systems Research, housed within the World Health Organization, convened an Advisory Group on Health Systems Research (HSR) Synthesis to bring together different stakeholders interested in HSR synthesis and its use in decision-making processes. We describe the rationale of the Advisory Group and the six areas of its work and reflects on its role in advancing the field of HSR synthesis. We argue in favour of greater cross-institutional collaborations, as well as capacity strengthening in low- and middle-income countries, to advance the science and practice of health systems research synthesis. We advocate for the integration of quasi-experimental study designs in reviews of effectiveness of health systems intervention and reforms. The Advisory Group also recommends adopting priority-setting approaches for HSR synthesis and increasing the use of findings from systematic reviews in health policy and decision-making.

Economic impacts of conditional cash transfer programmes: a systematic review and meta-analysis

The results of a systematic review of evidence on the effects of conditional cash transfer (CCT) programmes on household economic outcomes are presented. Out of 1076 original articles found through electronic and handsearches, 46 randomised and quasi-experimental impact evaluations were eligible for the review. The authors used statistical meta-analysis and analysis of programme mechanisms to explore heterogeneity in impacts between and within programmes. They conclude that, for households which benefited from those CCT programmes which have been rigorously evaluated, child labour decreased, particularly for boys, household consumption and investment increased and consumption smoothing improved. In addition, there were limited effects on girls’ labour and mixed effects on adult labour supply in beneficiary households. Limited evidence has been collected on locality-wide impacts in beneficiary communities.

The effects of microcredit on women's control over household spending in developing countries : a systematic review and meta-analysis

The main objective of this campbell systematic review was to provide a systematic review of the evidence on the effects of microcredit on womens control over household spending in developing countries. More specifically, we aimed to answer two related research questions: 1) what does the impact evaluative evidence say about the causal relationship between microcredit and specific dimensions of womens empowerment (womens control over household spending); and 2) what are the mechanisms which mediate this relationship. We prioritise depth of analysis over breadth, thus the scope of this review is narrower than previous systematic reviews on microfinance (stewart et al., 2010; duvendack et al. 2011; stewart et al., 2012). We focused on specific aspects of womens empowerment which allowed us to combine statistical meta-analysis and realist (context-mechanism-outcome) synthesis. From the different searches we identified an initial number of 310 papers that were selected for full text examination. Eventually, 29 papers were retained for further analysis, corresponding to 25 unique studies. In line with three recent other reviews on microfinance (stewart et al., 2010; duvendack et al., 2011; stewart et al. 2012) we found that the microcredit evidence base is extensive, yet most studies are weak methodologically. From those studies deemed comparable and of minimum acceptable quality, we concluded that overall there is no evidence for an effect of microcredit on womens control over household spending.

Quasi-experimental study designs series –Paper 9: Collecting Data from Quasi-Experimental Studies

OBJECTIVE To identify variables that must be coded when synthesizing primary studies that use quasi-experimental designs. STUDY DESIGN AND SETTING All quasi-experimental (QE) designs. RESULTS When designing a systematic review of QE studies, potential sources of heterogeneity-both theory-based and methodological-must be identified. We outline key components of inclusion criteria for syntheses of quasi-experimental studies. We provide recommendations for coding content-relevant and methodological variables and outlined the distinction between bivariate effect sizes and partial (i.e., adjusted) effect sizes. Designs used and controls used are viewed as of greatest importance. Potential sources of bias and confounding are also addressed. CONCLUSION Careful consideration must be given to inclusion criteria and the coding of theoretical and methodological variables during the design phase of a synthesis of quasi-experimental studies. The success of the meta-regression analysis relies on the data available to the meta-analyst. Omission of critical moderator variables (i.e., effect modifiers) will undermine the conclusions of a meta-analysis.

Quasi-experimental study designs series—paper 6: risk of bias assessment

OBJECTIVES Rigorous and transparent bias assessment is a core component of high-quality systematic reviews. We assess modifications to existing risk of bias approaches to incorporate rigorous quasi-experimental approaches with selection on unobservables. These are nonrandomized studies using design-based approaches to control for unobservable sources of confounding such as difference studies, instrumental variables, interrupted time series, natural experiments, and regression-discontinuity designs. STUDY DESIGN AND SETTING We review existing risk of bias tools. Drawing on these tools, we present domains of bias and suggest directions for evaluation questions. RESULTS The review suggests that existing risk of bias tools provide, to different degrees, incomplete transparent criteria to assess the validity of these designs. The paper then presents an approach to evaluating the internal validity of quasi-experiments with selection on unobservables. CONCLUSION We conclude that tools for nonrandomized studies of interventions need to be further developed to incorporate evaluation questions for quasi-experiments with selection on unobservables.