Maren Duvendack

High Noon for Microfinance Impact Evaluations: Re-Investigating the Evidence from Bangladesh

Recently, microfinance has come under increasing criticism raising questions of the validity of iconic studies which have justified the microfinance phenomenon. This paper applies propensity score matching (PSM), which has become widely used for the analysis of observational data, to the study by Pitt and Khandker (1998) which has been labelled the most rigorous evidence supporting claims that microfinance benefits the poorest especially when targeted on women. After carefully reconstructing the data we differentiate outcomes by gender of borrower, take account of borrowing from several formal and informal sources, and find that the mainly positive impacts of microfinance that we observe are shown by sensitivity analysis to be highly vulnerable to selection on unobservables, and we are therefore not convinced that the relationships between microfinance and outcomes are causal.

Assessing ‘what works’ in international development: meta-analysis for sophisticated dummies

Many studies of development interventions are individually unable to provide convincing conclusions because of low statistical significance, small size, limited geographical purview and so forth. Systematic reviews and meta-analysis are forms of research synthesis that combine studies of adequate methodological quality to produce more convincing conclusions. In the social sciences, study designs, types of analysis and methodological quality vary tremendously. Combining these studies for meta-analysis entails more demanding risk of bias assessments to ensure that only studies with largely appropriate methodological characteristics are included, and sensitivity analysis should be performed. In this article, we discuss assessing risk of bias and meta-analysis using such diverse studies.

Replication of Quantitative Work in Development Studies: Experiences and Suggestions

There is a growing demand for replications of authoritative works in development studies, which reflects recent trends in other social sciences as well as challenges to important quantitative works in development studies where replications have made contested contributions to understanding. At the same time, there is a strong trend within development towards adoption of medical models of evidence-based policy to find out what policies and interventions work. Replication is a key practice of medical (and natural science) research and was advocated frequently over several decades without success. This article addresses the incentives for replication going beyond a narrow focus on extrinsic rewards, reviews some significant examples, discusses behaviour during replication and draws lessons for replicators and replicatees.

The effects of microcredit on women's control over household spending in developing countries : a systematic review and meta-analysis

The main objective of this campbell systematic review was to provide a systematic review of the evidence on the effects of microcredit on womens control over household spending in developing countries. More specifically, we aimed to answer two related research questions: 1) what does the impact evaluative evidence say about the causal relationship between microcredit and specific dimensions of womens empowerment (womens control over household spending); and 2) what are the mechanisms which mediate this relationship. We prioritise depth of analysis over breadth, thus the scope of this review is narrower than previous systematic reviews on microfinance (stewart et al., 2010; duvendack et al. 2011; stewart et al., 2012). We focused on specific aspects of womens empowerment which allowed us to combine statistical meta-analysis and realist (context-mechanism-outcome) synthesis. From the different searches we identified an initial number of 310 papers that were selected for full text examination. Eventually, 29 papers were retained for further analysis, corresponding to 25 unique studies. In line with three recent other reviews on microfinance (stewart et al., 2010; duvendack et al., 2011; stewart et al. 2012) we found that the microcredit evidence base is extensive, yet most studies are weak methodologically. From those studies deemed comparable and of minimum acceptable quality, we concluded that overall there is no evidence for an effect of microcredit on womens control over household spending.

Meta-analysis of the Impact of Microcredit on Women’s Control over Household Decisions: Methodological Issues and Substantive Findings

Systematic reviews and meta-analysis have risen in popularity in international development to provide evidence on ‘what works’. This paper reports the findings of a meta-analysis to assess the impact of microcredit on women’s control over household spending to illustrate the challenges of conducting meta-analysis in the case of a diverse evidence base. We provide an assessment of methodological quality and present the findings of a meta-analysis. The results suggest that the effect sizes are small. Furthermore, the confidence that we can place in these findings is limited by the high level of heterogeneity within and between studies and the general reliance on non-experimental studies and statistical analyses which are not reported in sufficient detail to enable confident judgement as to their robustness.

Quasi-experimental study designs series –Paper 9: Collecting Data from Quasi-Experimental Studies

OBJECTIVE To identify variables that must be coded when synthesizing primary studies that use quasi-experimental designs. STUDY DESIGN AND SETTING All quasi-experimental (QE) designs. RESULTS When designing a systematic review of QE studies, potential sources of heterogeneity-both theory-based and methodological-must be identified. We outline key components of inclusion criteria for syntheses of quasi-experimental studies. We provide recommendations for coding content-relevant and methodological variables and outlined the distinction between bivariate effect sizes and partial (i.e., adjusted) effect sizes. Designs used and controls used are viewed as of greatest importance. Potential sources of bias and confounding are also addressed. CONCLUSION Careful consideration must be given to inclusion criteria and the coding of theoretical and methodological variables during the design phase of a synthesis of quasi-experimental studies. The success of the meta-regression analysis relies on the data available to the meta-analyst. Omission of critical moderator variables (i.e., effect modifiers) will undermine the conclusions of a meta-analysis.

Quasi-experimental study designs series—paper 10: synthesizing evidence for effects collected from quasi-experimental studies presents surmountable challenges

OBJECTIVE To outline issues of importance to analytic approaches to the synthesis of quasi-experiments (QEs) and to provide a statistical model for use in analysis. STUDY DESIGN AND SETTING We drew on studies of statistics, epidemiology, and social-science methodology to outline methods for synthesis of QE studies. The design and conduct of QEs, effect sizes from QEs, and moderator variables for the analysis of those effect sizes were discussed. RESULTS Biases, confounding, design complexities, and comparisons across designs offer serious challenges to syntheses of QEs. Key components of meta-analyses of QEs were identified, including the aspects of QE study design to be coded and analyzed. Of utmost importance are the design and statistical controls implemented in the QEs. Such controls and any potential sources of bias and confounding must be modeled in analyses, along with aspects of the interventions and populations studied. Because of such controls, effect sizes from QEs are more complex than those from randomized experiments. A statistical meta-regression model that incorporates important features of the QEs under review was presented. CONCLUSION Meta-analyses of QEs provide particular challenges, but thorough coding of intervention characteristics and study methods, along with careful analysis, should allow for sound inferences.

Micro-finance, women’s empowerment and fertility decline in Bangladesh: How important was women’s agency?

Abstract As Nobel Prize winner Amartya Sen has argued “[Bangladesh’s development achievements have] important lessons for other countries across the globe, [in particular a focus on] reducing gender inequality”. A major avenue through which this emphasis has been manifest lies, according to this narrative, in enhancements to women’s agency for instrumental and intrinsic reasons particularly through innovations in family planning and microfinance. The “Bangladesh paradox” of improved wellbeing despite low economic growth over the last four decades is claimed as a paradigmatic case of the spread of both modern family planning programmes and microfinance leading to women’s empowerment and fertility reduction. In this paper we show that the links between microfinance, empowerment and fertility reduction, are fraught with problems, and far from robust; hence the claimed causal links between microfinance and family planning via women’s empowerment needs to be further reconsidered.

DOES GOVERNMENT SPENDING AFFECT INCOME INEQUALITY? A META‐REGRESSION ANALYSIS

In this paper findings of a meta-regression analysis are presented exploring the effects of government spending on income inequality, with a particular focus on low- and middle-income countries. We identify a total of 84 separate studies containing over 900 estimates of the effect of one or more measures of spending on one or more measures of income inequality. The results show some evidence of a moderate negative relationship between government spending and income inequality, which is strongest for social welfare and other social spending, and when using the Gini coefficient or the top income share as the measure of inequality. However, both the size and direction of the estimated relationship between government spending and income inequality is affected by a range of other factors, including the control variables and estimation method used. We also find evidence of publication bias, in that negative estimates of the relationship appear to be under-reported in the literature.

Response to Chemin and to Pitt

Abstract We reply to the discussion and criticisms of Matthieu Chemin (MC) and Mark Pitt (MP) to our paper ((Duvendack and Palmer-Jones (DPJ)) (all this issue). MC clarifies many issues which now make replication pure probable (but not yet achieved), and MP identifies a number of problems with DPJ (some of which are shared with Chemin, 2008). Chemin (2008) made at least one crucial undocumented and unrealistic assumption, and did not document many of his variable constructions. MP correctly identifies inappropriate members of control groups, and other problems, but his claim that his propensity score matching (PSM) results provide support for Pitt and Khandkers (1998 – PnK) most important claim is misleading as it is not robust. We see no reason to change our conclusion that PnK is limited as an evaluation of microfinance by a weak research design which cannot be convincingly mitigated by the sophisticated methods used in PnK, or by PSM.