Gabriel Demombynes

When does rigorous impact evaluation make a difference? The case of the Millennium Villages

When is the rigorous impact evaluation of development projects a luxury, and when a necessity? This Paper studies one high-profile case: the Millennium Villages Project (MVP), an experimental and intensive package intervention to spark sustained local economic development in rural Africa. it illustrates the benefits of rigorous impact evaluation in this setting by showing that estimates of the projects effects depend heavily on the evaluation method. Comparing trends at the MVP intervention sites in Kenya, Ghana, and Nigeria to trends in the surrounding areas yields much more modest estimates of the projects effects than the before-versus-after comparisons published thus far by the MVP. Neither approach constitutes a rigorous impact evaluation of the MVP, which is impossible to perform due to weaknesses in the evaluation design of the projects initial phase. These weaknesses include the subjective choice of intervention sites, the subjective choice of comparison sites, the lack of baseline data on comparison sites, the small sample size, and the short time horizon. The authors describe how the next wave of the intervention could be designed to allow proper evaluation of the MVPs impact at little additional cost.

What has driven the decline of infant mortality in Kenya

Substantial declines in infant and under-5 mortality have taken place in recent years in many countries in Sub-Saharan Africa. Kenyas infant mortality rate has fallen by 7.6 percent per year, the fastest rate of decline among the 20 countries in the region for which recent Demographic and Health Survey data is available. Kenyas rate of postneonatal deaths per 1,000 live births fell by more than half over a five-year period, dropping from 47 to 22, as measured using data from the 2003 and 2008-09 Demographic and Health Surveys. Among the possible causes of the decline are various targeted new public health initiatives and improved access to water and sanitation. A Oaxaca-Blinder decomposition using Demographic and Health Survey data shows that the increased ownership of insecticide-treated bednets in endemic malaria zones explains 39 percent of the decline in postneonatal mortality and 58 percent of the decline in infant mortality. Changes in other observable candidate factors do not explain substantial portions of the decline. The portion of the decline not explained may be associated with generalized trends such as the overall improvement in living standards that has taken place with economic growth. The widespread ownership of insecticide-treated bednets in areas of Kenya where malaria is rare suggests that better targeting of insecticide-treated bednet provision programs could improve the cost-effectiveness of such programs.

How Good a Map? Putting Small Area Estimation to the Test

The authors examine the performance of small area welfare estimation. The method combines census and survey data to produce spatially disaggregated poverty and inequality estimates. To test the method, they compare predicted welfare indicators for a set of target populations with their true values. They construct target populations using actual data from a census of households in a set of rural Mexican communities. They examine estimates along three criteria: accuracy of confidence intervals, bias, and correlation with true values. The authors find that while point estimates are very stable, the precision of the estimates varies with alternative simulation methods. While the original approach of numerical gradient estimation yields standard errors that seem appropriate, some computationally less-intensive simulation procedures yield confidence intervals that are slightly too narrow. The precision of estimates is shown to diminish markedly if unobserved location effects at the village level are not well captured in underlying consumption models. With well specified models there is only slight evidence of bias, but the authors show that bias increases if underlying models fail to capture latent location effects. Correlations between estimated and true welfare at the local level are highest for mean expenditure and poverty measures and lower for inequality measures.

Collecting high frequency panel data in Africa using mobile phone interviews

Abstract As mobile phone ownership rates have risen in Africa, there is increased interest in using mobile telephony as a data collection platform. This paper draws on two pilot projects that use mobile phone interviews for data collection in Tanzania and South Sudan. In both cases, high frequency panel data have been collected on a wide range of topics in a manner that is cost effective, flexible and rapid. Attrition has been problematic in both surveys, but can be explained by the resource and organisational constraints that both surveys faced. We analyse the drivers of attrition to generate ideas for how to improve performance in future mobile phone surveys.

Challenges and opportunities of mobile phone-based data collection : evidence from South Sudan

The proliferation of mobile phones in developing countries has generated a wave of interest in collecting high-frequency socioeconomic surveys using this technology. This paper considers lessons from one such survey effort in a difficult environment -- the South Sudan Experimental Phone Survey, which gathered data on living conditions, access to services, and citizen attitudes via monthly interviews by phones provided to respondents. Non-response, particularly in later rounds of the survey, was a substantial problem, largely due to erratic functioning of the mobile network. However, selection due to non-response does not appear to have markedly affected survey results. Response rates were much higher for respondents who owned their own phones. Both compensation provided to respondents in the form of airtime and the type of phone (solar-charged or traditional) were varied experimentally. The type of phone was uncorrelated with response rates and, contrary to expectation, attrition was slightly higher for those receiving the higher level of compensation. The South Sudan Experimental Phone Survey experience suggests that mobile phones can be a viable means of data collection for some purposes, that calling people on their own phones is preferred to handing out phones, and that careful attention should be given to the potential for selective non-response.

The New Transparency in Development Economics: Lessons from the Millennium Villages Controversy

The Millennium Villages Project is a high profile, multi-country development project that has aimed to serve as a model for ending rural poverty in sub-Saharan Africa. The project became the subject of controversy when the methodological basis of early claims of success was questioned. The lively ensuing debate offers lessons on three recent mini-revolutions that have swept the field of development economics: the rising standards of evidence for measuring impact, the “open data” movement, and the growing role of the blogosphere in research debates. In this paper, Michael Clemens and Gabriel Demombynes discuss how a new transparency is changing the debate about what works.

What has driven the decline of infant mortality in Kenya in the 2000s

Substantial declines in early childhood mortality have taken place in many countries in Sub-Saharan Africa. Kenyas infant mortality rate fell by 7.6 percent per year between 2003 and 2008, the fastest rate of decline among the 20 countries in the region for which recent Demographic and Health Survey (DHS) data are available. The average rate of decline across all 20 countries was 3.6 percent per year. Among the possible causes of the observed decline in Kenya is a large-scale campaign to distribute insecticide-treated bednets (ITN) which started in 2004. A Oaxaca-Blinder decomposition using DHS data shows that the increased ownership of bednets in endemic malaria zones explains 79 percent of the decline in infant mortality. Although the Oaxaca-Blinder method cannot identify causal effects, given the wide evidence basis showing that ITN usage can reduce malaria prevalence and the huge surge in ITN ownership in Kenya, it is likely that the decomposition results reflect at least in part a causal effect. The widespread ownership of ITNs in areas of Kenya where malaria is rare suggests that better targeting of ITN provision could improve the cost-effectiveness of such programs.

Costing a Data Revolution

The lack of reliable development statistics for many poor countries has led the U.N. to call for a “data revolution†(United Nations, 2013). One fairly narrow but widespread interpretation of this revolution is for international aid donors to fund a coordinated wave of household surveys across the developing world, tracking progress on a new round of post-2015 Sustainable Development Goals. We use data from the International Household Survey Network (IHSN) to show (i) the supply of household surveys has accelerated dramatically over the past 30 years and that (ii) demand for survey data appears to be higher in democracies and more aid-dependent countries. We also show that given existing international survey programs, the cost to international aid donors of filling remaining survey gaps is manageable--on the order of

Connecting the Unobserved Dots: A Decomposition Analysis of Changes in Earnings Inequality in Urban Argentina, 1980-2002

300 million per year. We argue that any aid-financed expansion of household surveys should be complemented with (a) increased access to data through open data protocols, and (b) simultaneous support for the broader statistical system, including routine administrative data systems.

Crime and Local Inequality in South Africa

There are several possible explanations for the observed changes in inequality, the returns to education, and the gap between the wages of informal and formal salaried workers in Argentina over the period 1980-2002. Largely due to the lack of evidence for competing explanations, skill-biased technical change is the most likely explanation for the increases in the returns to education that occurred in the 1990s. Using a semi-parametric re-weighting variance decomposition technique and data from the Permanent Household Survey, the authors show that during the same period there was an increase in the returns to unobserved skill. This finding lends support to the hypothesis that skill-biased technical change has been a main driver of increases in inequality in Argentina. The pattern of changes suggests that the growth in returns to unobserved skill may have been partly responsible for the relative deterioration of informal salaried wages during the 1990s.