Justin Sandefur

Scaling Up What Works: Experimental Evidence on External Validity in Kenyan Education

The recent wave of randomized trials in development economics has provoked criticisms regarding external validity. We investigate two concerns – heterogeneity across beneficiaries and implementers – in a randomized trial of contract teachers in Kenyan schools. The intervention, previously shown to raise test scores in NGO-led trials in Western Kenya and parts of India, was replicated across all Kenyan provinces by an NGO and the government. Strong effects of short-term contracts produced in controlled experimental settings are lost in weak public institutions: NGO implementation produces a positive effect on test scores across diverse contexts, while government implementation yields zero effect. The data suggests that the stark contrast in success between the government and NGO arm can be traced back to implementation constraints and political economy forces put in motion as the program went to scale.

The Political Economy of Bad Data: Evidence from African Survey and Administrative Statistics

Across multiple African countries, discrepancies between administrative data and independent household surveys suggest official statistics systematically exaggerate development progress. We provide evidence for two distinct explanations of these discrepancies. First, governments misreport to foreign donors, as in the case of a results-based aid programme rewarding reported vaccination rates. Second, national governments are themselves misled by frontline service providers, as in the case of primary education, where official enrolment numbers diverged from survey estimates after funding shifted from user fees to per pupil government grants. Both syndromes highlight the need for incentive compatibility between data systems and funding rules.

Context Matters for Size: Why External Validity Claims and Development Practice do not Mix

In this paper we examine how policymakers and practitioners should interpret the impact evaluation literature when presented with conflicting experimental and non-experimental estimates of the same intervention across varying contexts. We show three things. First, as is well known, non-experimental estimates of a treatment effect comprise a causal treatment effect and a bias term due to endogenous selection into treatment. When non-experimental estimates vary across contexts any claim for external validity of an experimental result must make the assumption that (a) treatment effects are constant across contexts, while (b) selection processes vary across contexts. This assumption is rarely stated or defended in systematic reviews of evidence. Second, as an illustration of these issues, we examine two thoroughly researched literatures in the economics of education—class size effects and gains from private schooling—which provide experimental and non-experimental estimates of causal effects from the same context and across multiple contexts. We show that the range of “true” causal effects in these literatures implies OLS estimates from the right context are, at present, a better guide to policy than experimental estimates from a different context. Third, we show that in important cases in economics, parameter heterogeneity is driven by economy- or institution-wide contextual factors, rather than personal characteristics, making it difficult to overcome external validity concerns through estimation of heterogeneous treatment effects within a single localized sample. We conclude with recommendations for research and policy, including the need to evaluate programs in context, and avoid simple analogies to clinical medicine in which “systematic reviews” attempt to identify best-practices by putting most (or all) weight on the most “rigorous” evidence with no allowance for context.

Why Did Abolishing Fees Not Increase Public School Enrollment in Kenya

A large empirical literature has shown that user fees significantly deter public service utilization in developing countries. While most of these results reflect partial equilibrium analysis, we find that the nationwide abolition of public school fees in Kenya in 2003 led to no increase in net public enrollment rates, but rather a dramatic shift toward private schooling. Results suggest this divergence between partial- and general-equilibrium effects is partially explained by social interactions: the entry of poorer pupils into free education contributed to the exit of their more affluent peers.

The High Return to Private Schooling in a Low-Income Country

Existing studies from the United States, Latin America, and Asia provide scant evidence that private schools dramatically improve academic performance relative to public schools. Using data from Kenya — a poor country with weak public institutions — we find a large effect of private schooling on test scores, equivalent to one full standard deviation. This finding is robust to endogenous sorting of more able pupils into private schools. The magnitude of the effect dwarfs the impact of any rigorously-tested intervention to raise performance within public schools. Furthermore, nearly two-thirds of private schools operate at lower cost than the median government school.

Costing a Data Revolution

The lack of reliable development statistics for many poor countries has led the U.N. to call for a “data revolution†(United Nations, 2013). One fairly narrow but widespread interpretation of this revolution is for international aid donors to fund a coordinated wave of household surveys across the developing world, tracking progress on a new round of post-2015 Sustainable Development Goals. We use data from the International Household Survey Network (IHSN) to show (i) the supply of household surveys has accelerated dramatically over the past 30 years and that (ii) demand for survey data appears to be higher in democracies and more aid-dependent countries. We also show that given existing international survey programs, the cost to international aid donors of filling remaining survey gaps is manageable--on the order of

We Just Ran Twenty-Three Million Queries of the World Bank's Website

300 million per year. We argue that any aid-financed expansion of household surveys should be complemented with (a) increased access to data through open data protocols, and (b) simultaneous support for the broader statistical system, including routine administrative data systems.

The Impact of Gavi on Vaccination Rates: Regression Discontinuity Evidence

Much of the data underlying global poverty and inequality estimates is not in the public domain, but can be accessed in small pieces using the World Bank’s PovcalNet online tool. To overcome these limitations and reproduce this database in a format more useful to researchers, we ran approximately 23 million queries of the World Bank’s web site, accessing only information that was already in the public domain. This web scraping exercise produced 10,000 points on the cumulative distribution of income or consumption from each of 942 surveys spanning 127 countries over the period 1977 to 2012. This short note describes our methodology, briefly discusses some of the relevant intellectual property issues, and illustrates the kind of calculations that are facilitated by this data set, including growth incidence curves and poverty rates using alternative PPP indices.

The Returns to Formality and Informality in Urban Africa

Since 2001, an aid consortium known as Gavi has accounted for over half of vaccination expenditure in the 75 eligible countries with an initial per capita GNI below

Does Doing an Apprenticeship Pay Off? Evidence from Ghana

1,000. Regression discontinuity (RD) estimates show aid significantly displaced other immunization efforts and failed to increase vaccination rates for diseases covered by cheap, existing vaccines. For some newer and more expensive vaccines, i.e., Hib and rotavirus, we found large effects on vaccination and limited fungibility, though statistical significance is not robust. These RD estimates apply to middle-income countries near Gavis eligibility threshold, and cannot rule out differential effects for the poorest countries.