Christel M. J. Vermeersch

Impact evaluation in practice

Development programs and policies are typically designed to change outcomes such as raising incomes, improving learning, or reducing illness. Whether or not these changes are actually achieved is a crucial public policy question, but one that is not often examined. Impact evaluations are part of a broader agenda of evidence-based policy making. This growing global trend is marked by a shift in focus from inputs to outcomes and results, and is reshaping public policy.The first part of the book presents an overview of impact evaluation. Chapter 1 discusses why impact evaluation is important and how it fits within the context of ethical, evidence-based policy making. The authors contrast impact evaluation with monitoring, introduce the defining features of impact evaluation, and discuss complementary approaches, including cost-benefit analysis and cost-effectiveness analysis. The authors introduce a core focus of the book: namely, how a program’s available resources, eligibility criteria for selecting beneficiaries, and timing for mplementation serve to structure options in the selection of impact evaluation methods. Finally, the authors introduce different modalities of impact evaluation, such as prospective and retrospective evaluation, and efficacy versus effectiveness trials and conclude with a discussion on when to use impact evaluations. Chapter 2 discusses how to formulate evaluation questions and hypotheses that are useful for policy. These questions and hypotheses determine the focus of the evaluation. The authors also introduce the fundamental concept of a theory of change and the related use of results chains and performance indicators. Chapter 2 provides the first introduction to the fictional case study, the Health Insurance Subsidy Program (HISP), that is used throughout the book and in the accompanying material found on the Impact Evaluation in Practice website (www.worldbank .org/ieinpractice).

Labor market returns to an early childhood stimulation intervention in Jamaica

Help as hungry children helps young adults Supporters of early childhood interventions follow the rule “better early than late,” but so far theres been limited evidence that the rule applies to disadvantaged children in developing countries. Gertler et al. looked at the earnings of young adults in Jamaica, 20 years after, as toddlers, they were given 2 years of help from community health workers. The earnings of the treatment group caught up to those of a comparison group of well-fed children, but the control group of undernourished children that did not receive the health worker visits has lagged behind. Science, this issue p. 998 Encouraging greater mother-child interactions can compensate in the long term for poor nutrition. A substantial literature shows that U.S. early childhood interventions have important long-term economic benefits. However, there is little evidence on this question for developing countries. We report substantial effects on the earnings of participants in a randomized intervention conducted in 1986–1987 that gave psychosocial stimulation to growth-stunted Jamaican toddlers. The intervention consisted of weekly visits from community health workers over a 2-year period that taught parenting skills and encouraged mothers and children to interact in ways that develop cognitive and socioemotional skills. The authors reinterviewed 105 out of 129 study participants 20 years later and found that the intervention increased earnings by 25%, enough for them to catch up to the earnings of a nonstunted comparison group identified at baseline (65 out of 84 participants).

School meals, educational achievement, and school competition : evidence from a randomized evaluation

This paper examines the effects of subsidized school meals on school participation, educational achievement, and school finance in a developing country setting. The paper uses data from a program that was implemented in 25 randomly chosen preschools in a pool of 50. Childrens school participation was 30 percent higher in the treatment group than in the comparison group. The meals program led to higher curriculum test scores, but only in schools where the teacher was relatively experienced prior to the program. The school meals displaced teaching time and led to larger class sizes. Despite improved incentives, teacher absenteeism remained at a high level of 30 percent. Treatment schools raised their fees, and comparison schools close to treatment schools decreased their fees. Some of the price effects are due to a combination of capacity constraints and pupil transfers that would not happen if the school meals were offered in all schools. The intention-to-treat estimator of the effect of the randomized program incorporates those price effects, and therefore it should be considered a lower bound on the effect of generalized school meals. This insight on price effects generalizes to other randomized program evaluations.

Using performance incentives to improve health outcomes

This study examines the effect of performance incentives for health care providers to provide more and higher quality care in Rwanda on child health outcomes. The authors find that the incentives had a large and significant effect on the weight-for-age of children 0-11 months and on the height-for-age of children 24-49 months. They attribute this improvement to increases in the use and quality of prenatal and postnatal care. Consistent with theory, They find larger effects of incentives on services where monetary rewards and the marginal return to effort are higher. The also find that incentives reduced the gap between provider knowledge and practice of appropriate clinical procedures by 20 percent, implying a large gain in efficiency. Finally, they find evidence of a strong complementarity between performance incentives and provider skill.

Using provider performance incentives to increase HIV testing and counseling services in Rwanda

Paying for performance provides financial rewards to medical care providers for improvements in performance measured by utilization and quality of care indicators. In 2006, Rwanda began a pay for performance scheme to improve health services delivery, including HIV/AIDS services. Using a prospective quasi-experimental design, this study examines the schemes impact on individual and couples HIV testing. We find a positive impact of pay for performance on HIV testing among married individuals (10.2 percentage points increase). Paying for performance also increased testing by both partners by 14.7 percentage point among discordant couples in which only one of the partners is an AIDS patient.

Labor market returns to early childhood stimulation : a 20-year followup to an experimental intervention in Jamaica

This paper finds large effects on the earnings of participants from a randomized intervention that gave psychosocial stimulation to stunted Jamaican toddlers living in poverty. The intervention consisted of one-hour weekly visits from community Jamaican health workers over a 2-year period that taught parenting skills and encouraged mothers to interact and play with their children in ways that would develop their childrens cognitive and personality skills. The authors re-interviewed the study participants 20 years after the intervention. Stimulation increased the average earnings of participants by 42 percent. Treatment group earnings caught up to the earnings of a matched non-stunted comparison group. These findings show that psychosocial stimulation early in childhood in disadvantaged settings can have substantial effects on labor market outcomes and reduce later life inequality.

Impact Evaluation in Practice, Second Edition

The second edition of the Impact Evaluation in Practice handbook is a comprehensive and accessible introduction to impact evaluation for policy makers and development practitioners. First published in 2011, it has been used widely across the development and academic communities. The book incorporates real-world examples to present practical guidelines for designing and implementing impact evaluations. Readers will gain an understanding of impact evaluations and the best ways to use them to design evidence-based policies and programs. The updated version covers the newest techniques for evaluating programs and includes state-of-the-art implementation advice, as well as an expanded set of examples and case studies that draw on recent development challenges. It also includes new material on research ethics and partnerships to conduct impact evaluation. The handbook is divided into four sections: Part One discusses what to evaluate and why; Part Two presents the main impact evaluation methods; Part Three addresses how to manage impact evaluations; Part Four reviews impact evaluation sampling and data collection. Case studies illustrate different applications of impact evaluations. The book links to complementary instructional material available online, including an applied case as well as questions and answers. The updated second edition will be a valuable resource for the international development community, universities, and policy makers looking to build better evidence around what works in development.

More Time is Better: An Evaluation of the Full Time School Program in Uruguay

This paper estimates the impact of the full-time school program in Uruguay on standardized test scores of 6th grade students. The program lengthened the school day from a half day to a full day, and provided additional inputs to schools to make this possible, such as additional teachers and construction of classrooms. The program was not randomly placed, but targeted poor urban schools. Using propensity score matching, the authors construct a comparable group of schools, and show that students in very disadvantaged schools improved in their test scores by 0.07 of a standard deviation per year of participation in the full-time program in mathematics, and 0.04 in language. While the program is expensive, it may, if well targeted, help address inequalities in education in Uruguay, at an increase in cost per student not larger than the current deficit in spending between Uruguay and the rest of the region.

La evaluacion de impacto en la practica

Impact evaluation figures among a broad range of complementary methods that support evidence-based policy. Although this book focuses on quantitative impact evaluation methods, it begins by placing them in the broader results context, which also includes monitoring and other types of evaluation. Part 1 seeks to establish and quantify how an intervention affects the outcomes that are of interest to analysts and policy makers. Part 2 of this book explains what impact evaluations do, what questions they answer, what methods are available for conducting them, and the advantages and disadvantages of each. Part 3 focuses on the operational steps in managing or commissioning an impact evaluation. These steps constitute the building blocks of an impact evaluation that will answer the policy questions that have been formulated and estimate the causal impact of the program. Impact evaluations are a worthwhile investment for many programs and that, coupled with monitoring and other forms of evaluation, they allow for a clear understanding of the eff ectiveness of particular social policies.

Long-run effects of temporary incentives on medical care productivity

The adoption of new clinical practice patterns by medical care providers is often challenging, even when the patterns are believed to be efficacious and profitable. This paper uses a randomized field experiment to examine the effects of temporary financial incentives paid to medical care clinics for the initiation of prenatal care in the first trimester of pregnancy. The rate of early initiation of prenatal care was 34 percent higher in the treatment group than in the control group while the incentives were being paid, and this effect persisted at least 15 months and likely 24 months or more after the incentives ended. These results are consistent with a model where the incentives enable providers to address the fixed costs of overcoming organizational inertia in innovation, and suggest that temporary incentives may be effective at motivating improvements in long-run provider performance at a substantially lower cost than permanent incentives.