Mengli Song

Critical Issues and Common Pitfalls in Designing and Conducting Impact Studies in Education Lessons Learned From the What Works Clearinghouse (Phase I)

Drawing on our five years of experience developing WWC evidence standards and reviewing studies against those standards as well as current literature on the design of impact studies, we highlight in this paper some of the most critical issues and common pitfalls in designing and conducting impact studies in education, and provide practical guidance on addressing these issues and avoiding common pitfalls. This paper serves as a quick reference guide for researchers who design and conduct impact studies in education and related fields. Information presented in this paper may also help research users make better informed judgments about the quality of the research and the credibility of the evidence produced from the research.

Passing Reading First: Prominence and Processes in an Elite Policy Network

Reading became a hotbed of policy activity during the 1990s with both state and federal policymakers launching major initiatives to raise reading achievement. When President George W. Bush introduced his Reading First initiative in early 2001, we saw a unique opportunity to investigate the policy processes, antecedents, and unplanned events surrounding an important and potentially controversial initiative. We began with the assumption that major policy changes are typically made by powerful actors operating in relatively open issue networks. In regard to the Reading First legislation, however, we found a small clique of inside policy entrepreneurs who fashioned major changes in a short period. Upon recognizing this condition, we expanded the research purpose to describe the actions of this insider policy group.

Longitudinal Findings From the Early College High School Initiative Impact Study

This study is a randomized controlled trial that assessed the impact of Early College High Schools on students’ high school graduation, college enrollment, and college degree attainment, as well as students’ high school experiences using extant data and survey data. The study included 10 Early Colleges that enrolled students in Grades 9 to 12 in 2005 through 2011 and used a lottery for admissions, and 2,458 students who participated in those admission lotteries. The study time frame covered Grade 9 through 2 years post high school for all students and 4 years post high school for the oldest student cohort. It found that Early Colleges had positive impacts on college enrollment and college completion as well as students’ high school experiences.