Deven Carlson

A Multistate District-Level Cluster Randomized Trial of the Impact of Data-Driven Reform on Reading and Mathematics Achievement:

Analyzing mathematics and reading achievement outcomes from a district-level random assignment study fielded in over 500 schools within 59 school districts and seven states, the authors estimate the 1-year impacts of a data-driven reform initiative implemented by the Johns Hopkins Center for Data-Driven Reform in Education (CDDRE). CDDRE consultants work with districts to implement quarterly student benchmark assessments and provide district and school leaders with extensive training on interpreting and using the data to guide reform. Relative to a control condition, in which districts operated as usual without CDDRE services, the data-driven reform initiative caused statistically significant districtwide improvements in student mathematics achievement. The CDDRE intervention also had a positive effect on reading achievement, but the estimates fell short of conventional levels of statistical significance.

False Alarms and Missed Events: The Impact and Origins of Perceived Inaccuracy in Tornado Warning Systems

Theory and conventional wisdom suggest that errors undermine the credibility of tornado warning systems and thus decrease the probability that individuals will comply (i.e., engage in protective action) when future warnings are issued. Unfortunately, empirical research on the influence of warning system accuracy on public responses to tornado warnings is incomplete and inconclusive. This study adds to existing research by analyzing two sets of relationships. First, we assess the relationship between perceptions of accuracy, credibility, and warning response. Using data collected via a large regional survey, we find that trust in the National Weather Service (NWS; the agency responsible for issuing tornado warnings) increases the likelihood that an individual will opt for protective action when responding to a hypothetical warning. More importantly, we find that subjective perceptions of warning system accuracy are, as theory suggests, systematically related to trust in the NWS and (by extension) stated responses to future warnings. The second half of the study matches survey data against NWS warning and event archives to investigate a critical follow-up question--Why do some people perceive that their warning system is accurate, whereas others perceive that their system is error prone? We find that subjective perceptions are--in part-a function of objective experience, knowledge, and demographic characteristics. When considered in tandem, these findings support the proposition that errors influence perceptions about the accuracy of warning systems, which in turn impact the credibility that people assign to information provided by systems and, ultimately, public decisions about how to respond when warnings are issued.

The Determinants of Interdistrict Open Enrollment Flows: Evidence From Two States

Interdistrict open enrollment is the most widely used form of school choice in the United States. Through the theoretical lens of a utility maximization framework, this article analyzes the determinants of interdistrict open enrollment flows in Minnesota and Colorado. The authors’ empirical analysis employs an original data set that details open enrollment flows between all pairwise combinations of school districts within 100 miles of each other in these two states. These flows are merged with demographic and geographic data from the Common Core of Data and U.S. Census Bureau. The findings indicate that open enrollment flows are driven mainly by student achievement and structural characteristics of districts; distance plays a large constraining role. The results also suggest that most transfers take place between relatively high-achieving districts. The authors discuss the policy implications of these findings.

High-Stakes Choice Achievement and Accountability in the Nation’s Oldest Urban Voucher Program

This article considers the impact of a high-stakes testing and reporting requirement on students using publicly funded vouchers to attend private schools. We describe how such a policy was implemented during the course of a previously authorized multi-year evaluation of the Milwaukee Parental Choice Program, which provided us with data on voucher students before and after the reform, as well as on public school students who received no new policy treatment. Our results indicate substantial growth for voucher students in the first high-stakes testing year, particularly in mathematics, and for students with higher levels of earlier academic achievement. We discuss these results in the context of both the school choice and accountability literatures.

Student Neighborhoods, Schools, and Test Score Growth: Evidence from Milwaukee, Wisconsin.

Schools and neighborhoods are thought to be two of the most important contextual influences on student academic outcomes. Drawing on a unique data set that permits simultaneous estimation of neighborhood and school contributions to student test score gains, we analyze the distributions of these contributions to consider the relative importance of schools and neighborhoods in shaping student achievement outcomes. We also evaluate the sensitivity of estimated school and neighborhood contributions to the exclusion of an explicit measure of the other context, indicating the extent to which bias may exist in studies where either measure is unavailable. Taken together, results of these analyses provide substantial insight into the influences of two of the most important contextual settings in students’ lives.

Life After Vouchers: What Happens to Students Who Leave Private Schools for the Traditional Public Sector?

Few school choice evaluations consider students who leave such programs, and fewer still consider the effects of leaving these programs as policy-relevant outcomes. Using a representative sample of students from the citywide voucher program in Milwaukee, Wisconsin, we analyze more than 1,000 students who leave the program during a 4-year period. We show that low-performing voucher students tend to move from the voucher sector into lower performing and less effective public schools than the typical public school student attends, whereas high-performing students transfer to better public schools. In general, transferring students realize substantial achievement gains after moving to the public sector; these results are robust to multiple analytical approaches. This evidence has important implications for school choice policy and research.

Social Media and Severe Weather: Do Tweets Provide a Valid Indicator of Public Attention to Severe Weather Risk Communication?

AbstractEffective communication about severe weather requires that providers of weather information disseminate accurate and timely messages and that the intended recipients (i.e., the population at risk) receive and react to these messages. This article contributes to extant research on the second half of this equation by introducing a “real time” measure of public attention to severe weather risk communication based on the growing stream of data that individuals publish on social media platforms, in this case, Twitter. The authors develop a metric that tracks temporal fluctuations in tornado-related Twitter activity between 25 April 2012 and 11 November 2012 and assess the validity of the metric by systematically comparing fluctuations in Twitter activity to the issuance of tornado watches and warnings, which represent basic but important forms of communication designed to elicit, and therefore correlate with, public attention. The assessment finds that the measure demonstrates a high degree of converge...

Dynamic Participation in Interdistrict Open Enrollment

Interdistrict open enrollment is the nation’s largest and most widespread school choice program, but our knowledge of these programs is limited. Drawing on 5 years of student-level data from the universe of public school attendees in Colorado, we perform a three-stage analysis to examine the dynamics of student participation in the state’s interdistrict open enrollment program. First, we explore the characteristics of students who open enroll in a defined baseline year. Second, we analyze the characteristics of students who continue to participate in the program in subsequent years. Finally, we examine the characteristics of students who—conditional on not open enrolling in the defined-baseline year—choose to participate in the program in one or more subsequent years.

The Ineffectiveness of High School Graduation Credit Requirement Reforms: A Story of Implementation and Enforcement?

Graduation credit requirement reforms were expected to have a significant impact on the American educational landscape, but scholars have concluded that these reforms have exhibited less impact than expected on a wide range of educational outcomes. Drawing on Lipsky’s theory of street-level bureaucracy, we hypothesize that graduation requirement reforms have been relatively ineffective because of inconsistent implementation and enforcement at the local level. Using data from the National Education Longitudinal Study of 1988 (NELS:88) and the Education Longitudinal Study of 2002 (ELS:2002), we assess the viability of this hypothesized explanation. On the whole, the findings are consistent with our contention that inconsistent implementation and enforcement of graduation credit requirement policies contributes to the relative ineffectiveness of these policy reforms.

The Effect of Private School Vouchers on Political Participation: Experimental Evidence From New York City

ABSTRACT In 1997, the New York School Choice Scholarships Foundation Program (SCSF) randomly offered three-year scholarships to attend private schools to approximately 1,000 low-income families in New York City. In this paper we leverage exogenous variation generated by the SCSF to estimate the causal effect of the private school voucher offer—and the private school attendance it induced—on later-life voter registration and turnout outcomes. Our results demonstrate that the voucher intervention had no effect on registering to vote or voting in any of several elections for either the full sample or any of several demographic subgroups. Although unique aspects of the SCSF context impose limits on the scope of our results, they have clear implications for assessing the relative effectiveness of public and private schools in preparing democratic citizens. We close the paper by discussing the implications of the results for research and policy.