Brian M. Stecher

Standards-Based Accountability under No Child Left Behind: Experiences of Teachers and Administrators in Three States. MG-589-NSF.

notice appearing later in this work. This electronic representation of RAND intellectual property is provided for non-commercial use only. Permission is required from RAND to reproduce, or reuse in another form, any of our research documents. Limited Electronic Distribution Rights Visit RAND at www.rand.org Explore RAND Education View document details For More Information Purchase this document Browse Books & Publications Make a charitable contribution Support RAND This PDF document was made available from www.rand.org as a public service of the RAND Corporation.

Studying Large-Scale Reforms of Instructional Practice: An Example from Mathematics and Science

A number of challenges are encountered when evaluating a large-scale, multisite educational reform aimed at changing classroom practice. The challenges include substantial variability in implementation with little information on actual practice, lack of common, appropriate outcome measures, and the need to synthesize evaluation results across multiple study sites. This article describes an approach to addressing these challenges in the context of a study of the relationships between student achievement and instructional practices in the National Science Foundation’s Systemic Initiatives (SI) program. We gathered data from eleven SI sites and investigated relationships at the site level and pooled across sites using a planned meta-analytic approach. We found small but consistent positive relationships between teachers’ reported use of standards-based instruction and student achievement. The article also describes the ways in which we addressed the challenges discussed, as well as a number of additional obstacles that need to be addressed to improve future evaluations of large-scale reforms.

Classroom Assessment Practices, Teacher Judgments, and Student Achievement in Mathematics: Evidence from the ECLS

In this study we use data from the Early Childhood Longitudinal Survey third- and fifth-grade samples to investigate teacher judgments of student achievement, the extent to which they offer a similar picture of student mathematics achievement compared to standardized test scores, and whether classroom assessment practices moderate the relationship between the two measures. Results indicate that teacher ratings correlate strongly with standardized test scores; however, this relationship varies considerably across teachers, and this variation is associated with certain classroom assessment practices. Furthermore, the evidence suggests that teachers evaluate student performance not in absolute terms but relative to other students in the school and that they may adjust their grading for some students, perhaps with basis on perceived differences in need and/or ability.

Unintended Consequences of Test-Based Accountability When Testing in "Milepost" Grades

Educational accountability systems that use test scores as the primary measure of performance are in use in many states. Research has shown that such high-stakes testing can have negative consequences, including narrowing of the curriculum and overemphasizing decontextualized skills. This study examined the impact of a particular form of high-stakes testing, dubbed milepost testing, in which tests are only given at selected grade levels. Data were gathered in Kentucky at a time when state-mandated tests were given in some subjects in grades 4, 7, and 11, and in other subjects in grades 5, 8, and 11. This approach was used to reduce the cost and the burden of assessment and accountability. The study found that many of the effects identified in previous research were present when tests were only given in selected grade levels. This study documents substantial differences in practice between tested and nontested grades and draws implications for national testing proposals that are currently being debated.

The Cost of Science Performance Assessments in Large-Scale Testing Programs:

Estimates of the costs of including hands-on measures of science skills in large-scale assessment programs are drawn from a field trial involving more than 2,000 fifth- and sixth-grade students. These estimates include the resources needed to develop, administer, and score the tasks. They suggest that performance measures are far more expensive than typical multiple-choice tests for an equal amount of testing time, and the cost increases even further for an equally reliable score on an individual student. Because of the complexities of equipment and materials, hands-on measures in science are about three times more expensive than open-ended writing assessments. Alternative approaches to development and administration (such as using less expensive equipment and having the tasks administered by classroom teachers rather than trained Exercise Administrators) could reduce costs by up to 50%, but these practices may reduce the quality of the data obtained. However, including performance assessments in a state’s testing program may have many positive effects, including fostering standards-based educational reform and encouraging more effective teaching methods. The challenge is to determine whether these potential benefits actually exist and if they do, how they can be realized within the budget constraints of most testing programs.

Gender and Racial/Ethnic Differences on Performance Assessments in Science

We examined whether the differences in mean scores among gender and racial/ethnic groups on science performance assessments are comparable to the differences that are typically found among these groups on traditional multiple-choice tests. To do this, several hands-on science performance assessments and other measures were administered to over 2,000 students in grades five, six, and nine as part of a field test of California’s statewide testing program. Girls tended to have higher overall mean scores than boys on the performance measures, but boys tended to score higher than girls on certain types of questions within a performance task. In contrast, differences in mean scores among racial/ethnic groups on one type of test (or question) were comparable to the differences among these groups on the other measures studied. Overall, the results suggest that the type of science test used is unlikely to have much effect on gender or racial/ethnic differences in scores.

Team Pay for Performance Experimental Evidence From the Round Rock Pilot Project on Team Incentives

Education policymakers have shown increased interest in incentive programs for teachers based on the outcomes of their students. This article examines a program in which bonuses were awarded to teams of middle school teachers based on their collective contribution to student test score gains. The study employs a randomized controlled trial to examine effects of the bonus program over the course of an academic year, with the experiment repeated a second year, and finds no significant effects on the achievement of students or the attitudes and practices of teachers. The lack of effects of team-level pay for performance in this study is consistent with other recent experiments studying the short-term effects of bonus awards for individual performance or whole-school performance.

Using Structured Classroom Vignettes to Measure Instructional Practices in Mathematics

Large-scale educational studies frequently require accurate descriptions of classroom practices to judge implementation and impact. However, it can be difficult to obtain these descriptions in a timely, efficient manner. To address this problem, the authors developed a vignette-based measure of one aspect of mathematics instructional practice, reform-oriented instruction. Teachers read contextualized descriptions of teaching practices that varied in terms of reform-oriented instruction, and rated the degree to which the options corresponded to their own likely behaviors. Responses from 80 fourth-grade teachers yielded fairly consistent responses across two parallel vignettes and moderate correlations with other scales of reform-oriented instruction derived from classroom observations, surveys, and logs. The results suggested that the vignettes measure important aspects of reform-oriented instruction that are not captured by other measurement methods. Based on this work, it appears that vignettes can be a useful tool for research on instructional practice, but cognitive interviews with participating teachers provided insight into possible improvements to the items.

Standards-Based Accountability in the United States: Lessons Learned and Future Directions 1

Standards-based accountability (SBA) has been a primary driver of education policy in the United States for several decades. Although definitions of SBA vary, it typically includes standards that indicate what students are expected to know and be able to do, measures of student attainment of the standards, targets for performance on those measures, and a set of consequences for schools or educations based on performance. Research on SBA indicates that these policies have led to some of the consequences its advocates had hoped to achieve, such as an emphasis on equity and alignment of curriculum within and across grade levels, but that it has also produced some less desirable outcomes. This article summarizes the research on SBA in three areas: quality of standards, ways in which SBA has shaped educators’ practices, and effects on student achievement. The article identifies lessons learned from the implication of SBA in the United States and provides guidance for developing SBA systems that could promote beneficial outcomes for students.

Integrating findings from Surveys and Case Studies: Examples from a Study of Standards-Based Educational Reform

Researchers are increasingly using multiple research methods when studying large-scale school reforms, and the literature contains limited guidance about how to do this well. Over the past five years, the authors combined state-wide teacher surveys and exemplary school case studies to examine the impact of standards-based educational reforms in Kentucky and Washington. This paper uses examples from the study of the effects of the Washington education reform to explore how these methods can be used in complementary ways. It describes some of the benefits and shortcomings of the attempt to use both methods and makes recommendations for more effective integration of case study and survey methods in the future.