Albert E. Beaton

Interpreting Scales through Scale Anchoring.

The major purpose of the National Assessment of Educational Progress (NAEP) is to provide a means to compare groups of students both across and within assessment years. A complementary purpose of NAEP is to provide information about what these groups of students know and can do. This purpose has been addressed using the scale anchoring techniques described in this chapter. Scale anchoring involves a statistical component that identifies items that discriminate between successive points on the proficiency scale using specific item characteristics. It also involves a consensus component in which identified items are used by subject-area and educational experts to provide an interpretation of what groups of students at or close to the selected scale points know and can do.

Overview of the National Assessment of Educational Progress

This chapter gives an overview of the design and the statistical and psychometric analysis methods developed for use in the National Assessment of Educational Progress (NAEP). For more than 20 years, NAEP has provided information about the educational achievements of students in American schools. In recent years, NAEP has been gaining in prominence and has also been growing bigger and more complex. In 1990, an assessment of individual states was added to NAEP. Also, it is anticipated that the legislation that prohibits NAEP from reporting district and school results may be removed and that NAEP may return to annual rather than biennial assessments. In addition, future assessments will involve a larger number of innovative items, such as questions for which students must produce their own answers rather than selecting among specified options, tasks in which students are asked to read aloud, and portfolios that consist of classroom work produced over a period of time. NAEPs never-ending growth and evolution continue to provide new technological challenges to its statisticians and psychometricians.

A New Model for Student Support in High-Poverty Urban Elementary Schools Effects on Elementary and Middle School Academic Outcomes

Efforts to support children in schools require addressing not only academic issues, but also out-of-school factors that can affect students’ ability to succeed. This study examined academic achievement of students participating in City Connects, a student support intervention operating in high-poverty elementary schools. The sample included 7,948 kindergarten to fifth-grade students in a large urban district during 1999–2009. School- and student-level treatment effects on report card grades and standardized test scores in elementary through middle school were estimated. Propensity score methods accounted for pre-intervention group differences. City Connects students demonstrated higher report card scores than comparisons and scored higher on middle school English language arts and mathematics tests. This study provides evidence for the value of addressing out-of-school factors that impact student learning.

The Average Response Method of Scaling

The average response method (ARM) of scaling nonbinary data was developed to scale the data from the assessments of writing conducted by the National Assessment of Educational Progress (NAEP). The ARM applies linear models and multiple imputations technologies to characterize the predictive distribution of the person-level average of ratings over a pool of exercises when each person has responded to only a few of the exercises. The derivations of “plausible values” from the individual-level distributions of potential scale scores are given. Conditions are provided for the unbiasedness of estimates based on the plausible values, and the potential magnitude of the bias when the conditions are not met is indicated. Also discussed is how the plausible values allow for an accounting of the uncertainties due to the sampling of individuals and to the incomplete information on each sampled individual. The technique is illustrated using data from the assessment of writing.

Comparing cross-national student performance on TIMSS using different test items

Abstract The question addressed in this chapter is how “fair” the TIMSS tests were to the various participating countries. The Test-Curriculum Matching Analysis (TCMA) method was used to investigate how results might have changed if different subsets of TIMSS items were considered. The method computes the average proportion correct for each country on each selection of appropriate items. The results of the TCMA is a square matrix, with the rows representing the various results for each country and the columns representing the different items sets deemed appropriate for each country. The results suggested that the relative positions of the countries changed very little as a result of the item selection.

Providing Data for Educational Policy in an International Context: The Third International Mathematics and Science Study (TIMSS)

Policy-makers in many nations of the world are involved in educational reforms. In order to make effective educational decisions for the 21st century, policy-makers need information of a wide variety of kinds, for example, comparative performance data and curriculum information from other nations. National assessments can be valuable, but international surveys provide a broader base of information and allow countries to view their current status and planning within an international perspective. The purpose of this paper is to describe the goals of the Third International Mathematics and Science Study and the steps being followed to insure that the results from the study will meet the diverse informational needs of policy-makers.

Chapter 1: Overview of the National Assessment of Educational Progress

This chapter gives an overview of the design and the statistical and psychometric analysis methods developed for use in the National Assessment of Educational Progress (NAEP). For more than 20 years, NAEP has provided information about the educational achievements of students in American schools. In recent years, NAEP has been gaining in prominence and has also been growing bigger and more complex. In 1990, an assessment of individual states was added to NAEP. Also, it is anticipated that the legislation that prohibits NAEP from reporting district and school results may be removed and that NAEP may return to annual rather than biennial assessments. In addition, future assessments will involve a larger number of innovative items, such as questions for which students must produce their own answers rather than selecting among specified options, tasks in which students are asked to read aloud, and portfolios that consist of classroom work produced over a period of time. NAEP’s never-ending growth and evolution continue to provide new technological challenges to its statisticians and psychometricians.

Chapter 6: Interpreting Scales Through Scale Anchoring

The major purpose of the National Assessment of Educational Progress (NAEP) is to provide a means to compare groups of students both across and within assessment years. A complementary purpose of NAEP is to provide information about what these groups of students know and can do. This purpose has been addressed using the scale anchoring techniques described in this chapter. Scale anchoring involves a statistical component that identifies items that discriminate between successive points on the proficiency scale using specific item characteristics. It also involves a consensus component in which identified items are used by subject-area and educational experts to provide an interpretation of what groups of students at or close to the selected scale points know and can do.

Create News; Consortium for Research on Educational Accountability and Teacher Evaluation; Mathematics and Science Achievement in the Middle School Years: An International Perspective

This issue of CREATE NEWS contains an overview of the results of the Third International Mathematics and Science Study (TIMSS) for middle-school students. A description of the TIMSS project and general international results are presented here in an article authored by researchers at the TIMSS International Study Center at Boston College. More speci®c information about the performance of U.S. middle-school students, teachers, and curricula follows the international overview.

The National Assessment of Educational Progress

Publisher Summary The National Assessment of Educational Progress (NAEP) is a continuing federally mandated national survey of educational achievement. NAEP is intended primarily to estimate the performance of populations of students. They report their aggregated performance and trends in their performance to educational policy makers and the general public. The goals of NAEP are different from other testing programs and this has led to rather unusual features in its design. NAEP is a particularly well-documented testing program. In this chapter, the general design of NAEP is described. Since NAEP is a continuing program, its design is constantly changing to meet new information demands and budgetary restraints. Although the design details may vary, the components of a testing program are relatively fixed. The last section of the chapter is dedicated to the step of reporting. Reporting is the last step, where NAEP fulfills its aim of informing policy makers and the public. NAEP includes a substantial amount of innovative and complex technology. This technology contributes to NAEPs ability to produce readable, informative reports. These reports include not only the required population estimates but also estimates of the accuracy of its results.