Kristen Huff

The Promises and Challenges of Implementing Evidence-Centered Design in Large-Scale Assessment

The cornerstone of evidence-centered assessment design (ECD) is an evidentiary argument that requires that each target of measurement (e.g., learning goal) for an assessment be expressed as a claim to be made about an examinee that is relevant to the specific purpose and audience(s) for the assessment. The observable evidence required to warrant each claim is also articulated. In turn, the claims and evidence shape the design of assessment opportunities for students to demonstrate what they have learned, whether that opportunity is a classroom activity or a multiple-choice item on a high-stakes assessment. Once identified, the characteristics of these assessment opportunities are referred to as task models, each capable of generating multiple assessment tasks. Taken together, the claims, evidence, and task models constitute the evidentiary argument. The benefits and challenges of implementing ECD in the Advanced Placement Program are addressed.

Claims, Evidence, and Achievement-Level Descriptors as a Foundation for Item Design and Test Specifications.

Evidence-centered assessment design (ECD) explicates a transparent evidentiary argument to warrant the inferences we make from student test performance. This article describes how the vehicles for gathering student evidence—task models and test specifications—are developed. Task models, which are the basis for item development, flow directly from the claims we want to make about students and the observable evidence required to warrant those claims. The claims and evidence, and therefore, the task models, are directly connected to the underlying construct as defined by the achievement-level descriptors (ALDs). Test specifications must balance the claims and evidence that distinguish students at each achievement level with the psychometric characteristics necessary for reliability and comparability. The explicit relationships among claims, evidence, task models, items, achievement levels, and test specifications provide an evidentiary argument for making valid inferences about student test performance. The iterative nature, challenges, and resulting benefits of this evidence-centered approach to item design and test specifications are discussed.

Innovations in Setting Performance Standards for K–12 Test-Based Accountability

Standard setting is a systematic process that uses a combination of judgmental and empirical procedures to make recommendations about where on the score continuum cut scores should be placed. Cut scores divide the score scale into categories consistent with the descriptions of student performance associated with multiple levels of achievement. Critical decisions will be based on interpretations emerging from how students are categorized into each level. Ensuring that the methodology used to set the cut scores is sound and defensible is a prerequisite for valid interpretations of student performance. The proper functioning of any educational accountability system rests on the valid interpretation of student performance—as does the educational reform for which such an accountability system was put in place: ensuring that our students are achieving rigorous educational outcomes. That is, there is a direct and powerful connection between the outcomes of standard settings— the cut scores—and what happens in our classrooms on a daily basis. It is irrefutable that the procedures and results of standard setting need to be supported by sufficient validity evidence; as such, it is our due diligence as a field to invest in the continuous improvement of our methodologies and practice. Innovations in standard setting methodology in the first decade of this century have been prolific; the number of articles, chapters and presentations are too numerous to cite or review here, but the fact that there have been at least three books just on standard setting this decade is testimony to the amount of research and development in this area (Cizek, 2001; Cizek & Bunch, 2007; Zieky, Perie, & Livingston, 2008). In this paper, we question whether all of this work, effort, and resources has been misdirected given the less-than-adequate work on one of the most critical inputs to the standard setting process: the achievement level descriptions. In other words, like the crude adage “garbage in, garbage out,” innovations in standard setting methodology are attenuated (and not a very good use of our field’s limited resources) until some necessary preconditions are met. We outline these necessary preconditions, their promise for improving standard setting, their promise for improving the accountability system and the reform effort for