David Niemi

Relationships among Measures as Empirical Evidence of Validity: Incorporating Multiple Indicators of Achievement and School Context.

In this article we examine empirical evidence on the criterion, predictive, transfer, and fairness aspects of validity of a large-scale language arts performance assessment, referred to as the Performance Assignment (PA). We use multilevel models to avoid biased inferences that might result from the naturally nested data. Specifically, we examine the relationships of the assessment with the Stanford Achievement Test, 9th Edition and the California High School Exit Examination. The results indicate that the measures are related, that students demonstrate a degree of transfer, and that the language arts PA is relatively more fair than comparison assessments. The results are robust to various model specifications and demonstrate that benefits do not accrue to all students equally.

The effects of POWERSOURCE© assessments on middle-school students’ math performance

This paper describes results from field testing of middle-school math formative assessments alongside professional development and instructional resources. We employed a randomised, controlled design to address the question: Does using our formative assessment strategies improve student performance on assessments of key mathematical ideas relative to a comparison group? This study also provided data on the instructional sensitivity of the assessments, which is part of the validation needed for formative assessments. Teachers were recruited from two districts and seven middle schools. Nineteen treatment and 17 comparison group teachers and their students were included in study analyses. Scores on extended response and short-answer questions indicated that students in the treatment group performed better than students in the comparison group who received the formative assessments alone. These findings demonstrate both the feasibility and value of including performance task-types in a brief assessment context.

Language Arts Performance Assignments: Generalizability Studies of Local and Central Ratings.

We conducted generalizability studies to examine the extent to which ratings of language arts performance assignments, administered in a large, diverse, urban district to students in second through ninth grades, result in reliable and precise estimates of true student performance. The results highlight three important points when considering the use of performance assessments in large-scale settings: (a) Rater training may significantly impact reliability; (b) simple rater agreement indices do not provide enough information to assess the reliability of inferences about true student achievement; and (c) assessments adequate for relative judgments of student performance do not necessarily provide sufficient precision for absolute criterion-referenced decisions.