Raters’ perceptions of assessment criteria relevance

Abstract

Abstract This study adopts a novel approach to investigate perceptions of assessment criteria relevance in differentiating writing performance levels. Experienced writing assessors were asked to directly compare pairs of performances. For each comparison, assessors were asked to determine which performance was better and to record which aspects of writing were used to make determinations. To do so, assessors referred to a marking rubric containing 10 criteria. By triangulating direct/explicit qualitative comparisons of performances against indirect/implied comparisons using a rubric, the study aims to provide unique insights into the perceived relevance of criteria. Whereas the rubric presupposes relatively equal relevance of all criteria, in direct comparisons participating assessors perceive certain criteria relevant more often than others. In particular, conventions of writing, such as punctuation and spelling, were perceived relevant less often than authorial aspects. In combination with descriptions of qualitative gradations contained in the rubric itself, the information obtained from the study aims to assist with identifying and delineating criteria perceived relevant to developmental levels by the expert assessors. These and other findings have potential implications for selecting which criteria to use in narrative writing rubrics and they suggest advantages of granting assessors discretion in their selection and application of criteria.