Tine van Daal

Validity of comparative judgement to assess academic writing: examining implications of its holistic character and building on a shared consensus

Abstract Recently, comparative judgement has been introduced as an alternative method for scoring essays. Although this method is promising in terms of obtaining reliable scores, empirical evidence concerning its validity is lacking. The current study examines implications resulting from two critical assumptions underpinning the use of comparative judgement, namely: its holistic characteristic and how the final rank order reflects the shared consensus on what makes for a good essay. Judges’ justifications that underpin their decisions are qualitatively analysed to obtain insight into the dimensions of academic writing they take into account. The results show that most arguments are directly related to the competence description. However, judges also use their expertise in order to judge the quality of essays. Additionally, judges differ in terms of how they conceptualise writing quality, and regarding the extent to which they tap into their own expertise. Finally, this study explores diverging conceptualisation of misfitting judges.

The Complexity of Assessing Student Work Using Comparative Judgment: The Moderating Role of Decision Accuracy

Nowadays, comparative judgment is used to assess competences. Judges compare two pieces of student work and judge which of both is better regarding the competence assessed. Using these pairwise comparison data, students’ work is scaled according to its quality. Since student work is highly information loaded and heterogeneous of nature, this raises the question whether judges can handle this type of complex judgments? However, research into the complexity of comparative judgment and its relation with decision accuracy is currently lacking. Therefore, this study initiates a theoretical framework on the complexity of comparative judgment and relates it to decision accuracy. Based on this framework two hypotheses are formulated and their plausibility is examined. The first hypothesis states that the distance between two pieces of student work on the rank-order (rank-order distance) is negatively related to experienced complexity, irrespectively of decision accuracy. In contrast, hypothesis 2 expects decision accuracy to moderate the relation between rank-order distance and experienced complexity. A negative relation is expected for accurate decisions. Meanwhile, inaccurate decisions are assumed to result in higher experienced complexity than accurate decisions, irrespective of rank-order distance. In both hypotheses, judges are expected to vary in mean experienced complexity as well as in the strength of the expected relationship between rank-order distance and experienced complexity. Using an information-theoretic approach, both hypotheses are translated into a statistical model and their relative fit is assessed. All analyses are replicated on three samples. Sample 1 and 2 comprise CJ data on the assessment of writing, while sample 3 contain pairwise comparison data on the assessment of visual arts. Results unambiguously confirm the moderating role of decision accuracy (hypothesis 2). Inaccurate decisions are experienced as more complex than accurate decisions, irrespective of rank-order distance. Meanwhile, for accurate decisions, rank-order distance is negatively related to experienced complexity. In line with expectations, differences between judges are found in mean experienced complexity and in the strength of the relationship between rank-order distance and experienced complexity. Suggestions for further theory development are formulated based on the results of this study.

Differential effects of a long teacher training internship on students’ learning-to-teach patterns

To become a lifelong learner as a teacher, student teachers already have to learn how to direct their own learning during initial teacher education programmes. Previous empirical research has shown that student teachers differ in their patterns of learning-to-teach, but few is known about the changeability of these learning patterns throughout teacher education and the role of teacher training internships in this. In this study, the changes in student teachers’ patterns of learning-to-teach amongst pre-service teachers were investigated using a longitudinal design. 253 student teachers were asked to complete the ‘Inventory Learning To Teach Process’ questionnaire at two points during the last semester of a three-year teaching programme: immediately prior to and immediately following a long teacher training internship. The results of this study demonstrate that learning-to-teach patterns are subject to a relative degree of change. In particular, survival-oriented student teachers appear to have undergone a greater degree of change after the long teacher training internship.