Beverly Baker

Individual Differences in Rater Decision-Making Style: An Exploratory Mixed-Methods Study

Researchers of high-stakes, subjectively scored writing assessments have done much work to better understand the process that raters go through in applying a rating scale to a language performance to arrive at a score. However, there is still unexplained, systematic variability in rater scoring that resists rater training (see Hoyt & Kerns, 1999; McNamara, 1996; Weigle, 2002; Weir, 2005). The consideration of individual differences in rater cognition may explain some of this rater variability. This mixed-method exploratory case study (Yin, 2009) examined rater decision making in a high-stakes writing assessment for preservice teachers in Quebec, Canada, focussing on individual differences in decision-making style, or “stylistic differences in cognitive style that could affect decision-making” (Thunholm, 2004, p. 932). The General Decision Making Style Inventory questionnaire (Scott & Bruce, 1995) was administered to six raters of a high-stakes writing exam in Quebec, and information on the following rater behaviours was also collected for their potential for providing additional information on individual decision-making style (DMS): (a) the frequency at which a rater decides to defer his or her score, (b) the underuse of failing score levels, and (c) the comments provided by raters during the exam rating about their decisions (collected through “write-aloud” protocols; Gigerenzer & Hoffrage, 1995). The relative merits of each of these sources of data are discussed in terms of their potential for tapping into the construct of rater DMS. Although score effects of DMS have yet to be established, it is concluded that despite the exploratory nature of this study, there is potential for the consideration of individual sociocognitive differences in accounting for some rater variability in scoring.

The Certification of Teachers’ Language Competence in Quebec in French and English: Two Different Perspectives?

Language tools are reflections of the academic traditions of assessment developers, as well as the perceptions of language quality as held by a society. The certification of Quebec’s teachers offers an interesting case in point: the Québec Government’s ministère de l’Éducation, du Loisir et du Sport (MELS) requires that all students in university programs leading to teacher certification demonstrate their proficiency in the language of instruction—either French or English. The development and validation activities of the English and French examinations were systematically explored through a secondary document analysis to gain insight into possible cultural differences in perceptions of the construct of teacher language competence. Also revealed during this comparison were common concerns about these exams’ limitations and potential negative impacts, which include reinforcement of a narrow vision of language proficiency and exclusion of certain student populations.

The development of EFL examinations in Haiti: Collaboration and language assessment literacy development

Research was conducted during the delivery of a series of workshops on language assessment with Haitian teachers in the spring of 2013. The final products of these workshops were several revised national English examinations presented to the Haitian Ministry of Education and Professional Training (MENFP). The research goal was to examine the language assessment literacy (LAL) development of both teachers and language assessment specialists during this collaboration. Data included the compiled feedback from Haitian teachers on draft examinations during the workshops, as well as survey and interview responses immediately following the workshops. Results reveal the complementary expertise of teachers and specialists, which facilitated LAL development by both parties. Results also identified challenges in collaborative decision making and consensus building to be addressed in future projects.