Carolyn E. Turner

Systematic Effects in the Rating of Second-Language Speaking Ability: Test Method and Learner Discourse.

Major differences exist in two approaches to the study of second-language performance. Second-language-acquisition (SLA) research examines effects upon discourse, and is typically unconcerned with scores. Language-testing (LT) research investigates effects upon scores, generally without reference to discourse. Within a general framework of test taking and scoring, we report research from these two fields as it relates to questions of systematic effects on second-language tests. We then examine findings incidental to a test-development project. The findings were consistent with LT research into systematic effects of task and rater on ratings, and with SLA research into systematic effects of task on discourse. Using empirically derived scales as indicators of salient features of discourse, we infer that task type influences strategies for assessing language performance. Explanations for these joint findings are not afforded by either standard LT or SLA perspectives. There is no theory of method to explain how particular aspects of method affect discourse, how those discourse differences are then reflected in ratings and how task features influence the basis for judgement. We conclude that a full account of performance testing requires a paradigm that incorporates relationships that are not specified in either the major language-testing research tradition or the tradition of second-language-acquisition research.

Examining Washback in Second Language Education Contexts: A High Stakes Provincial Exam and the Teacher Factor in Classroom Practice in Quebec Secondary Schools

Abstract Research has helped us understand that high stakes tests can have some sort of influence at the micro level on classroom practice and activity. To date, much variation has been reported. This paper discusses washback in second language (L2) education contexts and specifically the teacher factor in classroom activity. It draws on four areas of research: washback research in second language (L2) and general education, the teacher factor in washback, alignment of assessment internal and external to the classroom, and mixed methods research. It provides teacher case study examples from a larger study probing into teacher practice and beliefs when dealing with a new high stakes English as a second language (ESL) provincial exit exam at the high school level. The participants are from the population of ESL secondary teachers in the French school system in the province of Quebec in Canada. The main research question is: How do teachers mediate between classroom assessment activity and preparing students for upcoming external exams? Results indicate that teachers used common overall approaches, but that there was variation in individual practice. When first introduced to the new exam material, teachers used a formative assessment approach. As the exam time neared, their practice evolved into a summative assessment approach. This phenomenon demonstrated an interfacing or ‘blurring’ of formative and summative assessment in an attempt to align classroom and external exam assessment. Implications are discussed pertaining to a coherent education system across curriculum, teaching, learning and assessment.

Identifying Second Language Speech Tasks and Ability Levels for Successful Nurse Oral Interaction with Patients in a Linguistic Minority Setting: An Instrument Development Project

One of the most demanding situations for members of linguistic minorities is a conversation between a health professional and a patient, a situation that frequently arises for linguistic minority groups in North America, Europe, and elsewhere. The present study reports on the construction of an oral interaction scale for nurses serving linguistic minorities in their second language (L2). A mixed methods approach was used to identify and validate a set of speech activities relating to nurse interactions with patients and to derive the L2 ability required to carry out those tasks. The research included an extensive literature review, the development of an initial list of speech tasks, and validation of this list with a nurse focus group. The retained speech tasks were then developed into a questionnaire and administered to 133 Quebec nurses who assessed each speech task for difficulty in an L2 context. Results were submitted to Rasch analysis and calibrated with reference to the Canadian Language Benchmarks, and the constructs underlying the speech tasks were identified through exploratory and confirmatory factor analyses. Results showed that speech tasks dealing with emotional aspects of caregiving and conveying health-specific information were reported as being the most demanding in terms of L2 ability, and the most strongly associated with L2 ability required for nurse–patient interactions. Implications are discussed with respect to the development and use of assessment instruments to facilitate L2 workplace training for health care professionals.

The underlying factor structure of L2 close test performance in Francophone, university-level students: causal modelling as an approach to construct validation

This study begins to address the question of cloze test construct validity through causal modelling. It investigates the underlying factor structure of L2 cloze test performance as explained by a theoretical model including the following hypothetical constructs: cloze-taking ability; knowledge of language; content domain; and knowledge of contextual constraints. Eight cloze tests reflecting the posited factors were constructed and administered to 182 Francophone, university-level students. The factors were examined separately and in combination through a model building process which included model fitting and model comparison. A model composed of three orthogonal factors was confirmed and accepted as the best explanation of the data. The results indicate that cloze performance is dependent upon language factors (a second language factor or a first language factor) and nonlinguistic specific knowledge related to cloze-taking ability that crosses over linguistic boundaries. Cloze has been considered as an overall L2 proficiency measure. This study empirically demonstrates that factors other than language are significantly contributing to cloze test performance. It also demonstrates the potential of a causal modelling approach.

The Impact of Communication and Collaboration Between Test Developers and Teachers on a High-Stakes ESL Exam: Aligning External Assessment and Classroom Practices

In Quebec the high-stakes Secondary Five ESL exit writing exam developed by the Education Ministry (MELS) is administered and corrected by classroom teachers. In this distinctive situation, the MELS works toward aligning classroom-based assessment (CBA) and the writing exam by making ongoing teacher involvement part of its development and validation process. This article presents data from Year 1 (2010) and Year 2 (2011) of a three-year study using a mixed-methods design to examine the consequences of this unique process. The participants from each administration of the exam include 2 MELS evaluation officials, 11 MELS-trained raters from the research marking centers, and approximately 500 students. Data consist of student scores rated by raters at the marking centers, interviews with raters, teacher and rater survey information, and notes from the marking centers. Comparative analyses of student scores, Rasch analyses of rater behavior, and content analyses of interviews, surveys, and notes were conducted. The findings show two-way benefits for teachers and MELS administrators, creating a value-added validation process: teacher comments for rubric and task revisions improved scoring and exam construct validity; increased teacher exam knowledge helped align CBA practices, enhancing student learning and enabling mentoring of colleagues in interpreting rubric criteria.

Book Review: Language program evaluation: theory and practice

Fulcher, G. 1996: Invalidating validity claims for the ACTFL oral rating scale.System24, 163–72. Fulcher, G. 1997: The testing of speaking in a second language. In Clapham, C. and Corson, D., editors, Encyclopaedia of Language and Education: Vol. 7. Language testing and assessment , Dordrecht: Kluwer Academic Publishers, 75–86. Henning, G. 1992: The ACTFL oral proficiency interview: Validity evidence.System20, 365–72. Mercier, L. 1933: Diverging trends in modern foreign language teaching and their possible reconciliation. French Review6, 370–86. Morrow, K. 1981: Communicative language testing: Revolution or evolution. In Alderson, C. and Hughes, A., editors, Issues in language testing. London: The British Council, 9–25. Reprinted from Brumfit, C.J. and Johnson, K., editors, 1979: The communicative approach to language teaching . Oxford: Oxford University Press. Spolsky, B.1995:Measured words: The development of objective language testing. Oxford: Oxford University Press. Wood, B. 1927: New York experiments with new-type modern language tests. New York: Macmillan.