Janna Fox

The test-takers' choice: an investigation of the effect of topic on language-test performance

A fundamental issue in validating topic-based tests of language proficiency is the effect of the topic on the test takers’ performance. Topic-based test developers must ensure that test takers are neither advantaged nor disadvantaged in terms of their test results when presented with a given test topic. We have termed this threat a ‘topic effect’ and argue that this topic effect may constitute a source of construct-irrelevant variance (Messick, 1989). We contend that investigating the possibility of a topic effect is a critical step in establishing the validity of all topic-based tests. This research investigates the potential presence of a topic effect for the Canadian Academic English Language (CAEL) Assessment using the mechanism of choice. The principal research question is to determine if test-takers given a choice of topic perform significantly differently than test-takers not given a choice. ESL university applicants (n = 254) were randomly assigned to one of two conditions: no choice of topic or choice among five topics. Overall Proficiency Level, Reading, Lecture and Essay scores were compared for the two conditions. Ordinal level data were analysed using the Mann Whitney U, Chi-Square and Kolmogorov-Smirnov tests. While the scores for the choice groups were usually higher than the scores for the no-choice groups, the differences were not statistically significant. Despite the lack of significance, we felt that the scores warranted closer examination. For the topic where the difference between the choice and no-choice groups was largest, a textual analysis of the essays was undertaken to look for instances of the use of information not provided in the test. Again, no difference was found between the choice and no-choice groups. The results provide support for the validity of inferences drawn from this test. Because choice is an essential element of the research design, a second focus of the study is to explore the advantages and disadvantages of the use of choice in language testing settings from the perspective of both the tester and the test-taker. The potential value of choice as a testing feature is discussed and a call for further research is made.

Did we take the same test? Differing accounts of the Ontario Secondary School Literacy Test by first and second language test‐takers

Within the context of increasing numbers of second language (L2) learners in Canadian schools and expanding standards‐driven testing frameworks, a passing score on the Ontario Secondary School Literacy Test (OSSLT) is a recently imposed secondary school graduation requirement in Ontario. There is evidence, however, that tests designed on the basis of first language (L1) populations may have lower reliability and validity for L2 students. This study elicited accounts of the OSSLT in 33 focus groups of 22 L1 students and 136 L2 students, attending 7 Ontario secondary schools, prior to and immediately after the March 2006 test administration. The results suggest important differences in L1 and L2 accounts of test constructs and suggest a gap between what is valued as literacy on the test and what is valued in classroom literacy practice, raising some concern regarding the test’s consequential validity. By examining how different groups of test‐takers interpret test constructs and the interaction between these interpretations, test design, and accounts of classroom practice, we may better address issues of fidelity in test construct representation (i.e., understand what may constitute construct under‐representation and construct‐irrelevant variance). This study highlights what may make a test more L2‐friendly, i.e. what supports (or impedes) L2 test performance. Although in the washback literature test‐taker accounts of tests have been the least researched, the results of this study suggest that such accounts have the potential to increase test fairness, enhance the validity of inferences drawn from test performance, improve the effectiveness of accommodation strategies, and promote positive washback.

Test decisions over time: tracking validity

This study took a grounded theory approach in investigating the relationship between the criteria indigenously drawn for an English for academic purposes (EAP) test and the outcomes of test decisions over time. Test outcomes were examined across three theoretical samples: (1) an EAP teacher-identified sample of misplaced students; (2) a database-generated sample that matched the key variables in the EAP teachers’ sample; and (3) a random sample of test-takers drawn from a single administration of the test. Although analysis revealed two sources of testing error, namely, under-specification of bands in the writing sub-test scale and an under-valuing of the listening sub-test in the overall weighted average of the test, it also revealed the relationship between key patterns of performance on the test and performance in university classrooms that signposted students at risk. As such, the study provides evidence of the usefulness of inquiry that examines the relationship between EAP test performance and the use of English as a mediating tool in academic performance.

Rethinking Second Language Admission Requirements: Problems with Language-Residency Criteria and the Need for Language Assessment and Support.

This study examines the academic performance of second language (L2) applicants who are admitted to a university on the basis of the number of years completed in English-medium secondary school. Such language-residency requirements are underexamined in the research literature. Results from the first 2 years of a 4-year longitudinal study indicate that there is no significant or meaningful difference in the performance of groups of students admitted to a university on the basis of 3, 4, or 5 years in English-medium high school. However, as a group, these L2 students underperform other groups and many are at risk. The results of this study are discussed in relation to research regarding time as a factor in language proficiency development and alternative models of language support. If universities are to continue using language residency as an indication of language proficiency, it is essential to provide these L2 applicants with language assessment that is tied to timely, supplemental instruction for students at risk.

The writing's on the board: The global and the local in teaching undergraduate mathematics through chalk talk

This article reports on an international study of the teaching of undergraduate mathematics in seven countries. Informed by rhetorical genre theory, activity theory, and the notion of Communities of Practice, this study explores a pedagogical genre at play in university mathematics lecture classrooms. The genre is mediational in that it is a tool employed in the activity of teaching. The data consist of audio/video-recorded lectures, observational notes, semistructured interviews, and written artifacts collected from 50 participants who differed in linguistic, cultural, and educational backgrounds; teaching experience; and languages of instruction. The study suggests that chalk talk, namely, writing out a mathematical narrative on the board while talking aloud, is the central pedagogical genre of the undergraduate mathematics lecture classroom. Pervasive pedagogical genres, like chalk talk, which develop within global disciplinary communities of practice, appear to override local differences across contexts of instruction. Better understanding these genres may lead to new insights regarding academic literacies and teaching.

Awareness Versus Production: Probing Students’ Antecedent Genre Knowledge:

This article explores the role of students’ prior, or antecedent, genre knowledge in relation to their developing disciplinary genre competence by drawing on an illustrative example of an engineering genre-competence assessment. The initial outcomes of this diagnostic assessment suggest that students’ ability to successfully identify and characterize rhetorical and textual features of a genre does not guarantee their successful writing performance in the genre. Although previous active participation in genre production (writing) seems to have a defining influence on students’ ability to write in the genre, such participation appears to be a necessary but insufficient precondition for genre-competence development. The authors discuss the usefulness of probing student antecedent genre knowledge early in communication courses as a potential source for macrolevel curriculum decisions and microlevel pedagogical adjustments in course design, and they propose directions for future research.

Biasing for the Best in Language Testing and Learning: An Interview With Merrill Swain

In response to the American Association of Applied Linguistics’ presentation of the Distinguished Scholarship and Service Award for extraordinary contributions to the fields of language learning, teaching, and testing in May 2004, Merrill Swain described her own work as “moving in cycles of research, theory and practice.” These spirals (“cycles with forward movement”), as Professor Swain described them, may explain in part why her work is consistently on the cutting-edge—questioning commonly held assumptions and shaping future inquiry. Throughout her career, Professor Swain has prompted teachers to rethink their teaching, generated awareness of best practices that support effective language learning, and challenged testers to develop better, fairer and more useful tests. From the early 1980s, Professor Swain recognized the importance of collaborative talk in language learning, informed by the landmark model of communicative language ability and use that she had elaborated with Michael Canale,1 and persuaded by her own research within early and late immersion programs. LANGUAGE ASSESSMENT QUARTERLY, 1(4), 235–251 Copyright

Language Testing Reconsidered

The editors gratefully acknowledge the support provided for publication of Language Testing Reconsidered by the Social Sciences and Humanities Research Council of Canada (SSHRC), the University of Ottawas Second Language Institute, and Carleton Universitys School of Linguistics and Applied Language Studies.

The cinematic art of teaching university mathematics: chalk talk as embodied practice

Abstract This article explores the multimodal nature of teaching university mathematics in international contexts. It focuses on the ‘cinematic’ art of teaching, applying a multimodal approach in the analysis of the pedagogical genre of ‘chalk talk’ as embodied disciplinary practice. The research draws on rhetorical genre studies and theories of situated learning and communities of practice. The data considered for the study consist of audio/video recorded lectures, observational notes, and semi-structured interviews collected from 50 participants teaching in 7 countries. Participants differ in linguistic, cultural, and educational backgrounds, teaching experience, and languages they use for instruction. The study suggests that a multimodal treatment of chalk talk as an embodied disciplinary pedagogical practice of teaching mathematics in the undergraduate lecture classroom allows researchers to further uncover the complexity of this genre. Better understanding the embodied pedagogical practices of the international mathematics CoP may lead to new insights regarding disciplinary-specific pedagogies.

Test Review: The Versant Spanish[TM] Test.

Center for Applied Linguistics. (2005b). BEST Plus trainer manual. Washington, DC: Center for Applied Linguistics. Center for Applied Linguistics. (2007, December). Effects of instructional hours and intensity of instruction on NRS level gain in listening and speaking. Washington, DC: CAL Digest. Fulcher, G. (2003). Testing second language speaking. London: Pearson Education. Luoma, S. (2004). Assessing speaking. Cambridge: Cambridge University Press. National Reporting Service. (2008). Characteristics of English literacy participants in adult education: 2000–200. Retrieved 14 August 2008, from http://www.nrsweb.org/reports/fast_facts.aspx van Lier, L. (1989). Reeling, writhing, drawling, stretching and fainting in coils: Oral proficiency interviews as conversation. TESOL Quarterly, 23, 489–508.