Tom Lumley

The effect of interlocutor and assessment mode variables in overseas assessments of speaking skills in occupational I settings

The increasing demand for performance assessment of speaking skills in second languages has led to logistic complications, for example, the delivery of tests in overseas locations. One solution to the problem has been to train native speaker interlocutors to carry out a series of oral interactions with the candidate, with assessment from audiorecordings of the test session postponed and conducted cen trally by a small team of trained raters. This technique is currently used in two large-scale occupationally related ESP tests administered internationally on behalf of the Australian government. But these procedures raise questions about the effect of such facets of the assessment situation as interlocutor variables and the quality of the audiotape recording. Recent developments in multifaceted Rasch measure ment have significantly broadened the possibilities for investigation of these issues. The research presented in this article investigates potential problems associated with the above approach to the offshore testing of speaking skills. Data from audiotape-based assessments of 70 offshore candidates from two administrations of the Occupational English Test, an advanced-level ESP test for health pro fessionals, are considered. In addition to multiple ratings of candidate perform ance, each recording is rated for perceptions of the competence of the interlocutor, the rapport established between the candidate and the interlocutor, and the audi bility of the interaction. These aspects of the assessment situation are treated as facets in a series of multifaceted Rasch analyses of the data. The results of the analysis reveal the effects of interlocutor variability and audio tape quality on ratings. The article concludes with an evaluation of the overall feasibility of the procedure, and implications for test administration are considered. The study is also a further demonstration of the application of multifaceted Rasch measurement in performance assessment settings.

Responses and replies

I read with interest the article by Lumley on reading subskills, his critique of my own work in this area and his claim that he is able to show ’very substantial agreement’ in matching skills to test items. My main interest was in the fact that, despite appearances to the contrary, Lumley’s work vindicates my own, both that done with Yasmeen Lukmani on the Bombay University Communication Skills Test, and subsequent work to which Lumley does not refer on the TEEP and ELTS tests. However, there are also several problems with Lumley’s own work, to which your readers’ attention needs to be drawn.

Book review : Hamp-Lyons, L., editor 1991: Assessing second language writing in academic contexts. Norwood, NJ: Ablex

In this book, Liz Hamp-Lyons presents a collection of articles addressing issues in the assessment of ESL writing in academic settings, particularly those concerned with the question of validity. Some articles present basic issues and are clearly intended for people with little familiarity with testing theory. Others, however, presuppose a greater breadth of experience with the concerns of language testing research. Following Hamp-Lyons’s useful overview of basic concepts and trends in language testing research (reliability, different types of validity, washback), the book is divided into six parts, dealing with the writer, the task, the reader, relating the assessment to the academic community, scoring and feedback, and accountability. In Part I (’The writer’), the first two articles (Ballard and Clanchy; Basham and Kwachka) focus on the issue of attitudes to knowledge and rhetorical styles. They both point out the need for greater tolerance of different cultural approaches to academic writing. However, this raises the question of whether it is feasible for the ESL practitioner, let alone the subject lecturer, to have sufficient breadth of knowledge to be familiar with the cultures of students from a wide range of backgrounds. And, even if it