Anthony Green

Washback to learning outcomes: a comparative study of IELTS preparation and university pre‐sessional language courses

This study investigated whether dedicated test preparation classes gave learners an advantage in improving their writing test scores. Score gains following instruction on a measure of academic writing skills—the International English Language Testing System (IELTS) academic writing test—were compared across language courses of three types; all designed for international students preparing for entry to UK universities. Course types included those with a test preparation focus, those designed to introduce students to academic writing in the university setting and those combining the two. In addition to IELTS academic writing test scores, data relating to differences in participants and practices across courses were collected through supplementary questionnaire and test instruments. To take account of the large number of moderating variables and non‐linearity in the data, a neural network approach was used in the analysis. Findings indicated no clear advantage for focused test preparation.

Watching for Washback: Observing the Influence of the International English Language Testing System Academic Writing Test in the Classroom

Previous studies of washback (the influence of a test on teaching and learning) have provided insights into the complexity of educational systems and test use, especially in relation to the role of the teacher, but have given insufficient attention to the relationship between observed practices and test design features. In this article a washback model is proposed that incorporates both test design and participant characteristics. The model is used to predict behaviour on preparation courses directed toward the Academic Writing component of the International English Language Testing System (IELTS) test. 197 learners and 20 teachers were observed over 51 classroom hours. These encompassed 22 IELTS preparation classes and, for comparison, 13 classes of English for academic purposes (EAP). Evidence was found for substantial areas of common practice between IELTS and other forms of EAP but also for some narrowing of focus in IELTS preparation classes that could be traced to test design features.

Can placement tests inform instructional decisions

Studies of placement tests are typically narrowly concerned with their validation as instruments for the efficient grouping of students. They rarely explore the assumption that placement test content can be related to classroom tasks and so inform instructional decisions. This study focuses on a trial version of the Global Placement Test (GPT), a measure of grammatical knowledge being developed for use in language schools worldwide. The central concern is the extent to which the GPT can enable schools to make valid inferences regarding mastery of the grammatical structures to be taught on the relevant courses and hence guide diagnostic intervention. In this study the GPT, together with another test specifically designed to measure knowledge of grammatical items expressing temporal relations in English, was administered to 1070 EFL (English as a Foreign Language) learners studying in the UK, Greece and Japan. The dimensionality of the GPT as a measurement scale was investigated by IRT and factor-analytic methods. The analysis suggests that item difficulty is affected more by item type than by any inherent linguistic difficulty of the element of grammatical competence being tested. The results cast doubt on whether quick grammar-based placement tests, which traditionally emphasize efficiency and practicality, can provide sufficient information on which to base diagnostic mediation or sequencing of instructional materials.

Empiricism versus Connoisseurship: Establishing the Appropriacy of Texts in Tests of Academic Reading.

Providers of tests of languages for academic purposes generally claim to provide evidence on the extent to which students are likely to be able to cope with the future demands of reading in specified real-life contexts. Such claims need to be supported by evidence that the texts employed in the test reflect salient features of the texts the test takers will encounter in the target situation as well as demonstrating the comparability of the cognitive processing demands of accompanying test tasks with target reading activities. This paper will focus on the issue of text comparability. For reasons of practicality, evidence relating to text characteristics is generally based on the expert judgement of individual test writers, arrived at through a holistic interpretation of test specifications. However, advances in automated textual analysis and a better understanding of the value of pooled qualitative judgement have now made it feasible to provide more quantitative approaches focusing analytically on a wide range of individual characteristics. This paper will employ these techniques to explore the comparability of texts used in a test of academic reading comprehension and key texts used by first-year undergraduates at a British university. It offers a principled means for test providers and test users to evaluate this aspect of test validity.

Developing assessment scales for large-scale speaking tests: a multiple-method approach

The process of constructing assessment scales for performance testing is complex and multi-dimensional. As a result, a number of different approaches, both empirically and intuitively based, are open to developers. In this paper we outline the approach taken in the revision of a set of assessment scales used with speaking tests, and present the value of combining methodologies to inform and refine scale development. We set the process in the context of the growing influence of the Common European Framework of Reference (Council of Europe 2001) and outline a number of stages in terms of the procedures followed and outcomes produced. The findings describe a range of data that was collected and analysed through a number of phases and used to inform the revision of the scales, including consultation with experts, and data-driven qualitative and quantitative research studies. The overall aim of the paper is to illustrate the importance of combining intuitive and data-driven scale construction methodologies, and to suggest a usable scale construction model for application or adaptation in a variety of contexts.

Re-fitting for a different purpose: A case study of item writer practices in adapting source texts for a test of academic reading:

The important yet under-researched role of item writers in the selection and adaptation of texts for high-stakes reading tests is investigated through a case study involving a group of trained item writers working on the International English Language Testing System (IELTS). In the first phase of the study, participants were invited to reflect in writing, and then audio-recorded in a semantic-differential-based joint discussion, on the processes they employed to generate test material. The group were next observed at a simulated item writers’ editing meeting to refine their texts and items for an IELTS reading test module. The participants’ written descriptions and recorded discussions provided rich data on how source texts were perceived, selected and adapted for the Test. The study reports findings from textual analyses using indices of readability and lexical density from the original material sourced by the item writers and their adapted versions for the test. Results from qualitative and quantitive analyses are discussed in terms of the implications for the IELTS reading module of editing actions such as: reducing redundancy and technical language, changing styles, deciding on potentially sensitive issues and relationships between texts and test items. The important issue of text authenticity in tests such as IELTS is also broached.

Requirements for Reference Level Descriptions for English

The founding purpose of the English Profile Programme is to answer the Council of Europes (2005) call for a set of Reference Level Descriptions (RLDs) for English linked to the Common European Framework of Reference for Languages (CEFR). The Council of Europe has issued guidelines setting out broad parameters for RLD development. This paper discusses how RLD might be developed for English in relation to the aims of the CEFR, incorporating consideration of critical voices, reports on the experiences of users of the CEFR and a review of currently operational RLDs for English: the Threshold series. On the basis of these sources, recommendations are made for the ongoing development of the English Profile Programme.

Prof Cyril Weir (1950–2018)

It is with deep sadness that we announce the death on September 28 of Cyril Weir, the Powdrill Research Professor in English Language Acquisition at the Centre for Research in English Language Learning and Assessment (CRELLA), University of Bedfordshire. Born on Merseyside in 1950, Cyril pursued his early enthusiasm for radical politics through an undergraduate degree at Reading University, followed by a Certificate in Education in Liberal Studies from Birmingham in 1972. He lectured in European Studies at Middlesex Polytechnic while studying for his Masters in Political Philosophy at Reading. Cyril first became involved in the world of English language education when he set out to fund further studies in History by working as a lecturer in EFL in Iran. Finding a new intellectual direction, he signed up for the Edinburgh course in Applied Linguistics and encountered the world of Language Testing in the inspiring shape of Alan Davies. On leaving Iran as the 1979 revolution took hold, Cyril found a position as a Research Officer at the Associated Examining Board (AEB: a UK-based examinations provider that is now part of AQA). He was tasked with developing a new test to screen international students entering UK universities. Adapting John Munby’s (1978) approach to needs analysis, Cyril carried out a comprehensive programme of observation and interviews to reveal how university students used language and their major sources of difficulty. This was the basis of his PhD (1983) from the Institute of Education under the supervision of Chris Brumfit and led to the AEB Test in English for Academic Purposes (TEAP). Both the original development process and innovative features of the test, such as the use of tasks that integrated reading, listening, and writing skills, remain influential 35 years later. A revised version of the test itself remains in use at the University of Reading as the TEEP. Although he returned to university teaching in 1983, Cyril was always keen that his ideas should serve practical ends. He found like-minded colleagues at the Centre for Applied Language Studies at the University of Reading where he teamed up with Arthur Hughes, Don Porter, and others to build the Test Development Unit, which he went on to direct from 1996. His books Communicative Language Testing (1990) and Understanding and Developing Language Tests (1993) together represented the most coherent case made for the needs-based approach to language test development that had emerged in the UK over the previous decade. Evaluation in ELT (with Jon Roberts) (1994) and Reading in a Second Language (with Sandy Urquhart) (1998) cemented his reputation. At Reading he directed and led numerous test and evaluation projects, including acting as the Senior UK Consultant on the College English Test and Test for English Majors projects in China, the Universities’ EAP Proficiency Test in Egypt, and on UK Overseas Development Administration (ODA) evaluation studies in Nepal, Guinea, and Ecuador. Clients and students alike appreciated his ability to combine his sharp insights with a clear sense of the steps required to put ideas into practice. In addition to teaching at Reading, he built up a collaborative relationship with the Association of Language Testers in Europe, working as a senior consultant and trainer. The annual Summer Course on Language Assessment that he developed at Reading in 1996, working with Barry O’Sullivan and Rita Green, was adopted as the ALTE Summer Course. Cyril served as the lead presenter from its inception in 2005 until 2016. Taking up a Professorship in ELT at the University of Surrey, Roehampton in 2000, Cyril developed his influential socio-cognitive framework for test development and validation, which

Linking Tests of English for Academic Purposes to the CEFR: The Score User’s Perspective

ABSTRACT The Common European Framework of Reference for Languages (CEFR) is widely used in setting language proficiency requirements, including for international students seeking access to university courses taught in English. When different language examinations have been related to the CEFR, the process is claimed to help score users, such as university admissions staff, to compare and evaluate these examinations as tools for selecting qualified applicants. This study analyses the linking claims made for four internationally recognised tests of English widely used in university admissions. It uses the Council of Europe’s (2009) suggested stages of specification, standard setting, and empirical validation to frame an evaluation of the extent to which, in this context, the CEFR has fulfilled its potential to “facilitate comparisons between different systems of qualifications.” Findings show that testing agencies make little use of CEFR categories to explain test content; represent the relationships between their tests and the framework in different terms; and arrive at conflicting conclusions about the correspondences between test scores and CEFR levels. This raises questions about the capacity of the CEFR to communicate competing views of a test construct within a coherent overarching structure.