Sauli Takala

Analysing Tests of Reading and Listening in Relation to the Common European Framework of Reference: The Experience of The Dutch CEFR Construct Project

The Common European Framework of Reference (CEFR) is intended as a reference document for language education including assessment. This article describes a project that investigated whether the CEFR can help test developers construct reading and listening tests based on CEFR levels. If the CEFR scales together with the detailed description of language use contained in the CEFR are not sufficient to guide test development at these various levels, then what is needed to develop such an instrument? The project methodology involved gathering expert judgments on the usability of the CEFR for test construction, identifying what might be missing from the CEFR, developing a frame for analysis of tests and specifications, and examining a range of existing test specifications and guidelines to item writers and sample test tasks for different languages at the 6 levels of the CEFR. Outcomes included a critical review of the CEFR, a set of compilations of CEFR scales and of test specifications at the different CEFR levels, and a series of frameworks or classification systems, which led to a Web-mounted instrument known as the Dutch CEFR Grid. Interanalyst agreement in using the Grid for analyzing test tasks was quite promising, but the Grids need to be improved by training and discussion before decisions on test task levels are made. The article concludes, however, that identifying separate CEFR levels is at least as much an empirical matter as it is a question of test content, either determined by test specifications or identified by any content classification system or grid.

Test fairness: a DIF analysis of an L2 vocabulary test

The purpose of this study is to analyse gender-uniform differential item functioning (DIF) in a second language (L2) vocabulary test with the tools of item response theory (the separate calibration t-method) and to study potential gender impact on the test performance measured by different item composites. The results of the study show that despite the fact that there are test items with indications of DIF in favour of either females or males, the test as a whole is not gender-biased. In spite of this, it was demonstrated that some item composites are gender-biased. In view of item bank building and use, it means that some of the tests constructed on the basis of an item bank might be biased if the item bank contains items with indication for DIF. Although the results of some empirical research suggest that the requirements for items with DIF to be excluded from the final test version may on the whole be too restrictive, this study demonstrated that the traditional advice of excluding biased items gains new significance in the light of item bank building and use since doing so will prevent possible biased item composites.

Relating examinations to the Common European Framework: a manual

This article deals with the linking of examinations to the Common European Framework of Reference (CEFR). Following the rapid adoption of the CEFR by the Council of Europe and the European Union as well as many national and local actors, there have been calls for guidance on how examinations could be made more transparent by means of the CEFR, especially by using its calibrated scales as a tool. The Council of Europe has recently produced a preliminary pilot Manual, which presents a procedure for doing this. The article describes the political and educational context of this work and presents the main procedures proposed in the Manual. Since the Manual is a preliminary document for discussion and feedback, the authors of the Manual join the discussion by presenting some critical reflections of their own. The article concludes by outlining two scenarios: one which predicts positive developments and the other which foresees less desirable consequences.

LANGUAGE POLICY AND PLANNING

The field of language policy and planning is clearly a sub-field within applied linguistics. It generally does not draw heavily on formal linguistics, except for aspects of corpus and status planning. However, it does draw extensively from a range of disciplines in order to plan, implement, and evaluate language policies that respond to the needs of stake holders of various types. Despite continuous development of the field, aspects of language policy and planning need to be developed further. One of the key areas where policy can be enhanced considerably is in the area of policy and planning evaluation. This direction of inquiry is also relevant to a number of other areas within applied linguistics.

Review of some recent Soviet research on vocabulary learning and teaching

This article presents a review of some recent Soviet research on vocabulary learning and teaching. It has been demonstrated in a number of experiments with young and older students, and using both first and second/foreign languages, that even young children can learn linguistic word analysis skills and can benefit from such teaching.