Ari Huhta

The Development of a Suite of Computer-Based Diagnostic Tests Based on the Common European Framework

DIALANG is an on-line language assessment system, which contains tests in 14 European languages and is based on the Common European Framework of Reference (CEFR). It is the first major testing system that is oriented towards diagnosing language skills and providing feedback to users rather than certifying their proficiency. This article describes the contents of Version 1 of DIALANG and the way in which the system works. This is followed by an account of the development of DIALANG tests and of the pilot testing and standard setting procedures. The results of the first analyses of items and self-assessment statements, and of the standard setting procedures, are reported. The article focuses on the results for English, but findings for some other languages are also touched upon.

Discursive construction of a high-stakes test: the many faces of a test-taker

As part of a larger project, we studied how a foreign language test got discursively constructed in the talk of upper-secondary-school leavers. A group of students were asked to keep an oral diary to record their ideas, feelings and experiences of preparing for and taking the test over the last spring term of school, as part of a high-stakes national examination. In addition, they took part in discussions either in pairs or groups of three after having learned about the final test results. After transcribing the data, drawing on a form of discourse analysis originally launched by a group of social psychologists, we identified (at least) four interpretative repertoires in the students’ accounts - with different constructions of themselves as test-takers, the test, and their performance in the test - including expectations and explanations for success or failure as well as credit or blame. The findings point to variation in the uses of these repertoires, not only from one context to another but also from moment to moment.

Assessing learners’ writing skills in a SLA study: Validating the rating process across tasks, scales and languages

There is still relatively little research on how well the CEFR and similar holistic scales work when they are used to rate L2 texts. Using both multifaceted Rasch analyses and qualitative data from rater comments and interviews, the ratings obtained by using a CEFR-based writing scale and the Finnish National Core Curriculum scale for L2 writing were examined to validate the rating process used in the study of the linguistic basis of the CEFR in L2 Finnish and English. More specifically, we explored the quality of the ratings and the rating scales across different tasks and across the two languages. As the task is an integral part of the data-gathering procedure, the relationship of task peformance across the scales and languages was also examined. We believe the kinds of analyses reported here are also relevant to other SLA studies that use rating scales in their data-gathering process.

Interaction of Language Policy and Assessment in Finland.

This paper reviews developments and future challenges in language policy, planning and assessment in Finland, where several important changes in legislation, curricula and assessment systems have recently taken place. Language proficiency requirement of immigrants and civil servants have been redefined, school curricula have been revised and new language examinations have been developed. The Common European Framework has been particularly influential. The paper also explores the tensions caused by changes, such as uncertainties in implementing new curricula in teaching and assessment, and differences between curricula and national examinations. The gate-keeping function of examinations also raises questions about social and educational consequences, especially for immigrants.

Chapter 2.03.02: Designing and Assessing L2 Writing Tasks Across CEFR Proficiency Levels

With the advent of the Common European Framework of Reference (CEFR) for the learning, teaching and assessment of modern languages, there have been renewed calls for the integration of the research perspectives of language testing and second language acquisition across Europe. The project Cefling was set up in 2006 with this purpose in mind. In the project our aim is to describe the features of language that L2 learners use at various levels of language proficiency defined by the CEFR scales. For this purpose, L2 Finnish and L2 English data were collected from young and adult L2 learners by using a set of communicative L2 writing tasks. In the course of the project, the different understandings of what the purpose of an L2 writing task is needed to be reconciled not only in the minds of researchers but also in research design. In what follows, we will discuss the issues involved in designing and assessing L2 tasks for SLA and language testing purposes by using the design and assessment procedures in the project as a case in point. We will also present some of our findings to illustrate how statistical procedures such as multifaceted Rasch analysis can be used to examine task difficulty.