Dan Douglas

Quantity and quality in speaking test performance

Scores on speaking tests are used as evidence of both learner language ability and the second language acquisition process, and most speaking tests include scoring rubrics to help ensure that ratings are reliable and reflect a theoretical construct of speaking ability. Nevertheless, it may be that similar ratings on a speaking test in fact represent qualitatively different learner performances. Such a situation would mean that interpretations of ability or acquisition process based on such test scores may not be valid. The purpose of this article is to investigate the hypothesis that similar quantitative scores on a speaking test represent qualitatively different performances. The results of the study raise a number of issues for further investigation.

LANGUAGE FOR SPECIFIC PURPOSES ASSESSMENT CRITERIA: WHERE DO THEY COME FROM?

Typically in assessment of Language for Specific Purposes (LSP), test content and methods are derived from an analysis of the target language use (TLU) situation. However, the criteria by which performances are judged are seldom derived from the same source. In this article, I argue that LSP assessment criteria should be derived from an analysis of the TLU situation, using the concept of indigenous assessment criteria (Jacoby, 1998). These criteria are defined as those used by subject specialists in assessing communicative performances of both novices and colleagues in academic, professional and vocational fields. Performance assessment practices are part of any professional culture, from formal, gatekeeping examination procedures, to informal, ongoing evaluation built into everyday interaction. I suggest a procedure for deriving assessment criteria from an analysis of the TLU situation and explore problems associated with doing so, recommending a ‘weak’ indigenous assessment hypothesis to assist in the development of LSP test assessment criteria and guide interpretations of test performance.

ASSESSING LANGUAGE USING COMPUTER TECHNOLOGY

In this article, we propose to follow up on the most recent ARAL survey article on trends in computer-based second language assessment (Jamieson, 2005) and review developments in the use of technology in the creation, delivery, and scoring of language tests. We will discuss the promise and threats associated with computer-based language testing, including the language construct in relation to computer-based delivery and response technologies; computer-based authoring options; current developments; scoring, feedback, and reporting systems; and validation issues.

ANALYZING ORAL PROFICIENCY TEST PERFORMANCE IN GENERAL AND SPECIFIC PURPOSE CONTEXTS

Abstract To investigate whether a field-specific oral proficiency test, constructed by manipulating test method facets, would be a better predictor of field-specific performance than a general purpose oral proficiency test, 31 Chinese chemistry graduate students were given three English tests: the field specific test, the general purpose test and a chemistry teaching performance test. Results suggested that when raters of the performance test were asked to recommend specifically whether or not a subject should be allowed to actually teach chemistry in a lab or classroom, the field-specific test was a better predictor than the general purpose test. The paper contains a theoretical discussion of field-specific language testing and guidelines for the construction of oral proficiency tests in specific purpose contexts.

Research Methodology in Contextually-Based Second Language Research.

We suggest a combined research methodology for studying SLA in real-life and important contexts, a methodology sensitive to the demands of such contexts. For studying IL learning in context, our suggested methodology combines and integrates aspects from three fields: grounded ethnography in ethnomethodol ogy ; subject-specialist informant procedures in language for specific purposes; and rhetorical/grammatical strategies in discourse analysis. We first present evidence for the importance of devising a research approach to contexually based SLA. Then we sketch the suggested research methodology and present two extended case studies which illustrate the methodology. We view such research as complimentary to universal approaches to core IL grammar. Finally, we sketch our theoretical approach, showing a possible link between research in universal and contextually-based SLA.

Performance Consistency in Second Language Acquisition and Language Testing Research: A Conceptual Gap.

Arguing from the premise that a language test is a special case of a second language acquisition (SLA) elicitation device, I suggest that SLA and language testing share much common ground in terms of research methods, which have similar properties in that they are both used to make systematic observations of language performances from which inferences can be made about the state of a learner’s interlanguage ability underlying the performance. However, I also argue that whereas the concept of demonstrating validity and reliability has been integrated into how language testing research is conducted, SLA researchers have generally failed to recognize the need to demonstrate these qualities. I compare examples of SLA and language testing research articles in terms of their treatment of validity and reliability and argue: • that it is important for SLA researchers to provide evidence that the methods they employ to elicit data are appropriate for the purposes intended; • that the procedures provide stable and consistent data; and, consequently • that the interpretations they make of the results are justified.

Developments in Language Testing

In many ways, the year 1990 represented a watershed in language testing. In that year, the 12th annual Language Testing Research Colloquium took as its theme “A new decade of language testing research: Collaboration and cooperation.” The 1990 Colloquium was also dedicated to the memory of Michael Canale, whose work during the previous decade laid the groundwork for much of that of the 1990s. In 1990, a seminar sponsored by the Regional Language Centre in Singapore, on Language Testing and Programme Evaluation, focused on many of the problems in the field of language testing that militated against the hoped for progress of the 1980s. At both of these meetings, and at others involving language testing that year, discussions were held concerning the formation of an international professional association of language testers. The result was the formation of the International Language Testing Association in 1992. Also in 1990, a language testing computer “bulletin board,” LTEST-L, was established on Listserv, so that the international community of language testing researchers and practitioners could hold electronic “conferences” and share information and data on a world-wide basis. Finally, a number of important books on language testing were published in 1990, including works by Bachman, Davies, Heaton, and Weir, which have already had an influence on language testing research and development in the first half of the decade.

Testing Listening Comprehension in the Context of the ACTFL Proficiency Guidelines.

This article discusses issues related to the testing of listening comprehension in the context of the 1986 ACTFL proficiency guidelines (American Council on the Teaching of Foreign Languages, 1987). After a review of current research on listening comprehension and development of listening tests, four issues are singled out for more detailed consideration: the meaning of context in listening testing, the concept of criterion-referenced tests, the notion of specific purposes in testing, and the use of technology in listening tests. A number of research issues are also discussed.

TAs on TV: Demonstrating Communication Strategies for International Teaching Assistants.

Abstract We present in this paper a technique for employing videotaped segments of actual classroom performances by U.S. and international TAs. This is a method for training ITAs in specific, definable skills in English language, U.S. academic culture and pedagogical performance, while at the same time developing communication strategies for coping with largely undefinable future communicative needs in their work as teaching assistants. After a brief discussion of previous research on lecture comprehension, focusing especially on the effect of discourse markers, we describe our procedure for developing and using the videotaped materials in an ITA training program. We conclude with a discussion of the evaluation of the materials and suggestions for further research.

LSP and Interlanguage: Some Empirical Studies.

Abstract It is argued that LSP and interlanguage studies in SLA need each other. A series of questions which should be open to empirical investigation are then presented. Some possible interrelations of LSP and IL are discussed. Finally, some of the questions are explored in terms of the empirical studies which appear in this issue of the Journal.