Nathan T. Carr

The factor structure of test task characteristics and examinee performance

The present study focuses on the task characteristics of reading passages and key sentences in a test of second language reading. Using a new methodological approach to describe variation in test task characteristics and explore how differences in these characteristics might relate to examinee performance, it posed the two following research questions: First, how do the characteristics of the texts used in a high-stakes test of English for Academic Purposes reading vary? Second, what relationships exist between its text characteristics and examinee performance? An expanded test task characteristics instrument was constructed, following Freedle and Kostin (1993) and Bachman et al. (1996), and adding a large number of syntactic features (Celce-Murcia and Larsen-Freeman, 1999). Ratings and numerical counts were compiled for three forms of the Test of English as a Foreign Language (TOEFL) Reading Comprehension Section. Taking items as the object of measurement, the results were then used in a series of exploratory and confirmatory factor analyses, along with IRT (item response theory) parameter estimates for the items in question.

Automated Scoring of Short-Answer Reading Items: Implications for Constructs

This article examines how the use of automated scoring procedures for short-answer reading tasks can affect the constructs being assessed. In particular, it highlights ways in which the development of scoring algorithms intended to apply the criteria used by human raters can lead test developers to reexamine and even refine the constructs they wish to assess. The article also points out examples of ways in which attempting to replicate human rating behavior while avoiding incidental construct alteration can pinpoint areas of the automated scoring process requiring further development. The examples discussed in this article illustrate the central point that construct definitions should always guide the development of scoring algorithms while the process of developing and refining such algorithms requires more rigorous construct definitions and can potentially push us to refine our constructs.

Using Microsoft Excel[R] to Calculate Descriptive Statistics and Create Graphs.

Descriptive statistics and appropriate visual representations of scores are important for all test developers, whether they are experienced testers working on large-scale projects, or novices working on small-scale local tests. Many teachers put in charge of testing projects do not know why they are important, however, and are utterly convinced that they lack the mathematical ability to address the matter anyway. This article begins by explaining why descriptives need to be calculated in the first place, and then discusses ways in which to display data visually, and how to do this using Microsoft Excel® spreadsheet software. The article then addresses three types of descriptive statistics: measures of central tendency, measures of dispersion, and indicators of the shape of the distribution. It discusses some basic points about interpreting them and then provides simple instructions for calculating them in Excel. The article assumes that readers are not particularly familiar with Excel and does not assume a high level of mathematical sophistication.

Book Review: Greater Than Lambda

This book was written with several audiences in mind—language teachers, curriculum developers, and graduate students in Teaching English to Speakers of Other Languages or other language education programs. It serves as a comprehensive guidebook for how to integrate a criterion referenced assessment program with a language program curriculum, and as such it dovetails nicely with Brown’s (1995) book on curriculum development. Although there is a considerable degree of overlap between this work and Brown’s (1996) Testing in Language Programs (no longer in print), in terms of the subjects covered, this is not a mere reworking of an earlier publication. Rather, it addresses many of the same subjects—both books are on language testing, after all—but does so from a different perspective and with a different emphasis. It also makes an excellent reference for those needing a book containing almost any criterionor norm-referenced reliability or item analysis statistic. Those using it for this purpose will find the fourth and fifth chapters the most helpful. The first chapter begins by contrasting the normand criterion-referenced testing (CRT) paradigms. This is followed by a discussion of what it is about language ability and its acquisition that makes criterion-referenced language tests different from that of CRTs of content knowledge. The chapter concludes with a brief listing of some of the issues that must be kept in mind when developing a CRT. The second chapter focuses on the implementation of CRT in the context of a well-plannedcurriculum. ItbeginswithanoverviewofBrown’s (1995)sixphasesof language curriculum development: conducting a needs analysis; setting goals and objectives; instituting testing—both normand criterion-referenced—for purposes LANGUAGE ASSESSMENT QUARTERLY, 2(3), 227–230 Copyright

A Review of Lertap (Laboratory of Educational Research Test Analysis Package) 5.2

Many people who work with test and survey data enter or store that data in Microsoft’s Excel (Microsoft Corporation, 1996) spreadsheet application. Although it cannot perform many of the analyses possible with true statistical applications such as SPSS, Excel nevertheless offers certain advantages, such as the ability to place column totals, averages, and other descriptive statistics directly following the data to which they refer. This makes Excel particularly useful for performing item analysis. For any but the simplest calculations, however, Excel requires mixing absolute and relative cell references, the bane of many an inexperienced user. This is what makes Lertap so valuable: It allows users to perform a wide array of classical item and reliability analyses in Excel without having to spend hours creating formulas and specifying ranges. For an Excel “power user,” Lertap can save a considerable amount of time. For a novice Excel user, on the other hand, Lertap could bring a number of difficult, tedious, or essentially impossible analyses within reach. Furthermore, the fact that all reports are formatted Excel spreadsheets means that the program’s output can be printed directly from Excel, with page breaks and page orientations easily previewed and changed. Results are easily transferred from Excel to Microsoft Word or PowerPoint documents as well. Lertap was developed by Larry Richard Nelson and is available from Assessment Systems Corporation. INTERNATIONAL JOURNAL OF TESTING, 4(2), 189–195 Copyright