Mary K. Enright

Items by Design: The Impact of Systematic Feature Variation on Item Statistical Characteristics

In this study we investigated the impact of systematic item feature variation on item statistical characteristics and the degree to which such information could be used as collateral information to supplement examinee performance data and reduce pretest sample size. Two families of word problem variants for the quantitative section of the Graduate Record Examinations General Test were generated by systematically manipulating item features. For rate problems, the item design features affected item difficulty (adjusted R2 = .90), item discrimination (adjusted R2 = .50), and guessing (adjusted R2 = .41). For probability problems the item design features affected difficulty (adjusted R2 = .61) but not discrimination or guessing. The results demonstrate the enormous potential of systematically creating item variants. The issue of how to develop a knowledge base that would support the systematic generation of a wider variety of quantitative problems is discussed.

Research Issues in High‐Stakes Communicative Language Testing: Reflections on TOEFL's New Directions

0 The TOEFL (Test of English as a Foreign Language) is a high-stakes language test intended to measure proficiency in English in an academic setting. A new TOEFL will become operational in the fall of 2005. In developing this new TOEFL, we decided to modify the test in three important ways (Jamieson, Jones, Kirsch, Mosenthal, & Taylor, 2000). One is to include more constructed-response or performance tasks (including speaking) that require the integration of more than one language skill (e.g., writing or speaking in response to listening and reading materials). A second is to develop test materials that more closely reflect the types of discourse that occur in academic settings. And a third is to provide information that will help test users better understand the meaning of test scores. Educational Testing Service carried out an extensive program of research to achieve these goals and to make sure that a new test would meet professional standards (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999). But as is often the case in doing research, addressing one question makes us aware of other unanswered questions. This discussion focuses both on the research Educational Testing Service carried out to support TOEFLs new directions and other issues yet to be explored.

ITEMS BY DESIGN: THE IMPACT OF SYSTEMATIC FEATURE VARIATION ON ITEM STATISTICAL CHARACTERISTICS

This study investigated the impact of systematic item feature variation on item statistical characteristics and the degree to which such information could be used as collateral information to supplement examinee performance data and reduce pretest sample size. Two families of word problem variants for the quantitative section of the Graduate Record Examination (GRE®) General Test were generated by systematically manipulating item features. For rate problems, the item design features affected item difficulty (Adj. R2 = .90), item discrimination (Adj. R2 = .50), and guessing (Adj. R2 = .41). For probability problems the item design features affected difficulty (Adj. R2 = .61), but not discrimination or guessing. The results demonstrate the enormous potential of systematically creating item variants. However, questions of how best to manage variants in item pools and to implement statistical procedures that use collateral information must still be resolved.