Hunter M. Breland

Comparability of TOEFL CBT Essay Prompts: Response-Mode Analyses

Eighty-three Test of English as a Foreign Language (TOEFL) writing prompts administered via computer-based testing between July 1998 and August 2000 were examined for differences attributable to the response mode (handwriting or word processing) chosen by examinees. Differences were examined statistically using polytomous logistic regression. A variable measuring English-language ability (ELA) was developed from the multiple-choice components of the TOEFL and used as a matching variable. Although there was little observed difference in mean writing scores, when examinees were matched on ELA, small differences were observed in effect sizes consistently favoring the handwriting response mode. This difference favoring the handwriting response mode occurred for all of the writing prompts analyzed, suggesting a general effect for response mode. Differences for individual writing prompts were small, however.

Comparability of TOEFL CBT Writing Prompts for Different Native Language Groups

This study has investigated the comparability of computer-based testing writing prompts in the Test of English as a Foreign LanguageTM (TOEFL) for examinees of different native language backgrounds. A total of 81 writing prompts introduced from July 1998 through August 2000 were examined using a 3-step logistic regression procedure for ordinal items. An English language ability (ELA) variable was created by summing the standardized TOEFL Reading, Listening, and Structure scale scores. This ELA variable was used to match examinees of East Asian (Chinese, Japanese, and Korean) and European (German, French, and Spanish) language groups. Although about one third of the 81 prompts were initially flagged because of statistically significant group effects; the effect sizes were too small for any of those flagged prompts to be classified as having an important group effect.

Investigating Uniform and Non-Uniform Gender DIF in Computer-Based ESL Writing Assessment

The objective of the present investigation was to examine the comparability of writing prompts for different gender groups in the context of the computer-based Test of English as a Foreign Language™ (TOEFL®-CBT). A total of 87 prompts administered from July 1998 through March 2000 were analyzed. An extended version of logistic regression for polytomous items was used to investigate both uniform and non-uniform gender effects. An English Language Ability variable was developed from the multiple-choice components of the TOEFL®-CBT examination and used as a matching variable. Initially, most of the prompts were flagged because of statistically significant uniform gender effects, with some prompts displaying non-uniform effects as well. Nevertheless, the effect sizes were too small for any of those flagged prompts to be classified as having an important group effect. These findings are discussed in relation to prompt content review, gender format differences, and second language learning theories.