Alan D. Mead

Equivalence of Computerized and Paper-and-Pencil Cognitive Ability Tests: A Meta-Analysis

The effects of the medium of test administration-paper and pencil versus computerized-were examined for timed power and speeded tests of cognitive abilities for populations of young adults and adults. Meta-analytic techniques were used to estimate the cross-mode correlation after correcting for measurement error. A total of 159 correlations was meta-analyzed: 123 from timed power tests and 36 from speeded tests. The corrected cross-mode correlation was found to be.91 when all correlations were analyzed simultaneously. Speededness was found to moderate the effects of administration mode in that the cross-mode correlation was estimated to be.97 for timed power tests but only.72 for speeded tests. No difference in equivalence was observed between adaptively and conventionally administered computerized tests

Fitting polytomous item response theory models to multiple-choice tests

This study examined how well current software implementations of four polytomous item response theory models fit several multiple-choice tests. The models were Bocks (1972) nominal model, Samejimas (1979) multiple-choice Model C, Thissen & Steinbergs (1984) multiple-choice model, and Levines (1993) maximum-likelihood formula scoring model. The parameters of the first three of these models were estimated with Thissens (1986) MULTILOG computer program; Williams & Levines (1993) FORSCORE program was used for Levines model. Tests from the Armed Services Vocational Aptitude Battery,the Scholastic Aptitude Test, and the American College Test Assessment were analyzed. The models were fit in estimation samples of approximately 3,000; cross-validation samples of approximately 3,000 were used to evaluate goodness of fit. Both fit plots and X2 statistics were used to determine the adequacy of fit. Bocks model provided surprisingly good fit; adding parameters to the nominal model did not yield improvements in fit. FORSCORE provided generally good fit for Levines nonparametric model across all tests. Index terms: Bocks nominal model, FORSCORE, maximum likelihood formula scoring, MULTILOG, polytomous IRT.

Assessment of the Measurement Equivalence of a Spanish Translation of the 16PF Questionnaire

The differential functioning of items and tests (DFIT) framework was used to examine the measurement equivalence of a Spanish translation of the Sixteen Personality Factor (16PF) Questionnaire. The questionnaire was administered in English to English-speaking Anglo-Americans and English-dominant Hispanic Americans and in Spanish to Spanish-dominant Hispanic Americans and Spanish-speaking Mexican nationals. As expected, the compensatory differential item functioning/differential test functioning (CDIF/DTF) procedure, which accounts for CDIF at the scale level, flagged fewer items as differential functioning than did the noncompensatory differential item functioning (NCDIF) procedure. Results did not support the hypothesis that DIF would be greatest in the Anglo versus Spanish-speaker comparison followed by the Hispanic versus Spanish-speaker comparison and least in the Anglo versus Hispanic comparison. Advantages of using the DFIT framework in assessing test translations, especially for test developers, are discussed.

An Introduction to Multistage Testing

This issue of Applied Measurement in Education contains a series of articles examining aspects of multistage testing (MST), also called computerized adaptive sequential testing (CAST). MST is a test administration method whereby examinees are routed to later tests on the basis of their performances on routing tests. Examinees who fare poorly on routing tests complete easier levels in subsequent stages. Thus, multistage tests are adaptive but otherwise have important differences as compared to computerized adaptive tests.