Frederic Robin

Using Cluster Analysis to Facilitate Standard Setting

Setting standards on tests remains an important and pervasive problem in educational and psychological testing. Traditional standard-setting methods have been criticized due to reliance on untested subjective judgment, lack of demonstrated reliability, and lack of external validation. In this article, we present a new procedure designed to help improve previous standard-setting methods. This procedure involves cluster analyzing test takers to discover examinee groups useful for (a) envisioning marginally competent performance as required in test-centered standard-setting methods or (b) defining borderline or contrasting groups used in examinee-centered methods. We applied the procedure to a state-wide mathematics proficiency test. The standards derived from the cluster analyses were compared with those established at the local level and with those derived from a more traditional borderline and contrasting groups analysis. We observed relative congruence across the local cutscores and those derived using c...

Detection Of Differential Item Functioning In Large-Scale State Assessments: A Study Evaluating A Two-Stage Approach

In differential item functioning (DIF) studies, examinees from different groups are typically ability matched, and then one or more statistical indices are used to compare performance on a set of test items. Typically, matching is on total test score (a criterion both observable and easily accessible), but it may be limited in value because if DIF is present, it is likely to distort test scores and potentially confound any item performance differences. Thus, some researchers have advocated iterative approaches for DIF detection. In this article, a two-stage methodology for evaluating DIF in large-scale state assessment data was explored. The findings illustrated the merit of iterative approaches for DIF detection. Items being flagged as DIF in the second stage were not necessarily the same items identified as DIF in the first stage and vice versa, and this finding was directly related to the amount of DIF found in the Stage 1 analyses.

19 – Item Response Models for the Analysis of Educational and Psychological Test Data

Publisher Summary This chapter introduces a number of widely used item response theory (IRT) models and describes briefly their application in test development and computer-adaptive testing. IRT models are central today in test development, test evaluation, and test data analysis. The chapter also emphasizes on the shortcomings of the classical test models. Much of classical test theory is concerned with the estimation and control of error in the testing process. Classical test models lack some desired features that can be fulfilled by IRT model. These features include item statistics that are not group dependent, examinee ability estimates that are not dependent on test difficulty, test models that provide a basis for matching test items to ability levels, and test models that are not based on implausible or difficult-to-meet assumptions. Finally, the chapter concludes by highlighting new directions for developing and using IRT models and identifying important issues requiring research.

Evaluating the Equivalence of Different Language Versions of a Credentialing Exam.

Many credentialing exams are administered in multiple languages with the assumption that the different language versions of the exam are equivalent. In this study, we illustrate how multidimensional scaling (MDS) and differential item functioning (DIF) procedures can be used to evaluate the equivalence of different language versions of an exam. Using data from an international credentialing program, examples of structural differences and DIF across languages are presented. Weighted MDS and the Delta-plot and standardization DIF detection methods appear to be effective for evaluating structural and item differences across language versions of an exam, even when sample sizes are modest.

Using Multidimensional Scaling To Assess the Dimensionality of Dichotomous Item Data.

In this study, we investigated the utility of multidimensional scaling (MDS) for assessing the dimensionality of dichotomous test data. Two MDS proximity measures were studied: one based on the PC statistic proposed by Chen and Davison (1996), the other based on inter-item Euclidean distances. Stouts (1987) test of essential unidimensionality (DIMTEST) was also used as a standard for comparison. Twenty different conditions of unidimensional and multidimensional data were simulated, varying the number of test items, correlations among dimensions, and type of data generation model (Rasch or two-parameter IRT model). DIMTEST performed best overall, but had some trouble detecting multidimensionality when the number of test items was small. The PC statistic correctly identified the dimensionality of the unidimensional data, whereas the use of Euclidean distances suggested the two-parameter unidimensional data were multidimensional. Both MDS procedures correctly identified multidimensionality under the low correlation conditions, but were generally unable to detect multidimensionality when the dimensions were highly correlated. Analysis of Euclidean distances were best for determining the precise dimensionality of the multidimensional data under the low correlation condition. Implications of the findings are discussed, and suggestions for future research are provided.

Rasch Scaling Program (RSP)

The Rasch Scaling Program ( RSP) is a software package for applying the one-parameter logistic (Rasch) model to dichotomously scored item response data. Applications of the model might include (1) estimating statistics for items in an item bank, (2) estimating latent trait scores ( θ), (3) conducting studies to identify bias in test items, (4) designing tests, and (5) identifying aberrant persons (i.e., persons with response patterns that are not consistent with predictions from the Rasch model). Special features of the software are the availability of several parameter estimation procedures, including conditional and marginal maximum likelihood ( MML ) estimation; an array of fit statistics for persons, items, and score distributions; the capability to handle incomplete data designs, such as designs that arise when equating or linking multiple forms of a test; and “caution indices” related to person fit. RSPwas completed in 1993 through the efforts of Cees Glas while at the Central Institution of Test Development ( CITO), Jules Ellis of the University of Nijmegen, staff members from the Interuniversity Expertise Center Pro GAMMA (a not-for-profit organization that develops and distributes computer software for the social sciences), and an interuniversity advisory group in The Netherlands. The software comes with a clearly written and comprehensive user’s manual. It contains a thorough introduction to the Rasch (1960/80) model, including explanations of the various estimation methods used, guidelines concerning model fit and associated fit statistics employed by RSP, and a discussion of item bias. Its description of the program interface and menu system is supplemented by clear figures and examples showing actual RSPscreens. The manual also provides step-by-step instructions for runningRSP.

An Exploratory Analysis of Differential Item Functioning and Its Possible Sources in a Higher Education Admissions Context.

ABSTRACT We analyzed a pool of items from an admissions test for differential item functioning (DIF) for groups based on age, socioeconomic status, citizenship, or English language status using Mantel-Haenszel and item response theory. DIF items were systematically examined to identify its possible sources by item type, content, and wording. DIF was primarily found in the citizenship group. As suggested by expert reviewers, possible sources of DIF in the direction of U.S. citizens was often in Quantitative Reasoning in items containing figures, charts, tables depicting real-world (as opposed to abstract) contexts. DIF items in the direction of non-U.S. citizens included “mathematical” items containing few words. DIF for the Verbal Reasoning items included geocultural references and proper names that may be differentially familiar for non-U.S. citizens. This study is responsive to foundational changes in the fairness section of the Standards for Educational and Psychological Testing, which now consider additional groups in sensitivity analyses, given the increasing demographic diversity in test-taker populations.