Bradley A. Hanson

Uniform DIF and DIF Defined by Differences in Item Response Functions

Uniform differential item functioning (DIF) exists when the statistical relationship between item response and group is constant for all levels of a matching variable. Two other types of DIF are defined based on differences in item response functions (IRFs) among the groups of examinees: unidirectional DIF (the IRFs do not cross) and parallel DIF (the IRFs are the same shape but shifted from one another by a constant, i.e., the IRFs differ only in location). It is shown that these three types of DIF are not equivalent and the relationships among them are examined in this paper for two item response categories, two groups, and an ideal continuous univariate matching variable. The results imply that unidirectional and parallel DIF which have been considered uniform DIF by several authors are not uniform DIF. For example, it is shown in this paper that parallel three-parameter logistic IRFs do not result in uniform DIF. It is suggested that the term “uniform DIF” be reserved for the condition in which the association between the item response and group is constant for all values of the matching variable, as distinguished from parallel and unidirectional DIF.

Reducing Bias in CAT Trait Estimation: A Comparison of Approaches

The use of a beta prior in trait estimation was extended to the maximum expected a posteriori (MAP) method of Bayesian estimation. This new method, called essentially unbiased MAP (EU-MAP), was compared with MAP (using a standard normal prior), essentially unbiased expected a posteriori, weighted likelihood, and maximum likelihood estimation methods. Comparisons were made based on the effects that the shape of prior distributions, different item bank characteristics, and practical constraints had on bias, standard error, and root-mean-square error (RMSE). Overall, EU-MAP performed best. This new method significantly reduced bias in fixed-length tests (though with a slight increase in RMSE) and performed reasonably well when a fixed posterior variance termination rule was used. Practical constraints had little effect on the bias of this method.

A Comparison of Bivariate Smoothing Methods in Common-Item Equipercentile Equating

The effectiveness of smoothing the bivariate distributions of common and noncommon item scores in the frequency estimation method of common-item equipercentile equating was exam ined. The mean squared error of equating was computed for several equating methods and sample sizes, for two sets of population bivariate distribu tions of equating and nonequating item scores defined using data from a professional licensure exam. Eight equating methods were compared: five equipercentile methods and three linear methods. One of the equipercentile methods was unsmoothed equipercentile equating. Four methods of smoothed equipercentile (SEP) equating were con sidered : two based on log-linear models, one based on the four-parameter beta binomial model, and one based on the four-parameter beta compound binomial model. The three linear equating methods were the Tucker method, the Levine Equally Reliable method, and the Levine Unequally Reliable method. The results indicated that smoothed distributions produced more accurate equating functions than the unsmoothed distributions, even for the largest sample size. Tucker linear equating produced more accurate results than SEP equating when the systematic error introduced by assuming a linear equating function was small relative to the random error of the methods of SEP equating.

Test equating under the multiple-choice model

This article presents a characteristic curve procedure for computing transformations of the item response theory (IRT) ability scale assuming the multiple-choice model. The multiple-choice model provides a realistic and informative approach to analyzing multiple-choice items in two important ways. First, the probability of guessing is a decreasing function of proficiency rather than a constant across different proficiency levels as in the three-parameter logistic model. Second, the model utilizes information from incorrect answers as well as from correct answers. The multiple-choice model includes many well-known IRT models as special cases, such as Bock’s nominal response model. Formulas needed to implement a characteristic curve method for scale transformation are presented for the multiple-choice model. The use of the characteristic curve method for the multiple-choice model is illustrated in an example equating American College Testing mathematics tests. In the process of deriving the scale transformation procedure for the multiplechoice model, corrections were made in some of the formulas presented by Baker for computing a scale transformation for the nominal response model.

A Note on Levine's Formula for Equating Unequally Reliable Tests Using Data from the Common Item Nonequivalent Groups Design.

Levine’s formula for equating unequally reliable tests using data collected in the common item nonequivalent groups equating design is an estimate of a linear function relating true scores on two test forms to be equated. Because a function relating true scores is applied to the observed score, it is not clear how the resulting converted observed score is in any sense comparable to the observed score it is being equated to. This article demonstrates that Levine’s formula can be interpreted as a method of moments estimate of an equating function that results in first order equity of the equated test score under a classical congeneric model.

A Comparison of Bootstrap Standard Errors of IRT Equating Methods for the Common-Item Nonequivalent Groups Design

The primary purpose of this study was to compare bootstrap standard errors of 5 item response theory (IRT) equating methods for the common-item nonequivalent groups design. For true-score (Method 1) and observed-score (Method 2) equating, IRT parameters were estimated separately, and a linear scaling transformation method was used to rescale the IRT parameter estimates for Form X onto the Form Y scale. For IRT chained true-score equating (Method 3), IRT parameters for Form X and Form Y were estimated separately, and then IRT chained true-score equating was performed. For the last 2 methods, IRT parameters for both forms were estimated simultaneously. Using the simultaneously estimated parameter estimates, IRT true-score (Method 4) and observed-score (Method 5) equatings were performed. For each method, the standard deviation was computed over 500 bootstrap replications to obtain the standard error of IRT equating at each raw score point for the new form. The estimated bootstrap standard errors for Methods 4 and 5 were slightly less than those for Methods 1 and 2. Method 3 produced the greatest standard errors. However, the standard errors for all 5 methods were small enough to suggest that standard errors of equating less than 0.1 standard deviation units could be obtained with any method, even with sample sizes of 500.

Standard errors of Levine linear equating

The delta method was used to derive standard errors (SEs) of the Levine observed score and Levine true score linear equating methods. SEs with a normality assumption as well as without a nor mality assumption were derived. Data from two forms of a test were used as an example to evalu ate the derived SEs of equating. Bootstrap SEs also were computed for the purpose of comparison. The SEs derived without the normality assumption and the bootstrap SEs were very close. For the skewed score distributions, the SEs derived with the normality assumption differed from the SEs derived without the normality assumption and the boot strap SEs.

Standard Errors of A Chain of Linear Equatings

A general delta method is described for computing the standard error (SE) of a chain of linear equatings. The general delta method derives the SEs directly from the moments of the score distributions obtained in the equating chain. The partial derivatives of the chain equating function needed for computing the SEs are de rived numerically. The method can be applied to equatings using the common-items nonequivalent populations design. Computer simulations were con ducted to evaluate the SEs of a chain of two equatings using the Levine and Tucker methods. The general delta method was more accurate than a method that as sumes the equating processes in the chain are statisti cally independent. Index terms: chain equating, delta method, equating, linear equating, standard error of equating.