T. C. Oshima

A Description and Demonstration of the Polytomous-DFIT Framework.

Raju, van der Linden, & Fleer (1995) proposed an item response theory based, parametric differential item functioning (DIF) and differential test functioning (DTF) procedure known as differential functioning of items and tests (DFIT). According to Raju et al., the DFIT framework can be used with unidimensional and multidimensional data that are scored dichotomously and/or polytomously. This study examined the polytomous-DFIT framework. Factors manipulated in the simulation were: (1) length of test (20 and 40 items), (2) focal group distribution, (3) number of DIF items, (4) direction of DIF, and (5) type of DIF. The findings provided promising results and indicated directions for future research. The polytomous DFIT framework was effective in identifying DTF and DIF for the simulated conditions. The DTF index did not perform as consistently as the DIF index. The findings are similar to those of unidimensional and multidimensional DFIT studies.

Effect of Sample Size, Number of Biased Items, and Magnitude of Bias on a Two-Stage Item Bias Estimation Method.

A two-stage procedure for estimating item bias was examined with six indexes of item bias and with the Mantel-Haenszel (MH) statistic; the sample size, the number of biased items, and the magnitude of the bias were varied. The second stage of the procedure did not identify substantial numbers of false positives (unbiased items identified as biased). However, the identification of true positives in the second stage was useful only when the magnitude of the bias was not small and the number of biased items was large (20% or 40% of the test). The weighted indexes tended to identify more true and false positives than their unweighted item response theory counterparts. Finally, the MH statistic identified fewer false positives, but did not identify small bias as well as the item response theory indexes

Multidimensionality and Item Bias in Item Response Theory

This paper demonstrates empirically how item bias indexes based on item response theory (IRT) identify bias that results from multidimensionality. When a test is multidimensional (MD) with a primary trait and a nuisance trait that affects a small portion of the test, item bias is defined as a mean difference on the nuisance trait between two groups. Results from a simulation study showed that although IRT-based bias indexes clearly distinguished multidimensionality from item bias, even with the presence of a between-group dif ference on the primary trait, the bias detection rate depended on the degree to which the item measured the nuisance trait, the values of MD discrimination, and the number of MD items. It was speculated that bias defined from the MD perspective was more likely to be detected when the test data met the essential unidimensionality assumption. Index

Type I Error Rates for Welch’s Test and James’s Second-Order Test Under Nonnormality and Inequality of Variance When There Are Two Groups

Type I error rates were estimated for three tests that compare means by using data from two independent samples: the independent samples t test, Welch’s approximate degrees of freedom test, and James’s second-order test. Type I error rates were estimated for skewed distributions, equal and unequal variances, equal and unequal sample sizes, and a range of total sample sizes. Welch’s test and James’s test have very similar Type I error rates and tend to control the Type I error rate as well or better than the independent samples t test does. The results provide guidance about the total sample sizes required for controlling Type I error rates.

Linking Multidimensional Item Calibrations.

Invariance of trait scales across changing samples of items and examinees is a central tenet of item re sponse theory (IRT). However, scales defined by most IRT models are truly invariant with respect to certain linear transformations of the parameters. The problem is to find the proper transformation that places separate calibrations on a common scale. A variety of proce dures for estimating transformations have been pro posed for unidimensional models. This paper explores some issues involved in extending and adapting unidi mensional linking procedures for use with multidimen sional IRT models.

Standardized Conditional "SEM": A Case for Conditional Reliability.

An examinee-level (or conditional) reliability is proposed for use in both classical test theory (CTT) and item response theory (IRT). The well-known group-level reliability is shown to be the average of conditional reliabilities of examinees in a group or a population. This relationship is similar to the known relationship between the square of the conditional standard error of measurement (SEM) and the square of the group-level SEM. The proposed conditional reliability is illustrated with an empirical data set in the CTT and IRT frameworks.

The Item Parameter Replication Method for Detecting Differential Functioning in the Polytomous DFIT Framework

The recent study of Oshima, Raju, and Nanda proposes the item parameter replication (IPR) method for assessing statistical significance of the noncompensatory differential item functioning (NCDIF) index within the differential functioning of items and tests (DFIT) framework. Previous Monte Carlo simulations have found that the appropriate cutoff values for determining statistical significance of NCDIF depend on sample size and the item response theory (IRT) model used for the analysis. The IPR method simplifies the process of identifying cutoff values that are tailored to a particular research setting. This approach has been shown to be effective for detecting differential item functioning (DIF) in dichotomous items. The current article extends the IPR method to the polytomous case.

Effect of Multiple Testing Adjustment in Differential Item Functioning Detection

In a typical differential item functioning (DIF) analysis, a significance test is conducted for each item. As a test consists of multiple items, such multiple testing may increase the possibility of making a Type I error at least once. The goal of this study was to investigate how to control a Type I error rate and power using adjustment procedures for multiple testing, which have been widely used in applied statistics. In the simulation, four distinct DIF methods were performed under various testing conditions. The methods were the Mantel–Haenszel (MH) method, the logistic regression (LR) procedure, the Differential Functioning Item and Test (DFIT) framework, and Lord’s chi-square test. As an adjustment procedure, the Bonferroni correction, Holm’s procedure, or the Benjamini and Hochberg (BH) false discovery rate was applied. The results showed the MH and the LR clearly benefited from Holm’s and the BH adjustments, whereas the DFIT and Lord’s chi-square test did not require adjustments for conditions under this study.

A SAS Program for Testing the Hypothesis of the Equal Means Under Heteroscedasticity: James's Second-Order Test

A SAS program is presented which computes Jamess second-order procedure for testing the hypothesis of the equality of J means under heteroscedasticity.

IPLINK: Multidimensional and Unidimensional Item Parameter Linking in Item Response Theory

Many applications of item response theory (IRT), such as test equating or differential item functioning, depend on successful linking procedures. IPLINK estimates linking coefficients that place item parameter estimates from separate calibrations onto a common trait metric for test data that are m-dimensional. The linking coefficient estimates are obtained by minimizing the differences between two sets of functions of item parameter estimates using one of the four methods described in Oshima, Davey, & Lee (1996). The four methods are: (1) the direct method, (2) the equated function method, (3) the test characteristic function method, and (4) the item characteristic function method. IPUNK is a Windows-based program. The user creates an initial file on an edit screen by selecting options, such as the number of trait dimensions and model type. The default settings for commonly used