M. Dolores Hidalgo

Differential Item Functioning Detection and Effect Size: A Comparison between Logistic Regression and Mantel-Haenszel Procedures

This article compares several procedures in their efficacy for detecting differential item functioning (DIF): logistic regression analysis, the Mantel-Haenszel (MH) procedure, and the modified Mantel-Haenszel procedure by Mazor, Clauser, and Hambleton. It also compares the effect size measures that these procedures provide. In this study, different conditions of item parameters (difficulty and discrimination) and DIF magnitude were manipulated. Furthermore, both uniform and nonuniform DIF conditions were simulated. Results suggest that logistic regression analysis generally detected more items with DIF than the standard MH procedure and the modified MH procedure for symmetrical nonuniform DIF. The DIF effect size measures based on logistic regression, however, appeared to be insensitive to the specified DIF conditions.

Efficacy of Effect Size Measures in Logistic Regression

Statistical techniques based on logistic regression (LR) are adequate for the detection of differential item functioning (DIF) in dichotomous items. Nevertheless, they return more false positives (FPs) than do other DIF detection techniques. This paper compares the efficacy of DIF detection using the LR significance test and the estimation of the effect size that these procedures provide using R2 of Nagelkerke. The variables manipulated were different conditions of sample size, focal and reference group sample size ratio, amount of DIF, test length and percentage of test items with DIF. In addition, examinee responses were generated to simulate both uniform and nonuniform DIF (symmetric and asymmetric). In all cases, dichotomous response tests were used. The results show that the use of R2 as a strategy for detecting DIF obtained lower correct detection percentages than those obtained from significance tests. Moreover, the LR significance test showed adequate control of FP rates, close to the nominal 5%, ...

EASY-DIF: Software for Analyzing Differential Item Functioning Using the Mantel- Haenszel and Standardization Procedures

Practitioners are increasingly becoming interested in Differential Item Functioning (DIF) forimproving the validity of test and scale interpretations. Among the statistical procedures availableto assess DIF in dichotomous and polytomous items, the Mantel-Haenszel chi-square (1959) andother standardization proceduresmay beparticularly attractive to practitioners.Some of the statis-tics associated with Mantel-Haenszel procedure can be performed using specific software such asEZ-DIF (Waller, 1998), DIFAS (Penfield, 2005), and MH-DIF (Fidalgo, 1994). However, usingthese programs requires being familiar with the statistics of DIF procedures in order to understandoutput. In addition, key characteristics of the Mantel-Haenszel procedures, such as the matchingstrategy (thick or thin), purification of the matching criteria, and so on, are not currently availablein all statistical software packages. As a result, EASY-DIF was developed to provide easy-to-usesoftware for performing the most common and useful MH and standardization procedures, witha view to guiding practitioners through the analyses and helping them interpret the output.EASY-DIFanalyzesuniformandnonuniformDIFforthetotalsampleandseparatelyforlow-performing and high-performing groups (Clauser, Mazor, & Hambleton, 1994). Users canexplore possible cancellation and amplification DIF effects by establishing different cut scoresfor each group. Up to six different matching strategies based on total score distributions can beimplemented: thinmatching,equal interval, percentage of totalsample, percentage of focalsam-ple, censored matching, and minimum cell frequency (Donoghue & Allen, 1993). In addition,users can use a procedure purification of matching criteria through the selection of a validsubtest.For dichotomous items, EASY-DIF computes the Mantel-Haenszel chi-square (Holland T Mantel & Haenszel, 1959), the Mantel-Haenszel common odds ratio (CamilliSMantel&Haenszel,1959),theMH-DeltaDIF,thestandarderrorofMH-DeltaComputer Program Exchange