Hongwen Guo

Accuracy of DIF Estimates and Power in Unbalanced Designs Using the Mantel–Haenszel DIF Detection Procedure

This study examined how much improvement was attainable with respect to accuracy of differential item functioning (DIF) measures and DIF detection rates in the Mantel–Haenszel procedure when employing focal and reference groups with notably unbalanced sample sizes where the focal group has a fixed small sample which does not satisfy the minimum DIF sample size requirement specified by the testing programs, while the reference group sample size far exceeds the minimum requirement. Results showed equivalent or better results with such unbalanced but large samples than with some of the currently used minimum DIF sample size conditions. DIF investigation, therefore, does not necessarily need to cease when the focal group does not meet the minimum sample size requirement. Some analytic explanations and guidelines for DIF investigations with unbalanced sample sizes are also provided.

A New Procedure for Detection of Students’ Rapid Guessing Responses Using Response Time

ABSTRACT Unmotivated test takers using rapid guessing in item responses can affect validity studies and teacher and institution performance evaluation negatively, making it critical to identify these test takers. The authors propose a new nonparametric method for finding response-time thresholds for flagging item responses that result from rapid-guessing behavior. Using data from a low-stakes assessment of college-level academic skills as an illustration, the authors evaluate and compare model fit and score estimation based on data sets cleaned by both new and existing methods. Flagging rapid-guessing responses is found to generally improve model fit, item parameter estimation, and score estimation, as in the literature. This new method, based on both response time and response accuracy, shows promise in detecting rapid guessing and in improving efficiency of the flagging process when built into data analysis.

Nonparametric Item Response Curve Estimation With Correction for Measurement Error

Nonparametric or kernel regression estimation of item response curves (IRCs) is often used in item analysis in testing programs. These estimates are biased when the observed scores are used as the regressor because the observed scores are contaminated by measurement error. Accuracy of this estimation is a concern theoretically and operationally. This study investigates the deconvolution kernel estimation of IRCs, which corrects for the measurement error in the regressor variable. A comparison of the traditional kernel estimation and the deconvolution estimation of IRCs is carried out using both simulated and operational data. It is found that, in item analysis, the traditional kernel estimation is comparable to the deconvolution kernel estimation in capturing important features of the IRC.