Randall D. Penfield

Assessing Differential Item Functioning among Multiple Groups: A Comparison of Three Mantel-Haenszel Procedures.

It is often the case in performing a differential item functioning (DIF) analysis that comparisons are made between a single reference group and multiple focal groups. Conducting a separate test of DIF for each focal group has several undesirable qualities: (a) the Type I error rate will exceed the intended nominal level if the level of significance for each individual test is not appropriately adjusted, (b) the power may not be as high as a single test that assesses DIF among all groups simultaneously, and (c) substantial time and computing resources are required. These drawbacks are potentially avoided by using a procedure that has the capacity to assess DIF across all groups simultaneously. In this study I compare the performance of three methods of assessing DIF across multiple demographic groups; the Mantel-Haenszel chi-square statistic with no adjustment to the alpha level, the Mantel-Haenszel chi-square statistic with a Bonferroni adjusted alpha level, and the Generalized Mantel-Haenszel statistic (GMH) that offers a single test of significance across all groups. Simulations were conducted in which there was a single reference group and 1, 2, 3, and 4 focal groups, having from 1 to all of the focal groups in a given condition experiencing DIF. Additional conditions that were varied included group size, focal group ability distribution, and magnitude of matching criterion contamination. The results suggest that GMH is in general the most appropriate procedure because its Type I error rate remained at the nominal level of 0.05, and its power was consistently among the highest.

Confidence Interval Coverage for Cohen's Effect Size Statistic:

Kelley compared three methods for setting a confidence interval (CI) around Cohens standardized mean difference statistic: the noncentral-t-based, percentile (PERC) bootstrap, and biased-corrected and accelerated (BCA) bootstrap methods under three conditions of nonnormality, eight cases of sample size, and six cases of population effect size (ES) magnitude. Kelley recommended the BCA bootstrap method. The authors expand on his investigation by including additional cases of nonnormality. Like Kelley, they find that under many conditions, the BCA bootstrap method works best; however, they also find that in some cases of nonnormality, the method does not control probability coverage. The authors also define a robust parameter for ES and a robust sample statistic, based on trimmed means and Winsorized variances, and cite evidence that coverage probability for this parameter is good over the range of nonnormal distributions investigated when the PERC bootstrap method is used to set CIs for the robust ES.

Applying a Score Confidence Interval to Aiken's Item Content-Relevance Index

Item content-relevance is an important consideration for researchers when developing scales used to measure psychological constructs. Aiken (1980) proposed a statistic, V, that can be used to summarize item content-relevance ratings obtained from a panel of expert judges. This article proposes the application of the Score confidence interval to Aikens V statistic to improve the inference of the unknown population value of V. The application of the Score confidence interval to V is described, a numerical example is provided, and a demonstration of the Score confidence interval is presented for ratings obtained in the development of a scale measuring life skills.

An Approach for Categorizing DIF in Polytomous Items

A widely used approach for categorizing the level of differential item functioning (DIF) in dichotomous items is the scheme proposed by Educational Testing Service (ETS) based on a transformation of the Mantel-Haeszel common odds ratio. In this article two classification schemes for DIF in polytomous items (referred to as the P1 and P2 schemes) are proposed that parallel the criteria set forth in the ETS scheme for dichotomous items. The theoretical equivalence of the P1 and P2 schemes to the ETS scheme is described, and the results of a simulation study conducted to examine the empirical equivalence of the P1 and P2 schemes to the ETS scheme are presented.

Modeling DIF Effects Using Distractor-Level Invariance Effects: Implications for Understanding the Causes of DIF

In 2008, Penfield showed that measurement invariance across all response options of a multiple-choice item (correct option and the J distractors) can be modeled using a nominal response model that included a differential distractor functioning (DDF) effect for each of the J distractors. This article extends this concept to consider how the differential item functioning (DIF) effect (i.e., the conditional between-group differences in the probability of correct response) is determined by the J DDF effects. In particular, this article shows how the DIF effect can be modeled as a function of the J DDF effects and thus reveals the conditions that must hold for uniform DIF, nonuniform DIF, and crossing DIF to exist. The results provide insight into the potential item-level properties that lead to uniform, nonuniform, and crossing DIF. The findings may shed light on the etiology of different forms of DIF, which may help analysts target the particular causes of the DIF effect.

Identifying Differential Item Functioning of Rating Scale Items With the Rasch Model: An Introduction and an Application

This study (a) provided a conceptual introduction to differential item functioning (DIF), (b) introduced the multifaceted Rasch rating scale model (MRSM) and an associated statistical procedure for identifying DIF in rating scale items, and (c) applied this procedure to previously collected data from American coaches who responded to the coaching efficacy scale (CES; Feltz, Chase, Moritz, & Sullivan, 1999). In this study, an item displayed DIF if coaches from different groups were more or less likely to endorse that item once coaches were matched on the efficacy of interest, where Motivation, Game Strategy, Technique, and Character Building efficacies defined coaching efficacy. Coach gender and level coached were selected as the grouping variables. None of the Technique and Character Building items exhibited DIF based on coach gender or level coached. One of the Motivation items and one of the Game Strategy items exhibited DIF based on coach gender. Two of the Motivation items exhibited DIF based on level coached.

Urban Elementary Teachers’ Perspectives on Teaching Science to English Language Learners

This descriptive study examined urban elementary school teachers’ perceptions of their science content knowledge, science teaching practices, and support for language development of English language learners. Also examined were teachers’ perceptions of organizational supports and barriers associated with teaching science to nonmainstream students. The study involved 221 third- through fifth-grade teachers from 15 urban elementary schools in a large school district. The teachers completed a survey in the spring of 2005. The internal consistency reliability estimates, Cronbach α, for scales created from the survey items were within an acceptable range. The teachers reported that they were generally knowledgeable about science topics at their grade level and that they taught science to promote students’ understanding and inquiry. In contrast, the teachers reported rarely discussing student diversity in their own teaching or with other teachers at their schools. The teachers identified specific organizational supports and barriers in teaching science with diverse student groups at both the school and classroom levels.

Further Refinements in the Measurement of Exercise Imagery: The Exercise Imagery Inventory

The factorial and construct validity of the Exercise Imagery Inventory (EII) were assessed with 3 separate samples of participants. In Phase 1, a 41-item measure was administered to 504 undergraduate students. Exploratory factor analysis supported a 4-factor model that explained 65% of the variance. In Phase 2, a 19-item measure was administered to a sample of 509 individuals to assess 4- and 5-factor models. During Phase 3, 724 participants completed the EII, the Leisure Time Exercise Questionnaire (Godin & Shephard, 1985), and the Barriers Self-Efficacy Scale (McAuley, 1992) to further test the factor structure of the EII and correlations with exercise-related constructs. Our results supported a 4-factor model to explain the latent structure of the 19-item scale. The 4 exercise imagery factors were labeled Appearance- Health imagery, Exercise Technique, Exercise Self-efficacy, and Exercise Feelings. The EII subscales were positively correlated with exercise behavior and exercise self-efficacy.

Effect Sizes and their Intervals: The Two-Level Repeated Measures Case.

Probability coverage for eight different confidence intervals (CIs) of measures of effect size (ES) in a two-level repeated measures design was investigated. The CIs and measures of ES differed with regard to whether they used least squares or robust estimates of central tendency and variability, whether the end critical points of the interval were obtained using a theoretical or an empirical sampling distribution, and whether the ESs used a pooled or nonpooled estimate of error variability. These intervals were compared when data were obtained from both normal and nonnormal distributions and when the population magnitude of effect, size of sample, and variance heterogeneity were varied. Itwas found that the ESs and intervals that used robust estimators and critical values were obtained through a bootstrap method better at controlling the probability coverage (i.e., within [.925, .975]).

Using a Taxonomy of Differential Step Functioning to Improve the Interpretation of DIF in Polytomous Items: An Illustration

The assessment of differential item functioning (DIF) in polytomous items addresses between-group differences in measurement properties at the item level, but typically does not inform which score levels may be involved in the DIF effect. The framework of differential step functioning (DSF) addresses this issue by examining between-group differences in the measurement properties at each step underlying the polytomous response variable. The pattern of the DSF effects across the steps of the polytomous response variable can assume several different forms, and the different forms can have different implications for the sensitivity of DIF detection and the final interpretation of the causes of the DIF effect. In this article we propose a taxonomy of DSF forms, establish guidelines for using the form of DSF to help target and guide item content review and item revision, and provide procedural rules for using the frameworks of DSF and DIF in tandem to yield a comprehensive assessment of between-group measurement equivalence in polytomous items.