Minjeong Jeon

Modeling Differential Item Functioning Using a Generalization of the Multiple-Group Bifactor Model.

The authors present a generalization of the multiple-group bifactor model that extends the classical bifactor model for categorical outcomes by relaxing the typical assumption of independence of the specific dimensions. In addition to the means and variances of all dimensions, the correlations among the specific dimensions are allowed to differ between groups. By including group-specific difficulty parameters, the model can be used to assess differential item functioning (DIF) for testlet-based tests. The model encompasses various item response models for polytomous data by allowing for different link functions, and it includes testlet and second-order models as special cases. Importantly, by assuming that the testlet dimensions are conditionally independent given the general dimension, the authors show, using a graphical model framework, that the integration over all latent variables can be carried out through a sequence of computations in two-dimensional subspaces, making full-information maximum likelihood estimation feasible for high-dimensional problems and large datasets. The importance of relaxing the orthogonality assumption and allowing for a different covariance structure of the dimensions for each group is demonstrated in the context of the assessment of DIF. Through a simulation study, it is shown that ignoring between-group differences in the structure of the multivariate latent space can result in substantially biased estimates of DIF.

A generalized item response tree model for psychological assessments.

A new item response theory (IRT) model with a tree structure has been introduced for modeling item response processes with a tree structure. In this paper, we present a generalized item response tree model with a flexible parametric form, dimensionality, and choice of covariates. The utilities of the model are demonstrated with two applications in psychological assessments for investigating Likert scale item responses and for modeling omitted item responses. The proposed model is estimated with the freely available R package flirt (Jeon et al., 2014b).

Modeling Answer Changes on Test Items

The probability of test takers changing answers upon review of their initial choices is modeled. The primary purpose of the model is to check erasures on answer sheets recorded by an optical scanner for numbers and patterns that may be indicative of irregular behavior, such as teachers or school administrators changing answer sheets after their students have finished the test or test takers communicating with each other about their initial responses. A statistical test based on the number of erasures is derived from the model. Besides, it is shown how to analyze the residuals under the model to check for suspicious patterns of erasures. The use of the two procedures is illustrated for an empirical data set from a large-scale assessment. The robustness of the model with respect to less than optimal opportunities for regular test takers to review their responses is investigated.

Flexible Item Response Theory Modeling With FLIRT

The free R software package FLIRT (Flexible Item Response Theory) is introduced. As the acronym of the package indicates, FLIRT offers flexible modeling of item response data. Flexibility of FLIRT comes from its general statistical framework. By conceptualizing Item Response Theory (IRT) models as generalized linear and nonlinear mixed models (Rijmen, Tuerlinckx, De Boeck, & Kuppens, 2003), various types of IRT models can be understood and constructed with FLIRT by simply selecting a parametric form, the number of dimensions, the number of item and person covariates, the number of person groups, a link function for different types of response variables, and so on. A major attraction of FLIRT is that users can easily build, explore, and estimate a variety of one and two parameter logistic (1PL and 2PL, respectively) and bifactor IRT models using a variety of modeling options (currently three parameter logistic [3PL] models are not available). A unique strength of FLIRT is its ability to incorporate explanatory features in traditional measurement models. Therefore, different hypotheses on item and person parameters can be modeled and tested. Furthermore, FLIRT is a dedicated IRT software package and provides IRT-friendly specifications of models and interpretations of outputs. Another strength of FLIRT comes from its efficient maximum likelihood (ML) estimation using a modified expectation-maximization (EM) algorithm based on graphical model theory (Lauritzen, 1995; Rijmen, Vansteelandt, & De Boeck, 2008). The modified EM algorithm implements the expectation (E) step in an efficient way such that computations can be carried out in lower dimensional latent spaces. Additional computational efficiency is achieved by adopting adaptive quadrature (Pinheiro & Bates, 1995; Rabe-Hesketh, Skrondal, & Pickles, 2005) for numerical integration. FLIRT is thus relatively faster than other software packages based on regular ML methods. The gain in computational efficiency can be nontrivial, in particular for high-dimensional models (Jeon, Rijmen, & Rabe-Hesketh, 2013).

A Third-Order Item Response Theory Model for Modeling the Effects of Domains and Subdomains in Large-Scale Educational Assessment Surveys

Second-order item response theory models have been used for assessments consisting of several domains, such as content areas. We extend the second-order model to a third-order model for assessments that include subdomains nested in domains. Using a graphical model framework, it is shown how the model does not suffer from the curse of multidimensionality. We apply unidimensional, second-order, and third-order item response models to the 2007 Trends in International Mathematics and Science Study. Our findings suggest that deviations from unidimensionality are more pronounced at the content domain level than at the cognitive domain level and that deviations from unidimensionality at the content domain level become negligible after taking into account topic areas.

Profile-Likelihood Approach for Estimating Generalized Linear Mixed Models With Factor Structures

In this article, the authors suggest a profile-likelihood approach for estimating complex models by maximum likelihood (ML) using standard software and minimal programming. The method works whenever setting some of the parameters of the model to known constants turns the model into a standard model. An important class of models that can be estimated this way is generalized linear mixed models with factor structures. Such models are useful in educational research, for example, for estimation of value-added teacher or school effects with persistence parameters and for analysis of large-scale assessment data using multilevel item response models with discrimination parameters. The authors describe the profile-likelihood approach, implement it in the R software, and apply the method to longitudinal data and binary item response data. Simulation studies and comparison with gllamm show that the profile-likelihood method performs well in both types of applications. The authors also briefly discuss other types of models that can be estimated using the profile-likelihood idea.

Fitting an item response theory model with random item effects across groups by a variational approximation method

AbstractPurpose. Data from international educational assessments conducted in many countries are mostly analyzed using item response theory. The assumption that all items behave the same in all countries is often not tenable. The variability of item parameters across countries can be taken into account by assuming that the item parameters are random effects (De Jong et al. in J. Consum. Res. 34:260–278, 2007; De Jong and Steenkamp in Psychometrika 75:3–32, 2010). However, the complex latent structure of such a model, with latent variables both at the item and the person level, renders maximum likelihood estimation computationally challenging. We describe a variational estimation technique that consists of approximating the likelihood function by a computationally tractable lower bound. Methods. A mean field approximation to the posterior distribution of the latent variables was used. The update equations were derived for the specific case of discrete random effects and implemented in a Maximization Maximization algorithm (Neal and Hinton in M.I. Jordan (Ed.) Learning in Graphical Models, Kluwer Academic, Dordrecht, pp. 355–368, 1998). Parameter recovery was investigated in a simulation study. The method was also applied to the Progress in International Reading Study of 2006. Results. The model parameters were recovered well under all conditions of the simulation study. In the application, the estimated variances of the random item effects showed a high positive correlation with traditional measures for the lack of item invariance across groups. Conclusions. The mean field approximation and variational methods in general offer a computationally tractable alternative to exact maximum likelihood estimation.

Multidimensionality in the measurement of math-specific anxiety and its relationship with mathematical performance

Traditionally, mathematical anxiety has been utilized as a unidimensional construct. However, math-specific anxiety may have distinguishable factors, and taking these factors into account may better illuminate the relationship between anxiety and mathematics performance. Drawing from the Western Reserve Reading and Math Project (N = 244 children, mean age = 12.28 years), the present study examined math-specific anxiety and mathematics problem evaluation, utilizing a structural equation modeling approach with an item-level measurement model structure. Results suggested math-specific anxiety tapped into three factors: anxiety about performing mathematical calculations, anxiety about math in classroom situations, and anxiety about math tests. Among the three math anxiety factors, only calculation anxiety was significantly and negatively related to math performance while holding other anxiety factors constant. Implications for the measurement of math-specific anxiety are discussed.

Recent developments in maximum likelihood estimation of MTMM models for categorical data.

Maximum likelihood (ML) estimation of categorical multitrait-multimethod (MTMM) data is challenging because the likelihood involves high-dimensional integrals over the crossed method and trait factors, with no known closed-form solution. The purpose of the study is to introduce three newly developed ML methods that are eligible for estimating MTMM models with categorical responses: Variational maximization-maximization (e.g., Rijmen and Jeon, 2013), alternating imputation posterior (e.g., Cho and Rabe-Hesketh, 2011), and Monte Carlo local likelihood (e.g., Jeon et al., under revision). Each method is briefly described and its applicability for MTMM models with categorical data are discussed.

Decision qualities of Bayes factor and p value-based hypothesis testing.

The purpose of this article is to investigate the decision qualities of the Bayes factor (BF) method compared with the p value-based null hypothesis significance testing (NHST). The performance of the 2 methods is assessed in terms of the false- and true-positive rates, as well as the false-discovery rates and the posterior probabilities of the null hypothesis for 2 different models: an independent-samples t test and an analysis of variance (ANOVA) model with 2 random factors. Our simulation study results showed the following: (a) The common BF > 3 criterion is more conservative than the NHST &agr; = .05 criterion, and it corresponds better with the &agr; = .01 criterion. (b) An increasing sample size has a different effect on the false-positive rate and the false-discovery rate, depending on whether the BF or NHST approach is used. (c) When effect sizes are randomly sampled from the prior, power curves tend to be flat compared with when effect sizes are prespecified. (d) The larger the scale factor (or the wider the prior), the more conservative the inferential decision is. (e) The false-positive and true-positive rates of the BF method are very sensitive to the scale factor when the effect size is small. (f) While the posterior probabilities of the null hypothesis ideally follow from the BF value, they can be surprisingly high using NHST. In general, these findings were consistent independent of which of the 2 different models was used.