Alberto Maydeu-Olivares

Factor Analysis with Ordinal Indicators: A Monte Carlo Study Comparing DWLS and ULS Estimation

Factor analysis models with ordinal indicators are often estimated using a 3-stage procedure where the last stage involves obtaining parameter estimates by least squares from the sample polychoric correlations. A simulation study involving 324 conditions (1,000 replications per condition) was performed to compare the performance of diagonally weighted least squares (DWLS) and unweighted least squares (ULS) in the procedures third stage. Overall, both methods provided accurate and similar results. However, ULS was found to provide more accurate and less variable parameter estimates, as well as more precise standard errors and better coverage rates. Nevertheless, convergence rates for DWLS are higher. Our recommendation is therefore to use ULS, and, in the case of nonconvergence, to use DWLS, as this method might converge when ULS does not.

Estimation of IRT graded response models: limited versus full information methods.

The performance of parameter estimates and standard errors in estimating F. Samejimas graded response model was examined across 324 conditions. Full information maximum likelihood (FIML) was compared with a 3-stage estimator for categorical item factor analysis (CIFA) when the unweighted least squares method was used in CIFAs third stage. CIFA is much faster in estimating multidimensional models, particularly with correlated dimensions. Overall, CIFA yields slightly more accurate parameter estimates, and FIML yields slightly more accurate standard errors. Yet, across most conditions, differences between methods are negligible. FIML is the best election in small sample sizes (200 observations). CIFA is the best election in larger samples (on computational grounds). Both methods failed in a number of conditions, most of which involved 200 observations, few indicators per dimension, highly skewed items, or low factor loadings. These conditions are to be avoided in applications.

Item Response Modeling of Forced-Choice Questionnaires

Multidimensional forced-choice formats can significantly reduce the impact of numerous response biases typically associated with rating scales. However, if scored with classical methodology, these questionnaires produce ipsative data, which lead to distorted scale relationships and make comparisons between individuals problematic. This research demonstrates how item response theory (IRT) modeling may be applied to overcome these problems. A multidimensional IRT model based on Thurstone’s framework for comparative data is introduced, which is suitable for use with any forced-choice questionnaire composed of items fitting the dominance response model, with any number of measured traits, and any block sizes (i.e., pairs, triplets, quads, etc.). Thurstonian IRT models are normal ogive models with structured factor loadings, structured uniquenesses, and structured local dependencies. These models can be straightforwardly estimated using structural equation modeling (SEM) software Mplus. A number of simulation studies are performed to investigate how latent traits are recovered under various forced-choice designs and provide guidelines for optimal questionnaire design. An empirical application is given to illustrate how the model may be applied in practice. It is concluded that when the recommended design guidelines are met, scores estimated from forced-choice questionnaires with the proposed methodology reproduce the latent traits well.

Asymptotically Distribution Free (Adf) Interval Estimation of Coefficient Alpha

The point estimate of sample coefficient alpha may provide a misleading impression of the reliability of the test score. Because sample coefficient alpha is consistently biased downward, it is more likely to yield a misleading impression of poor reliability. The magnitude of the bias is greatest precisely when the variability of sample alpha is greatest (small population reliability and small sample size). Taking into account the variability of sample alpha with an interval estimator may lead to retaining reliable tests that would be otherwise rejected. Here, the authors performed simulation studies to investigate the behavior of asymptotically distribution-free (ADF) versus normal-theory interval estimators of coefficient alpha under varied conditions. Normal-theory intervals were found to be less accurate when item skewness >1 or excess kurtosis >1. For sample sizes over 100 observations, ADF intervals are preferable, regardless of item skewness and kurtosis. A formula for computing ADF confidence intervals for coefficient alpha for tests of any size is provided, along with its implementation as an SAS macro.

How IRT can solve problems of ipsative data in forced-choice questionnaires

In multidimensional forced-choice (MFC) questionnaires, items measuring different attributes are presented in blocks, and participants have to rank order the items within each block (fully or partially). Such comparative formats can reduce the impact of numerous response biases often affecting single-stimulus items (aka rating or Likert scales). However, if scored with traditional methodology, MFC instruments produce ipsative data, whereby all individuals have a common total test score. Ipsative scoring distorts individual profiles (it is impossible to achieve all high or all low scale scores), construct validity (covariances between scales must sum to zero), criterion-related validity (validity coefficients must sum to zero), and reliability estimates. We argue that these problems are caused by inadequate scoring of forced-choice items and advocate the use of item response theory (IRT) models based on an appropriate response process for comparative data, such as Thurstones law of comparative judgment. We show that when Thurstonian IRT modeling is applied (Brown & Maydeu-Olivares, 2011), even existing forced-choice questionnaires with challenging features can be scored adequately and that the IRT-estimated scores are free from the problems of ipsative data.

Goodness-of-Fit Assessment of Item Response Theory Models

The article provides an overview of goodness-of-fit assessment methods for item response theory (IRT) models. It is now possible to obtain accurate p-values of the overall fit of the model if bivariate information statistics are used. Several alternative approaches are described. As the validity of inferences drawn on the fitted model depends on the magnitude of the misfit, if the model is rejected it is necessary to assess the goodness of approximation. With this aim in mind, a class of root mean squared error of approximation (RMSEA) is described, which makes it possible to test whether the model misfit is below a specific cutoff value. Also, regardless of the outcome of the overall goodness-of-fit assessment, a piece-wise assessment of fit should be performed to detect parts of the model whose fit can be improved. A number of statistics for this purpose are described, including a z statistic for residual means, a mean-and-variance correction to Pearsons X 2 statistic applied to each bivariate subtable separately, and the use of z statistics for residual cross-products.

Fitting a Thurstonian IRT model to forced-choice data using Mplus

To counter response distortions associated with the use of rating scales (a.k.a. Likert scales), items can be presented in a comparative fashion, so that respondents are asked to rank the items within blocks (forced-choice format). However, classical scoring procedures for these forced-choice designs lead to ipsative data, which presents psychometric challenges that are well described in the literature. Recently, Brown and Maydeu-Olivares (Educational and Psychological Measurement 71: 460–502, 2011a) introduced a model based on Thurstone’s law of comparative judgment, which overcomes the problems of ipsative data. Here, we provide a step-by-step tutorial for coding forced-choice responses, specifying a Thurstonian item response theory model that is appropriate for the design used, assessing the model’s fit, and scoring individuals on psychological attributes. Estimation and scoring is performed using Mplus, and a very straightforward Excel macro is provided that writes full Mplus input files for any forced-choice design. Armed with these tools, using a forced-choice design is now as easy as using ratings.

Target Rotations and Assessing the Impact of Model Violations on the Parameters of Unidimensional Item Response Theory Models

Reise, Cook, and Moore proposed a “comparison modeling” approach to assess the distortion in item parameter estimates when a unidimensional item response theory (IRT) model is imposed on multidimensional data. Central to their approach is the comparison of item slope parameter estimates from a unidimensional IRT model (a restricted model), with the item slope parameter estimates from the general factor in an exploratory bifactor IRT model (the unrestricted comparison model). In turn, these authors suggested that the unrestricted comparison bifactor model be derived from a target factor rotation. The goal of this study was to provide further empirical support for the use of target rotations as a method for deriving a comparison model. Specifically, we conducted Monte Carlo analyses exploring (a) the use of the Schmid–Leiman orthogonalization to specify a viable initial target matrix and (b) the recovery of true bifactor pattern matrices using target rotations as implemented in Mplus. Results suggest that to the degree that item response data conform to independent cluster structure, target rotations can be used productively to establish a plausible comparison model.

Disentangling impulsiveness, aggressiveness and impulsive aggression: An empirical approach using self-report measures

There is confusion in the literature concerning the concept of impulsive aggression. Based on previous research, we hypothesize that impulsivity and aggression may be related, though not as closely as to consider them the same construct. So, our aim was to provide empirical evidence of the relationship between the impulsivity and aggressiveness constructs when considered as traits. Two widely used questionnaires [Barratts Impulsiveness Scale (BIS) and Aggression Questionnaire-Refined (AQ-R)] were administered to 768 healthy respondents. Product-moment and canonical correlations were then calculated. In addition, a principal components analysis was conducted to explore whether impulsive aggression can be defined phenotypically as the expression of a single trait. The common variance between impulsivity and aggressiveness was never higher than 42%. The principal components analysis reveals that one component is not enough to represent all the variables. In conclusion, our results show that impulsivity and aggressiveness are two separate, although related constructs. This is particularly important in view of the misconceptions in the literature.

Assessing Approximate Fit in Categorical Data Analysis

A family of Root Mean Square Error of Approximation (RMSEA) statistics is proposed for assessing the goodness of approximation in discrete multivariate analysis with applications to item response theory (IRT) models. The family includes RMSEAs to assess the approximation up to any level of association of the discrete variables. Two members of this family are RMSEA2, which uses up to bivariate moments, and the full information RMSEAn. The RMSEA2 is estimated using the M2 statistic of Maydeu-Olivares and Joe (2005, 2006), whereas for maximum likelihood estimation, RMSEAn is estimated using Pearsons X2 statistic. Using IRT models, we provide cutoff criteria of adequate, good, and excellent fit using the RMSEA2. When the data are ordinal, we find a strong linear relationship between the RMSEA2 and the Standardized Root Mean Squared Residual goodness-of-fit index. We are unable to offer cutoff criteria for the RMSEAn as its population values decrease as the number of variables and categories increase.