Victoria Savalei

When Can Categorical Variables Be Treated as Continuous? A Comparison of Robust Continuous and Categorical SEM Estimation Methods under Suboptimal Conditions.

A simulation study compared the performance of robust normal theory maximum likelihood (ML) and robust categorical least squares (cat-LS) methodology for estimating confirmatory factor analysis models with ordinal variables. Data were generated from 2 models with 2-7 categories, 4 sample sizes, 2 latent distributions, and 5 patterns of category thresholds. Results revealed that factor loadings and robust standard errors were generally most accurately estimated using cat-LS, especially with fewer than 5 categories; however, factor correlations and model fit were assessed equally well with ML. Cat-LS was found to be more sensitive to sample size and to violations of the assumption of normality of the underlying continuous variables. Normal theory ML was found to be more sensitive to asymmetric category thresholds and was especially biased when estimating large factor loadings. Accordingly, we recommend cat-LS for data sets containing variables with fewer than 5 categories and ML when there are 5 or more categories, sample size is small, and category thresholds are approximately symmetric. With 6-7 categories, results were similar across methods for many conditions; in these cases, either method is acceptable.

Redefine Statistical Significance

We propose to change the default P-value threshold for statistical significance from 0.05 to 0.005 for claims of new discoveries.

Assessing Mediational Models: Testing and Interval Estimation for Indirect Effects

Theoretical models specifying indirect or mediated effects are common in the social sciences. An indirect effect exists when an independent variables influence on the dependent variable is mediated through an intervening variable. Classic approaches to assessing such mediational hypotheses (Baron & Kenny, 1986; Sobel, 1982) have in recent years been supplemented by computationally intensive methods such as bootstrapping, the distribution of the product methods, and hierarchical Bayesian Markov chain Monte Carlo (MCMC) methods. These different approaches for assessing mediation are illustrated using data from Dunn, Biesanz, Human, and Finn (2007). However, little is known about how these methods perform relative to each other, particularly in more challenging situations, such as with data that are incomplete and/or nonnormal. This article presents an extensive Monte Carlo simulation evaluating a host of approaches for assessing mediation. We examine Type I error rates, power, and coverage. We study normal and nonnormal data as well as complete and incomplete data. In addition, we adapt a method, recently proposed in statistical literature, that does not rely on confidence intervals (CIs) to test the null hypothesis of no indirect effect. The results suggest that the new inferential method—the partial posterior p value—slightly outperforms existing ones in terms of maintaining Type I error rates while maximizing power, especially with incomplete data. Among confidence interval approaches, the bias-corrected accelerated (BC a ) bootstrapping approach often has inflated Type I error rates and inconsistent coverage and is not recommended; In contrast, the bootstrapped percentile confidence interval and the hierarchical Bayesian MCMC method perform best overall, maintaining Type I error rates, exhibiting reasonable power, and producing stable and accurate coverage rates.

Small Sample Statistics for Incomplete Nonnormal Data: Extensions of Complete Data Formulae and a Monte Carlo Comparison.

Incomplete nonnormal data are common occurrences in applied research. Although these 2 problems are often dealt with separately by methodologists, they often cooccur. Very little has been written about statistics appropriate for evaluating models with such data. This article extends several existing statistics for complete nonnormal data to incomplete data and evaluates their performance via a Monte Carlo study. The focus is on statistics that also perform well in small samples. The following statistics are defined and studied: corrected residual-based statistic, residual-based F statistic, scaled chi-square, adjusted chi-square, Bartlett-corrected scaled chi-square, and Swain-corrected scaled chi-square. Both Type I error rates and power are studied with missing completely at random nonnnormally distributed data and varying degrees of nonnormality. Sample size, model size, and number of variables containing missingness are also varied. For power comparisons, both minor and major model misspecifications are considered. Two statistics had the best Type I error control and power: the adjusted chi-square and Bartlett-corrected chi-square. These statistics are recommended to practitioners. It is concluded that model fit can be assessed reliably and with sufficient power even at the intersection of all 3 problems: incomplete data, nonnormality, and small sample size.

Constrained versus unconstrained estimation in structural equation modeling.

Recently, R. D. Stoel, F. G. Garre, C. Dolan, and G. van den Wittenboer (2006) reviewed approaches for obtaining reference mixture distributions for difference tests when a parameter is on the boundary. The authors of the present study argue that this methodology is incomplete without a discussion of when the mixtures are needed and show that they only become relevant when constrained difference tests are conducted. Because constrained difference tests can hide important model misspecification, a reliable way to assess global model fit under constrained estimation would be needed. Examination of the options for assessing model fit under constrained estimation reveals that no perfect solutions exist, although the conditional approach of releasing a degree of freedom for each active constraint appears to be the most methodologically sound one. The authors discuss pros and cons of constrained and unconstrained estimation and their implementation in 5 popular structural equation modeling packages and argue that unconstrained estimation is a simpler method that is also more informative about sources of misfit. In practice, researchers will have trouble conducting constrained difference tests appropriately, as this requires a commitment to ignore Heywood cases. Consequently, mixture distributions for difference tests are rarely appropriate.

A Two-Stage Approach to Missing Data: Theory and Application to Auxiliary Variables

A well-known ad-hoc approach to conducting structural equation modeling with missing data is to obtain a saturated maximum likelihood (ML) estimate of the population covariance matrix and then to use this estimate in the complete data ML fitting function to obtain parameter estimates. This 2-stage (TS) approach is appealing because it minimizes a familiar function while being only marginally less efficient than the full information ML (FIML) approach. Additional advantages of the TS approach include that it allows for easy incorporation of auxiliary variables and that it is more stable in smaller samples. The main disadvantage is that the standard errors and test statistics provided by the complete data routine will not be correct. Empirical approaches to finding the right corrections for the TS approach have failed to provide unequivocal solutions. In this article, correct standard errors and test statistics for the TS approach with missing completely at random and missing at random normally distributed data are developed and studied. The new TS approach performs well in all conditions, is only marginally less efficient than the FIML approach (and is sometimes more efficient), and has good coverage. Additionally, the residual-based TS statistic outperforms the FIML test statistic in smaller samples. The TS method is thus a viable alternative to FIML, especially in small samples, and its further study is encouraged.

A Statistically Justified Pairwise ML Method for Incomplete Nonnormal Data: A Comparison With Direct ML and Pairwise ADF

This article proposes a new approach to the statistical analysis of pairwise-present covariance structure data. The estimator is based on maximizing the complete data likelihood function, and the associated test statistic and standard errors are corrected for misspecification using Satorra-Bentler corrections. A Monte Carlo study was conducted to compare the proposed method (pairwise maximum likelihood [ML]) to 2 other methods for dealing with incomplete nonnormal data: direct ML estimation with the Yuan-Bentler corrections for nonnormality (direct ML) and the asymptotically distribution free (ADF) method applied to available cases (pairwise ADF). Data were generated from a 4-factor model with 4 indicators per factor; sample size varied from 200 to 5,000; data were either missing completely at random (MCAR) or missing at random (MAR); and the proportion of missingness was either 15% or 30%. Measures of relative performance included model fit, relative accuracy in parameter estimates and their standard errors, and efficiency of parameter estimates. The results generally favored direct ML over either of the pairwise methods, except at N = 5,000, when ADF outperformed both ML methods with MAR data. The inferior performance of the 2 pairwise methods was primarily due to inflated test statistics. Among the unexpected findings was that ADF did better at estimating factor covariances in all conditions, and that MCAR data presented more problems for all methods than did MAR data, in terms of convergence, performance of test statistics, and relative accuracy of parameter estimates.

Is the ML Chi-Square Ever Robust to Nonnormality? A Cautionary Note With Missing Data

Normal theory maximum likelihood (ML) is by far the most popular estimation and testing method used in structural equation modeling (SEM), and it is the default in most SEM programs. Even though this approach assumes multivariate normality of the data, its use can be justified on the grounds that it is fairly robust to the violations of the distributional assumptions under some conditions. In support of this claim, a large literature exists outlining conditions under which the ML chi-square retains its asymptotic distribution even under nonnormality. The most important of these conditions is specifying a model in which the exogenous variables presumed to be uncorrelated (e.g., factors and errors in a confirmatory factor analysis model) are also statistically independent. The goal of this article is to point out that these conditions were developed for complete data, and in fact no longer ensure robustness when the data are both nonnormal and incomplete. This lack of robustness is illustrated both mathematically and empirically. Violation becomes more severe when the data are highly nonnormal and when a higher proportion of data is missing. It is concluded that if the proportion of missing data is greater than about 10%, robustness of the ML chi-square with incomplete nonnormal data cannot be counted on, even if the necessary assumptions such as independence are made. Other approaches to model testing are to be preferred in this case.

The performance of robust test statistics with categorical data

This paper reports on a simulation study that evaluated the performance of five structural equation model test statistics appropriate for categorical data. Both Type I error rate and power were investigated. Different model sizes, sample sizes, numbers of categories, and threshold distributions were considered. Statistics associated with both the diagonally weighted least squares (cat-DWLS) estimator and with the unweighted least squares (cat-ULS) estimator were studied. Recent research suggests that cat-ULS parameter estimates and robust standard errors slightly outperform cat-DWLS estimates and robust standard errors (Forero, Maydeu-Olivares, & Gallardo-Pujol, 2009). The findings of the present research suggest that the mean- and variance-adjusted test statistic associated with the cat-ULS estimator performs best overall. A new version of this statistic now exists that does not require a degrees-of-freedom adjustment (Asparouhov & Muthén, 2010), and this statistic is recommended. Overall, the cat-ULS estimator is recommended over cat-DWLS, particularly in small to medium sample sizes.

Understanding Robust Corrections in Structural Equation Modeling

Robust corrections to standard errors and test statistics have wide applications in structural equation modeling (SEM). The original SEM development, due to Satorra and Bentler (1988, 1994), was to account for the effect of nonnormality. Muthén (1993) proposed corrections to accompany certain categorical data estimators, such as cat-LS or cat-DWLS. Other applications of robust corrections exist. Despite the diversity of applications, all robust corrections are constructed using the same underlying rationale: They correct for inefficiency of the chosen estimator. The goal of this article is to make the formulas behind all types of robust corrections more intuitive. This is accomplished by building an analogy with similar equations in linear regression and then by reformulating the SEM model as a nonlinear regression model.