Myeongsun Yoon

Detecting Violations of Factorial Invariance Using Data-Based Specification Searches: A Monte Carlo Study.

In testing factorial invariance, researchers have often used a reference variable strategy in which the factor loading for a variable (i.e., reference variable) is fixed to 1 for identification. This commonly used method can be misleading if the chosen reference variable is actually a noninvariant item. This simulation study suggests an alternative method for testing factorial invariance and evaluates the performance of the method in specification searches based on the modification index. The results of the study showed that the proposed specification searches performed well when the number of noninvariant variables was relatively small and this performance improved as sample size increased and the size of group differences increased. When the number of noninvariant variables was relatively large, however, the method rarely succeeded in detecting the noninvariant items in the specification searches. Implications of the findings are discussed along with the limitations of the study.

Testing Measurement Invariance: A Comparison of Multiple-Group Categorical CFA and IRT

This study investigated two major approaches in testing measurement invariance for ordinal measures: multiple-group categorical confirmatory factor analysis (MCCFA) and item response theory (IRT). Unlike the ordinary linear factor analysis, MCCFA can appropriately model the ordered-categorical measures with a threshold structure. A simulation study under various conditions was conducted for the comparison of MCCFA and IRT with respect to the power to detect the lack of invariance across groups. Both MCCFA and IRT showed reasonable power to identify the noninvariant item when differential item functioning (DIF) was large. The false positive rates were relatively high in both methods, however. The adjustment of critical values improved the performance of MCCFA by reducing false positive rates substantially and yet yielding adequate power. Alternative model fit indexes of MCCFA were also examined and they were found to be reliable to detect DIF, in general.

Testing Measurement Invariance Using MIMIC Likelihood Ratio Test With a Critical Value Adjustment

Multiple-indicators multiple-causes (MIMIC) modeling is often used to test a latent group mean difference while assuming the equivalence of factor loadings and intercepts over groups. However, this study demonstrated that MIMIC was insensitive to the presence of factor loading noninvariance, which implies that factor loading invariance should be tested through other measurement invariance testing techniques. MIMIC modeling is also used for measurement invariance testing by allowing a direct path from a grouping covariate to each observed variable. This simulation study with both continuous and categorical variables investigated the performance of MIMIC in detecting noninvariant variables under various study conditions and showed that the likelihood ratio test of MIMIC with Oort adjustment not only controlled Type I error rates below the nominal level but also maintained high power across study conditions.

Testing Factorial Invariance in Multilevel Data: A Monte Carlo Study

Testing factorial invariance has recently gained more attention in different social science disciplines. Nevertheless, when examining factorial invariance, it is generally assumed that the observations are independent of each other, which might not be always true. In this study, we examined the impact of testing factorial invariance in multilevel data, especially when the dependency issue is not taken into account. We considered a set of design factors, including number of clusters, cluster size, and intraclass correlation (ICC) at different levels. The simulation results showed that the test of factorial invariance became more liberal (or had inflated Type I error rate) in terms of rejecting the null hypothesis of invariance held between groups when the dependency was not considered in the analysis. Additionally, the magnitude of the inflation in the Type I error rate was a function of both ICC and cluster size. Implications of the findings and limitations are discussed.

Covariances between regression coefficient estimates in a single mediator model

This study presents formulae for the covariances between parameter estimates in a single mediator model. These covariances are necessary to build confidence intervals (CI) for effect size measures in mediation studies. We first analytically derived the covariances between the parameter estimates in a single mediator model. Using the derived covariances, we computed the multivariate-delta standard errors, and built the 95% CIs for the effect size measures. A simulation study evaluated the accuracy of the standard errors as well as the Type I error, power, and coverage of the CIs using various parameter values and sample sizes. Finally, we presented a numerical example and a SAS MACRO that calculates the CIs for the effect size measures.

Within-Level Group Factorial Invariance With Multilevel Data: Multilevel Factor Mixture and Multilevel MIMIC Models

This study suggests 2 approaches to factorial invariance testing with multilevel data when the groups are at the within level: multilevel factor mixture model for known classes (ML FMM) and multilevel multiple indicators multiple causes model (ML MIMIC). The adequacy of the proposed approaches was investigated using Monte Carlo simulations. Additionally, the performance of different types of model selection criteria for determining factorial invariance or detecting item noninvariance was examined. Generally, both ML FMM and ML MIMIC demonstrated acceptable performance with high true positive and low false positive rates, but the performance depended on the fit statistics used for model selection under different simulation conditions.

Specification Search for Identifying the Correct Mean Trajectory in Polynomial Latent Growth Models.

This study investigated the optimal strategy for model specification search under the latent growth modeling (LGM) framework, specifically on searching for the correct polynomial mean or average growth model when there is no a priori hypothesized model in the absence of theory. In this simulation study, the effectiveness of different starting models on the search of the true mean growth model was investigated in terms of the mean and within-subject variance-covariance (V-C) structure model. The results showed that specifying the most complex (i.e., unstructured) within-subject V-C structure with the use of LRT, ΔAIC, and ΔBIC achieved the highest recovery rate (>85%) of the true mean trajectory. Implications of the findings and limitations are discussed.

Tracing Student Responsiveness to Intervention With Early Literacy Skills Indicators: Do They Reflect Growth Toward Text Reading Outcomes?

This study investigated four widely-used early literacy skills indicators in reflecting growth toward first-grade text reading skills. Examining the progress of 101 students across kindergarten and first grade, Letter Naming Fluency (LNF) and Nonsense Word Fluency (NWF) were more accurate than Initial Sounds Fluency and Phoneme Segmentation Fluency in discriminating between students grouped according to successful or unsuccessful first-grade reading outcomes. LNF and NWF slope also discriminated between groups, but graphed observed scores suggested potential problems in identifying students with persistently low achievement. Results suggest the need for continued refinement of early literacy skills measures for instructional decision-making.

Comparisons of Three Empirical Methods for Partial Factorial Invariance: Forward, Backward, and Factor-Ratio Tests

When factorial invariance is violated, a possible first step in locating the source of violation(s) might be to pursue partial factorial invariance (PFI). Two commonly used methods for PFI are sequential use of the modification index (backward MI method) and the factor-ratio test. In this study, we propose a simple forward method using the confidence interval (forward CI method). We compare the performances of the aforementioned 3 methods under various simulated PFI conditions. Results indicate that the forward CI method using 99% CIs has the highest perfect recovery rates and the lowest Type I error rates. A performance that is competitive with this is that produced by the backward method with the more conservative criterion (MI = 6.635). Consistently delivering the poorest performance, regardless of the chosen confidence level, was the factor-ratio test. Also discussed are the work’s contribution, implications, and limitations.

Screening Assessment Within a Multi-Tiered System of Support: Current Practices, Advances, and Next Steps

Within multi-tiered systems of support (MTSS), screening assessments play an important role in identifying students who are in need of supplemental support strategies. In this chapter, the authors review the tools and methods commonly used in MTSS for academic skills screening, identify limitations with these practices, and highlight potential areas of improvement regarding assessment methods and content of screening tools, decision-making processes used to identify students in need of support, and methods used for evaluating screening tools. A set of recommendations and directions for future work are offered for advancing screening assessment and improving decision-making processes in schools with MTSS.