Dubravka Svetina

Evaluation of Parallel Analysis Methods for Determining the Number of Factors.

Population and sample simulation approaches were used to compare the performance of parallel analysis using principal component analysis (PA-PCA) and parallel analysis using principal axis factoring (PA-PAF) to identify the number of underlying factors. Additionally, the accuracies of the mean eigenvalue and the 95th percentile eigenvalue criteria were examined. The 95th percentile criterion was preferable for assessing the first eigenvalue using either extraction method. In assessing subsequent eigenvalues, PA-PCA tended to perform as well as or better than PA-PAF for models with one factor or multiple minimally correlated factors; the relative performance of the mean eigenvalue and the 95th percentile eigenvalue criteria depended on the number of variables per factor. PA-PAF using the mean eigenvalue criterion generally performed best if factors were more than minimally correlated or if one or more strong general factors as well as group factors were present.

Assessing the Hypothesis of Measurement Invariance in the Context of Large-Scale International Surveys.

In the field of international educational surveys, equivalence of achievement scale scores across countries has received substantial attention in the academic literature; however, only a relatively recent emphasis on scale score equivalence in nonachievement education surveys has emerged. Given the current state of research in multiple-group models, findings regarding these recent measurement invariance investigations were supported with research that was limited in scope to few groups and relatively small sample sizes. To that end, this study uses data from one large-scale survey as a basis for examining the extent to which typical fit measures used in multiple-group confirmatory factor analysis are suitable for detecting measurement invariance in a large-scale survey context. Using measures validated in a smaller scale context and an empirically grounded simulation study, our findings indicate that many typical measures and associated criteria are either unsuitable in a large group and varied sample-size context or should be adjusted, particularly when the number of groups is large. We provide specific recommendations and discuss further areas for research.

A generalized dimensionality discrepancy measure for dimensionality assessment in multidimensional item response theory.

A generalized dimensionality discrepancy measure is introduced to facilitate a critique of dimensionality assumptions in multidimensional item response models. Connections between dimensionality and local independence motivate the development of the discrepancy measure from a conditional covariance theory perspective. A simulation study and a real-data analysis demonstrate the utility of the discrepancy measures application at multiple levels of analysis in a posterior predictive model checking framework.

Defining and Comparing the Reading Comprehension Construct: A Cognitive-Psychometric Modeling Approach.

As a construct definition, the current study develops a cognitive model describing the knowledge, skills, and abilities measured by critical reading test items on a high-stakes assessment used for selection decisions in the United States. Additionally, in order to establish generalizability of the construct meaning to other similarly structured tests designed for international populations and distinct uses, the skills invoked during a reading comprehension test for English learners from a previous study are contrasted to those in the present study. The results obtained using rule-space methodology suggest that the most difficult skills on reading comprehension items pertain to complex cognitive processes (e.g., understanding implicit ideas), while skills tapping into basic cognitive processes (e.g., word meaning) are mastered with ease by both populations. However, variations across tests in the impact of various cognitive skills on test scores suggest that the differences in construct meaning be considered when interpreting and comparing test scores. Cognitive-psychometric modeling approaches such as those applied in this study prove to be useful in substantively examining score interpretation and construct generalizability.

A Framework for Dimensionality Assessment for Multidimensional Item Response Models.

A framework is introduced for considering dimensionality assessment procedures for multidimensional item response models. The framework characterizes procedures in terms of their confirmatory or exploratory approach, parametric or nonparametric assumptions, and applicability to dichotomous, polytomous, and missing data. Popular and emerging approaches are situated within the proposed framework and illustrated via analyses of item response data from the National Assessment of Educational Progress Science Assessment.

An Overview of Software for Conducting Dimensionality Assessment in Multidimensional Models.

An overview of popular software packages for conducting dimensionality assessment in multidimensional models is presented. Specifically, five popular software packages are described in terms of their capabilities to conduct dimensionality assessment with respect to the nature of analysis (exploratory or confirmatory), types of data (dichotomous, ordered polytomous, continuous, missingness), technical details, and statistics used for dimensionality assessment. Following descriptions of existing software packages, several promising potentially broadly applicable approaches are described that have been proposed but are not yet implemented in widely available software.

Measurement Invariance in International Surveys: Categorical Indicators and Fit Measure Performance

ABSTRACT In spite of the challenges inherent in making dozens of comparisons across heterogeneous populations, a relatively recent interest in scale-score equivalence for non-achievement measures in an international context has emerged. Until recently, operational procedures for establishing measurement invariance using multiple-groups analyses were typically supported with research that was limited in scope to few groups and relatively small sample sizes. Recent research that examined situations more representative of international surveys recommended some revisions to typically used fit measures. The current study extends this research and evaluates the performance of several fit measures when data are assumed to have an ordered categorical, rather than the typically assumed continuous, scale. Using a simulation study based on empirical results, findings indicated that classic measures and associated criteria were either unsuitable in a large-group and varied sample-size context or should be adjusted, particularly when observed variables were not normally distributed. We provide specific recommendations for revising currently used criteria for evaluating overall and relative fit based on the chi-square test, root mean-squared error of approximation, and comparative fit index (CFI).

Assessing Dimensionality of Noncompensatory Multidimensional Item Response Theory With Complex Structures

The purpose of this study was to investigate the effect of complex structure on dimensionality assessment in noncompensatory multidimensional item response models using dimensionality assessment procedures based on DETECT (dimensionality evaluation to enumerate contributing traits) and NOHARM (normal ogive harmonic analysis robust method). Five methods were evaluated: two DETECT-based methods—exploratory and cross-validated—and three NOHARM-based methods: root mean square residual (RMSR), χ G / D 2 , and approximate likelihood ratio (ALR). The results suggested that the studied methods had varying degree of success in correctly counting the number of dimensions and in meaningfully labeling sets of items as dimension-like. In two-dimensional, shorter tests, χ G / D 2 and ALR largely outperformed RMSR- and DETECT-based methods in conditions with small and medium sample sizes, across all levels of complexity and correlations. Lengthening of the test in two-dimensional conditions led to most notably improved accuracy in determining the correct number of dimensions for NOHARM-based RMSR, whereas for the remaining methods, increase in the number of items had differential effect. DETECT-based methods, on the other hand, had more success in labeling sets of items as dimension-like in conditions with two dimensions, irrespective of the test length, suggesting the items that ought to be together were more often grouped together. Performance of the methods was further evaluated in conditions with larger dimensionality (i.e., 3) and conditions with the increased number of items (i.e., longer tests).

Parameter Recovery in Multidimensional Item Response Theory Models Under Complexity and Nonnormality

Information about the psychometric properties of items can be highly useful in assessment development, for example, in item response theory (IRT) applications and computerized adaptive testing. Although literature on parameter recovery in unidimensional IRT abounds, less is known about parameter recovery in multidimensional IRT (MIRT), notably when tests exhibit complex structures or when latent traits are nonnormal. The current simulation study focuses on investigation of the effects of complex item structures and the shape of examinees’ latent trait distributions on item parameter recovery in compensatory MIRT models for dichotomous items. Outcome variables included bias and root mean square error. Results indicated that when latent traits were skewed, item parameter recovery was generally adversely impacted. In addition, the presence of complexity contributed to decreases in the precision of parameter recovery, particularly for discrimination parameters along one dimension when at least one latent trait was generated as skewed.

Examining Similarities and Differences Among Parent–Teacher Reports of Spanish–English Productive Vocabulary

PURPOSE The purposes of the present study were to (a) explore the relationship between parent and teacher reports of childrens bilingual (Spanish-English) productive vocabulary and (b) examine similarities and differences among parent-teacher reports. Word categories were examined to determine the nature of similarities and differences. METHOD Parents and teachers of eleven Spanish-English bilinguals ( Mage = 44.5 months) completed the MacArthur-Bates Communicative Development Inventory upper extension and an experimental version of a congruent Spanish form at 2 time points. Percent agreement, kappa coefficient, and Spearmans rho were employed to estimate overall interrater agreement and agreement on specific word categories. RESULTS Results indicated inconsistent levels of overall agreement across measures and forms. Higher levels of parent-teacher agreement were observed on Spanish forms at either time point using Spearmans rho coefficient and kappa, whereas percent agreement was higher on English forms. Limited overlap of high agreement between parents and teachers was found on word categories across indices. Unique contributions of reporters were observed. CONCLUSION This work underscores the utility of multiple informants of bilingual childrens productive vocabulary. Combined and unique contributions of parent and teacher reporters may inform the language development of preschool-age bilingual children as productive vocabulary skills develop and change.