Dorothy T. Thayer

EM algorithms for ML factor analysis

The details of EM algorithms for maximum likelihood factor analysis are presented for both the exploratory and confirmatory models. The algorithm is essentially the same for both cases and involves only simple least squares regression operations; the largest matrix inversion required is for aq ×q symmetric matrix whereq is the matrix of factors. The example that is used demonstrates that the likelihood for the factor analysis model may have multiple modes that are not simply rotations of each other; such behavior should concern users of maximum likelihood factor analysis and certainly should cast doubt on the general utility of second derivatives of the log likelihood as measures of precision of estimation.

The Kernel method of test equating

and Notation.- and Notation.- The Kernel Method of Test Equating: Theory.- Data Collection Designs.- Kernel Equating: Overview, Pre-smoothing, and Estimation of r and s.- Kernel Equating: Continuization and Equating.- Kernel Equating: The SEE and the SEED.- Kernel Equating versus Other Equating Methods.- The Kernel Method of Test Equating: Applications.- The Equivalent-Groups Design.- The Single-Group Design.- The Counterbalanced Design.- The NEAT Design: Chain Equating.- The NEAT Design: Post-Stratification Equating.

Univariate and Bivariate Loglinear Models for Discrete Test Score Distributions

The well-developed theory of exponential families of distributions is applied to the problem of fitting the univariate histograms and discrete bivariate frequency distributions that often arise in the analysis of test scores. These models are powerful tools for many forms of parametric data smoothing and are particularly well-suited to problems in which there is little or no theory to guide a choice of probability models, e.g., smoothing a distribution to eliminate roughness and zero frequencies in order to equate scores from different tests. Attention is given to efficient computation of the maximum likelihood estimates of the parameters using Newtons Method and to computationally efficient methods for obtaining the asymptotic standard errors of the fitted frequencies and proportions. We discuss tools that can be used to diagnose the quality of the fitted frequencies for both the univariate and the bivariate cases. Five examples, using real data, are used to illustrate the methods of this paper.

Evaluating the Magnitude of Differential Item Functioning in Polytomous Items

Several recent studies have investigated the application of statistical inference procedures to the analysis of differential item functioning (DIF) in polytomous test items that are scored on an ordinal scale. Mantel’s extension of the Mantel-Haenszel test is one of several hypothesis-testing methods for this purpose. The development of descriptive statistics for characterizing DIF in polytomous test items has received less attention. As a step in this direction, two possible standard error formulas for the polytomous DIF index proposed by Dorans and Schmitt were derived. These standard errors, as well as associated hypothesis-testing procedures, were evaluated though application to simulated data. The standard error that performed better is based on Mantel’s hypergeometric model. The alternative standard error, based on a multinomial model, tended to yield values that were too small.

A Simulation Study of Methods for Assessing Differential Item Functioning in Computerized Adaptive Tests.

Simulated data were used to investigate the performance of modified versions of the Mantel-Haenszel method of differential item functioning (DIF) analysis in computerized adaptive tests (CATs). Each simulated examinee received 25 items from a 75-item pool. A three-parameter logistic item response theory (IRT) model was assumed, and examinees were matched on expected true scores based on their CAT responses and estimated item parameters. The CAT-based DIF statistics were found to be highly correlated with DIF statistics based on nonadaptive administration of all 75 pool items and with the true magnitudes of DIF in the simulation. Average DIF statistics and average standard errors also were examined for items with various characteristics. Finally, a study was conducted of the accuracy with which the modified Mantel-Haenszel procedure could identify CAT items with substantial DIF using a classification system now implemented by some testing programs. These additional analyses provided further evidence that the CAT-based DIF procedures performed well. More generally, the results supported the use of IRT-based matching variables in DIF analysis. Index terms: adaptive testing, computerized adaptive testing, differential item functioning, item bias, item response theory.

Application of an Empirical Bayes Enhancement of Mantel-Haenszel Differential Item Functioning Analysis to a Computerized Adaptive Test.

This study used a simulation to investigate the applicability to computerized adaptive test data of a differential item functioning (DIF) analysis method developed by Zwick, Thayer, and Lewis. The approach involves an empirical Bayes (EB) enhancement of the popular Mantel-Haenszel (MH) DIF analysis method. Results showed the performance of the EB DIF approach to be quite promising, even in extremely small samples. In particular, the EB procedure was found to achieve roughly the same degree of stability for samples averaging 117 and 40 members in the two examinee groups as did the ordinary MH for samples averaging 240 in each of the two groups. Overall, the EB estimates tended to be closer to their target values than did the ordinary MH statistics in terms of root mean square residuals; the EB statistics were also more highly correlated with the target values than were the MH statistics. When combined with a loss-function-based decision rule, the EB method is better at detecting DIF than conventional approaches, but it has a higher Type I error rate.

More on EM for ML Factor Analysis.

We address several issues that are raised by Bentler and Tanakas [1983] discussion of Rubin and Thayer [1982]. Our conclusions are: standard methods do not completely monitor the possible existence of multiple local maxima; summarizing inferential precision by the standard output based on second derivatives of the log likelihood at a maximum can be inappropriate, even if there exists a unique local maximum; EM and LISREL can be viewed as complementary, albeit not entirely adequate, tools for factor analysis.

A SIMULATION STUDY OF METHODS FOR ASSESSING DIFFERENTIAL ITEM FUNCTIONING IN COMPUTER‐ADAPTIVE TESTS

Simulated data were used to investigate the performance of modified versions of the Mantel-Haenszel and standardization methods of differential item functioning (DIF) analysis in computer-adaptive tests (CATs). Each “examinee” received 25 items out of a 75-item pool. A three-parameter logistic item response model was assumed, and examinees were matched on expected true scores based on their CAT responses and on estimated item parameters. Both DIF methods performed well. The CAT-based DIF statistics were highly correlated with DIF statistics based on nonadaptive administration of all 75 pool items and with the true magnitudes of DIF in the simulation. DIF methods were also investigated for “pretest items,” for which item parameter estimates were assumed to be unavailable. The pretest DIF statistics were generally well-behaved and also had high correlations with the true DIF. The pretest DIF measures, however, tended to be slightly smaller in magnitude than their CAT-based counterparts. Also, in the case of the Mantel-Haenszel approach, the pretest DIF statistics tended to have somewhat larger standard errors than the CAT DIF statistics.

Section Pre-Equating in the Presence of Practice Effects

Section pre-equating (SPE) is a new method for equating tests that was developed in response to test disclosure legislation. It is designed to equate a new test to an old test prior to the actual use of the new test, and makes extensive use of experimental sections of a testing instrument. This paper extends the theory behind SPE to allow for the effects of practice on both the old and the new tests, and gives a unified and generalized account of SPE.

Undesirable optimality results in multiple testing

A number of authors have considered the problem of making multiple comparisons among level-one parameters in multilevel models. This is a setting in which Bayesian procedures have a natural sampling theory interpretation, and where a natural justification for methods that control a directional version of the false discovery rate may be found. However, a basic desirable characteristic of multiple comparison procedures, namely that they should be more conservative than corresponding ‘per-comparison’ procedures, appears to be violated by some optimal procedures that have been developed in a multilevel setting. This concern is illustrated in the context of a very simple multilevel model, namely one-way, random-effects analysis of variance.