Kyung T. Han

IRTPRO 2.1 for Windows (Item Response Theory for Patient-Reported Outcomes).

This article reviews a new item response theory (IRT) model estimation program, IRTPRO 2.1, for Windows that is capable of unidimensional and multidimensional IRT model estimation for existing and user-specified constrained IRT models for dichotomously and polytomously scored item response data.

SimulCAT: Windows Software for Simulating Computerized Adaptive Test Administration.

Most, if not all, computerized adaptive testing (CAT) programs use simulation techniques to develop and evaluate CAT program administration and operations, but such simulation tools are rarely available to the public. Up to now, several software tools have been available to conduct CAT simulations for research purposes; however, these existing tools, for the most part, oversimplify the CAT algorithms and are not powerful enough to simulate operational CAT environments. SimulCAT, a new CAT simulation software tool, was developed to serve various purposes ranging from fundamental CAT research to technical CAT program evaluations. The new CAT simulation software tool offers many advantages, including a wide range of itemselection algorithms, adaptability for a variety of CAT administration environments, and a userfriendly graphical interface, among others, as described in the following commentary.

A Review of Commercial Software Packages for Multidimensional IRT Modeling

In this study, the authors evaluate several commercially available multidimensional item response theory (MIRT) software packages, including IRTPRO 2.1, Mplus 7.1, FlexMIRT, and EQSIRT, as well as their built-in estimation algorithms, and compare them for their performance in MIRT model estimation. The study examines the performance of model parameter recovery via a series of simulations based on four approaches for latent structuring—within-item MIRT, between-item MIRT, a mixture of within- and between-item MIRT and a bifactor model. The simulation studies focused on realistic conditions and models that researchers and practitioners are likely to encounter in practice. The results showed that the studied software packages recovered the item parameters reasonably well but differed greatly in terms of the types of data and models they could handle and also the run time required for estimation completion.

Item Pocket Method to Allow Response Review and Change in Computerized Adaptive Testing

Most computerized adaptive testing (CAT) programs do not allow test takers to review and change their responses because it could seriously deteriorate the efficiency of measurement and make tests vulnerable to manipulative test-taking strategies. Several modified testing methods have been developed that provide restricted review options while limiting the trade-off in CAT efficiency. The extent to which these methods provided test takers with options to review test items, however, still was quite limited. This study proposes the item pocket (IP) method, a new testing approach that allows test takers greater flexibility in changing their responses by eliminating restrictions that prevent them from moving across test sections to review their answers. A series of simulations were conducted to evaluate the robustness of the IP method against various manipulative test-taking strategies. Findings and implications of the study suggest that the IP method may be an effective solution for many CAT programs when the IP size and test time limit are properly set.

Maximum Likelihood Score Estimation Method With Fences for Short-Length Tests and Computerized Adaptive Tests

A critical shortcoming of the maximum likelihood estimation (MLE) method for test score estimation is that it does not work with certain response patterns, including ones consisting only of all 0s or all 1s. This can be problematic in the early stages of computerized adaptive testing (CAT) administration and for tests short in length. To overcome this challenge, test practitioners often set lower and upper bounds of theta estimation and truncate the score estimation to be one of those bounds when the log likelihood function fails to yield a peak due to responses consisting only of 0s or 1s. Even so, this MLE with truncation (MLET) method still cannot handle response patterns in which all harder items are correct and all easy items are incorrect. Bayesian-based estimation methods such as the modal a posteriori (MAP) method or the expected a posteriori (EAP) method can be viable alternatives to MLE. The MAP or EAP methods, however, are known to result in estimates biased toward the center of a prior distribution, resulting in a shrunken score scale. This study introduces an alternative approach to MLE, called MLE with fences (MLEF). In MLEF, several imaginary “fence” items with fixed responses are introduced to form a workable log likelihood function even with abnormal response patterns. The findings of this study suggest that, unlike MLET, the MLEF can handle any response patterns and, unlike both MAP and EAP, results in score estimates that do not cause shrinkage of the theta scale.

A Review of DIMPACK Version 1.0 Conditional Covariance–Based Test Dimensionality Analysis Package

DIMPACK Version 1.0 for assessing test dimensionality based on a nonparametric conditional covariance approach is reviewed. This software was originally distributed by Assessment Systems Corporation and now can be freely accessed online. The software consists of Windows-based interfaces of three components: DIMTEST, DETECT, and CCPROX/HAC, which conduct hypothesis test for unidimensionality, cluster items, and perform hierarchical cluster analysis, respectively. Two simulation studies were conducted to evaluate the software in confirming test unidimensionality (a Type I error study) and detecting multidimensionality (a statistical power study). The results suggested that different data always be used in selecting assessment subtest items independent of calculating the DIMTEST statistic. The Type I error rate was excessively inflated otherwise. The statistical power was found low when sample size was small or the dimensions were highly correlated. It is suggested that some major changes be made to the software before it can be successfully useful among practitioners.

Examining Test Speededness by Native Language

When power tests include a time limit, it is important to assess the possibility of speededness for examinees. Past research on differential speededness has examined gender and ethnic subgroups in the United States on paper and pencil tests. When considering the needs of a global audience, research regarding different native language speakers is warranted. Data from a computerized adaptive test administered globally in English were used to compare different measures of time pressure and speed for 15 language groups compared to native-speaking English examinees. Results indicated that examinees whose native language was Korean had issues with speededness when many of the metrics were considered, but when test score was controlled, there were no relevant differences for any of the languages compared to English.