Ratna Nandakumar

Refinements of Stout’s Procedure for Assessing Latent Trait Unidimensionality

This article provides a detailed investigation of Stout’s statistical procedure (the computer program DIMTEST) for testing the hypothesis that an essentially unidimensional latent trait model fits observed binary item response data from a psychological test. One finding was that DIMTEST may fail to perform as desired in the presence of guessing when coupled with many high-discriminating items. A revision of DIMTEST is proposed to overcome this limitation. Also, an automatic approach is devised to determine the size of the assessment subtests. Further, an adjustment is made on the estimated standard error of the statistic on which DIMTEST depends. These three refinements have led to an improved procedure that is shown in simulation studies to adhere closely to the nominal level of signficance while achieving considerably greater power. Finally, DIMTEST is validated on a selection of real data sets.

A handbook of introductory statistical methods

Definitions Sample Mean and Variance: Coefficient of Variation Sample and Population Relative Frequency Distributions Probability (Continuous Variates) Population Mean and Variance: Sample Estimates The Simple Linear Transformation of a Variate The Addition and Subtraction of Independent Variates The Distribution of the Sample Mean Statistical Models Normal (Gaussian) Distributions The Standard Normal (Gaussian) Reference Distribution T-Distributions Confidence Intervals for the Population Mean of a Normal Distribution and a General Formulation Chi-Squared (X2) Reference Distributions Confidence Intervals for the Population Variance of a Normal Distribution Hypothesis Testing (Significance Testing) Hypothesis Testing of the Mean of a Normal Distribution: The Two-Tailed T-Test Hypothesis Testing for the Variance of a Normal Distribution: The Two Tailed X2 Test Confidence Intervals and Hypothesis Tests Related Testing One-Sided Alternative Hypotheses Type I and Type II Errors and Power.

Assessing Unidimensionality of Polytomous Data

This study investigated the performance of Poly-DIMTEST (PD) to assess unidimensionality of test data produced by polytomous items. Two types of polytomous data were considered: (1) tests in which all items had the same number of response categories, and (2) tests in which items had a mixed number of response categories. Test length, sample size, and the type of correlation matrix (used in factor analysis for selecting ATI subset items) were varied in Type I error analyses. For the power study, the correlation between Os and the item-0 loadings were also varied. The results showed that PD was able to confirm unidimensionality for unidimensional simulated test data, with the average observed level of significance slightly below the nominal level. PD was also highly effective in detecting lack of unidimensionality in various two-dimensional tests. As expected, power increased as the sample size and test length increased, and the correlation between the Os decreased. The results also demonstrated that use of Pearson correlations to select ATI items led to equally good or better performance than using polychoric correlations; therefore Pearson correlations are recommended for future use.

Assessing Essential Unidimensionality of Real Data.

The capability of DIMTEST in assessing essential unidimensionality of item responses to real tests was investigated. DIMTEST found that some test data fit an essentially unidimensional model and other data did not. Essentially unidimensional test data identified by DIMTEST then were combined to form two-dimensional test data. The power of Stouts statistic T was examined for these two- dimensional data. DIMTEST results on real tests replicated findings from simulated tests—T dis criminated well between essentially unidimensional and multidimensional tests. T was also highly sen sitive to major traits and insensitive to relatively minor traits that influenced item responses.

MULTISIB: A Procedure to Investigate DIF When a Test is Intentionally Two-Dimensional:

MULTISIB is proposed as a statistical test for assessing differential item functioning (DIF) of intentionally two-dimensional test data, such as a mathematics test designed to measure algebra and geometry. MULTISIB is based on the multidimensional model of DIF as presented in Shealy & Stout (1993), and is a direct extension of SIBTEST, its unidimensional counterpart. For an intentionally two-dimensional test, DIF is appropriately modeled to result from secondary dimensional influence from other than the two intended dimensions. Simulation studies were used to assess the performance of MULTISIB to detect DIF in intentionally two-dimensional tests. These results indicate that MULTISIB exhibited reasonably good adherence to the nominal level of significance and good power. Moreover, for each DIF model the average amount of DIF estimated over the 100 simulations of the model by MULTISIB was close to the true value, confirming its relative lack of statistical estimation bias in assessing true DIF. In addition, the simulation studies supported the importance of using the regression correction to adjust the scores on the studied item due to impact and the importance of matching examinees on two subtest scores instead of the total test score.

DIMTEST: A Fortran Program for Assessing Dimensionality of Binary Item Responses

DIMTEST is a statistical test developed by Stout (1987), and refined by Nandakumar & Stout (in press; see also Nandakumar, in press). DIMTEST tests the hypothesis that the model underlying a matrix of binary item responses, generated by administering a test to a specific examinee population, is essentially unidimensional (essential dimensionality is a mathematical formulation of the existence of one dominant latent dimension).

Evaluation of the CATSIB DIF Procedure in a Pretest Setting

A new procedure, CATSIB, for assessing differential item functioning (DIF) on computerized adaptive tests (CATs) is proposed. CATSIB, a modified SIBTEST procedure, matches test takers on estimated ability and controls for impact-induced Type 1 error inflation by employing a CAT version of the SIBTEST “regression correction.” The performance of CATSIB in terms of detection of DIF in pretest items was evaluated in a simulation study. Simulated test takers were adaptively administered 25 operational items from a pool of 1,000 and were linearly administered 16 pretest items that were evaluated for DIF. Sample size varied from 250 to 500 in each group. Simulated impact levels ranged from a 0- to 1-standard-deviation difference in mean ability levels. The results showed that CATSIB with the regression correction displayed good control over Type 1 error, whereas CATSIB without the regression correction displayed impact-induced Type 1 error inflation. With 500 test takers in each group, power rates were exceptionally high (84% to 99%) for values of DIF at the boundary between moderate and large DIF. For smaller samples of 250 test takers in each group, the corresponding power rates ranged from 47% to 95%. In addition, in all cases, CATSIB was very accurate in estimating the true values of DIF, displaying at most only minor estimation bias.

A Comparison of Difficulty Levels of Vocabulary in First Grade Basal Readers for Preschool Dual Language Learners and Monolingual English Learners

The present study investigated preschoolers’ knowledge of vocabulary that appears in first grade basal readers by applying Rasch modeling to data from a researcher-developed receptive picture vocabulary assessment administered to 238 children. Levels of word difficulty for dual language learners (DLLs) and monolingual English learners (MELs) were compared. A total of 60 target words were selected from the glossaries of basal readers, and two test forms of 30 words each were created with four illustrations per word plate. Rasch analyses carried out on the entire preschool sample and on separate samples of DLLs and MELs showed that the ranking of target words by difficulty was similar for DLLs and MELs, but the groups differed in mastery of target words. Language status and level of general vocabulary knowledge were stronger predictors of word difficulty than age for this sample of preschoolers. Findings suggest MELs and DLLs learn English words in a similar order, but further research is needed to support this conclusion.

Differential item functioning in the Patient Reported Outcomes Measurement Information System Pediatric Short Forms in a sample of children and adolescents with cerebral palsy

The present study examined the Patient Reported Outcomes Measurement Information System (PROMIS) Mobility, Fatigue, and Pain Interference Short Forms (SFs) in children and adolescents with cerebral palsy (CP) for the presence of differential item functioning (DIF) relative to the original calibration sample.

Bayesian Prior Choice in IRT Estimation Using MCMC and Variational Bayes

This study investigated the impact of three prior distributions: matched, standard vague, and hierarchical in Bayesian estimation parameter recovery in two and one parameter models. Two Bayesian estimation methods were utilized: Markov chain Monte Carlo (MCMC) and the relatively new, Variational Bayesian (VB). Conditional (CML) and Marginal Maximum Likelihood (MML) estimates were used as baseline methods for comparison. Vague priors produced large errors or convergence issues and are not recommended. For both MCMC and VB, the hierarchical and matched priors showed the lowest root mean squared errors (RMSEs) for ability estimates; RMSEs of difficulty estimates were similar across estimation methods. For the standard errors (SEs), MCMC-hierarchical displayed the largest values across most conditions. SEs from the VB estimation were among the lowest in all but one case. Overall, VB-hierarchical, VB-matched, and MCMC-matched performed best. VB with hierarchical priors are recommended in terms of their accuracy, and cost and (subsequently) time effectiveness.