Ivo W. Molenaar

The many null distributions of person fit indices

This paper deals with the situation of an investigator who has collected the scores ofn persons to a set ofk dichotomous items, and wants to investigate whether the answers of all respondents are compatible with the one parameter logistic test model of Rasch. Contrary to the standard analysis of the Rasch model, where all persons are kept in the analysis and badly fittingitems may be removed, this paper studies the alternative model in which a small minority ofpersons has an answer strategy not described by the Rasch model. Such persons are called anomalous or aberrant. From the response vectors consisting ofk symbols each equal to 0 or 1, it is desired to classify each respondent as either anomalous or as conforming to the model. As this model is probabilistic, such a classification will possibly involve false positives and false negatives. Both for the Rasch model and for other item response models, the literature contains several proposals for a person fit index, which expresses for each individual the plausibility that his/her behavior follows the model. The present paper argues that such indices can only provide a satisfactory solution to the classification problem if their statistical distribution is known under the null hypothesis that all persons answer according to the model. This distribution, however, turns out to be rather different for different values of the persons latent trait value. This value will be called “ability parameter”, although our results are equally valid for Rasch scales measuring other attributes.As the true ability parameter is unknown, one can only use its estimate in order to obtain an estimated person fit value and an estimated null hypothesis distribution. The paper describes three specifications for the latter: assuming that the true ability equals its estimate, integrating across the ability distribution assumed for the population, and conditioning on the total score, which is in the Rasch model the sufficient statistic for the ability parameter.Classification rules for aberrance will be worked out for each of the three specifications. Depending on test length, item parameters and desired accuracy, they are based on the exact distribution, its Monte Carlo estimate and a new and promising approximation based on the moments of the person fit statistic. Results for the likelihood person fit statistic are given in detail, the methods could also be applied to other fit statistics. A comparison of the three specifications results in the recommendation to condition on the total score, as this avoids some problems of interpretation that affect the other two specifications.

Stochastic Ordering Using the Latent Trait and the Sum Score in Polytomous IRT Models.

In a restricted class of item response theory (IRT) models for polytomous items the unweighted total score has monotone likelihood ratio (MLR) in the latent traitϑ. MLR implies two stochastic ordering (SO) properties, denoted SOM and SOL, which are both weaker than MLR, but very useful for measurement with IRT models. Therefore, these SO properties are investigated for a broader class of IRT models for which the MLR property does not hold.In this study, first a taxonomy is given for nonparametric and parametric models for polytomous items based on the hierarchical relationship between the models. Next, it is investigated which models have the MLR property and which have the SO properties. It is shown that all models in the taxonomy possess the SOM property. However, counterexamples illustrate that many models do not, in general, possess the even more useful SOL property.

Selection of unidimensional scales from a multidimensional item bank in the polytomous Mokken IRT model

An automated item selection procedure for selecting unidimensional scales of polytomous items from multi dimensional datasets is developed for use in the context of the Mokken item response theory model of monotone homogeneity (Mokken & Lewis, 1982). The selection procedure is directly based on the selection procedure proposed by Mokken (1971, p. 187) and relies heavily on the scalability coefficient H (Loevinger, 1948; Molenaar, 1991). New theoretical results relating the latent model structure to H are provided. The item selec tion procedure requires selection of a lower bound for H. A simulation study determined ranges of H for which the unidimensional item sets were retrieved from multidimensional datasets. If multidimensionality is suspected in an empirical dataset, well-chosen lower bound values can be used effectively to detect the unidi mensional scales.

Some improved diagnostics for failure of the Rasch model

Although several goodness of fit tests have been developed for the Rasch model for dichotomous items, most of them are of a global, asymptotic, and confirmatory type. This paper, based on ideas from a recent thesis by Van den Wollenberg, offers some suggestions for local, small sample, and exploratory techniques: difficulty plots for person groups scoring right and wrong on a specific item, a slope test per item based on a binomial distribution per score group, and a unidimensionality check based on an extended hypergeometric distribution per score group.

Estimation of Item Parameters

The Introduction of this chapter sketches the problem of the estimation of item parameters and the notation in the ease of incomplete data. Then the joint, conditional, and marginal maximum likelihood methods are discussed. A final section briefly mentions a few other methods not based on likelihoods.

A multidimensional item response model: Constrained latent class analysis using the gibbs sampler and posterior predictive checks

In this paper it will be shown that a certain class of constrained latent class models may be interpreted as a special case of nonparametric multidimensional item response models. The parameters of this latent class model will be estimated using an application of the Gibbs sampler. It will be illustrated that the Gibbs sampler is an excellent tool if inequality constraints have to be taken into consideration when making inferences. Model fit will be investigated using posterior predictive checks. Checks for manifest monotonicity, the agreement between the observed and expected conditional association structure, marginal local homogeneity, and the number of latent classes will be presented.

Reliability of test scores in nonparametric item response theory

Three methods for estimating reliability are studied within the context of nonparametric item response theory. Two were proposed originally by Mokken (1971) and a third is developed in this paper. Using a Monte Carlo strategy, these three estimation methods are compared with four “classical” lower bounds to reliability. Finally, recommendations are given concerning the use of these estimation methods.

Mokken scale analysis for polychotomous items: theory, a computer program and an empirical application

This paper contains three subjects. First, an extension of Mokkens nonparametric item response models from dichotomous items to items with two or more ordered answer categories is proposed. Second, a computer program to analyze multicategory item scores is presented. This program is called MSP. The analyses by means of MSP are based on the multicategory extension of Mokkens theory. Finally, an application of MSP to empirical multicategory test data is presented in order to illuminate its possibilities.

Psychometric properties of the RAND-36 among three chronic disease (multiple sclerosis, rheumatic diseases and COPD) in the Netherlands

Objective: In this article, psychometric properties both of the total RAND-36 and of its subscales, such as unidimensionality, differential item functioning (DIF or item bias), homogeneity and reliabilities, are examined. Methods: The data from populations with three chronic illnesses, multiple sclerosis (n = 448), rheumatism (n = 336) and COPD (n = 259), have been collected in different parts of the Netherlands. The main technique used was Mokken scale analysis for polytomous items. Results: All subscales of the RAND-36 appeared to be unidimensional. For the sub scales ‘mental health’ and ‘general health perceptions’ some minor indications of DIF for the different chronic illnesses were found. Reliabilities of almost all subscales in all subpopulations were higher than 0.80, while the homogeneities of almost all subscales in all subpopulations were higher than 0.50, indicating ‘strong unidimensional, hierarchical scales’. Conclusions: In general, the subscales of the RAND-36 can be used to compare persons with different chronic illnesses. The subscale ‘general health perceptions’ did not function as well as would be preferred.

Polytomous IRT models and monotone likelihood ratio of the total score

In a broad class of item response theory (IRT) models for dichotomous items the unweighted total score has monotone likelihood ratio (MLR) in the latent traitθ. In this study, it is shown that for polytomous items MLR holds for the partial credit model and a trivial generalization of this model. MLR does not necessarily hold if the slopes of the item step response functions vary over items, item steps, or both. MLR holds neither for Samejimas graded response model, nor for nonparametric versions of these three polytomous models. These results are surprising in the context of Graysons and Huynhs results on MLR for nonparametric dichotomous IRT models, and suggest that establishing stochastic ordering properties for nonparametric polytomous IRT models will be much harder.