Rob R. Meijer

Methodology Review: Evaluating Person Fit.

Person-fit methods based on classical test theory-and item response theory (IRT), and methods investigating particular types of response behavior on tests, are examined. Similarities and differences among person-fit methods and their advantages and disadvantages are discussed. Sound person-fit methods have been derived for the Rasch model. For other IRT models, the empirical and theoretical distributions differ for most person-fit statistics when used with short and moderate length tests. The detection rate of person-fit statistics depends on the type of misfitting item-score patterns, test length, and trait levels. The usefulness of person-fit statistics for improving measurement depends on the application.

Analyzing psychopathology items: A case for nonparametric item response theory modeling

The authors discuss the applicability of nonparametric item response theory (IRT) models to the construction and psychometric analysis of personality and psychopathology scales, and they contrast these models with parametric IRT models. They describe the fit of nonparametric IRT to the Depression content scale of the Minnesota Multiphasic Personality Inventory--2 (J. N. Butcher, W. G. Dahlstrom, J. R. Graham, A. Tellegen, & B. Kaemmer, 1989). They also show how nonparametric IRT models can easily be applied and how misleading results from parametric IRT models can be avoided. They recommend the use of nonparametric IRT modeling prior to using parametric logistic models when investigating personality data.

Computerized Adaptive Testing: Overview and Introduction.

Use of computerized adaptive testing (CAT) has increased substantially since it was first formulated in the 1970s. This paper provides an overview of CAT and introduces the contributions to this Special Issue. The elements of CAT discussed here include item selection procedures, estimation of the latent trait, item exposure, measurement precision, and item bank development. Some topics for future research are also presented.

A Bayesian Approach to Person Fit Analysis in Item Response Theory Models

A Bayesian approach to the evaluation of person fit in item response theory (IRT) models is presented. In a posterior predictive check, the observed value on a discrepancy variable is positioned in its posterior distribution. In a Bayesian framework, a Markov chain Monte Carlo procedure can be used to generate samples of the posterior distribution of the parameters of interest. These draws can also be used to compute the posterior predictive distribution of the discrepancy variable. The procedure is worked out in detail for the three-parameter normal ogive model, but it is also shown that the procedure can be directly generalized to many other IRT models. Type I error rate and the power against some specific model violations are evaluated using a number of simulation studies. Index terms: Bayesian statistics, item response theory, person fit, model fit, 3-parameter normal ogive model, posterior predictive check, power studies, Type I error.

On the consistency of individual classification using short scales

Short tests containing at most 15 items are used in clinical and health psychology, medicine, and psychiatry for making decisions about patients. Because short tests have large measurement error, the authors ask whether they are reliable enough for classifying patients into a treatment and a nontreatment group. For a given certainty level, proportions of correct classifications were computed for varying test length, cut-scores, item scoring, and choices of item parameters. Short tests were found to classify at most 50% of a group consistently. Results were much better for tests containing 20 or 40 items. Small differences were found between dichotomous and polytomous (5 ordered scores) items. It is recommended that short tests for high-stakes decision making be used in combination with other information so as to increase reliability and classification consistency.

The Null Distribution of Person-Fit Statistics for Conventional and Adaptive Tests.

Several person-fit statistics have been proposed to detect item score patterns that do not fit an item response theory model. To classify response patterns as misfitting, the distribution of a person-fit statistic is needed. The theoretical null distributions of several fit statistics have been derived for paper-and-pencil (P&P) tests. However, it is unknown whether these distributions also hold for computerized adaptive tests (CAT). A three-part simulation study was conducted. In the first study, the theoretical distribution of the l z statistic across trait. θlevels for CAT and P&P tests was investigated. The distribution of the l* z statistic proposed by Snijders (in press) was also investigated. Results indicated that the distribution of both l z and l* z differed from the theoretical distribution in CAT. The second study examined the distributions of l z and l* z using simulation. These simulated distributions, when based on O [UNKNOWN], were found to be problematic in CAT. In the third study, the detection rates of l* z and l z were compared. The rates for both statistics were found to be similar in most cases.

The number of Guttman errors as a simple and powerful person-fit statistic

A number of studies have examined the power of several statistics that can be used to detect examinees with unexpected (nonfitting) item score patterns, or to determine person fit. This study compared the power of the U3 statistic with the power of one of the sim plest person-fit statistics, the sum of the number of Guttman errors. In most cases studied, (a weighted version of) the latter statistic performed as well as the U3 statistic. Counting the number of Guttman errors seems to be a useful and simple alternative to more complex statistics for determining person fit. Index terms: aberrance detection, appropriateness measure ment, Guttman errors, nonparametric item response theory, person fit.

Global, Local, and Graphical Person-Fit Analysis Using Person-Response Functions.

Person-fit statistics test whether the likelihood of a respondents complete vector of item scores on a test is low given the hypothesized item response theory model. This binary information may be insufficient for diagnosing the cause of a misfitting item-score vector. The authors propose a comprehensive methodology for person-fit analysis in the context of nonparametric item response theory. The methodology (a) includes H. Van der Fliers (1982) global person-fit statistic U3 to make the binary decision about fit or misfit of a persons item-score vector, (b) uses kernel smoothing (J. O. Ramsay, 1991) to estimate the person-response function for the misfitting item-score vectors, and (c) evaluates unexpected trends in the person-response function using a new local person-fit statistic (W. H. M. Emons, 2003). An empirical data example shows how to use the methodology for practical person-fit analysis.

A comparison of the person response function and the Iz person-fit statistic

In the past several decades, there has been an increasing interest in the development of statistics used to identify examinees who respond to test items in a manner that is divergent from the underlying test model. One statistic that has received a great deal of attention is the lz index (Drasgow, Levine, & Williams, 1985). Trabin & Weiss (1983) developed a slightly different approach for identifying model-divergent response patterns, based on the discrepancy between observed and expected person response functions (PRFs). Here, the PRF method was compared theoretically and empirically to 1z. Although the results suggest that the performance of lz was, in most conditions studied, superior to the PRF method, the PRF method was useful in investigating why a response vector was model divergent. The PRF method can be used along with lz to identify different types of model divergent response patterns.

Trait Level Estimation for Nonfitting Response Vectors

Item responses that do not fit an item response theory model may cause the latent trait value, 0, to be inaccurately estimated. Although in many studies the proportion of nonmodel-fitting response vectors (NRvs) identified (i.e., the detection rate) has been investigated, less is known about the severity of the inaccuracy of the estimated 0 (0) in relation to the rate at which response patterns are classified as not fitting a model using person-fit statistics. In the present study, three scoring methods-maximum likelihood estimation (MLE), expected a posterior (EAP) estimation, and biweight estimation (BIw)-were used to estimate 0 when NRVS were present. The detection rate of the lz person-fit statistic (Drasgow, Levine, & Williams, 1985) was also investigated. It was found that NRVS influenced the value of 8, and this depended heavily on the type of misfit and the 8 level. It was also found that the BIW scoring method reduced the bias in 0 and improved the detection rate compared to MLE and EAP for examinees located at the extreme ends of the 8 continuum. Results of this study extended the results obtained by Reise (1995) and showed that the low detection rates found in his study were due to the particular kind of procedure used to generate NRVS.