David Thissen

Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the Patient-Reported Outcomes Measurement Information System (PROMIS)

Background:The construction and evaluation of item banks to measure unidimensional constructs of health-related quality of life (HRQOL) is a fundamental objective of the Patient-Reported Outcomes Measurement Information System (PROMIS) project. Objectives:Item banks will be used as the foundation for developing short-form instruments and enabling computerized adaptive testing. The PROMIS Steering Committee selected 5 HRQOL domains for initial focus: physical functioning, fatigue, pain, emotional distress, and social role participation. This report provides an overview of the methods used in the PROMIS item analyses and proposed calibration of item banks. Analyses:Analyses include evaluation of data quality (eg, logic and range checking, spread of response distribution within an item), descriptive statistics (eg, frequencies, means), item response theory model assumptions (unidimensionality, local independence, monotonicity), model fit, differential item functioning, and item calibration for banking. Recommendations:Summarized are key analytic issues; recommendations are provided for future evaluations of item banks in HRQOL assessment.

Likelihood-Based Item-Fit Indices for Dichotomous Item Response Theory Models.

New goodness-of-fit indices are introduced for dichotomous item response theory (IRT) models. These indices are based on the likelihoods of number-correct scores derived from the IRT model, and they provide a direct comparison of the modeled and observed frequencies for correct and incorrect responses for each number-correct score. The behavior of Pearson’s X 2 (S-X 2) and the likelihood ratio G 2 (S-G 2) was assessed in a simulation study and compared with two fit indices similar to those currently in use (Q1-X 2 and Q 1-G 2). The simulations included three conditions in which the simulating and fitting models were identical and three conditions involving model misspecification. S-X 2 performed well, with Type I error rates close to the expected .05 and .01 levels. Performance of this index improved with increased test length. S-G 2 tended to reject the null hypothesis too often, as did Q 1-X 2 and Q 1-G 2. The power of S-X 2 appeared to be similar for all test lengths, but varied depending on the type of model misspecification.

Local Dependence Indexes for Item Pairs Using Item Response Theory.

Four statistics are proposed for the detection of local dependence (LD) among items analyzed using item response theory. Among them, the X2 and G2 LD indexes are of special interest. Simulated data are used to study the distribution and sensitivity of these statistics under the null condition, as well as under conditions in which LD is introduced. The results show that under the null condition of local independence, both the X2 and G2 LD indexes have distributions very similar to the X2 distribution with 1 degree of freedom. Under the locally dependent conditions, both indexes appear to be sensitive in detecting LD or multidimensionality among items. When compared to Q3, another statistic often used to detect LD, these new statistics are somewhat less powerful for underlying LD, equally powerful for surface LD, and better behaved in the null case.

A TAXONOMY OF ITEM RESPONSE MODELS

A number of models for categorical item response data have been proposed in recent years. The models appear to be quite different. However, they may usefully be organized as members of only three distinct classes, within which the models are distinguished only by assumptions and constraints on their parameters. “Difference models” are appropriate for ordered responses, “divide-by-total” models may be used for either ordered or nominal responses, and “left-side added” models are used for multiple-choice responses with guessing. The details of the taxonomy and the models are described in this paper.

More information from fewer questions: The factor structure and item properties of the original and Brief Fear of Negative Evaluation scale

Statistical methods designed for categorical data were used to perform confirmatory factor analyses and item response theory (IRT) analyses of the Fear of Negative Evaluation scale (FNE; D. Watson & R. Friend, 1969) and the Brief FNE (BFNE; M. R. Leary, 1983). Results suggested that a 2-factor model fit the data better for both the FNE and the BFNE, although the evidence was less strong for the FNE. The IRT analyses indicated that although both measures had items with good discrimination, the FNE items discriminated only at lower levels of the underlying construct, whereas the BFNE items discriminated across a wider range. Convergent validity analyses indicated that the straightforwardly-worded items on each scale had significantly stronger relationships with theoretically related measures than did the reverse-worded items. On the basis of all analyses, usage of the straightforwardly-worded BFNE factor is recommended for the assessment of fear of negative evaluation.

On the relationship between the higher-order factor model and the hierarchical factor model

The relationship between the higher-order factor model and the hierarchical factor model is explored formally. We show that the Schmid-Leiman transformation produces constrained hierarchical factor solutions. Using a generalized Schmid-Leiman transformation and its inverse, we show that for any unconstrained hierarchical factor model there is an equivalent higher-order factor model with direct effects (loadings) on the manifest variables from the higher-order factors. Therefore, the class of higher-order factor models (without direct effects of higher-order factors) is nested within the class of unconstrained hierarchical factor models. In light of these formal results, we discuss some implications for testing the higher-order factor model and the issue of general factor. An interesting aspect concerning the efficient fitting of the higher-order factor model with direct effects is noted.

Quick and Easy Implementation of the Benjamini-Hochberg Procedure for Controlling the False Positive Rate in Multiple Comparisons

Williams, Jones, and Tukey (1999) showed that a sequential approach to controlling the false discovery rate in multiple comparisons, due to Benjamini and Hochberg (1995), yields much greater power than the widely used Bonferroni technique that limits the familywise Type I error rate. The Benjamini-Hochberg (B-H) procedure has since been adopted for use in reporting results from the National Assessment of Educational Progress (NAEP), as well as in other research applications. This short note illustrates that the B-H procedure is extremely simple to implement using widely available spreadsheet software. Given its easy implementation, it is feasible to include the B-H procedure in introductory instruction in inferential statistics, augmenting or replacing the Bonferroni technique.

A Response Model for Multiple-Choice Items

We introduce an extended multivariate logistic response model for multiple choice items; this model includes several earlier proposals as special cases. The discussion includes a theoretical development of the model, a description of the relationship between the model and data, and a marginal maximum likelihood estimation scheme for the item parameters. Comparisons of the performance of different versions of the full model with more constrained forms corresponding to previous proposals are included, using likelihood ratio statistics and empirical data.

Further Investigation of the Performance of S - X2: An Item Fit Index for Use With Dichotomous Item Response Theory Models

This study presents new findings on the utility of S - X 2 as an item fit index for dichotomous item response theory models. Results are based on a simulation study in which item responses were generated and calibrated for 100 tests under each of 27 conditions. The item fit indices S - X 2 and Q 1 - X 2 were calculated for each item. ROC curves were constructed based on the hit and false alarm rates of the two indices. Examination of these curves indicated that in general, the performance of S - X 2 improved with test length and sample size. The performance of S - X 2 was superior to that of Q 1 - X 2 under most but not all conditions. Results from this study imply that S - X 2 may be a useful tool in detecting the misfit of one item contained in an otherwise well-fitted test, lending additional support to the utility of the index for use with dichotomous item response theory models. Index Terms: item response theory, S - X 2, Q 1 - X, model = data fit, item fit index.

Marginal maximum likelihood estimation for the one-parameter logistic model

Two algorithms are described for marginal maximum likelihood estimation for the one-parameter logistic model. The more efficient of the two algorithms is extended to estimation for the linear logistic model. Numerical examples of both procedures are presented.