William P. Skorupski

A Comparison of Approaches for Improving the Reliability of Objective Level Scores.

This study is an evaluation of the psychometric issues associated with estimating objective level scores, often referred to as ‘‘subscores.’’ The article begins by introducing the concepts of reliability and validity for subscores from statewide achievement tests. These issues are discussed with reference to popular scaling techniques, classical test theory, and item response theory. Methods for increasing the reliability of subscore estimates that have been suggested in literature are then reviewed. Based on this review, an empirical study comparing some of the more promising procedures was conducted. Test score data from a large statewide testing program were analyzed in this study. The comparison of subscore augmentation approaches found that generally all methods were very successful in dramatically increasing the reliability of subscore estimates. However, this increase was accompanied by near-perfect correlations among the subscore estimates. This finding called into question the validity of the resultant subscores, and therefore the usefulness of the subscore augmentation process. Implications for practice are discussed.

A Method for Imputing Response Options for Missing Data on Multiple-Choice Assessments

When missing values are present in item response data, there are a number of ways one might impute a correct or incorrect response to a multiple-choice item. There are significantly fewer methods for imputing the actual response option an examinee may have provided if he or she had not omitted the item either purposely or accidentally. This article applies the multiple-choice model, a multiparameter logistic model that allows for in-depth distractor analyses, to impute response options for missing data in multiple-choice items. Following a general introduction of the issues involved with missing data, the article describes the details of the multiple-choice model and demonstrates its use for multiple imputation of missing item responses. A simple simulation example is provided to demonstrate the accuracy of the imputation method by comparing true item difficulties (p values) and item–total correlations (r values) to those estimated after imputation. Missing data are simulated according to three different types of missing mechanisms: missing completely at random, missing at random, and missing not at random.

Expanding Secondary Attainment in the United States, 1940–80

Abstract In this article, the authors examine the growth of secondary school attainment in the United States between 1940 and 1980, exploring several different conceptual frames of reference. Using state-level data, the authors identify a diffusion model of educational expansion using a fixed-effects panel regression approach. This method is used to analyze change over time, with particular attention to evaluating nonlinear processes of growth. The authors consider the effect of a number of correlates on changing patterns of enrollment in the postwar era. Regional differences in attainment diminished during each decade, and a limited number of social and economic developments appear to have influenced rising enrollment, although most attainment growth appears to have been linked to a self-generating process of diffusion.

The Effects of Small Sample Size on Identifying Polytomous DIF Using the Liu-Agresti Estimator of the Cumulative Common Odds Ratio

This study is an evaluation of the behavior of the Liu—Agresti estimator of the cumulative common odds ratio when identifying differential item functioning (DIF) with polytomously scored test items using small samples. The Liu—Agresti estimator has been proposed by Penfield and Algina as a promising approach for the study of polytomous DIF but no simulation study focusing on small samples has analyzed this estimator regarding effect size, Type I error, and power rates. The article begins with a description of this estimator in the context of polytomous DIF. Then it presents the methods and results for a simulation study in which three factors are manipulated: between-group difference in ability distribution, form of DIF introduced into the polytomous item, and sample size. The results of this study indicate that for samples smaller than 200, very little power was observed for statistical significance testing; however, Type I error rates were close to nominal levels, and the recovery of the log odds ratio as an effect size was relatively unaffected by sample size for high discriminating items. Implications for practice are discussed.

A Bayesian approach to estimating variance components within a multivariate generalizability theory framework

In many behavioral research areas, multivariate generalizability theory (mG theory) has been typically used to investigate the reliability of certain multidimensional assessments. However, traditional mG-theory estimation—namely, using frequentist approaches—has limits, leading researchers to fail to take full advantage of the information that mG theory can offer regarding the reliability of measurements. Alternatively, Bayesian methods provide more information than frequentist approaches can offer. This article presents instructional guidelines on how to implement mG-theory analyses in a Bayesian framework; in particular, BUGS code is presented to fit commonly seen designs from mG theory, including single-facet designs, two-facet crossed designs, and two-facet nested designs. In addition to concrete examples that are closely related to the selected designs and the corresponding BUGS code, a simulated dataset is provided to demonstrate the utility and advantages of the Bayesian approach. This article is intended to serve as a tutorial reference for applied researchers and methodologists conducting mG-theory studies.

Standard Errors and Confidence Intervals from Bootstrapping for Ramsay-Curve Item Response Theory Model Item Parameters.

Ramsay-curve item response theory (RC-IRT) is a nonparametric procedure that estimates the latent trait using splines, and no distributional assumption about the latent trait is required (Woods & Thissen, 2006). Description of this procedure can be found, for example, in the technical manual of RCLOG v.2, software for RC-IRT (Woods, 2006b). For item parameters of the twoparameter logistic (2-PL), three-parameter logistic (3-PL), and polytomous IRT models, RC-IRT can provide more accurate estimates than the commonly used marginal maximum likelihood estimation (MMLE) when the latent trait is not normally distributed (Woods, 2006a, 2007, 2008). However, standard errors (SEs) for the item parameter estimates have not been developed in RC-IRT as no analytical solution is readily available (Woods, 2006a, 2007, 2008; Woods & Lin, 2009). In such cases, bootstrapping provides an alternative way to estimate SEs. Using bootstrapping, the observed sample is treated as the pseudopopulation from which n repeated random samples are drawn with replacement. The same estimation procedure is employed on each random sample and the point estimates are retained. Then, the SE of a particular parameter estimate is the standard deviation of the retained estimates, and the associated confidence interval (CI) can be determined by two percentiles. For example, a 95% CI can be determined by the range between the 2.5th and 97.5th percentiles. In this research, bootstrapping was utilized to estimate SEs and CIs for item parameters in the 2-PL model, and the performance of bootstrapping was compared with that of MMLE.