Jiahe Qian

Achieving a Stable Scale for an Assessment with Multiple Forms: Weighting Test Samples in IRT Linking

In the quality control of an assessment with multiple forms, one goal is to attain a stable scale across time. Variability and seasonality across examinee samples and test conditions could cause variation in IRT linking and equating procedures and twist the “sampling exchangeability” in the Draper–Lindley–de Finetti (DLD) measurement validity framework. As an initial exploration of optimal design in linking, we intended to obtain an improved sampling design for invariant Stocking–Lord test characteristic curve (TCC) linking across testing seasons. We applied statistical weighting techniques, such as raking and poststratification, to yield a weighted sample distribution that is consistent with the target population distribution. To assess the weighting effects on linking, we first selected multiple subsamples from an original sample; then, we compared the linking parameters from subsamples with those from the original sample. The results showed that the linking parameters from the weighted sample yielded smaller mean square errors (MSE) than those from the unweighted subsample. The developed techniques can be applied to (1) assessments such as GRE® and TOEFL® with variability and seasonality among multiple forms and (2) assessments such as state assessments with linking decisions based on small initial data.

Linear Prediction of a True Score From a Direct Estimate and Several Derived Estimates

Statistical prediction problems often involve both a direct estimate of a true score and covariates of this true score. Given the criterion of mean squared error, this study determines the best linear predictor of the true score given the direct estimate and the covariates. Results yield an extension of Kelley’s formula for estimation of the true score to cases in which covariates are present. The best linear predictor is a weighted average of the direct estimate and of the linear regression of the direct estimate onto the covariates. The weights depend on the reliability of the direct estimate and on the multiple correlation of the true score with the covariates. One application of the best linear predictor is to use essay features provided by computer analysis and an observed holistic score of an essay provided by a human rater to approximate the true score corresponding to the holistic score.

Optimal Sampling Design for IRT Linking with Bimodal Data

Optimal sampling designs for an IRT linking with improved efficiency are often sought in analyzing assessment data. In practice, the skill distribution of an assessment sample may be bimodal, and this warrants special consideration when trying to create these designs. In this study we explore optimal sampling designs for IRT linking of bimodal data. Our design paradigm is modeled and presents a formal setup for optimal IRT linking. In an optimal sampling design, the sample structure of bimodal data is treated as being drawn from a stratified population. The optimum search algorithm proposed is used to adjust the stratum weights and form a weighted compound sample that minimizes linking errors. The initial focus of the current study is the robust mean–mean transformation method, though the model of IRT linking under consideration is adaptable to generic methods.