David A. Frisbie

Estimating Reliability Under a Generalizability Theory Model for Test Scores Composed of Testlets

Previous studies have indicated that the reliability of test scores composed of testlets might be overestimated by conventional item-based reliability estimation methods (Anastasi, 1988; Sireci, Thissen, & Wainer, 1991; Thorndike, 1951; Wainer, 1995; Wainer & Thissen, 1996). We designed this study to investigate the appropriateness and implications of using a generalizability theory (G-theory) approach to estimating the reliability of scores from tests composed of testlets. The magnitude of overestimation from using Cronbachs alpha based on item scores in this situation was found to be about 0.04 relative to the testlet approach with G-theory. The generalizability coefficients based on varying numbers of passages and a fixed total number of items were found to be more variable than when the number of passages was fixed, the total number of items was fixed, and the number of items per passage varied. Therefore, manipulating the number of passages is a more productive way to obtain efficient measurement pr...

The Relative Appropriateness of Eight Measurement Models for Analyzing Scores from Tests Composed of Testlets.

It has been shown that fundamental assumptions associated with conventional one-factor measurement models are frequently violated in analyses of scores from a test composed of testlets. Eight different measurement models were conceptualized for this kind of situation, and the goodness of fit of each model was examined. Conventional essentially tauequivalent and congeneric models present worse model fit to data and overestimate the reliability when testlets are involved. The one-factor congeneric model with correlated error specifications seems to be the best measurement model for a test composed of testlets if dichotomously scored items are used as the unit of analysis. However, in estimating score reliability for tests composed of testlets, the one-factor essentially tauequivalent model with correlated error specifications also provides good estimates. Measurement models using passage (testlet) scores would be alternatives for analyzing scores from tests composed of testlets when passage (testlet) scores are used as the unit of analysis.

The Effect of Test Item Familiarization on Achievement Test Scores.

One method of constructing test batteries that are used over a range of age or grade levels involves overlapping some of the same items across consecutive test levels. For example, some items that appear in the last half of a 3rd-grade vocabulary test also appear in the first half of the 4th-grade vocabulary test. Over consecutive years of testing, do these overlapping items familiarize students with specific test content to such an extent that students become advantaged during subsequent test administrations? Such an added advantage, if it exists, would clearly decrease the validity of the subsequent test scores. In this study, we used both overlapping and nonoverlapping-item conditions, controlled for test content and grade level, and focused on whether there is an effect on achievement test scores due to item familiarization. No effects were detected.