Jodi M. Casabianca

Effect of Observation Mode on Measures of Secondary Mathematics Teaching.

Classroom observation of teachers is a significant part of educational measurement; measurements of teacher practice are being used in teacher evaluation systems across the country. This research investigated whether observations made live in the classroom and from video recording of the same lessons yielded similar inferences about teaching. Using scores on the Classroom Assessment Scoring System–Secondary (CLASS-S) from 82 algebra classrooms, we explored the effect of observation mode on inferences about the level or ranking of teaching in a single lesson or in a classroom for a year. We estimated the correlation between scores from the two observation modes and tested for mode differences in the distribution of scores, the sources of variance in scores, and the reliability of scores using generalizability and decision studies for the latter comparisons. Inferences about teaching in a classroom for a year were relatively insensitive to observation mode. However, time trends in the raters’ use of the score scale were significant for two CLASS-S domains, leading to mode differences in the reliability and inferences drawn from individual lessons. Implications for different modes of classroom observation with the CLASS-S are discussed.

Comparing computer adaptive testing stopping rules under the generalized partial-credit model

An important consideration of any computer adaptive testing (CAT) program is the criterion used for ending item administration—the stopping rule, which ensures that all examinees are assessed to the same standard. Although various stopping rules exist, none of them have been compared under the generalized partial-credit model (Muraki in Applied Psychological Measurement, 16, 159–176, 1992). In this simulation study we compared the performance of three variable-length stopping rules—standard error (SE), minimum information (MI), and change in theta (CT)—both in isolation and in combination with requirements of minimum and maximum numbers of items, as well as a fixed-length stopping rule. Each stopping rule was examined under two termination criteria—one a more lenient requirement (SE = 0.35, MI = 0.56, CT = 0.05), and one more stringent (SE = 0.30, MI = 0.42, CT = 0.02). The simulation design also included content-balancing and exposure controls, aspects of CAT that have been excluded in previous research comparing variable-length stopping rules. The minimum-information stopping rule produced biased theta estimates and varied greatly in measurement quality across the theta distribution. The absolute-change-in-theta stopping rule had strong performance when paired with a lower criterion and a minimum test length. The standard error stopping rule consistently provided the best balance of measurement precision and operational efficiency and was based on the fewest number of administered items necessary to obtain accurate and precise theta estimates, particularly when it was paired with a maximum-number-of-items stopping rule.

Estimating the Latent Trait Distribution with Loglinear Smoothing Models

This research provides a framework for specifying the latent trait distribution using the family of loglinear smoothing models. Connections between loglinear smoothing models and the standard normal, normal, and direct estimation of the distribution are the beginning of this framework. The framework is important because it gives the connection between the standard approaches and loglinear smoothing models so that parsimonious (smoothing) models providing adequate representation of the distribution can be identified. Future extensions will include additional complex models for estimating the latent trait distribution.