Duanli Yan

Bayesian Network Models for Local Dependence among Observable Outcome Variables.

Bayesian network models offer a large degree of flexibility for modeling dependence among observables (item outcome variables) from the same task, which may be dependent. This article explores four design patterns for modeling locally dependent observations: (a) no context—ignores dependence among observables; (b) compensatory context—introduces a latent variable, context, to model task-specific knowledge and use a compensatory model to combine this with the relevant proficiencies; (c) inhibitor context—introduces a latent variable, context, to model task-specific knowledge and use an inhibitor (threshold) model to combine this with the relevant proficiencies; (d) compensatory cascading—models each observable as dependent on the previous one in sequence. This article explores the four design patterns through experiments with simulated and real data. When the proficiency variable is categorical, a simple Mantel-Haenszel procedure can test for local dependence. Although local dependence can cause problems in the calibration, if the models based on these design patterns are successfully calibrated to data, all the design patterns appear to provide very similar inferences about the students. Based on these experiments, the simpler no context design pattern appears more stable than the compensatory context model, while not significantly affecting the classification accuracy of the assessment. The cascading design pattern seems to pick up on dependencies missed by other models and should be explored with further research.

Displays for Comparing a Given State to Many Others

Abstract This article presents a graphical display useful for comparing a chosen individual (e.g., a single state in a state-by-state survey) with the others in the population. The display immediately shows the ranking of the individual in the population, which other individuals are “significantly” higher than the reference, and which are “significantly” lower. The confidence bars are optimized for making comparisons with the reference individual. We illustrate this display with examples from the National Assessment for Educational Progress (NAEP) and the Third International Mathematics and Science Study (TIMSS).

Adaptive Testing With Regression Trees in the Presence of Multidimensionality

It is unrealistic to suppose that standard item response theory (IRT) models will be appropriate for all the new and currently considered computer-based tests. In addition to developing new models, we also need to give attention to the possibility of constructing and analyzing new tests without the aid of strong models. Computerized adaptive testing currently relies heavily on IRT. Alternative, empirically based, nonparametric adaptive testing algorithms exist, but their properties are little known. This article introduces a nonparametric, tree-based algorithm for adaptive testing and shows that it may be superior to conventional, IRT-based adaptive testing in cases where the IRT assumptions are not satisfied. In particular, it shows that the tree-based approach clearly outperformed (one-dimensional) IRT when the pool was strongly two-dimensional.

An Introduction to Evidence-Centered Design

This chapter provides a brief introduction to evidence-centered assessment design. Although assessment design is an important part of this book, we do not tackle it in a formal way until Part III. Part I builds up a class of mathematical models for scoring an assessment, and Part II discusses how the mathematical models can be refined with data. Although throughout the book there are references to cognitive processes that the probability distributions model, the full discussion of assessment design follows the discussion of the more mathematical issues.

Learning in Models with Fixed Structure

The preceding chapters have described an approach to assessment design and analysis that exploits the advantages of Bayesian networks. This chapter addresses the problem of estimating these distributions or parameters they are modeled in terms of. It lays out a general Bayesian framework for expressing educational measurement models in these terms. It then describes and illustrates two estimation approaches: Bayes modal estimation via the expectation–maximization (EM) algorithm and Markov Chain Monte Carlo (MCMC) estimation.

An Illustrative Example

The preceding chapters described how to build the Bayesian networks, choosing parameterizations for the conditional probability tables that quantify the network, learning the parameters of Bayes nets given a body of assessment data, and evaluating how well a proposed network fits the data. This chapter reviews these concepts in terms of an example: the mixed number subtraction example of Tatsuoka (1983).

The Future of Bayesian Networks in Educational Assessment

Bayesian networks offer an approach that scales up from familiar assessment designs and purposes to the wide array of new kinds of assessments made possible by advances in technology, cognitive psychology, and learning sciences. The previous chapters laid out the graphical, statistical, and assessment design foundations for assessment accordingly. This final chapter offers thoughts and provides pointers to topics that lay beyond the scope of the book. They include a quick look at various applications in educational assessment, developments in Bayesian networks that should find uses in assessment, integrating assessment with instruction with dynamic Bayesian networks.

Examples of Simulations Using catR

The previous chapter focused on the description of the R package catR as a tool for simulation studies on CAT processes. This chapter presents several practical illustrations of catR using two real item banks, one for dichotomously scored items and one for polytomously scored items. After a brief description of both item banks, several examples are displayed with the same organization.

An Overview of Computerized Multistage Testing

In this chapter, we present a brief overview of computerized multistage testing theory, including test design, test assembly, item bank, module selection, routing, scoring and linking, and exposure and security. We also provide a summary of the IRT-based module selection process, as well as the tree-based multistage testing.

An Overview of Computerized Adaptive Testing

In this chapter, we present a brief overview of computerized adaptive testing theory, including test design, test assembly, item bank, item selection, scoring and equating, content balance, item exposure and security. We also provide a summary of the IRT-based item selection process with a list of the commonly used item selection methods, as well as a brief outline of the tree-based adaptive testing.