Robert A. Henson

Test Construction for Cognitive Diagnosis

Although cognitive diagnostic models (CDMs) can be useful in the analysis and interpretation of existing tests, little has been developed to specify how one might construct a good test using aspects of the CDMs. This article discusses the derivation of a general CDM index based on Kullback-Leibler information that will serve as a measure of how informative an item is for the classification of examinees. The effectiveness of the index is examined for items calibrated using the deterministic input noisy “and” gate model (DINA) and the reparameterized unified model (RUM) by implementing a simple heuristic to construct a test from an item bank. When compared to randomly constructed tests from the same item bank, the heuristic shows significant improvement in classification rates.

Cognitive Diagnostic Assessment for Education: The Fusion Model Skills Diagnosis System

INTRODUCTION There is a long history of calls for combining cognitive science and psychometrics (Cronbach, 1975; Snow & Lohman, 1989). The U.S. standards movement, begun more than 20 years ago (McKnight et al., 1987; National Council of Teachers of Mathematics, 1989), sought to articulate public standards for learning that would define and promote successful performance by all students; establish a common base for curriculum development and instructional practice; and provide a foundation for measuring progress for students, teachers and programs. The standards movement provided the first widespread call for assessment systems that directly support learning. For success, such systems must satisfy a number of conditions having to do with cognitive-science–based design, psychometrics, and implementation. This chapter focuses on the psychometric aspects of one particular system that builds on a carefully designed test and a user-selected set of relevant skills measured by that test to assess student mastery of each of the chosen skills. This type of test-based skills level assessment is called skills diagnosis . The system that the chapter describes in detail is called the Fusion Model system . This chapter focuses on the statistical and psychometric aspects of the Fusion Model system, with skills diagnosis researchers and practitioners in mind who may be interested in working with this system. We view the statistical and psychometric aspects as situated within a comprehensive framework for diagnostic assessment test design and implementation.

OCLUS: An Analytic Method for Generating Clusters with Known Overlap

AbstractThe primary method for validating cluster analysis techniques is throughMonte Carlo simulations that rely on generating data with known cluster structure (e.g., Milligan 1996). This paper defines two kinds of data generation mechanisms with cluster overlap, marginal and joint; current cluster generation methods are framed within these definitions. An algorithm generating overlapping clusters based on shared densities from several different multivariate distributions is proposed and shown to lead to an easily understandable notion of cluster overlap. Besides outlining the advantages of generating clusters within this framework, a discussion is given of how the proposed data generation technique can be used to augment research into current classification techniques such as finite mixture modeling, classification algorithm robustness, and latent profile analysis.

Cognitive diagnostic attribute-level discrimination indices

Cognitive diagnostic models (CDMs) model the probability of correctly answering an item as a function of an examinees attribute mastery pattern. Because estimation of the mastery pattern involves more than a continuous measure of ability, reliability concepts introduced by classical test theory and item response theory do not apply. The cognitive diagnostic index (CDI) measures an items overall discrimination power, which indicates an items usefulness in examinee attribute pattern estimation. Because of its relationship with correct classification rates, the CDI was shown to be instrumental in cognitively diagnostic test assembly. This article generalizes the CDI to attribute-level discrimination indices for an item. Two different attribute-level discrimination indices are defined; their relationship with correct classification rates is explored using Monte Carlo simulations. There are strong relationships between the defined attribute indices and correct classification rates. Thus, one important potential application of these indices is test assembly from a CDM-calibrated item bank.

Testing Person Fit in Cognitive Diagnosis

In cognitive diagnosis, the test-taking behavior of some examinees may be idiosyncratic so that their test scores may not reflect their true cognitive abilities as much as that of more typical examinees. Statistical tests are developed to recognize the following: (a) nonmasters of the required attributes who correctly answer the item (spuriously high scores) and (b) masters of the attributes who fail to correctly answer the item (spuriously low scores). For a person, nonzero probability of aberrant behavior is tested as the alternative hypothesis, against normal behavior as the null hypothesis. The two generalized likelihood ratio test statistics used, with the null hypothesis parameter on the boundary of the parameter space in each, have asymptotic distributions of a 50:50 mixture of a chi-square distribution with one degree of freedom and a degenerate distribution that is a constant of 0 under the null hypothesis. Simulation results, primarily based on the DINA model (deterministic inputs, noisy ‘‘AND’’ gate), are used to investigate the following: (a) how accurately the statistical tests identify normal/aberrant behaviors, (b) how the power of the tests depends on the length of the cognitive exam and the degree of the inclination toward aberrance, and (c) how sensitive the tests are to inaccurate estimation of model parameters.

Robustness of Hierarchical Modeling of Skill Association in Cognitive Diagnosis Models.

Several types of parameterizations of attribute correlations in cognitive diagnosis models use the reduced reparameterized unified model. The general approach presumes an unconstrained correlation matrix with K(K−1)/2 parameters, whereas the higher order approach postulates K parameters, imposing a unidimensional structure on the correlation matrix between the latent skills. This article investigates the differences in performance between the correlational structure parameterizations (a general structure, a higher order single-factor structure, and a baseline uniform distributional approach constraining the attributes to be independent) across a wide variety of simulated multidimensional attribute spaces. Results suggest that the correlational approaches perform equally well with respect to classification and item parameter estimation accuracy, regardless of the violations of the assumptions of the higher order model. Findings suggest the general robustness of the higher order model and the associated estimation procedure. The three approaches are also used to analyze a real-world test; results suggest that such tests can be analyzed effectively by the higher order algorithm.

Using Deterministic, Gated Item Response Theory Model to detect test cheating due to item compromise.

The Deterministic, Gated Item Response Theory Model (DGM, Shu, Unpublished Dissertation. The University of North Carolina at Greensboro, 2010) is proposed to identify cheaters who obtain significant score gain on tests due to item exposure/compromise by conditioning on the item status (exposed or unexposed items). A “gated” function is introduced to decompose the observed examinees’ performance into two distributions (the true ability distribution determined by examinees’ true ability and the cheating distribution determined by examinees’ cheating ability). Test cheaters who have score gain due to item exposure are identified through the comparison of the two distributions. Hierarchical Markov Chain Monte Carlo is used as the model’s estimation framework. Finally, the model is applied in a real data set to illustrate how the model can be used to identify examinees having pre-knowledge on the exposed items.

Using Neural Network Analysis to Define Methods of DINA Model Estimation for Small Sample Sizes

The DINA model is a commonly used model for obtaining diagnostic information. Like many other Diagnostic Classification Models (DCMs), it can require a large sample size to obtain reliable item and examinee parameter estimation. Neural Network (NN) analysis is a classification method that uses a training dataset for calibration. As a result, if this training dataset is determined theoretically, as was the case in Gierl’s attribute hierarchical method (AHM), the NN analysis does not have any sample size requirements. However, a NN approach does not provide traditional item parameters of a DCM or allow for item responses to influence test calibration. In this paper, the NN approach will be implemented for the DINA model estimation to explore its effectiveness as a classification method beyond its use in AHM. The accuracy of the NN approach across different sample sizes, item quality and Q-matrix complexity is described in the DINA model context. Then, a Markov Chain Monte Carlo (MCMC) estimation algorithm and Joint Maximum Likelihood Estimation is used to extend the NN approach so that item parameters associated with the DINA model are obtained while allowing examinee responses to influence the test calibration. The results derived by the NN, the combination of MCMC and NN (NN MCMC) and the combination of JMLE and NN are compared with that of the well-established Hierarchical MCMC procedure and JMLE with a uniform prior on the attribute profile to illustrate their strength and weakness.

Diagnostic Classification Models: Thoughts and Future Directions

In general, I feel that diagnostic classification modeling (DCMs; also commonly referred to as cognitive diagnosis models) holds great potential in low stake situations because of the promise of providing more detailed information related to those attributes (sometimes called skills, abilities, or traits) that an individual should improve. Thus, diagnostic modeling provides a tool that can aid in the development of tailored lesson plans (for a student or a class), which could save students and teachers time. However, this true potential of diagnostic modeling has yet to be seen. Most papers in the literature seem to have focused on theoretical issues where the application is used mostly as an example of the methodology. In addition, most of these example applications do not provide a typical low stakes situation, but instead represent nearly ideal situations with large sample sizes that provide for the demonstration of a new methodology. Probably the most common example of such a data set is in math (e.g., mixed fraction subtraction which has been used by Tatsuoka, 1990; de la Torre and Douglas, 2004; Templin, Henson, & Douglas, 2008; Henson, Templin, & Willse, 2008), although other examples that do not use a math assessment exist. I believe that this direction of the literature is due to an initial demand for DCMs to address many technical questions related to issues that have already been addressed by alterative competing theories of test assessment. That is, new methodologies tend to be evaluated based on those characteristics that are most familiar and so much of the DCM literature was attempting to catch up to what had already been developed in methodologies, such as factor analysis and item response theory. Although this direction was necessary as a first step toward establishing a very basic set of statistical principles, the growing emphasis of the methodology for DCMs is now on providing evidence that these models, in application, can provide the information that has been promised. Specifically, the field must now address the question, “Can a DCM be applied in an interdisciplinary setting, such as a typical teaching setting, where alternative models have already proven useful?” For such a question to be answered, DCMs must be accessible to a broader audience that is not familiar with the typical jargon used to describe these models. Therefore, a set of papers must be written describing the models that are available, how they differ, and how they can be estimated, among other issues, in a way that is accessible to this audience. The paper, “Unique Characteristics of Diagnostic Classification Models: A Comprehensive Review of the Current State-of-the-Art” by Drs. Rupp and Templin (2008) addresses this concern by providing an ambitious summary of many of these basic concepts. Specifically, the authors have provided: (i) a definition of DCMs, including when they are appropriate; (ii) a summary of a basic set of core models; (iii) the methods used to estimate these models and the available software;

Graphical Representations of Items and Tests That are Measuring Multiple Abilities

This article compares graphical representations of items and tests for four different multidimensional item response theory (MIRT) models: compensatory logistic model, the noncompensatory logistic model, a noncompensatory diagnostic model (DINA), and a compensatory diagnostic model (CRUM/GDM). Graphical representations can provide greater insight for measurement specialists and item/test developers about the validity and reliability of the multidimensional tests. They also can provide a link between quantitative analyses and substantive interpretations of the score scale and inform the test development process.