Jonathan Templin

Unique Characteristics of Diagnostic Classification Models: A Comprehensive Review of the Current State-of-the-Art

Diagnostic classification models (DCM) are frequently promoted by psychometricians as important modelling alternatives for analyzing response data in situations where multivariate classifications of respondents are made on the basis of multiple postulated latent skills. In this review paper, a definitional boundary of the space of DCM is developed, core DCM within this space are reviewed, and their defining features are compared and contrasted with those of other latent variable models. The models to which DCM are compared include unrestricted latent class models, multidimensional factor analysis models, and multidimensional item response theory models. Attention is paid to both statistical considerations of model structure, as well as substantive considerations of model use.

Fibromyalgia : The Role of Sleep in Affect and in Negative Event Reactivity and Recovery

OBJECTIVE Fibromyalgia (FM) syndrome is a chronic pain condition characterized by diffuse muscle pain, increased negative mood, and sleep disturbance. Until recently, sleep disturbance in persons with FM has been modeled as the result of the disease process or its associated pain. The current study examined sleep disturbance (i.e., sleep duration and sleep quality) as a predictor of daily affect, stress reactivity, and stress recovery. DESIGN AND MEASURES A hybrid of daily diary and ecological momentary assessment methodology was used to evaluate the psychosocial functioning of 89 women with FM. Participants recorded numeric ratings of pain, fatigue, and positive and negative affect 3 times throughout the day for 30 consecutive days. At the end of each day, participants completed daily diary records of positive and negative life events. In addition, participants reported on their sleep duration and sleep quality each morning. RESULTS After accounting for the effects of positive events, negative events, and pain on daily affect scores, it was found that sleep duration and quality were prospectively related to affect and fatigue. Furthermore, the effects of inadequate sleep on negative affect were cumulative. In addition, an inadequate amount of sleep prevented affective recovery from days with a high number of negative events. CONCLUSIONS These results lend support to the hypothesis that sleep is a component of allostatic load and has an upstream role in daily functioning.

Are the sources of interest the same for everyone? Using multilevel mixture models to explore individual differences in appraisal structures.

How does personality influence the relationship between appraisals and emotions? Recent research suggests individual differences in appraisal structures: people may differ in an emotions appraisal pattern. We explored individual differences in interests appraisal structure, assessed as the within-person covariance of appraisals with interest. People viewed images of abstract visual art and provided ratings of interest and of interests appraisals (novelty–complexity and coping potential) for each picture. A multilevel mixture model found two between-person classes that reflected distinct within-person appraisal styles. For people in the larger class (68%), the novelty–complexity appraisal had a stronger effect on interest; for people in the smaller class (32%), the coping potential appraisal had a stronger effect. People in the larger class were significantly higher in appetitive traits related to novelty seeking (e.g., sensation seeking, openness to experience, and trait curiosity), suggesting that the appraisal classes have substantive meaning. We conclude by discussing the value of within-person mixture models for the study of personality and appraisal.

Hierarchical Diagnostic Classification Models: A Family of Models for Estimating and Testing Attribute Hierarchies

Although latent attributes that follow a hierarchical structure are anticipated in many areas of educational and psychological assessment, current psychometric models are limited in their capacity to objectively evaluate the presence of such attribute hierarchies. This paper introduces the Hierarchical Diagnostic Classification Model (HDCM), which adapts the Log-linear Cognitive Diagnosis Model to cases where attribute hierarchies are present. The utility of the HDCM is demonstrated through simulation and by an empirical example. Simulation study results show the HDCM is efficiently estimated and can accurately test for the presence of an attribute hierarchy statistically, a feature not possible when using more commonly used DCMs. Empirically, the HDCM is used to test for the presence of a suspected attribute hierarchy in a test of English grammar, confirming the data is more adequately represented by hierarchical attribute structure when compared to a crossed, or nonhierarchical structure.

Perspectives on Methodological Issues

In this chapter the authors have surveyed the methodological perspectives seen as important for assessing twenty-first century skills. Some of those issues are specific to twenty-first century skills, but the majority would apply more generally to the assessment of other psychological and educational variables. The narrative of the paper initially follows the logic of assessment development, commencing by defining constructs to be assessed, designing tasks that can be used to generate informative student responses, coding/valuing of those responses, delivering the tasks and gathering the responses, and modeling the responses in accordance with the constructs. The paper continues with a survey of the strands of validity evidence that need to be established, and a discussion of specific issues that are prominent in this context, such as the need to resolve issues of generality versus contextual specificity; the relationships of classroom to large-scale assessments; and the possible roles for technological advances in assessing these skills. There is also a brief segment discussing some issues that arise with respect to specific types of variables involved in the assessment of twenty-first century skills. The chapter concludes with a listing of particular challenges that are regarded as being prominent at the time of writing. There is an annexure that describes specific approaches to assessment design that are useful in the development of new assessments.

Measuring the Reliability of Diagnostic Classification Model Examinee Estimates

Over the past decade, diagnostic classification models (DCMs) have become an active area of psychometric research. Despite their use, the reliability of examinee estimates in DCM applications has seldom been reported. In this paper, a reliability measure for the categorical latent variables of DCMs is defined. Using theory-and simulation-based results, we show how DCMs uniformly provide greater examinee estimate reliability than IRT models for tests of the same length, a result that is a consequence of the smaller range of latent variable values examinee estimates can take in DCMs. We demonstrate this result by comparing DCM and IRT reliability for a series of models estimated with data from an end-of-grade test, culminating with a discussion of how DCMs can be used to change the character of large scale testing, either by shortening tests that measure examinees unidimensionally or by providing more reliable multidimensional measurement for tests of the same length.

Robustness of Hierarchical Modeling of Skill Association in Cognitive Diagnosis Models.

Several types of parameterizations of attribute correlations in cognitive diagnosis models use the reduced reparameterized unified model. The general approach presumes an unconstrained correlation matrix with K(K−1)/2 parameters, whereas the higher order approach postulates K parameters, imposing a unidimensional structure on the correlation matrix between the latent skills. This article investigates the differences in performance between the correlational structure parameterizations (a general structure, a higher order single-factor structure, and a baseline uniform distributional approach constraining the attributes to be independent) across a wide variety of simulated multidimensional attribute spaces. Results suggest that the correlational approaches perform equally well with respect to classification and item parameter estimation accuracy, regardless of the violations of the assumptions of the higher order model. Findings suggest the general robustness of the higher order model and the associated estimation procedure. The three approaches are also used to analyze a real-world test; results suggest that such tests can be analyzed effectively by the higher order algorithm.

Combining item response theory and diagnostic classification models: a psychometric model for scaling ability and diagnosing misconceptions.

Traditional testing procedures typically utilize unidimensional item response theory (IRT) models to provide a single, continuous estimate of a student’s overall ability. Advances in psychometrics have focused on measuring multiple dimensions of ability to provide more detailed feedback for students, teachers, and other stakeholders. Diagnostic classification models (DCMs) provide multidimensional feedback by using categorical latent variables that represent distinct skills underlying a test that students may or may not have mastered. The Scaling Individuals and Classifying Misconceptions (SICM) model is presented as a combination of a unidimensional IRT model and a DCM where the categorical latent variables represent misconceptions instead of skills. In addition to an estimate of ability along a latent continuum, the SICM model provides multidimensional, diagnostic feedback in the form of statistical estimates of probabilities that students have certain misconceptions. Through an empirical data analysis, we show how this additional feedback can be used by stakeholders to tailor instruction for students’ needs. We also provide results from a simulation study that demonstrate that the SICM MCMC estimation algorithm yields reasonably accurate estimates under large-scale testing conditions.

A Latent Transition Analysis Model for Assessing Change in Cognitive Skills

Latent transition analysis (LTA) was initially developed to provide a means of measuring change in dynamic latent variables. In this article, we illustrate the use of a cognitive diagnostic model, the DINA model, as the measurement model in a LTA, thereby demonstrating a means of analyzing change in cognitive skills over time. An example is presented of an instructional treatment on a sample of seventh-grade students in several classrooms in a Midwestern school district. In the example, it is demonstrated how hypotheses could be framed and then tested regarding the form of the change in different groups within the population. Both manifest and latent groups also are defined and used to test additional hypotheses about change specific to particular subpopulations. Results suggest that the use of a DINA measurement model expands the utility of LTA to practical problems in educational measurement research.

Intra-Individual Variability of Physical Activity in Older Adults With and Without Mild Alzheimer's Disease.

Physical activity shows promise for protection against cognitive decline in older adults with and without Alzheimer’s disease (AD). To better understand barriers to adoption of physical activity in this population, a clear understanding of daily and weekly activity patterns is needed. Most accelerometry studies report average physical activity over an entire wear period without considering the potential importance of the variability of physical activity. This study evaluated individual differences in the amount and intra-individual variability of physical activity and determined whether these differences could be predicted by AD status, day of wear, age, gender, education, and cardiorespiratory capacity. Physical activity was measured via accelerometry (Actigraph GT3X+) over one week in 86 older adults with and without AD (n = 33 and n = 53, respectively). Mixed-effects location-scale models were estimated to evaluate and predict individual differences in the amount and intra-individual variability of physical activity. Results indicated that compared to controls, participants with AD averaged 21% less activity, but averaged non-significantly greater intra-individual variability. Women and men averaged similar amounts of physical activity, but women were significantly less variable. The amount of physical activity differed significantly across days of wear. Increased cardiorespiratory capacity was associated with greater average amounts of physical activity. Investigation of individual differences in the amount and intra-individual variability of physical activity provided insight into differences by AD status, days of monitor wear, gender, and cardiovascular capacity. All individuals regardless of AD status were equally consistent in their physical activity, which may have been due to a highly sedentary sample and/or the early disease stage of those participants with AD. These results highlight the value of considering individual differences in both the amount and intra-individual variability of physical activity.