Shuliang Ding

Dichotomous and Polytomous Q Matrix Theory

Besides specifying the Q matrix correctly, an extended Q matrix theory should provide some answers to following questions: (a) how to find the universal set of knowledge states (KSs), (b) how to use an algorithm to calculate the set of ideal response patterns (IRPs), (c) how to construct the test Q matrix which establishes one-to-one mapping from the set of KSs to the set of IRPs, and (d) how to mine the hierarchy relation hidden in the Q matrix. In this note, a dichotomous and polytomous Q matrix theory is established and the properties of the polytomous Q matrix are explored to help address the above questions.

Estimating Classification Accuracy and Consistency Indices for Multidimensional Latent Ability

For criterion-referenced tests, classification consistency and accuracy are viewed as important indicators for evaluating reliability and validity of classification results. Numerous procedures have been proposed in the framework of unidimensional item response theory (UIRT) to estimate these indices. Some of these were based on total sum scores, others on latent trait estimates. However, there exist very few attempts to develop them in the framework of multidimensional item response theory (MIRT). Based on previous studies, the aim of this study is first to estimate the consistency and accuracy indices of multidimensional ability estimates from a single administration of a criterion-referenced test. We also examined how Monte Carlo sample size, sample size, test length, and the correlation between the different abilities affect the estimate quality. Comparative analysis of simulation results indicated that the new indices are very desirable to evaluate test-retest consistency and correct classification rate of different decision rules.

Irreplaceability of a Reachability Matrix

Q-matrix is a critical concept for cognitive diagnosis. Reachability matrix R is special in its two important properties: (1) any column in the Q-matrix can be expressed by a linear combination of R’s columns; (2) there is a one-to-one mapping from the set of knowledge states to the set of ideal response patterns if R is a sub-matrix of the test Q-matrix. It is proved that these properties are irreplaceable, i.e., no other upper triangular Q-matrices have these properties.

An EM-Based Method for Q-Matrix Validation:

With the purpose to assist the subject matter experts in specifying their Q-matrices, the authors used expectation–maximization (EM)–based algorithm to investigate three alternative Q-matrix validation methods, namely, the maximum likelihood estimation (MLE), the marginal maximum likelihood estimation (MMLE), and the intersection and difference (ID) method. Their efficiency was compared, respectively, with that of the sequential EM-based δ method and its extension (ς2), the γ method, and the nonparametric method in terms of correct recovery rate, true negative rate, and true positive rate under the deterministic-inputs, noisy “and” gate (DINA) model and the reduced reparameterized unified model (rRUM). Simulation results showed that for the rRUM, the MLE performed better for low-quality tests, whereas the MMLE worked better for high-quality tests. For the DINA model, the ID method tended to produce better quality Q-matrix estimates than other methods for large sample sizes (i.e., 500 or 1,000). In addition, the Q-matrix was more precisely estimated under the discrete uniform distribution than under the multivariate normal threshold model for all the above methods. On average, the ς2 and ID method with higher true negative rates are better for correcting misspecified Q-entries, whereas the MLE with higher true positive rates is better for retaining the correct Q-entries. Experiment results on real data set confirmed the effectiveness of the MLE.

An Exploratory Discrete Factor Loading Method for Q-Matrix Specification in Cognitive Diagnostic Models

The Q-matrix is usually unknown for many existing tests. If the Q-matrix is specified by subject matter experts but contains a large amount of misspecification, it will be difficult for the recovery of a high-quality Q-matrix through a validation method, because the performance of the validation method relies on the quality of a provisional Q-matrix. Under these two situations above, an exploratory technique is necessary. The purpose of this study is to explore a simple method for Q-matrix specification, called a discretized factor loading (DFL) method, in which exploratory factor analysis regarding latent attributes as latent factors is used to estimate a factor loading matrix after which a discretization process is employed on the factor loading matrix to obtain a binary Q-matrix. A series of simulation studies were conducted to investigate the performance of the DFL method under various conditions. The simulation results showed that the DFL method can provide a high-quality provisional Q-matrix.

Bayesian Network for Modeling Uncertainty in Attribute Hierarchy

In the attribute hierarchy method, cognitive attributes are assumed to be organized hierarchically. Content specialists usually conduct a task analysis on a sample of items to specify the cognitive attributes required by the correctly answered items, and to order these attributes to create an attribute hierarchy. However, the problem-solving performance of experts and novices was almost certain to be different. Additionally, experts’ knowledge is highly organized in deeply integrated schemas, while a novice views domain knowledge and problem-solving knowledge separately. Thus, this may bring uncertainty into the attribute hierarchy and lead to different attribute hierarchies for a test. Formally, a Bayesian network is a probabilistic graphical model that represents a set of random latent attributes or variables and their conditional dependencies via a directed acyclic graph. For example, a Bayesian network can be used to represent the probabilistic relationships between latent attributes in the attribute hierarchy. The purpose of this study is to apply Bayesian network for modeling uncertainty in an attribute hierarchy. The Bayesian network created from the attribute hierarchy, which is regarded as a flexible high-order model, is incorporated into three cognitive diagnostic models. The new model has an advantage of taking an account of subjectivity of the attribute hierarchy specified by experts with the uncertainty of item responses. Fraction subtraction data were analyzed to evaluate the performance of the new model.

A Cognitive Diagnosis Method Based on Mahalanobis Distance

Cognitive diagnosis methods (CDMs) is very important for cognitive diagnosis, the primary purpose for CDMs is to classify examinees into mutually exclusive categories. Although there exist many CDMs, researchers propose many better new CDMs. Among them, the generalized distance discrimination (GDD) and the Hamming distance discrimination (HDD) receive more and more attention for their advantages of simple and easy to use, high classification accuracy, thus, Mahalanobis distance discrimination (MDD), a generalized CDM is introduced. GDD and HDD are its special cases. Mahalanobis distance (MD) is employed for MDD to calculate the distance between an examinee’s observed response pattern (ORP) and all kinds of ideal response pattern (IRP). The Shannon entropy is specified as covariance. According to the principle of minimum distance and designing test blueprint, IRP can be bijection mapped to the state of knowledge. Under dichotomous model, the pattern match ratio and average attribute match ratio are selected as the criteria for evaluating the classification accuracy. The Monte Carlo simulation study shows that the performance of MDD is better than GDD and HDD.

Different Expressions of a Knowledge State and Their Applications

Based on the Augment algorithm, any column of Q matrix can be expressed as a Boolean union of some columns of reachability matrix R, but the expression is not unique. There are two different expressions for a column of the reduced Q matrix, say x, a redundant expression of x and a concise expression of x. When a test length is short, the redundant expression of a knowledge state can be used to simplify the proof of an important property of the reachability matrix R in the design of cognitive diagnostic test, and provides a novel method to specify Q matrix. This specification method can be employed to deal with the polytomous Q matrix.

An Extension of Rudner-Based Consistency and Accuracy Indices for Multidimensional Item Response Theory

Although the field of multidimensional item response theory (MIRT) has enjoyed tremendous growth over recent years, solutions to some problems remain to be studied. One case in point is the estimate of classification accuracy and consistency indices. There have been a few research studies focusing on these indices based on total scores under MIRT. The purposes of this study are to extend Rudner-based index for MIRT under complex decision rules and to compare it with the Guo-based index and the Lee-based index. The Rudner-based index assumes that an ability estimation error follows a multivariate normal distribution around each examinee’s ability estimate, and a simple Monte Carlo method is used to estimate accuracy and consistency indices. The simulation results showed that the Rudner-based index worked well under various conditions. Finally, conclusions are described along with thoughts for future research.

Application Study on Online Multistage Intelligent Adaptive Testing for Cognitive Diagnosis

“On-the-fly assembled multistage adaptive Testing (OMST)” provides some unique advantages for both Computerized Adaptive Testing (CAT) and Multistage Testing (MST). In OMST, not one but multiple items are assembled on the fly into one unit in each stage. We apply the idea of OMST to Cognitive Diagnosis CAT (CD-CAT), name it as Online Multistage Intelligent Adaptive Testing (OMIAT), which aims to accurately estimate both examinees’ latent ability level and their knowledge state (KS) simultaneously. A simulation study was conducted to five different item selection methods in CD-CAT: OMIAT method, Shannon Entropy (SHE) method, Aggregate standardized information (ASI) method, Maximum Fisher Information (MFI) method, and random method. The result shows that: (1) both the OMIAT and the ASI methods can not only measure the ability level with precision, but also classify the examinee’s KS with accuracy. In most cases, the OMIAT method is superior to the ASI method in terms of the evaluation criteria, especially when the number of attributes, which is required to respond correctly to the item, is small (<=2). (2) The pattern classification correct rate of the SHE method is always the highest and that of the OMIAT method is always second, but the item exposure rate and the time consumption of the OMIAT method is far superior to those of the SHE method.