Lihong Song

Estimating Classification Accuracy and Consistency Indices for Multidimensional Latent Ability

For criterion-referenced tests, classification consistency and accuracy are viewed as important indicators for evaluating reliability and validity of classification results. Numerous procedures have been proposed in the framework of unidimensional item response theory (UIRT) to estimate these indices. Some of these were based on total sum scores, others on latent trait estimates. However, there exist very few attempts to develop them in the framework of multidimensional item response theory (MIRT). Based on previous studies, the aim of this study is first to estimate the consistency and accuracy indices of multidimensional ability estimates from a single administration of a criterion-referenced test. We also examined how Monte Carlo sample size, sample size, test length, and the correlation between the different abilities affect the estimate quality. Comparative analysis of simulation results indicated that the new indices are very desirable to evaluate test-retest consistency and correct classification rate of different decision rules.

An EM-Based Method for Q-Matrix Validation:

With the purpose to assist the subject matter experts in specifying their Q-matrices, the authors used expectation–maximization (EM)–based algorithm to investigate three alternative Q-matrix validation methods, namely, the maximum likelihood estimation (MLE), the marginal maximum likelihood estimation (MMLE), and the intersection and difference (ID) method. Their efficiency was compared, respectively, with that of the sequential EM-based δ method and its extension (ς2), the γ method, and the nonparametric method in terms of correct recovery rate, true negative rate, and true positive rate under the deterministic-inputs, noisy “and” gate (DINA) model and the reduced reparameterized unified model (rRUM). Simulation results showed that for the rRUM, the MLE performed better for low-quality tests, whereas the MMLE worked better for high-quality tests. For the DINA model, the ID method tended to produce better quality Q-matrix estimates than other methods for large sample sizes (i.e., 500 or 1,000). In addition, the Q-matrix was more precisely estimated under the discrete uniform distribution than under the multivariate normal threshold model for all the above methods. On average, the ς2 and ID method with higher true negative rates are better for correcting misspecified Q-entries, whereas the MLE with higher true positive rates is better for retaining the correct Q-entries. Experiment results on real data set confirmed the effectiveness of the MLE.

An Exploratory Discrete Factor Loading Method for Q-Matrix Specification in Cognitive Diagnostic Models

The Q-matrix is usually unknown for many existing tests. If the Q-matrix is specified by subject matter experts but contains a large amount of misspecification, it will be difficult for the recovery of a high-quality Q-matrix through a validation method, because the performance of the validation method relies on the quality of a provisional Q-matrix. Under these two situations above, an exploratory technique is necessary. The purpose of this study is to explore a simple method for Q-matrix specification, called a discretized factor loading (DFL) method, in which exploratory factor analysis regarding latent attributes as latent factors is used to estimate a factor loading matrix after which a discretization process is employed on the factor loading matrix to obtain a binary Q-matrix. A series of simulation studies were conducted to investigate the performance of the DFL method under various conditions. The simulation results showed that the DFL method can provide a high-quality provisional Q-matrix.

Bayesian Network for Modeling Uncertainty in Attribute Hierarchy

In the attribute hierarchy method, cognitive attributes are assumed to be organized hierarchically. Content specialists usually conduct a task analysis on a sample of items to specify the cognitive attributes required by the correctly answered items, and to order these attributes to create an attribute hierarchy. However, the problem-solving performance of experts and novices was almost certain to be different. Additionally, experts’ knowledge is highly organized in deeply integrated schemas, while a novice views domain knowledge and problem-solving knowledge separately. Thus, this may bring uncertainty into the attribute hierarchy and lead to different attribute hierarchies for a test. Formally, a Bayesian network is a probabilistic graphical model that represents a set of random latent attributes or variables and their conditional dependencies via a directed acyclic graph. For example, a Bayesian network can be used to represent the probabilistic relationships between latent attributes in the attribute hierarchy. The purpose of this study is to apply Bayesian network for modeling uncertainty in an attribute hierarchy. The Bayesian network created from the attribute hierarchy, which is regarded as a flexible high-order model, is incorporated into three cognitive diagnostic models. The new model has an advantage of taking an account of subjectivity of the attribute hierarchy specified by experts with the uncertainty of item responses. Fraction subtraction data were analyzed to evaluate the performance of the new model.

Different Expressions of a Knowledge State and Their Applications

Based on the Augment algorithm, any column of Q matrix can be expressed as a Boolean union of some columns of reachability matrix R, but the expression is not unique. There are two different expressions for a column of the reduced Q matrix, say x, a redundant expression of x and a concise expression of x. When a test length is short, the redundant expression of a knowledge state can be used to simplify the proof of an important property of the reachability matrix R in the design of cognitive diagnostic test, and provides a novel method to specify Q matrix. This specification method can be employed to deal with the polytomous Q matrix.

An Extension of Rudner-Based Consistency and Accuracy Indices for Multidimensional Item Response Theory

Although the field of multidimensional item response theory (MIRT) has enjoyed tremendous growth over recent years, solutions to some problems remain to be studied. One case in point is the estimate of classification accuracy and consistency indices. There have been a few research studies focusing on these indices based on total scores under MIRT. The purposes of this study are to extend Rudner-based index for MIRT under complex decision rules and to compare it with the Guo-based index and the Lee-based index. The Rudner-based index assumes that an ability estimation error follows a multivariate normal distribution around each examinee’s ability estimate, and a simple Monte Carlo method is used to estimate accuracy and consistency indices. The simulation results showed that the Rudner-based index worked well under various conditions. Finally, conclusions are described along with thoughts for future research.

New Item-Selection Methods for Balancing Test Efficiency Against Item-Bank Usage Efficiency in CD-CAT

Cognitive diagnostic computerized adaptive testing (CD-CAT) is a popular mode of online testing for cognitive diagnostic assessment (CDA). A key issue in CD-CAT programs is item-selection methods. Existing popular methods can achieve high measurement efficiencies but fail to yield balanced item-bank usage. Diagnostic tests often have low stakes, so item overexposure may not be a major concern. However, item underexposure leads to wasted time and money on item development, and high test overlap leads to intense practice effects, which in turn threaten test validity. The question is how to improve item-bank usage without sacrificing too much measurement precision (i.e., the correct recovery of knowledge states) in CD-CAT, which is the major purpose of this study. We have developed several item-selection methods that successfully meet this goal. In addition, we have investigated the Kullback–Leibler expected discrimination (KL-ED) method that considers only measurement precision except for item-bank usage.