Debora de Chiusole

Assessing Parameter Invariance in the BLIM: Bipartition Models

In knowledge space theory, the knowledge state of a student is the set of all problems he is capable of solving in a specific knowledge domain and a knowledge structure is the collection of knowledge states. The basic local independence model (BLIM) is a probabilistic model for knowledge structures. The BLIM assumes a probability distribution on the knowledge states and a lucky guess and a careless error probability for each problem. A key assumption of the BLIM is that the lucky guess and careless error probabilities do not depend on knowledge states (invariance assumption). This article proposes a method for testing the violations of this specific assumption. The proposed method was assessed in a simulation study and in an empirical application. The results show that (1) the invariance assumption might be violated by the empirical data even when the model’s fit is very good, and (2) the proposed method may prove to be a promising tool to detect invariance violations of the BLIM.

Modeling missing data in knowledge space theory.

Missing data are a well known issue in statistical inference, because some responses may be missing, even when data are collected carefully. The problem that arises in these cases is how to deal with missing data. In this article, the missingness is analyzed in knowledge space theory, and in particular when the basic local independence model (BLIM) is applied to the data. Two extensions of the BLIM to missing data are proposed: The former, called ignorable missing BLIM (IMBLIM), assumes that missing data are missing completely at random; the latter, called missing BLIM (MissBLIM), introduces specific dependencies of the missing data on the knowledge states, thus assuming that the missing data are missing not at random. The IMBLIM and the MissBLIM modeled the missingness in a satisfactory way, in both a simulation study and an empirical application, depending on the process that generates the missingness: If the missing data-generating process is of type missing completely at random, then either IMBLIM or MissBLIM provide adequate fit to the data. However, if the pattern of missingness is functionally dependent upon unobservable features of the data (e.g., missing answers are more likely to be wrong), then only a correctly specified model of the missingness distribution provides an adequate fit to the data.

Modeling Skill Dependence in Probabilistic Competence Structures

Abstract In knowledge space theory (KST) a competence structure is a set theoretical representation of the dependencies among a given set of skills. A probabilistic model for skill dependence is proposed which respects a precise correspondence requirement between set theoretical and probabilistic representations of skill dependence. An empirical application on integer subtraction problems at the primary school shows that the proposed model fits the data pretty well. Moreover, in a comparison with an unrestricted skill based version of the basic local independence model (BLIM), the proposed model fits better than this last, indicating that the restrictions implied by the correspondence requirement are not too strong.

The Gain-Loss Model: Bias of the Parameter Estimates

Abstract The gain-loss model is a formal model developed within Knowledge Space Theory. It consists of five parameters (initial probabilities of the skills, effects of learning objects on gaining and losing skills, careless error and lucky guess probabilities of the items) that are estimated by maximum likelihood. Three simulation studies show that high values of both initial and final probabilities of an item lead to a systematic overestimation of the lucky guess parameter of that item. A re-parameterization of the model is proposed, in which a joint probability of lucky guess is introduced.

Naïve Tests of Basic Local Independence Model's Invariance.

The basic local independence model (BLIM) is a probabilistic model for knowledge structures, characterized by the property that lucky guess and careless error parameters of the items are independent of the knowledge states of the subjects. When fitting the BLIM to empirical data, a good fit can be obtained even when the invariance assumption is violated. Therefore, statistical tests are needed for detecting violations of this specific assumption. This work provides an extension to theoretical results obtained by de Chiusole, Stefanutti, Anselmi, and Robusto (2013), showing that statistical tests based on the partitioning of the empirical data set into two (or more) groups are not adequate for testing the BLIMs invariance assumption. A simulation study confirms the theoretical results.

The assessment of knowledge and learning in competence spaces: The gain-loss model for dependent skills

The gain-loss model (GaLoM) is a formal model for assessing knowledge and learning. In its original formulation, the GaLoM assumes independence among the skills. Such an assumption is not reasonable in several domains, in which some preliminary knowledge is the foundation for other knowledge. This paper presents an extension of the GaLoM to the case in which the skills are not independent, and the dependence relation among them is described by a well-graded competence space. The probability of mastering skill s at the pretest is conditional on the presence of all skills on which s depends. The probabilities of gaining or losing skill s when moving from pretest to posttest are conditional on the mastery of s at the pretest, and on the presence at the posttest of all skills on which s depends. Two formulations of the model are presented, in which the learning path is allowed to change from pretest to posttest or not. A simulation study shows that models based on the true competence space obtain a better fit than models based on false competence spaces, and are also characterized by a higher assessment accuracy. An empirical application shows that models based on pedagogically sound assumptions about the dependencies among the skills obtain a better fit than models assuming independence among the skills.

A class of k-modes algorithms for extracting knowledge structures from data.

One of the most crucial issues in knowledge space theory is the construction of the so-called knowledge structures. In the present paper, a new data-driven procedure for large data sets is described, which overcomes some of the drawbacks of the already existing methods. The procedure, called k-states, is an incremental extension of the k-modes algorithm, which generates a sequence of locally optimal knowledge structures of increasing size, among which a “best” model is selected. The performance of k-states is compared to other two procedures in both a simulation study and an empirical application. In the former, k-states displays a better accuracy in reconstructing knowledge structures; in the latter, the structure extracted by k-states obtained a better fit.

An Upgrading Procedure for Adaptive Assessment of Knowledge.

In knowledge space theory, existing adaptive assessment procedures can only be applied when suitable estimates of their parameters are available. In this paper, an iterative procedure is proposed, which upgrades its parameters with the increasing number of assessments. The first assessments are run using parameter values that favor accuracy over efficiency. Subsequent assessments are run using new parameter values estimated on the incomplete response patterns from previous assessments. Parameter estimation is carried out through a new probabilistic model for missing-at-random data. Two simulation studies show that, with the increasing number of assessments, the performance of the proposed procedure approaches that of gold standards.

Testing the actual equivalence of automatically generated items

If the automatic item generation is used for generating test items, the question of how the equivalence among different instances may be tested is fundamental to assure an accurate assessment. In the present research, the question was dealt by using the knowledge space theory framework. Two different ways of considering the equivalence among instances are proposed: The former is at a deterministic level and it requires that all the instances of an item template must belong to exactly the same knowledge states; the latter adds a probabilistic level to the deterministic one. The former type of equivalence can be modeled by using the BLIM with a knowledge structure assuming equally informative instances; the latter can be modeled by a constrained BLIM. This model assumes equality constraints among the error parameters of the equivalent instances. An approach is proposed for testing the equivalence among instances, which is based on a series of model comparisons. A simulation study and an empirical application show the viability of the approach.