Jaime Gálvez

A data-driven technique for misconception elicitation

When a quantitative student model is constructed, one of the first tasks to perform is to identify the domain concepts assessed In general, this task is easily done by the domain experts In addition, the model may include some misconceptions which are also identified by these experts Identifying these misconceptions is a difficult task, however, and one which requires considerable previous experience with the students In fact, sometimes it is difficult to relate these misconceptions to the elements in the knowledge diagnostic system which feeds the student model In this paper we present a data-driven technique which aims to help elicit the domain misconceptions It also aims to relate these misconceptions with the assessment activities (e.g exercises, problems or test questions), which assess the subject in question.

Data-driven student knowledge assessment through ill-defined procedural tasks

The Item Response Theory (IRT) is a statistical mechanism successfully used since the beginning of the 20th century to infer student knowledge through tests. Nevertheless, existing well-founded techniques to assess procedural tasks are generally complex and applied to well-defined tasks. In this paper, we describe how, using a set of techniques we have developed based on IRT, it is possible to infer declarative student knowledge through procedural tasks. We describe how these techniques have been used with undergraduate students, in the object oriented programming domain, through ill-defined procedural exercises.

A SOA-Based Framework for Constructing Problem Solving Environments

In this paper we present a framework for constructing problem solving environments for assessing procedural knowledge, i.e. the students ability to apply his/her knowledge in order to accomplish a task. Our proposal combines the most recent technologies for web-based development (e.g. service oriented architectures, JSF, JBoss rules, etc.) with a well-founded theory to make sound student knowledge estimations and to carry out diagnosis adaptively.

Statistical Techniques to Explore the Quality of Constraints in Constraint-Based Modeling Environments

One of the most popular student modeling approaches is Constraint-Based Modeling (CBM). It is an efficient approach that can be easily applied inside an Intelligent Tutoring System (ITS). Even with these characteristics, building new ITSs requires carefully designing the domain model to be taught because different sources of errors could affect the efficiency of the system. In this paper a novel mechanism for studying the quality of the elements in the domain model of CBM systems is presented. This mechanism combines CBM with the Item Response Theory (IRT), a data-driven technique for automatic assessment. The goal is to improve the quality of the elements that are used in problem solving environments for assessment or instruction. In this paper we propose a set of statistical techniques, i.e., the analysis of the point-biserial correlation, the Cronbach’s alpha and the information function, to explore the quality of constraints. Two different tools have been used to test this approach: a problem solving environment designed to assess students in project investment analysis; and an independent component that performs assessments using CBM and IRT. Results suggest that the three methods produce consistent diagnosis and may be complementary in some cases. In the experiments we have carried out they were able to detect faulty, bad and good quality constraints.

Data calibration for statistical-based assessment in constraint-based tutors

Item Response Theory models for constraint-based intelligent tutoring systems.Data-driven assessment of problem solving tasks.Data filtering criteria for Item Response Theory parameters estimation.Best model fit selection criteria. Intelligent Tutoring Systems (ITSs) are one of a wide range of learning environments, where the main activity is problem solving. One of the most successful approaches for implementing ITSs is Constraint-Based Modeling (CBM). Constraint-based tutors have been successfully used as drill-and-practice environments for learning. More recently CBM tutors have been complemented with a model derived from the field of Psychometrics. The goal of this synergy is to provide CBM tutors with a data-driven and sound mechanism of assessment, which mainly consists in applying the principles of Item Response Theory (IRT). The result of this synergy is, therefore, a formal approach that allows carrying out assessments of performance on problem solving tasks. Several previous studies were conducted proving the validity and utility of this combined approach with small groups of students, in short periods of time and using systems designed specifically for assessment purposes. In this paper, the approach has been extended and adapted to deal with a large set of students who used an ITS over a long period of time. The main research questions addressed in this paper are: (1) Which IRT models are more suitable to be used in a constrained-based tutor? (2) Can data collected from the ITS be used as a source for calibrating the constraints characteristic curves? (3) Which is the best strategy to assemble data for calibration? To answer these questions, we have analyzed three years of data from SQL-Tutor.