Kurt VanLehn

Repair theory: A generative theory of bugs in procedural skills

This paper describes a generative theory of bugs. It claims that all bugs of a procedural skill can be derived by a highly constrained form of problem solving acting on incomplete procedures. These procedures are characterized by formal deletion operations that model incomplete learning and forgetting. The problem solver and the deletion operator have been constrained to make it impossible to derive “star-bugs”—algorithms that are so absurd that expert diagnosticians agree that the alogorithm will never be observed as a bug. Hence, the theory not only generates the observed bugs, it fails to generate star-bugs. The theory has been tested on an extensive data base of bugs for multidigit subtraction that was collected with the aid of the diagnostic systems buggy and debuggy. In addition to predicting bug occurrence, by adoption of additional hypotheses, the theory also makes predictions about the frequency and stability of bugs, as well as the occurrence of certain latencies in processing time during testing. Arguments are given that the theory can be applied to domains other than subtraction and that it can be extended to provide a theory of procedural learning that accounts for bug acquisition. Lastly, particular care has been taken to make the theory principled so that it can not be tailored to fit any possible data.

The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems

This article is a review of experiments comparing the effectiveness of human tutoring, computer tutoring, and no tutoring. “No tutoring” refers to instruction that teaches the same content without tutoring. The computer tutoring systems were divided by their granularity of the user interface interaction into answer-based, step-based, and substep-based tutoring systems. Most intelligent tutoring systems have step-based or substep-based granularities of interaction, whereas most other tutoring systems (often called CAI, CBT, or CAL systems) have answer-based user interfaces. It is widely believed as the granularity of tutoring decreases, the effectiveness increases. In particular, when compared to No tutoring, the effect sizes of answer-based tutoring systems, intelligent tutoring systems, and adult human tutors are believed to be d = 0.3, 1.0, and 2.0 respectively. This review did not confirm these beliefs. Instead, it found that the effect size of human tutoring was much lower: d = 0.79. Moreover, the effect size of intelligent tutoring systems was 0.76, so they are nearly as effective as human tutoring.

Using Bayesian Networks to Manage Uncertainty in Student Modeling

When a tutoring system aims to provide students with interactive help, it needs to know what knowledge the student has and what goals the student is currently trying to achieve. That is, it must do both assessment and plan recognition. These modeling tasks involve a high level of uncertainty when students are allowed to follow various lines of reasoning and are not required to show all their reasoning explicitly. We use Bayesian networks as a comprehensive, sound formalism to handle this uncertainty. Using Bayesian networks, we have devised the probabilistic student models for Andes, a tutoring system for Newtonian physics whose philosophy is to maximize student initiative and freedom during the pedagogical interaction. Andes’ models provide long-term knowledge assessment, plan recognition, and prediction of students’ actions during problem solving, as well as assessment of students’ knowledge and understanding as students read and explain worked out examples. In this paper, we describe the basic mechanisms that allow Andes’ student models to soundly perform assessment and plan recognition, as well as the Bayesian network solutions to issues that arose in scaling up the model to a full-scale, field evaluated application. We also summarize the results of several evaluations of Andes which provide evidence on the accuracy of its student models.

Why Do Only Some Events Cause Learning During Human Tutoring

Developers of intelligent tutoring systems would like to know what human tutors do and which activities are responsible for their success in tutoring. We address these questions by comparing episodes where tutoring does and does not cause learning. Approximately 125 hr of tutorial dialog between expert human tutors and physics students are analyzed to see what features of the dialog are associated with learning. Successful learning appears to require that the student reach an impasse. When students were not at an impasse, learning was uncommon regardless of the tutorial explanations employed. On the other hand, once students were at an impasse, tutorial explanations were sometimes associated with learning. Moreover, for different types of knowledge, different types of tutorial explanations were associated with learning different types of knowledge.

Scaffolding Deep Comprehension Strategies Through Point&Query, AutoTutor, and iSTART

It is well-documented that most students do not have adequate proficiencies in inquiry and metacognition, particularly at deeper levels of comprehension that require explanatory reasoning. The proficiencies are not routinely provided by teachers and normal tutors so it is worthwhile to turn to computer-based learning environments. This article describes some of our recent computer systems that were designed to facilitate explanation-centered learning through strategies of inquiry and metacognition while students learn science and technology content. Point&Query augments hypertext, hypermedia, and other learning environments with question-answer facilities that are under the learner control. AutoTutor and iSTART use animated conversational agents to scaffold strategies of inquiry, metacognition, and explanation construction. AutoTutor coaches students in generating answers to questions that require explanations (e.g., why, what-if, how) by holding a mixed-initiative dialogue in natural language. iSTART models and coaches students in constructing self-explanations and in applying other metacomprehension strategies while reading text. These systems have shown promising results in tests of learning gains and learning strategies.

The Architecture of Why2-Atlas: A Coach for Qualitative Physics Essay Writing

The Why2-Atlas system teaches qualitative physics by having students write paragraph-long explanations of simple mechanical phenomena. The tutor uses deep syntactic analysis and abductive theorem proving to convert the students essay to a proof. The proof formalizes not only what was said, but the likely beliefs behind what was said. This allows the tutor to uncover misconceptions as well as to detect missing correct parts of the explanation. If the tutor finds such a flaw in the essay, it conducts a dialogue intended to remedy the missing or misconceived beliefs, then asks the student to correct the essay. It often takes several iterations of essay correction and dialogue to get the student to produce an acceptable explanation. Pilot subjects have been run, and an evaluation is in progress. After explaining the research questions that the system addresses, the bulk of the paper describes the systems architecture and operation.

Learning one subprocedure per lesson

Abstract sierra is a program that learns procedures incrementally from examples, where an example is a sequence of actions. sierra learns by completing explanations. Whenever the current procedure is inadequate for explaining (parsing) the current example, sierra formulates a new subprocedure whose instantiation completes the explanation (parse tree). The key to sierra s success lies in supplying a small amount of extra information with the examples. Instead of giving it a set of examples, under which conditions correct learning is provably impossible, it is given a sequence of “lessons,” where a lesson is a set of examples that is guaranteed to introduce only one subprocedure. This permits unbiased learning, i.e., learning without a priori, heuristic preferences concerning the outcome.

Student assessment using Bayesian nets

We describe OLAE as an assessment tool that collects data from students solving problems in introductory college physics, analyses that data with probabilistic methods that determine what knowledge the student is using, and flexibly presents the results of analysis. For each problem, OLAE automatically creates a Bayesian net that relates knowledge, represented as first-order rules, to particular actions, such as written equations. Using the resulting Bayesian network, OLAE observes a students behavior and computes the probabilities that the student knows and uses each of the rules.

Toward a theory of impasse-driven learning

Learning is widely viewed as a knowledge communication process coupled with a knowledge compilation process (Anderson, 1985). The communication process interprets instruction, thereby incorporating new information from the environment into the mental structures of the student. Knowledge compilation occurs with practice. It transforms the initial mental structures into a form that makes performance faster and more accurate. Moreover, the transformed mental structures are less likely to be forgotten. At one time, psychology concerned itself exclusively with the compilation process by using such simple stimuli (e.g., nonsense syllables) that the effects of the communication process could be ignored. The work presented here uses more complicated stimuli, the calculational procedures of ordinary arithmetic. For such stimuli, the effects of the knowledge communication process cannot be ignored. Later in this chapter it is shown that certain types of miscommunication can cause students to have erroneous conceptions. The long-term objective of the research reported here is to develop a theory of the neglected half of learning, knowledge communication. The experimental methods employed are designed to show the effects of knowledge communication and hide the effects of knowledge compilation. Consequently, whenever the term learning appears, it is intended to mean knowledge communication.

Rule acquisition events in the discovery of problem-solving strategies

Although there are many machine-learning programs that can acquire new problem-solving strategies, it is not known exactly how their processes will manifest themselves in human behavior, if at all. In order to find out, a line-by-line protocol analysis was conducted of a subject discovering problem-solving strategies. A model was developed that could explain 96% of the lines in the protocol. On this analysis, the subjects learning was confined to 11 rule acquisition events, wherein she temporarily abandoned her normal problem solving and focused on improving her strategic knowledge. Further analysis showed that: (1) Not all rule acquisition events are triggered by impasses; (2) Rules are acquired gradually, both because of competition between new and old rules, and because of the subjects apparently deliberate policy of gradual generalization. (3) This subject took a scientific approach to strategy discovery, even planning and conducting small experiments.