Lishan Zhang

Evaluation of a meta-tutor for constructing models of dynamic systems

Abstract Modelling is an important skill to acquire, but it is not an easy one for students to learn. Existing instructional technology has had limited success in teaching modelling. We have applied a recently developed technology, meta-tutoring, to address the important problem of teaching model construction. More specifically, we have developed and evaluated a system that has two parts, a tutor and a meta-tutor. The tutor is a simple step-based tutoring system that can give correct/incorrect feedback on students steps and can demonstrate steps for students when asked. Because deep modelling requires difficult analyses of the quantitative relationships in a given system, we expected, and found, that students tended to avoid deep modelling by abusing the tutors help. In order to increase the frequency of deep modelling, we added a meta-tutor that coached students to follow a learning strategy that decomposed the overall modelling problem into a series of “atomic” modelling problems. We conducted three experiments to test the effectiveness of the meta-tutor. The results indicate that students who studied with meta-tutor did indeed engage in more deep modelling practices. However, when the meta-tutor and tutor were turned off, students tended to revert to shallow modelling. Thus, the next stage of the research is to add an affective agent that will try to persuade students to persist in using the taught strategies even when the meta-tutoring and tutoring have ceased.

The Affective Meta-Tutoring Project: Lessons Learned

The Affective Meta-Tutoring system is comprised of 1 a tutor that teaches system dynamics modeling, 2 a meta-tutor that teaches good strategies for learning how to model from the tutor, and 3 an affective learning companion that encourages students to use the learning strategy that the meta-tutor teaches. The affective learning companions messages are selected by using physiological sensors and log data to determine the students affective state. Evaluations compared the learning gains of three conditions: the tutor alone, the tutor plus meta-tutor and the tutor, meta-tutor and affective learning companion.

Defining the Behavior of an Affective Learning Companion in the Affective Meta-tutor Project

Research in affective computing and educational technology has shown the potential of affective interventions to increase student’s self-concept and motivation while learning. Our project aims to investigate whether the use of affective interventions in a meta-cognitive tutor can help students achieve deeper modeling of dynamic systems by being persistent in their use of meta-cognitive strategies during and after tutoring. This article is an experience report on how we designed and implemented the affective intervention. (The meta-tutor is described in a separate paper.) We briefly describe the theories of affect underlying the design and how the agent’s affective behavior is defined and implemented. Finally, the evaluation of a detector-driven categorization of student behavior, that guides the agent’s affective interventions, against a categorization performed by human coders, is presented.

A System Architecture for Affective Meta Intelligent Tutoring Systems

Intelligent Tutoring Systems ITSs constitute an alternative to expert human tutors, providing direct customized instruction and feedback to students. ITSs could positively impact education if adopted on a large scale, but doing that requires tools to enable their mass production. This circumstance is the key motivation for this work. We present a component-based approach for a system architecture for ITSs equipped with meta-tutoring and affective capabilities. We elicited the requirements that those systems might address and created a system architecture that models their structure and behavior to drive development efforts. Our experience applying the architecture in the incremental implementation of a four-year project is discussed.

Evaluation of a Meta-tutor for Constructing Models of Dynamic Systems

While modeling dynamic systems in an efficient manner is an important skill to acquire for a scientist, it is a difficult skill to acquire. A simple step-based tutoring system, called AMT, was designed to help students learn how to construct models of dynamic systems using deep modeling practices. In order to increase the frequency of deep modeling and reduce the amount of guessing/gaming, a meta-tutor coaching students to follow a deep modeling strategy was added to the original modeling tool. This paper presents the results of two experiments investigating the effectiveness of the meta-tutor when compared to the original software. The results indicate that students who studied with the meta-tutor did indeed engage more in deep modeling practices.

How do machine-generated questions compare to human-generated questions?

Science instructors need questions for use in exams, homework assignments, class discussions, reviews, and other instructional activities. Textbooks never have enough questions, so instructors must find them from other sources or generate their own questions. In order to supply biology instructors with questions for college students in introductory biology classes, two algorithms were developed. One generates questions from a formal representation of photosynthesis knowledge. The other collects biology questions from the web. The questions generated by these two methods were compared to questions from biology textbooks. Human students rated questions for their relevance, fluency, ambiguity, pedagogy, and depth. Questions were also rated by the authors according to the topic of the questions. Although the exact pattern of results depends on analytic assumptions, it appears that there is little difference in the pedagogical benefits of each class, but the questions generated from the knowledge base may be shallower than questions written by professionals. This suggests that all three types of questions may work equally well for helping students to learn.

Can a Non-Cognitive Learning Companion Increase the Effectiveness of a Meta-Cognitive Learning Strategy?

This project aimed to improve students’ learning and task performance using a non-cognitive learning companion in the context of both a tutor and a meta-tutor. The tutor taught students how to construct models of dynamic systems and the meta-tutor taught students a learning strategy. The non-cognitive learning companion was designed to increase students’ effort and persistence in using the learning strategy. It decided when to intervene and what to say using both log data and affective state monitoring via a facial expression camera and a posture sensor. Experiments with high school students showed that the non-cognitive learning companion increased students’ learning and performance. However, it had no effect on performance during a transfer phase in which the learning companion, meta-tutor, and tutor were all absent. The transfer phase null effect must be interpreted with caution due to low power, a possible floor effect, and other issues.

Adaptively selecting biology questions generated from a semantic network

ABSTRACT The paper describes a biology tutoring system with adaptive question selection. Questions were selected for presentation to the student based on their utilities, which were estimated from the chance that the student’s competence would increase if the questions were asked. Competence was represented by the probability of mastery of a set of biology knowledge components. Tasks were represented and selected based on which knowledge components they addressed. Unlike earlier work, where the knowledge components and their relationships to the questions were defined by domain experts, this project demonstrated that the knowledge components, questions and their relationships could all be generated from a semantic network. An experiment found that students using our adaptive question selection had reliably larger learning gains than students who received questions in a mal-adaptive order.