John Champaign

Correlating skill and improvement in 2 MOOCs with a student's time on tasks

Because MOOCs offer complete logs of student activities for each student there is hope that it may be possible to find out which activities are the most useful for learning. We start this quest by examining correlations between time spent on specific course resources and various measures of student performance: score on assessments, skill as defined by Item Response Theory, improvement in skill over the period of the course, and conceptual improvement as measured by a pre-post test. We study two MOOCs offered on edX.org by MIT faculty: Circuits and Electronics (6.002x) and Mechanics Review (8.MReV). Surprisingly, we find strong negative correlations in 6.002x between student skill and resource use; we attribute these findings to the fact that students with higher initial skills can do the exercises faster and with less time spent on instructional resources. We find weak or slightly negative correlations between relative improvement and resource use in 6.002x. The correlations with learning are stronger for conceptual knowledge in 8.MReV than with relative improvement, but similar for all course activities (except that eText checkpoint questions correlate more strongly with relative improvement). Clearly, the wide distribution of demographics and initial skill in MOOCs challenges us to isolate the habits of learning and resource use that correlate with learning for different students.

Coping with poor advice from peers in peer-based intelligent tutoring: the case of avoiding bad annotations of learning objects

In this paper, we examine a challenge that arises in the application of peer-based tutoring: coping with inappropriate advice from peers. We examine an environment where students are presented with those learning objects predicted to improve their learning (on the basis of the success of previous, like-minded students) but where peers can additionally inject annotations. To avoid presenting annotations that would detract from student learning (e.g. those found confusing by other students) we integrate trust modeling, to detect over time the reputation of the annotation (as voted by previous students) and the reputability of the annotator. We empirically demonstrate, through simulation, that even when the environment is populated with a large number of poor annotations, our algorithm for directing the learning of the students is effective, confirming the value of our proposed approach for student modeling. In addition, the research introduces a valuable integration of trust modeling into educational applications.

Empowering Patients and Caregivers to Manage Healthcare Via Streamlined Presentation of Web Objects Selected by Modeling Learning Benefits Obtained by Similar Peers

In this article, we introduce a framework for selecting web objects (texts, videos, simulations) from a large online repository to present to patients and caregivers, in order to assist in their healthcare. Motivated by the paradigm of peer-based intelligent tutoring, we model the learning gains achieved by users when exposed to specific web objects in order to recommend those objects most likely to deliver benefit to new users. We are able to show that this streamlined presentation leads to effective knowledge gains, both through a process of simulated learning and through a user study, for the specific application of caring for children with autism. The value of our framework for peer-driven content selection of health information is emphasized through two additional roles for peers: attaching commentary to web objects and proposing subdivided objects for presentation, both of which are demonstrated to deliver effective learning gains, in simulations. In all, we are offering an opportunity for patients to navigate the deep waters of excessive online information towards effective management of healthcare, through content selection influenced by previous peer experiences.

AIED 2013 Simulated Learners Workshop

In their landmark paper VanLehn, Ohlsson and Nason [1] delineate three roles for simulated learners in learning systems: (i) to provide an environment in which human teachers can practise; (ii) to embed simulated learners as part of the learning environment; (iii) to provide an environment for exploring and testing learning system design issues. The second of these roles has been much explored in AIED, with the development of pedagogical agents [2] that can serve, for example, as learning companions [3] or disturbing agents, or even as tutors. In contrast, there is a paucity of research into either the first or third role for simulated learners. The main research touching on the first role is the development of teachable agents in a reciprocal learning context [4], but this is more of a pedagogical strategy for learners than it is a practice environment for teachers. As to the third role, even though VanLehn et al strongly argued that simulated learners could be used to provide both quick and deep insights about learners and pedagogy at the formative evaluation stage of the design of a learning system, there has not been much subsequent research into this role for simulated learners. There has been recent interest in opening up this third line of research again.

Ecological content sequencing: from simulated students to an effective user study

In this paper, we present an algorithm for reasoning about the sequencing of content for students in an intelligent tutoring system, influenced by McCallas ecological approach. We record with each learning object those students who experienced the object, together with their initial and final states of knowledge, and then use these interactions to reason about the most effective lesson to show future students based on their similarity to previous students. We validate our approach through a novel method of validation, providing details of the model of learning used in the simulation and the results obtained in order to demonstrate the value of our model. Beyond confirmation through simulations of student learning, we report on a study with human users and expand on a previous pilot study. We demonstrate the effectiveness of our algorithms for selection of learning objects to solidify the overall defence of our approach.

A framework to restrict viewing of peer commentary on Web objects based on trust modeling

In this paper, we present a framework aimed at assisting users in coping with the deluge of information within social networks. We focus on the scenario where a user is trying to digest feedback provided on a Web document (or a video) by peers. In this context, it is ideal for the user to be presented with a restricted view of all the commentary, namely those messages that are most beneficial in increasing the user’s understanding of the document. Operating within the computer science subfield of artificial intelligence, the centerpiece of our approach is a modeling of the trustworthiness of the person leaving commentary (the annotator), determined on the basis of ratings provided by peers, adjusted by a modeling of the similarity of those peers to the current user. We compare three competing formulae for restricting what is shown to users which vary in the extent to which they integrate trust modeling, to emphasize the value of this component. By simulating the knowledge gains achieved by users (inspired by methods used in peer-based intelligent tutoring), we are able to validate the effectiveness of our algorithms. Overall, we offer a framework to make the Social Web a viable source of information, through effective modeling of the credibility of peers. When peers are misguided or deceptive, our approach is able to remove these messages from consideration, for the user.

Simulated Learners

Simulated learners will play an increasingly important role in the design of learning environments in the coming years. Here, current research and remaining issues are considered.

Peer-Based intelligent tutoring systems: a corpus-oriented approach

Our work takes as a starting point McCallas proposed ecological approach for the design of peer-based intelligent tutoring systems and proposes three distinct directions for research The first is to develop an algorithm for selecting appropriate content (learning objects) to present to a student, based on previous learning experiences of like-minded students The second is to build on this research by also having students leaving explicit annotations on learning objects to convey refinements of their understanding to subsequent students; the challenge is to intelligently match students to those annotations that will be most beneficial for their tutoring The third is to develop methods for intelligently extracting learning objects from a repository of knowledge, in a manner that may be customized to the needs of specific students In order to develop our research we are exploring the specific application of assisting health care workers via peer-based intelligent tutoring.