Ryan S. Baker

Learning analytics and educational data mining: towards communication and collaboration

Growing interest in data and analytics in education, teaching, and learning raises the priority for increased, high-quality research into the models, methods, technologies, and impact of analytics. Two research communities -- Educational Data Mining (EDM) and Learning Analytics and Knowledge (LAK) have developed separately to address this need. This paper argues for increased and formal communication and collaboration between these communities in order to share research, methods, and tools for data mining and analysis in the service of developing both LAK and EDM fields.

The Beginning of a Beautiful Friendship? Intelligent Tutoring Systems and MOOCs

A key challenge in ITS research and development is to support tutoring at scale, for example by embedding tutors in MOOCs. An obstacle to at-scale deployment is that ITS architectures tend to be complex, not easily deployed in browsers without significant server-side processing, and not easily embedded in a learning management system (LMS). We present a case study in which a widely used ITS authoring tool suite, CTAT/TutorShop, was modified so that tutors can be embedded in MOOCs. Specifically, the inner loop (the example-tracing tutor engine) was moved to the client by reimplementing it in JavaScript, and the tutors were made compatible with the LTI e-learning standard. The feasibility of this general approach to ITS/MOOC integration was demonstrated with simple tutors in an edX MOOC “Data Analytics and Learning.”

Adapting to when students game an intelligent tutoring system

It has been found in recent years that many students who use intelligent tutoring systems game the system, attempting to succeed in the educational environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. In this paper, we introduce a system which gives a gaming student supplementary exercises focused on exactly the material the student bypassed by gaming, and which also expresses negative emotion to gaming students through an animated agent. Students using this system engage in less gaming, and students who receive many supplemental exercises have considerably better learning than is associated with gaming in the control condition or prior studies.

A review of recent advances in learner and skill modeling in intelligent learning environments

In recent years, learner models have emerged from the research laboratory and research classrooms into the wider world. Learner models are now embedded in real world applications which can claim to have thousands, or even hundreds of thousands, of users. Probabilistic models for skill assessment are playing a key role in these advanced learning environments. In this paper, we review the learner models that have played the largest roles in the success of these learning environments, and also the latest advances in the modeling and assessment of learner skills. We conclude by discussing related advancements in modeling other key constructs such as learner motivation, emotional and attentional state, meta-cognition and self-regulated learning, group learning, and the recent movement towards open and shared learner models.

Educational Data Mining and Learning Analytics

In recent years, two communities have grown around a joint interest on how big data can be exploited to benefit education and the science of learning: Educational Data Mining and Learning Analytics. This article discusses the relationship between these two communities, and the key methods and approaches of educational data mining. The article discusses how these methods emerged in the early days of research in this area, which methods have seen particular interest in the EDM and learning analytics communities, and how this has changed as the field matures and has moved to making significant contributions to both educational research and practice.

Modeling and understanding students' off-task behavior in intelligent tutoring systems

We present a machine-learned model that can automatically detect when a student using an intelligent tutoring system is off-task, i.e., engaged in behavior which does not involve the system or a learning task. This model was developed using only log files of system usage (i.e. no screen capture or audio/video data). We show that this model can both accurately identify each students prevalence of off-task behavior and can distinguish off-task behavior from when the student is talking to the teacher or another student about the subject matter. We use this model in combination with motivational and attitudinal instruments, developing a profile of the attitudes and motivations associated with off-task behavior, and compare this profile to the attitudes and motivations associated with other behaviors in intelligent tutoring systems. We discuss how the model of off-task behavior can be used within interactive learning environments which respond to when students are off-task.

Developing a generalizable detector of when students game the system

Some students, when working in interactive learning environments, attempt to “game the system”, attempting to succeed in the environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. In this paper, we present a system that can accurately detect whether a student is gaming the system, within a Cognitive Tutor mathematics curricula. Our detector also distinguishes between two distinct types of gaming which are associated with different learning outcomes. We explore this detector’s generalizability, and find that it transfers successfully to both new students and new tutor lessons.

Affective states and state tests: investigating how affect throughout the school year predicts end of year learning outcomes

In this paper, we investigate the correspondence between student affect in a web-based tutoring platform throughout the school year and learning outcomes at the end of the year, on a high-stakes mathematics exam. The relationships between affect and learning outcomes have been previously studied, but not in a manner that is both longitudinal and finer-grained. Affect detectors are used to estimate student affective states based on post-hoc analysis of tutor log-data. For every student action in the tutor the detectors give us an estimated probability that the student is in a state of boredom, engaged concentration, confusion, and frustration, and estimates of the probability that they are exhibiting off-task or gaming behaviors. We ran the detectors on two years of log-data from 8th grade student use of the ASSISTments math tutoring system and collected corresponding end of year, high stakes, state math test scores for the 1,393 students in our cohort. By correlating these data sources, we find that boredom during problem solving is negatively correlated with performance, as expected; however, boredom is positively correlated with performance when exhibited during scaffolded tutoring. A similar pattern is unexpectedly seen for confusion. Engaged concentration and frustration are both associated with positive learning outcomes, surprisingly in the case of frustration.

Educational Data Mining and Learning Analytics: Applications to Constructionist Research

AbstractConstructionism can be a powerful framework for teaching complex content to novices. At the core of constructionism is the suggestion that by enabling learners to build creative artifacts that require complex content to function, those learners will have opportunities to learn this content in contextualized, personally meaningful ways. In this paper, we investigate the relevance of a set of approaches broadly called “educational data mining” or “learning analytics” (henceforth, EDM) to help provide a basis for quantitative research on constructionist learning which does not abandon the richness seen as essential by many researchers in that paradigm. We suggest that EDM may have the potential to support research that is meaningful and useful both to researchers working actively in the constructionist tradition but also to wider communities. Finally, we explore potential collaborations between researchers in the EDM and constructionist traditions; such collaborations have the potential to enhance the ability of constructionist researchers to make rich inferences about learning and learners, while providing EDM researchers with many interesting new research questions and challenges.

Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill

We present work toward automatically assessing and estimating science inquiry skills as middle school students engage in inquiry within a physical science microworld. Towards accomplishing this goal, we generated machine-learned models that can detect when students test their articulated hypotheses, design controlled experiments, and engage in planning behaviors using two inquiry support tools. Models were trained using labels generated through a new method of manually hand-coding log files, “text replay tagging”. This approach led to detectors that can automatically and accurately identify these inquiry skills under student-level cross-validation. The resulting detectors can be applied at run-time to drive scaffolding intervention. They can also be leveraged to automatically score all practice attempts, rather than hand-classifying them, and build models of latent skill proficiency. As part of this work, we also compared two approaches for doing so, Bayesian Knowledge-Tracing and an averaging approach that assumes static inquiry skill level. These approaches were compared on their efficacy at predicting skill before a student engages in an inquiry activity, predicting performance on a paper-style multiple choice test of inquiry, and predicting performance on a transfer task requiring data collection skills. Overall, we found that both approaches were effective at estimating student skills within the environment. Additionally, the models’ skill estimates were significant predictors of the two types of inquiry transfer tests.