Jaclyn Ocumpaugh

Towards automatically detecting whether student learning is shallow

Recent research has extended student modeling to infer not just whether a student knows a skill or set of skills, but also whether the student has achieved robust learning --- learning that leads the student to be able to transfer their knowledge and prepares them for future learning (PFL). However, a student may fail to have robust learning in two fashions: they may have no learning, or they may have shallow learning (learning that applies only to the current skill, and does not support transfer or PFL). Within this paper, we present an automated detector which is able to identify shallow learners, who are likely to need different intervention than students who have not yet learned at all. This detector is developed using a step regression approach, with data from college students learning introductory genetics from an intelligent tutoring system.

Using Video to Automatically Detect Learner Affect in Computer-Enabled Classrooms

Affect detection is a key component in intelligent educational interfaces that respond to students’ affective states. We use computer vision and machine-learning techniques to detect students’ affect from facial expressions (primary channel) and gross body movements (secondary channel) during interactions with an educational physics game. We collected data in the real-world environment of a school computer lab with up to 30 students simultaneously playing the game while moving around, gesturing, and talking to each other. The results were cross-validated at the student level to ensure generalization to new students. Classification accuracies, quantified as area under the receiver operating characteristic curve (AUC), were above chance (AUC of 0.5) for all the affective states observed, namely, boredom (AUC = .610), confusion (AUC = .649), delight (AUC = .867), engagement (AUC = .679), frustration (AUC = .631), and for off-task behavior (AUC = .816). Furthermore, the detectors showed temporal generalizability in that there was less than a 2% decrease in accuracy when tested on data collected from different times of the day and from different days. There was also some evidence of generalizability across ethnicity (as perceived by human coders) and gender, although with a higher degree of variability attributable to differences in affect base rates across subpopulations. In summary, our results demonstrate the feasibility of generalizable video-based detectors of naturalistic affect in a real-world setting, suggesting that the time is ripe for affect-sensitive interventions in educational games and other intelligent interfaces.

Modeling how incoming knowledge, persistence, affective states, and in-game progress influence student learning from an educational game

This study investigated the relationships among incoming knowledge, persistence, affective states, in-game progress, and consequently learning outcomes for students using the game Physics Playground. We used structural equation modeling to examine these relations. We tested three models, obtaining a model with good fit to the data. We found evidence that both the pretest and the in-game measure of student performance significantly predicted learning outcome, while the in-game measure of performance was predicted by pretest data, frustration, and engaged concentration. Moreover, we found evidence for two indirect paths from engaged concentration and frustration to learning, via the in-game progress measure. We discuss the importance of these findings, and consider viable next steps concerning the design of effective learning supports within game environments. We model relations among various student variables and learning outcome in a game.Pretest and in-game performance significantly predict learning outcome.In-game performance is predicted by pretest data, frustration, and engagement.Two indirect paths involving frustration and engagement predict learning.

Extending Log-Based Affect Detection to a Multi-User Virtual Environment for Science

The application of educational data mining (EDM) techniques to interactive learning software is increasingly being used to broaden the range of constructs typically incorporated in student models, moving from traditional assessment of student knowledge to the assessment of engagement, affect, strategy, and metacognition. Researchers are also broadening the range of environments within which these constructs are assessed. In this study, we develop sensor-free affect detection for EcoMUVE, an immersive multi-user virtual environment that teaches middle-school students about casualty in ecosystems. In this study, models were constructed for five different educationally-relevant affective states (boredom, confusion, delight, engaged concentration, and frustration). Such models allow us to examine the behaviors most closely associated with particular affective states, paving the way for the design of adaptive personalization to improve engagement and learning.

Towards general models of effective science inquiry in virtual performance assessments

Recent interest in online assessment of scientific inquiry has led to several new online systems that attempt to assess these skills, but producing models that detect when students are successfully practising these skills can be challenging. In this paper, we study models that assess student inquiry in an immersive virtual environment, where a student navigates an avatar around a world, speaking to in-game characters, collecting samples and conducting scientific tests with those samples in the virtual laboratory. To this goal, we leverage log file data from nearly 2000 middle school students using virtual performance assessment VPA, a software system where students practice inquiry skills in different virtual scenarios. We develop models of student interaction within VPA that predict whether a student will successfully conduct scientific inquiry. Specifically, we identify behaviours that lead to distinguishing causal from non-causal factors to identify a correct final conclusion and to design a causal explanation about these conclusions. We then demonstrate that these models can be adapted with minimal effort between VPA scenarios. We conclude with discussions of how these models serve as a tool for better understanding scientific inquiry in virtual environments and as a platform for the future design of evidence-based interventions.

HART: the human affect recording tool

This paper evaluates the Human Affect Recording Tool (HART), a Computer Assisted Direct Observation (CADO) application that facilitates scientific sampling. HART enforces an established method for systematic direct observation in Educational Data Mining (EDM) research, the Baker Rodrigo Ocumpaugh Monitoring Protocol [25] [26]. This examination provides insight into the design of HART for rapid data collection for both formative classroom assessment and educational research. It also discusses the possible extension of these tools to other domains of affective computing and human computer interaction.

Field Observations of Engagement in Reasoning Mind

This study presents Quantitative Field Observations (QFOs) of educationally relevant affect and behavior among students at three schools using Reasoning Mind, a game-based software system designed to teach elementary-level mathematics. High levels of engagement are observed. Possible causes for these high levels of engagement are considered, including the interactive pedagogical agent and other design elements.

Cross-System Transfer of Machine Learned and Knowledge Engineered Models of Gaming the System

Replicable research on the behavior known as gaming the system, in which students try to succeed by exploiting the functionalities of a learning environment instead of learning the material, has shown it is negatively correlated with learning outcomes. As such, many have developed models that can automatically detect gaming behaviors, towards deploying them in online learning environments. Both machine learning and knowledge engineering approaches have been used to create models for a variety of software systems, but the development of these models is often quite time consuming. In this paper, we investigate how well different kinds of models generalize across learning environments, specifically studying how effectively four gaming models previously created for the Cognitive Tutor Algebra tutoring system function when applied to data from two alternate learning environments: the scatterplot lesson of Cognitive Tutor Middle School and ASSISTments. Our results suggest that the similarity between the systems our model are transferred between and the nature of the approach used to create the model impact transfer to new systems.

Contextual factors affecting hint utility

BackgroundInteractive learning environments often provide help strategies to facilitate learning. Hints, for example, help students recall relevant concepts, identify mistakes, and make inferences. However, several studies have shown cases of ineffective help use. Findings from an initial study on the availability of hints in a mathematics problem-solving activity showed that early access to on-demand hints were linked to lack of performance improvements and longer completion times in students answering problems for summer work. The same experimental methodology was used in the present work with a different student sample population collected during the academic year to check for generalizability.ResultsResults from the academic year study showed that early access to on-demand-hints in an online mathematics assignment significantly improved student performance compared to students with later access to hints, which was not observed in the summer study. There were no differences in assignment completion time between conditions, which had been observed in the summer study and has been attributed to engagement in off-task activities. Although the summer and academic year studies were internally valid, there were significantly more students in the academic year study who did not complete their assignment. The sample populations differed significantly by student characteristics and external factors, possibly contributing to differences in the findings. Notable contextual factors that differed included prior knowledge, grade level, and assignment deadlines.ConclusionsContextual differences influence hint effectiveness. This work found varying results when the same experimental methodology was conducted on two separate sample populations engaged in different learning settings. Further work is needed, however, to better understand how on-demand hints generalize to other learning contexts. Despite its limitations, the study shows how randomized controlled trials can be used to better understand the effectiveness of instructional designs applied in online learning systems that cater to thousands of learners across diverse student populations. We hope to encourage additional research that will validate the effectiveness of instructional designs in different learning contexts, paving the way for the development of robust and generalizable designs.

Temporal Generalizability of Face-Based Affect Detection in Noisy Classroom Environments

The goal of this paper was to explore the possibility of generalizing face-based affect detectors across multiple days, a problem which plagues physiological-based affect detection. Videos of students playing an educational physics game were collected in a noisy computer-enabled classroom environment where students conversed with each other, moved around, and gestured. Trained observers provided real-time annotations of learning-centered affective states (e.g., boredom, confusion) as well as off-task behavior. Detectors were trained using data from one day and tested on data from different students on another day. These cross-day detectors demonstrated above chance classification accuracy with average Area Under the ROC Curve (AUC, .500 is chance level) of .658, which was similar to within-day (training and testing on data collected on the same day) AUC of .667. This work demonstrates the feasibility of generalizing face-based affect detectors across time in an ecologically valid computer-enabled classroom environment.