Maria Ofelia Clarissa Z. San Pedro

Affective states and state tests: investigating how affect throughout the school year predicts end of year learning outcomes

In this paper, we investigate the correspondence between student affect in a web-based tutoring platform throughout the school year and learning outcomes at the end of the year, on a high-stakes mathematics exam. The relationships between affect and learning outcomes have been previously studied, but not in a manner that is both longitudinal and finer-grained. Affect detectors are used to estimate student affective states based on post-hoc analysis of tutor log-data. For every student action in the tutor the detectors give us an estimated probability that the student is in a state of boredom, engaged concentration, confusion, and frustration, and estimates of the probability that they are exhibiting off-task or gaming behaviors. We ran the detectors on two years of log-data from 8th grade student use of the ASSISTments math tutoring system and collected corresponding end of year, high stakes, state math test scores for the 1,393 students in our cohort. By correlating these data sources, we find that boredom during problem solving is negatively correlated with performance, as expected; however, boredom is positively correlated with performance when exhibited during scaffolded tutoring. A similar pattern is unexpectedly seen for confusion. Engaged concentration and frustration are both associated with positive learning outcomes, surprisingly in the case of frustration.

Detecting carelessness through contextual estimation of slip probabilities among students using an intelligent tutor for mathematics

A student is said to have committed a careless error when a students answer is wrong despite the fact that he or she knows the answer (Clements, 1982). In this paper, educational data mining techniques are used to analyze log files produced by a cognitive tutor for Scatterplots to derive a model and detector for carelessness. Bayesian Knowledge Tracing and its variant, the Contextual-Slip-and-Guess Estimation, are used to model and predict carelessness behavior in the Scatterplot Tutor. The study examines as well the robustness of this detector to a major difference in the tutors interface, namely the presence or absence of an embodied conversational agent, as well as robustness to data from a different school setting (USA versus Philippines).

Towards an Understanding of Affect and Knowledge from Student Interaction with an Intelligent Tutoring System

Csikszentmihalyi’s Flow theory states that a balance between challenge and skill leads to high engagement, overwhelming challenge leads to anxiety or frustration, and insufficient challenge leads to boredom. In this paper, we test this theory within the context of student interaction with an intelligent tutoring system. Automated detectors of student affect and knowledge were developed, validated, and applied to a large data set. The results did not match Flow theory: boredom was more common for poorly-known material, and frustration was common both for very difficult material and very easy material. These results suggest that design for optimal engagement within online learning may require further study of the factors leading students to become bored on difficult material, and frustrated on very well-known material.

Carelessness and Affect in an Intelligent Tutoring System for Mathematics

We investigate the relationship between students’ affect and their frequency of careless errors while using an Intelligent Tutoring System for middle school mathematics. A student is said to have committed a careless error when the student’s answer is wrong despite knowing the skill required to provide the correct answer. We operationalize the probability that an error is careless through the use of an automated detector, developed using educational data mining, which infers the probability that an error involves carelessness rather than not knowing the relevant skill. This detector is then applied to log data produced by high-school students in the Philippines using a Cognitive Tutor for scatterplots. We study the relationship between carelessness and affect, triangulating between the detector of carelessness and field observations of affect. Surprisingly, we find that carelessness is common among students who frequently experience engaged concentration. This finding implies that a highly engaged student may paradoxically become overconfident or impulsive, leading to more careless errors. In contrast, students displaying confusion or boredom make fewer careless errors. Further analysis over time suggests that confused and bored students have lower learning overall. Thus, their mistakes appear to stem from a genuine lack of knowledge rather than carelessness.

Learning, moment-by-moment and over the long term

The development of moment-by-moment learning graphs (MBMLGs), which plot predictions about the probability that a student learned a skill at a specific time, has already helped to improve our understanding of how student performance during the learning process relates to robust learning [1]. In this study, we extend this work to study year-end learning outcomes and to account for differences in learning on original questions and within knowledge-construction scaffolds. We discuss which quantitative features of moment-by-moment learning in these two contexts are predictive of the longer-term outcomes, and conclude with potential implications for instruction.

An Integrated Look at Middle School Engagement and Learning in Digital Environments as Precursors to College Attendance

Middle school is an important phase in the academic trajectory, which plays a major role in the path to successful post-secondary outcomes such as going to college. Despite this, research on factors leading to college-going choices do not yet utilize the extensive fine-grained data now becoming available on middle school learning and engagement. This paper uses interaction-based data-mined assessments of student behavior, academic emotions and knowledge from a middle school online learning environment, and evaluates their relationships with different outcomes in high school and college. The data-mined measures of student behavior, emotions, and knowledge are used in three analyses: (1) to develop a prediction model of college attendance; (2) to evaluate their relationships to intermediate outcomes on the path to college attendance such as math and science course-taking during high school; (3) to develop an overall path model between the educational experiences students have during middle school, their high school experiences, and their eventual college attendance. This gives a richer picture of the cognitive and non-cognitive mechanisms that students experience throughout varied phases in their years in school, and how they may be related to one another. Such understanding may provide educators with information about students’ trajectories within the college pipeline.