Sidney D’Mello

Mind wandering while reading easy and difficult texts.

Mind wandering is a phenomenon in which attention drifts away from the primary task to task-unrelated thoughts. Previous studies have used self-report methods to measure the frequency of mind wandering and its effects on task performance. Many of these studies have investigated mind wandering in simple perceptual and memory tasks, such as recognition memory, sustained attention, and choice reaction time tasks. Manipulations of task difficulty have revealed that mind wandering occurs more frequently in easy than in difficult conditions, but that it has a greater negative impact on performance in the difficult conditions. The goal of this study was to examine the relation between mind wandering and task difficulty in a high-level cognitive task, namely reading comprehension of standardized texts. We hypothesized that reading comprehension may yield a different relation between mind wandering and task difficulty than has been observed previously. Participants read easy or difficult versions of eight passages and then answered comprehension questions after reading each of the passages. Mind wandering was reported using the probe-caught method from several previous studies. In contrast to the previous results, but consistent with our hypothesis, mind wandering occurred more frequently when participants read difficult rather than easy texts. However, mind wandering had a more negative influence on comprehension for the difficult texts, which is consistent with the previous data. The results are interpreted from the perspectives of the executive-resources and control-failure theories of mind wandering, as well as with regard to situation models of text comprehension.

A Motivationally Supportive Affect-Sensitive AutoTutor

This chapter describes a fully automated affect-sensitive Intelligent Tutoring System (ITS) called the Affective AutoTutor. AutoTutor is an ITS that helps students learn topics in Newtonian physics, computer literacy, and critical thinking via natural language dialogues that simulate the dialogue patterns observed in human–human tutoring. AutoTutor uses state-of-the-art natural language understanding mechanisms to model learners’ cognitive states and plan its dialogue moves in a manner that is sensitive to these states. While the original AutoTutor is sensitive to learners’ cognitive states, the affect-sensitive tutor is responsive to their affective states as well. This Affective tutor automatically detects learners’ boredom, confusion, and frustration by monitoring conversational cues, gross body language, and facial features. The sensed affective states guide the tutor’s responses in a manner that helps learners regulate their negative emotions. The tutor also synthesizes affect via the verbal content of its responses and the facial expressions and speech of an embodied pedagogical agent. An experiment comparing the affect-sensitive and nonaffective tutors indicated that the affective tutor improved learning for low-domain knowledge learners, particularly at deeper levels of comprehension.

Toward Fully Automated Person-Independent Detection of Mind Wandering

Mind wandering is a ubiquitous phenomenon where attention involuntary shifts from task-related processing to task-unrelated thoughts. Mind wandering has negative effects on performance, hence, intelligent interfaces that detect mind wandering can intervene to restore attention to the current task. We investigated the use of eye gaze and contextual cues to automatically detect mind wandering during reading with a computer interface. Participants were pseudo-randomly probed to report mind wandering instances while an eye tracker recorded their gaze during a computerized reading task. Supervised machine learning techniques detected positive responses to mind wandering probes from gaze and context features in a user-independent fashion. Mind wandering was predicted with an accuracy of 72% (expected accuracy by chance was 62%) when probed at the end of a page and an accuracy of 59% (chance was 50%) when probed in the midst of reading a page. Possible improvements to the detectors and applications are discussed.

Emotions During the Learning of Difficult Material

Abstract Students experience a variety of emotions (or cognitive-affective states) when they are assigned difficult material to learn or problems to solve. We have documented the emotions that occur while college students learn and reason about topics in science and technology. The predominant learning-centered emotions are confusion, frustration, boredom, engagement/flow, curiosity, anxiety, delight, and surprise. A cognitive disequilibrium framework provides a reasonable explanation of why and how these emotions arise during difficult tasks. The student is in the state of cognitive disequilibrium when confronting impasses and obstacles, which launches a trajectory of cognitive-affective processes until equilibrium is restored, disequilibrium is dampened, or the student disengages from the task. Most of our work has been conducted in computerized learning environments (such as AutoTutor and Operation Acquiring Research Investigative and Evaluative Skills! (ARIES)) that help students learn with pedagogical agents that hold conversations in natural language. An emotion-sensitive AutoTutor detects students emotions and adaptively responds in ways to enhance learning and motivation.

The Temporal and Dynamic Nature of Self-Regulatory Processes during Independent and Externally Assisted Hypermedia Learning.

This study examined the temporal and dynamic nature of students’ self-regulatory processes while learning about the circulatory system with hypermedia. A total of 74 undergraduate students were randomly assigned to 1 of 2 conditions: independent learning or externally assisted learning. Participants in the independent learning condition used a hypermedia environment to learn about the circulatory system on their own, while participants in the externally assisted condition used the same hypermedia environment, but were given prompts and feedback from a human tutor during the session to facilitate their self-regulatory behavior. Previously published pretest–posttest shifts toward more mature understanding of the circulatory system indicate that the externally assisted condition leads to greater learning. The present article uses think-aloud data during learning to explore process issues in light of models of self-regulated learning and conditions of engagement that may affect those processes. Results indicate that access to a human tutor influences the deployment of regulatory processes, intervals of use, and temporal dependencies. For example, there is significantly more planning during the final time interval of the learning session in the externally assisted condition; students in both conditions deploy more learning strategies in the first and second time intervals, compared to the last two time intervals. Additionally, in the externally assisted condition participants were more likely to shift from planning to monitoring and less likely to shift from learning strategies to planning. We discuss theoretical, conceptual, and methodological issues pertaining to these results, as well as implications for future research and the design of adaptive hypermedia systems.

What Emotions Do Novices Experience during Their First Computer Programming Learning Session

We conducted a study to track the emotions, their behavioral correlates, and relationship with performance when novice programmers learned the basics of computer programming in the Python language. Twenty-nine participants without prior programming experience completed the study, which consisted of a 25 minute scaffolding phase (with explanations and hints) and a 15 minute fadeout phase (no explanations or hints) with a computerized learning environment. Emotional states were tracked via retrospective self-reports in which learners viewed videos of their faces and computer screens recorded during the learning session and made judgments about their emotions at approximately 100 points. The results indicated that flow/engaged (23%), confusion (22%), frustration (14%), and boredom (12%) were the major emotions students experienced, while curiosity, happiness, anxiety, surprise, anger, disgust, fear, and sadness were comparatively rare. The emotions varied as a function of instructional scaffolds and were systematically linked to different student behaviors (idling, constructing code, running code). Boredom, flow/engaged, and confusion were also correlated with performance outcomes. Implications of our findings for affect-sensitive learning interventions are discussed.

Significant Accomplishments, New Challenges, and New Perspectives

This concluding chapter provides an integrative summative evaluation of the various threads of interdisciplinary research described in this book. After reflecting on the recent emphasis on emotions in seemingly disparate fields such as cognitive psychology, computer science, and education, we synthesize some of the important milestones achieved in the still nascent field of affect-sensitive learning technologies. These defining accomplishments include (a) an infusion of theories on emotions and learning, (b) the identification of affective states that are relevant to learning along with some of their antecedents and consequents, (c) the advance of automated affect detection systems, and (d) the emergence of some of the first fully automated affect-sensitive learning environments. Next, we highlight some of the open problems and promising areas for future research. These include (a) obtaining coherence among multiple levels of analysis, (b) modeling complex interactions between affective traits, moods, affect-elicitation events, and emotions, (c) incorporating temporal dependencies and affective dynamics into models of emotion, (d) reconceptualizing existing affect detection systems, (e) revisiting reactive emotion regulation strategies, (f) the need for proactive emotionally intelligent strategies, and (g) the importance of broadening the scope of affect and learning research so that next-generation learning technologies are consistent with the learning styles of the twenty-first century and beyond.

MIKI: a speech enabled intelligent kiosk

We introduce MIKI, a three-dimensional, directory assistance-type digital persona displayed on a prominently-positioned 50 inch plasma unit housed at the FedEx Institute of Technology at the University of Memphis. MIKI, which stands for Memphis Intelligent Kiosk Initiative, guides students, faculty and visitors through the Institute’s maze of classrooms, labs, lecture halls and offices through graphically-rich, multidimensional, interactive, touch and voice sensitive digital content. MIKI differs from other intelligent kiosk systems by its advanced natural language understanding capabilities that provide it with the ability to answer informal verbal queries without the need for rigorous phraseology. This paper describes, in general, the design, implementation, and observations of visitor reactions to the Intelligent Kiosk.

Theoretical Perspectives on Affect and Deep Learning

This chapter focuses on connections between affect and cognition that are prevalent during deep learning. Deep learning occurs when a person attempts to comprehend difficult material, to solve a difficult problem, and to make a difficult decision. We emphasize theoretical perspectives that highlight the importance of cognitive disequilibrium to deep learning and problem solving. Cognitive disequilibrium occurs when there are obstacles to goals, interruptions of organized action sequences, impasses, system breakdowns, contradictions, anomalous events, dissonance, incongruities, negative feedback, uncertainty, and deviations from norms, and novelty. Cognitive disequilibrium launches a trajectory of cognitive and affective processes such as confusion and frustration until equilibrium is restored or disequilibrium is dampened via effortful problem solving and impasse resolution. We discuss the role of cognitive and task constraints in dictating the time-course of cognitive disequilibrium and affiliated affective states such as surprise, delight, confusion, and frustration. We conclude by discussing how these states and processes are mediated by self-concepts, goals, meta-knowledge, social interaction, and the learning environment.

It’s Written on Your Face: Detecting Affective States from Facial Expressions while Learning Computer Programming

We built detectors capable of automatically recognizing affective states of novice computer programmers from student-annotated videos of their faces recorded during an introductory programming tutoring session. We used the Computer Expression Recognition Toolbox (CERT) to track facial features based on the Facial Action Coding System, and machine learning techniques to build classification models. Confusion/Uncertainty and Frustration were distinguished from all other affective states in a student-independent fashion at levels above chance (Cohen’s kappa = .22 and .23, respectively), but detection accuracies for Boredom, Flow/Engagement, and Neutral were lower (kappas = .04, .11, and .07). We discuss the differences between detection of spontaneous versus fixed (polled) judgments as well as the features used in the models.