Kristy Elizabeth Boyer

Balancing Cognitive and Motivational Scaffolding in Tutorial Dialogue

A key challenge in the design of tutorial dialogue systems is identifying tutorial strategies that can effectively balance the tradeoffs between cognitive and affective student outcomes. This balance is problematic because the precise nature of the interdependence between cognitive and affective strategies is not well understood. Furthermore, previous studies suggest that some cognitive and motivational goals are at odds with one another because a tutorial strategy designed to maximize one may negatively impact the other. This paper reports on a tutorial dialogue study that investigates motivational strategies and cognitive feedback. It was found that the choice of corrective tutorial strategy makes a significant difference in the outcomes of both student learning gains and self-efficacy gains.

A development environment for distributed synchronous collaborative programming

While collaborative approaches in the classroom have been shown to be highly beneficial for students of computer science, obstacles inherent in todays academic environment often prevent collocated collaborative approaches from being implemented. One solution to the collocation problem may lie with tools that facilitate distributed collaboration. This paper presents R IPPLE (Remote Interactive Pair Programming and Learning Environment), a development environment for distributed synchronous collaborative programming. R IPPLE is an open source software tool. Initial user tests demonstrate positive responses from students, and the potential for long term learning, motivation, and retention benefits is significant. In addition to its benefits for students, R IPPLE is a tool for computing education researchers who wish to collect data on collaborative programming.

Automatically Recognizing Facial Indicators of Frustration: A Learning-centric Analysis

Affective and cognitive processes form a rich substrate on which learning plays out. Affective states often influence progress on learning tasks, resulting in positive or negative cycles of affect that impact learning outcomes. Developing a detailed account of the occurrence and timing of cognitive-affective states during learning can inform the design of affective tutorial interventions. In order to advance understanding of learning-centered affect, this paper reports on a study to analyze a video corpus of computer-mediated human tutoring using an automated facial expression recognition tool that detects fine-grained facial movements. The results reveal three significant relationships between facial expression, frustration, and learning: (1) Action Unit 2 (outer brow raise) was negatively correlated with learning gain, (2) Action Unit 4 (brow lowering) was positively correlated with frustration, and (3) Action Unit 14 (mouth dimpling) was positively correlated with both frustration and learning gain. Additionally, early prediction models demonstrated that facial actions during the first five minutes were significantly predictive of frustration and learning at the end of the tutoring session. The results represent a step toward a deeper understanding of learning-centered affective states, which will form the foundation for data-driven design of affective tutoring systems.

Characterizing the effectiveness of tutorial dialogue with hidden markov models

Identifying effective tutorial dialogue strategies is a key issue for intelligent tutoring systems research. Human-human tutoring offers a valuable model for identifying effective tutorial strategies, but extracting them is a challenge because of the richness of human dialogue. This paper addresses that challenge through a machine learning approach that 1) learns tutorial strategies from a corpus of human tutoring, and 2) identifies the statistical relationships between student outcomes and the learned strategies. We have applied hidden Markov modeling to a corpus of annotated task-oriented tutorial dialogue to learn one model for each of two effective human tutors. We have identified significant correlations between the automatically extracted tutoring modes and student learning outcomes. This work has direct applications in authoring data-driven tutorial dialogue system behavior and in investigating the effectiveness of human tutoring.

Modeling Dialogue Structure with Adjacency Pair Analysis and Hidden Markov Models

Automatically detecting dialogue structure within corpora of human-human dialogue is the subject of increasing attention. In the domain of tutorial dialogue, automatic discovery of dialogue structure is of particular interest because these structures inherently represent tutorial strategies or modes, the study of which is key to the design of intelligent tutoring systems that communicate with learners through natural language. We propose a methodology in which a corpus of human-human tutorial dialogue is first manually annotated with dialogue acts. Dependent adjacency pairs of these acts are then identified through X2 analysis, and hidden Markov modeling is applied to the observed sequences to induce a descriptive model of the dialogue structure.

The Additive Value of Multimodal Features for Predicting Engagement, Frustration, and Learning during Tutoring

Detecting learning-centered affective states is difficult, yet crucial for adapting most effectively to users. Within tutoring in particular, the combined context of student task actions and tutorial dialogue shape the students affective experience. As we move toward detecting affect, we may also supplement the task and dialogue streams with rich sensor data. In a study of introductory computer programming tutoring, human tutors communicated with students through a text-based interface. Automated approaches were leveraged to annotate dialogue, task actions, facial movements, postural positions, and hand-to-face gestures. These dialogue, nonverbal behavior, and task action input streams were then used to predict retrospective student self-reports of engagement and frustration, as well as pretest/posttest learning gains. The results show that the combined set of multimodal features is most predictive, indicating an additive effect. Additionally, the findings demonstrate that the role of nonverbal behavior may depend on the dialogue and task context in which it occurs. This line of research identifies contextual and behavioral cues that may be leveraged in future adaptive multimodal systems.

Predicting facial indicators of confusion with hidden Markov models

Affect plays a vital role in learning. During tutoring, particular affective states may benefit or detract from student learning. A key cognitiveaffective state is confusion, which has been positively associated with effective learning. Although identifying episodes of confusion presents significant challenges, recent investigations have identified correlations between confusion and specific facial movements. This paper builds on those findings to create a predictive model of learner confusion during task-oriented human-human tutorial dialogue. The model leverages textual dialogue, task, and facial expression history to predict upcoming confusion within a hidden Markov modeling framework. Analysis of the model structure also reveals meaningful modes of interaction within the tutoring sessions. The results demonstrate that because of its predictive power and rich qualitative representation, the model holds promise for informing the design of affective-sensitive tutoring systems.

How Early Does the CS Gender Gap Emerge?: A Study of Collaborative Problem Solving in 5th Grade Computer Science

Elementary computer science has gained increasing attention within the computer science education research community. We have only recently begun to explore the many unanswered questions about how young students learn computer science, how they interact with each other, and how their skill levels and backgrounds vary. One set of unanswered questions focuses on gender equality for young computer science learners. This paper examines how the gender composition of collaborative groups in elementary computer science relates to student achievement. We report on data collected from an in-school 5th grade computer science elective offered over four quarters in 2014-2015. We found a significant difference in the quality of artifacts produced by learner groups depending upon their gender composition, with groups of all female students performing significantly lower than other groups. Our analyses suggest important factors that are influential as these learners begin to solve computer science problems. This new evidence of gender disparities in computer science achievement as young as ten years of age highlights the importance of future study of these factors in order to provide effective, equitable computer science education to learners of all ages.

Identifying Effective Moves in Tutoring: On the Refinement of Dialogue Act Annotation Schemes

The rich natural language dialogue that is exchanged between tutors and students has inspired many successful lines of research on tutorial dialogue systems. Yet, today’s tutorial dialogue systems do not regularly achieve the same level of student learning gain as has been observed with expert human tutors. Implementing models directly informed by, and even machine-learned from, human-human tutorial dialogue is highly promising. With this goal in mind, this paper makes two contributions to tutorial dialogue systems research. First, it presents a dialogue act annotation scheme that is designed specifically to address a common weakness within dialogue act tag sets, namely, their dominance by a single large majority dialogue act class. Second, using this new fine-grained annotation scheme, the paper describes important correlations uncovered between tutor dialogue acts and student learning gain within a corpus of tutorial dialogue for introductory computer science. These findings can inform the design of future tutorial dialogue systems by suggesting ways in which systems can adapt at a fine-grained level to student actions.

Embodied Affect in Tutorial Dialogue: Student Gesture and Posture

Recent years have seen a growing recognition of the central role of affect and motivation in learning. In particular, nonverbal behaviors such as posture and gesture provide key channels signaling affective and motivational states. Developing a clear understanding of these mechanisms will inform the development of personalized learning environments that promote successful affective and motivational outcomes. This paper investigates posture and gesture in computer-mediated tutorial dialogue using automated techniques to track posture and hand-to-face gestures. Annotated dialogue transcripts were analyzed to identify the relationships between student posture, student gesture, and tutor and student dialogue. The results indicate that posture and hand-to-face gestures are significantly associated with particular tutorial dialogue moves. Additionally, two-hands-to-face gestures occurred significantly more frequently among students with low self-efficacy. The results shed light on the cognitive-affective mechanisms that underlie these nonverbal behaviors. Collectively, the findings provide insight into the interdependencies among tutorial dialogue, posture, and gesture, revealing a new avenue for automated tracking of embodied affect during learning.