Elaine Farrow

Using Natural Language Processing to Analyze Tutorial Dialogue Corpora Across Domains Modalities

Our research goal is to investigate whether previous findings and methods in the area of tutorial dialogue can be generalized across dialogue corpora that differ in domain (mechanics versus electricity in physics), modality (spoken versus typed), and tutor type (computer versus human). We first present methods for unifying our prior coding and analysis methods. We then show that many of our prior findings regarding student dialogue behaviors and learning not only generalize across corpora, but that our methodology yields additional new findings. Finally, we show that natural language processing can be used to automate some of these analyses.

BEETLE II: Deep natural language understanding and automatic feedback generation for intelligent tutoring in basic electricity and electronics

Within STEM domains, physics is considered to be one of the most difficult topics to master, in part because many of the underlying principles are counter-intuitive. Effective teaching methods rely on engaging the student in active experimentation and encouraging deep reasoning, often through the use of self-explanation. Supporting such instructional approaches poses a challenge for developers of Intelligent Tutoring Systems. We describe a system that addresses this challenge by teaching conceptual knowledge about basic electronics and electricity through guided experimentation with a circuit simulator and reflective dialogue to encourage effective self-explanation. The Basic Electricity and Electronics Tutorial Learning Environment (BEETLE II) advances the state of the art in dynamic adaptive feedback generation and natural language processing (NLP) by extending symbolic NLP techniques to support unrestricted student natural language input in the context of a dynamically changing simulation environment in a moderately complex domain. This allows contextually-appropriate feedback to be generated “on the fly” without requiring curriculum designers to anticipate possible student answers and manually author multiple feedback messages. We present the results of a system evaluation. Our curriculum is highly effective, achieving effect sizes of 1.72 when comparing pre- to post-test learning gains from our system to those of a no-training control group. However, we are unable to demonstrate that dynamically generated feedback is superior to a non-NLP feedback condition. Evaluation of interpretation quality demonstrates its link with instructional effectiveness, and provides directions for future research and development.

Intelligent tutoring with natural language support in the BEETLE II system

We present Beetle II, a tutorial dialogue system designed to accept unrestricted language input and support experimentation with different tutorial planning and dialogue strategies. Our first system evaluation used two different tutoring policies and demonstrated that BEETLE II can be successfully used as a platform to study the impact of different approaches to tutoring. In the future, the system can also be used to experiment with a variety of parameters that may affect learning in intelligent tutoring systems.

Pilot randomised controlled trial of Help4Mood, an embodied virtual agent-based system to support treatment of depression

Introduction Help4Mood is an interactive system with an embodied virtual agent (avatar) to assist in self-monitoring of patients receiving treatment for depression. Help4Mood supports self-report and biometric monitoring and includes elements of cognitive behavioural therapy. We aimed to evaluate system use and acceptability, to explore likely recruitment and retention rates in a clinical trial and to obtain an estimate of potential treatment response with a view to conducting a future randomised controlled trial (RCT). Methods We conducted a pilot RCT of Help4Mood in three centres, in Romania, Spain and Scotland, UK. Patients with diagnosed depression (major depressive disorder) and current mild/moderate depressive symptoms were randomised to use the system for four weeks in addition to treatment as usual (TAU) or to TAU alone. Results Twenty-seven individuals were randomised and follow-up data were obtained from 21 participants (12/13 Help4Mood, 9/14 TAU). Half of participants randomised to Help4Mood used it regularly (more than 10 times); none used it every day. Acceptability varied between users. Some valued the emotional responsiveness of the system, while others found it too repetitive. Intention to treat analysis showed a small difference in change of Beck Depression Inventory II (BDI-2) scores (Help4Mood –5.7 points, TAU –4.2). Post-hoc on-treatment analysis suggested that participants who used Help4Mood regularly experienced a median change in BDI-2 of –8 points. Conclusion Help4Mood is acceptable to some patients receiving treatment for depression although none used it as regularly as intended. Changes in depression symptoms in individuals who used the system regularly reached potentially meaningful levels.

Combining Semantic Interpretation and Statistical Classification for Improved Explanation Processing in a Tutorial Dialogue System

We present an approach for combining symbolic interpretation and statistical classification in the natural language processing (NLP) component of a tutorial dialogue system. Symbolic NLP approaches support dynamic generation of context-adaptive natural language feedback, but lack robustness. In contrast, statistical classification approaches are robust to ill-formed input but provide less detail for context-specific feedback generation. We describe a system design that combines symbolic interpretation with statistical classification to support context-adaptive, dynamically generated natural language feedback, and show that the combined system significantly improves interpretation quality while retaining the adaptivity benefits of a symbolic interpreter.

The “DeMAND” coding scheme: A “common language” for representing and analyzing student discourse

We propose that a set of five dimensions forms a foundation underlying a number of prevalent theoretical perspectives on learning. We show how student contributions to instructional dialogue can be reliably annotated with these dimensions. Finally, we provide preliminary validation evidence for our coding scheme and illustrate the potential value of such an approach to analyzing student behavior in tutorial dialogue.

Adaptive Tutorial Dialogue Systems Using Deep NLP Techniques

We present tutorial dialogue systems in two different domains that demonstrate the use of dialogue management and deep natural language processing techniques. Generation techniques are used to produce natural sounding feedback adapted to student performance and the dialogue history, and context is used to interpret tentative answers phrased as questions.

Interpretation and generation in a knowledge-based tutorial system

We discuss how deep interpretation and generation can be integrated with a knowledge representation designed for question answering to build a tutorial dialogue system. We use a knowledge representation known to perform well in answering exam-type questions and show that to support tutorial dialogue it needs additional features, in particular, compositional representations for interpretation and structured explanation representations.

Generating Narratives from Personal Digital Data: Triptychs

The need for users to make sense of their growing mass of personal digital data presents a challenge to Design and HCI researchers. There is a growing interest in using narrative techniques to support the interpretation and understanding of such data. In this early study we explore methods of selecting images from personal Instagram accounts in the form of a triptych (a sequence of three images) in order to create a sense of narrative. We present a brief description of the algorithms behind image selection, evaluate how effective they are in creating a sense of narrative, and discuss the wider implications of our work. Results show that semantic tagging, a dynamic programming algorithm, and a simple narrative structure produced triptychs which were significantly more story-like, with a significantly more coherent order, than a random selection, or a neutral sequence of images.

Beetle-Grow: An Effective Intelligent Tutoring System for Data Collection

We present the Beetle-Grow intelligent tutoring system, which combines active experimentation, self-explanation, and formative feedback using natural language interaction. It runs in a standard web browser and has a fresh, engaging design. The underlying back-end system has previously been shown to be highly effective in teaching basic electricity and electronics concepts. Beetle-Grow has been designed to capture student interaction and indicators of learning in a form suitable for data mining, and to support future work on building tools for interactive tutoring that improve after experiencing interaction with students, as human tutors do.