Debora Field

OpenEssayist: a supply and demand learning analytics tool for drafting academic essays

This paper focuses on the use of a natural language analytics engine to provide feedback to students when preparing an essay for summative assessment. OpenEssayist is a real-time learning analytics tool, which operates through the combination of a linguistic analysis engine that processes the text in the essay, and a web application that uses the output of the linguistic analysis engine to generate the feedback. We outline the system itself and present analysis of observed patterns of activity as a cohort of students engaged with the system for their module assignments. We report a significant positive correlation between the number of drafts submitted to the system and the grades awarded for the first assignment. We can also report that this cohort of students gained significantly higher overall grades than the students in the previous cohort, who had no access to OpenEssayist. As a system that is content free, OpenEssayist can be used to support students working in any domain that requires the writing of essays.

A prototype for a conversational companion for reminiscing about images

This paper describes an initial prototype of the Companions project (www.companions-project.org): the Senior Companion (SC), designed to be a platform to display novel approaches to:(1)The use of Information Extraction (IE) techniques to extract the content of incoming dialogue utterances after an ASR phase. (2)The conversion of the input to RDF form to allow the generation of new facts from existing ones, under the control of a Dialogue Manager (DM), that also has access to stored knowledge and knowledge accessed in real time from the web, all in RDF form. (3)A DM expressed as a stack and network virtual machine that models mixed initiative in dialogue control. (4)A tuned dialogue act detector based on corpus evidence. The prototype platform was evaluated, and we describe this; it is also designed to support more extensive forms of emotion detection carried by both speech and lexical content, as well as extended forms of machine learning. We describe preliminary studies and results for these, in particular a novel approach to enabling reinforcement learning for open dialogue systems through the detection of emotion in the speech signal and its deployment as a form of a learned DM, at a higher level than the DM virtual machine and able to direct the SCs responses to a more emotionally appropriate part of its repertoire.

Sarcasm, Deception, and Stating the Obvious: Planning Dialogue without Speech Acts

This paper presents an alternative to the ‘speech acts with STRIPS’ approach to implementing dialogue a fully implemented AI planner which generates and analyses the semantics of utterances using a single linguistic act for all contexts. Using this act, the planner can model problematic conversational situations, including felicitous and infelicitous instances of bluffing, lying, sarcasm, and stating the obvious. The act has negligible effects, and its precondition can always be proved. ‘Speaker maxims’ enable the speaker to plan to deceive, as well as to generate implicatures, while ‘hearer maxims’ enable the hearer to recognise deceptions, and interpret implicatures. The planner proceeds by achieving parts of the constructive proof of a goal. It incorporates an epistemic theorem prover, which embodies a deduction model of belief, and a constructive logic.

Speech Acts, Epistemic Planning and Grice's Maxims

Work on speech acts has generally involved the introduction of sets of different actions such as informing, reminding, bluffing and lying. These actions have different preconditions and effects, and hence can be used to achieve a wide variety of different real-world goals. The problem is that they tend to have indistinguishable surface forms. As such, it is extremely difficult for the hearer to decide which action she thinks has been performed, and it is therefore also extremely difficult for the speaker to be confident about how the hearer will respond. We will show how to achieve complex goals on the basis of a very simple set of linguistic actions. These actions have clearly marked surface forms, and hence can easily be distinguishable by a hearer. In order to do this, we have developed an epistemic planner with a number of interesting features, and with a number of optimisations that relate directly to aspects of the task at hand.

Generating context-sensitive ECA responses to user barge-in interruptions

We present an Embodied Conversational Agent (ECA) that incorporates a context-sensitive mechanism for handling user barge-in. The affective ECA engages the user in social conversation, and is fully implemented. We will use actual examples of system behaviour to illustrate. The ECA is designed to recognise and be empathetic to the emotional state of the user. It is able to detect, react quickly to, and then follow up with considered responses to different kinds of user interruptions. The design of the rules which enable the ECA to respond intelligently to different types of interruptions was informed by manually analysed real data from human–human dialogue. The rules represent recoveries from interruptions as two-part structures: an address followed by a resumption. The system is robust enough to manage long, multi-utterance turns by both user and system, which creates good opportunities for the user to interrupt while the ECA is speaking.

'Sorry' seems to be the hardest word

We are interested in the ways that language is used to achieve a variety of goals, where the same utterance may have vastly different consequences in different situations. This is closely related to the topic of creativity in language. The fact that the same utterance can be used to achieve a variety of goals opens up the possibility of using it to achieve new goals. The current paper concentrates largely on an implemented system for exploring how the effects of an utterance depend on the situation in which it is produced, but we will end with some speculations about how how utterances can come to have new kinds of uses.

Functional, Frustrating and Full of Potential: Learners’ Experiences of a Prototype for Automated Essay Feedback

OpenEssayist is an automated feedback system designed to support university students as they write essays for assessment. A first generation prototype of this system was tested on a cohort of postgraduate distance learners at the UK Open University from September to December 2013. A case study approach was used to examine three participants’ experiences of the prototype. Findings from the case studies offered insight into how different users may perceive the usefulness, future potential and end-user of such a tool. This study has important implications for the next phase of development, when the role of OpenEssayist in supporting students’ learning will need to be more clearly understood.

What Does a ‘Good’ Essay Look Like? Rainbow Diagrams Representing Essay Quality

This paper reports on an essay-writing study using a technical system that has been developed to generate automated feedback on academic essays. The system operates through the combination of a linguistic analysis engine, which processes the text in the essay, and a web application that uses the output of the linguistic analysis engine to generate the feedback. In this paper we focus on one particular visual representation produced by the system, namely “rainbow diagrams”. Using the concept of a reverse rainbow, diagrams are produced which visually represent how concepts are interlinked between the essay introduction (violet nodes) and conclusion (red nodes), and how concepts are linked and developed across the whole essay – thus a measure of how cohesive the essay is as a whole. Using a bank of rainbow diagrams produced from real essays, we rated the diagrams as belonging to high-, medium- or low-scoring essays according to their structure, and compared this rating to the actual marks awarded for the essays. On the basis of this we can conclude that a significant relationship exists between an essay’s rainbow diagram structure and the mark awarded. This finding has vast implications, as it is relatively easy to show users what the diagram for a “good” essay looks like. Users can then compare this to their own work before submission so that they can make necessary changes and so improve their essay’s structure, without concerns over plagiarism. Thus the system is a valuable tool that can be utilised across academic disciplines.

What Types of Essay Feedback Influence Implementation: Structure Alone or Structure and Content?

Students have varying levels of experience and understanding, and need support to inform them of expectations and guide their learning efforts. Feedback is critical in this process. This study focused on the effects of providing different types of feedback on participants’ written essays and on participants’ motivations for learning using measures of motivation and self-efficacy. We examined whether participants performed differently in subsequent essays after receiving feedback on structure alone or on structure and content; whether their self-reported levels of motivation and attitudes to learning were related to essay performance; and whether the difference in type of feedback affected their self-reported levels of motivation and attitudes to learning. Findings revealed no significant difference in marks between those receiving feedback on structure alone and those receiving feedback on structure and content. Even so, using feedback to highlight structural elements of essay writing can have a positive impact on future essay performance.

The pragmatics of margin comments: An empirical study

This paper describes the design and rationale behind a classification scheme for English margin comments. The scheme’s design was informed by pragmatics and pedagogy theory, and by observations made from a corpus of 24,387 margin comments from assessed university assignments. The purpose of the scheme is to computationally explore content and form relationships between margin comments and the passages to which they point. The process of designing the scheme resulted in the conclusion that margin comments require more work to understand than utterances do, and that they are more prone to being misunderstood.