Zahra Rahimi

Automatic Scoring of an Analytical Response-To-Text Assessment

In analytical writing in response to text, students read a complex text and adopt an analytic stance in their writing about it. To evaluate this type of writing at scale, an automated approach for Response to Text Assessment RTA is needed. With the long-term goal of producing informative feedback for students and teachers, we design a new set of interpretable features that operationalize the Evidence rubric of RTA. When evaluated on a corpus of essays written by students in grades 4-6, our results show that our features outperform baselines based on well-performing features from other types of essay assessments.

Incorporating Coherence of Topics as a Criterion in Automatic Response-to-Text Assessment of the Organization of Writing

This paper presents an investigation of score prediction for the Organization dimension of an assessment of analytical writing in response to text. With the long-term goal of producing feedback for students and teachers, we designed a task-dependent model that aligns with the scoring rubric and makes use of the source material. Our experimental results show that our rubric-based model performs as well as baselines on datasets from grades 6-8. On shorter and noisier essays from grades 5-6, the rubric-based model performs better than the baselines. Further, we show that the baseline model (lexical chaining) can be improved if we extend it with information from the source text for shorter and noisier data.

Topic based creation of a persian-english comparable corpus

One of the most important issues in cross language information retrieval (CLIR) is where to obtain the translation knowledge. Multilingual corpora are valuable resources for this purpose, but few studies have been done on constructing multilingual corpora in Persian language. In this study, we propose a method to construct a Persian- English comparable corpus using two independent news collections and based on date and topic criteria. Unlike most existing methods which use publication dates as the main basis for aligning documents, we also consider date-independent alignments: alignments based only on topics and concept similarities. In order to avoid low quality alignments, we cluster the collections based on their topics prior to alignments which allows us to align similar documents whose publication dates are distant. Evaluation results show the high quality of constructed corpus and the possibility of extracting high quality association knowledge from the corpus for the task of CLIR.

Assessing Students’ Use of Evidence and Organization in Response-to-Text Writing: Using Natural Language Processing for Rubric-Based Automated Scoring

This paper presents an investigation of score prediction based on natural language processing for two targeted constructs within analytic text-based writing: 1) students’ effective use of evidence and, 2) their organization of ideas and evidence in support of their claim. With the long-term goal of producing feedback for students and teachers, we designed a task-dependent model, for each dimension, that aligns with the scoring rubric and makes use of the source material. We believe the model will be meaningful and easy to interpret given the writing task. We used two datasets of essays written by students in grades 5–6 and 6–8. Our experimental results show that our task-dependent model (consistent with the rubric) performs as well as if not outperforms competitive baselines. We also show the potential generalizability of the rubric-based model by performing cross-corpus experiments. Finally, we show that the predictive utility of different feature groups in our rubric-based modeling approach is related to how much each feature group covers a rubric’s criteria.

The Teams Corpus and Entrainment in Multi-Party Spoken Dialogues

When interacting individuals entrain, they begin to speak more like each other. To support research on entrainment in cooperative multi-party dialogues, we have created a corpus where teams of three or four speakers play two rounds of a cooperative board game. We describe the experimental design and technical infrastructure used to collect our corpus, which consists of audio, video, transcriptions, and questionnaire data for 63 teams (47 hours of audio). We illustrate the use of our corpus as a novel resource for studying team entrainment by 1) developing and evaluating teamlevel acoustic-prosodic entrainment measures that extend existing dyad measures, and 2) investigating relationships between team entrainment and participation dominance.

Creating a Wikipedia-based Persian-English word association dictionary

One of the most important issues in cross language information retrieval is how to cross the language barrier between the query and the documents. Different translation resources have been studied for this purpose. In this research, we study using Wikipedia for query translation by constructing a Wikipedia-based bilingual association dictionary. We use English and Persian Wikipedia inter-language links to align related titles and then mine word by word associations between the two languages using the extracted alignments. We use the mined word association dictionary for translating queries in Persian-English cross language information retrieval. Our experimental results on Hamshari corpus show that the proposed method is effective in extracting word associations and that Persian Wikipedia is a promising translation resource. Using the association dictionary, we can improve the pure dictionary-based method, where the only translation resource is a bilingual dictionary, by 33.6% and its recall by 26.2%.

Automatically Extracting Topical Components for a Response-to-Text Writing Assessment

We investigate automatically extracting multiword topical components to replace information currently provided by experts that is used to score the Evidence dimension of a writing in response to text assessment. Our goal is to reduce the amount of expert effort and improve the scalability of an automatic scoring system. Experimental results show that scoring performance using automatically extracted data-driven topical components is promising.

Turn-Taking Behavior in a Human Tutoring Corpus.

Analysis of turn-taking in tutoring dialogues can be helpful to understand the procedure of tutoring and also the influence with regard to demographics between students and the tutor. In this research, we analyze turn-taking behavior between students in a human-human spoken tutoring system. Our approach is to learn turn-taking models using dialog activity state sequences and then we measure the association of these models with students’ demographic features (gender and education). The experimental results show that female students speak simultaneously longer with the tutor than male students, female activities are less than male activities and also the tutor speaks longer with students who have lower pre-test score.

Acoustic-Prosodic Entrainment in Multi-party Spoken Dialogues: Does Simple Averaging Extend Existing Pair Measures Properly?

Linguistic entrainment, the tendency of interlocutors to become similar to each other during spoken interaction, is an important characteristic of human speech. Implementing linguistic entrainment in spoken dialogue systems helps to improve the naturalness of the conversation, likability of the agents, and dialogue and task success. The first step toward implementation of such systems is to design proper measures to quantify entrainment. Multi-party entrainment and multi-party spoken dialogue systems have received less attention compared to dyads. In this study, we analyze an existing approach of extending pair measures to team-level entrainment measures, which is based on simple averaging of pairs. We argue that although simple averaging is a good starting point to measure team entrainment, it has several weaknesses in terms of capturing team-specific behaviors specifically related to convergence.