Maryam Sadat Mirzaei

Partial and synchronized captioning: A new tool to assist learners in developing second language listening skill

This paper introduces a novel captioning method, partial and synchronized captioning (PSC), as a tool for developing second language (L2) listening skills. Unlike conventional full captioning, which provides the full text and allows comprehension of the material merely by reading, PSC promotes listening to the speech by presenting a selected subset of words, where each word is synched to its corresponding speech signal. In this method, word-level synchronization is realized by an automatic speech recognition (ASR) system, dedicated to the desired corpora. This feature allows the learners to become familiar with the correspondences between words and their utterances. Partialization is done by automatically selecting words or phrases likely to hinder listening comprehension. In this work we presume that the incidence of infrequent or specific words and fast delivery of speech are major barriers to listening comprehension. The word selection criteria are thus based on three factors: speech rate, word frequency and specificity. The thresholds for these features are adjusted to the proficiency level of the learners. The selected words are presented to aid listening comprehension while the remaining words are masked in order to keep learners listening to the audio. PSC was evaluated against no-captioning and full-captioning conditions using TED videos. The results indicate that PSC leads to the same level of comprehension as the full-captioning method while presenting less than 30% of the transcript. Furthermore, compared with the other methods, PSC can serve as an effective medium for decreasing dependence on captions and preparing learners to listen without any assistance.

Conversation Envisioning to Train Inter-cultural Interactions

Conversation is an integral part of human’s relationship, which involves a large amount of tacit information to be uncovered. In this paper, we introduce the idea of conversation envisioning to disclose the tacit information beneath our conversation. We employ virtual reality for graphic recording (VRGR) to allow both observers and participants to visualize their thoughts in the conversation and to provide a training tool to acquire inter-cultural interactions using situated conversations. We focus on a bargaining scenario to highlight the tacitness of our conversations and use VRGR to make an in-depth analysis of the scenario. The proposed framework allows for performing a detailed analysis of the conversation and collecting different interpretations to provide timely assistance for realizing smoother cross-cultural conversations.

Conversation Envisioning Framework for Situated Conversation

This paper introduces virtual reality conversation envisioning (VRCE) framework as an effective approach for analyzing situated conversations. The goal of VRCE is to raise understanding of shared information, facilitate common ground formation and promote smooth communication, especially in cross-cultural interactions. In this method, a situated conversational scenario is reconstructed in a VR environment to enable detailed analysis from first and third person view, empowered by flexible traverse in the time dimension. This framework allows participants and meta-participants (observers) to actively engage in the envisioning process in VR. A conversation description language (CDL) is introduced for encoding the obtained interpretations and developing a conversation envisioner. We focused on a bargaining scenario as a situated conversation with rich cultural practices. Preliminary experiments with this scenario indicated the effectiveness of VRCE to achieve better reasoning about the situation and received positive participant feedback.

Exploiting automatic speech recognition errors to enhance partial and synchronized caption for facilitating second language listening

Abstract This paper addresses the viability of using Automatic Speech Recognition (ASR) errors as the predictor of difficulties in speech segments, thereby exploiting them to improve Partial and Synchronized Caption (PSC), which we have proposed to train second language (L2) listening skill by encouraging listening over reading. The system uses ASR technology to make word-level text-to-speech synchronization and generates a partial caption. The baseline system determines difficult words based on three features: speech rate, word frequency and specificity. While it encompasses most of the difficult words, it does not cover a wide range of features that hinder L2 listening. Therefore, we propose the use of ASR systems as a model of L2 listeners and hypothesize that ASR errors can predict challenging speech segments for these learners. Among different cases of ASR errors, annotation results suggest the usefulness of four categories of homophones, minimal pairs, negatives, and breached boundaries for L2 listeners. A preliminary experiment with L2 learners focusing on these four categories of the ASR errors revealed that these cases highlight the problematic speech regions for L2 listeners. Based on the findings, the PSC system is enhanced to incorporate these kinds of useful ASR errors. An experiment with L2 learners demonstrated that the enhanced version of PSC is not only preferable, but also more helpful to facilitate the L2 listening process.