Cosmin Munteanu

The effect of speech recognition accuracy rates on the usefulness and usability of webcast archives

The widespread availability of broadband connections has led to an increase in the use of Internet broadcasting (webcasting). Most webcasts are archived and accessed numerous times retrospectively. In the absence of transcripts of what was said, users have difficulty searching and scanning for specific topics. This research investigates user needs for transcription accuracy in webcast archives, and measures how the quality of transcripts affects user performance in a question-answering task, and how quality affects overall user experience. We tested 48 subjects in a within-subjects design under 4 conditions: perfect transcripts, transcripts with 25% Word Error Rate (WER), transcripts with 45% WER, and no transcript. Our data reveals that speech recognition accuracy linearly influences both user performance and experience, shows that transcripts with 45% WER are unsatisfactory, and suggests that transcripts having a WER of 25% or less would be useful and usable in webcast archives.

Situational Ethics: Re-thinking Approaches to Formal Ethics Requirements for Human-Computer Interaction

Most Human-Computer Interaction (HCI) researchers are accustomed to the process of formal ethics review for their evaluation or field trial protocol. Although this process varies by country, the underlying principles are universal. While this process is often a formality, for field research or lab-based studies with vulnerable users, formal ethics requirements can be challenging to navigate -- a common occurrence in the social sciences; yet, in many cases, foreign to HCI researchers. Nevertheless, with the increase in new areas of research such as mobile technologies for marginalized populations or assistive technologies, this is a current reality. In this paper we present our experiences and challenges in conducting several studies that evaluate interactive systems in difficult settings, from the perspective of the ethics process. Based on these, we draft recommendations for mitigating the effect of such challenges to the ethical conduct of research. We then issue a call for interaction researchers, together with policy makers, to refine existing ethics guidelines and protocols in order to more accurately capture the particularities of such field-based evaluations, qualitative studies, challenging lab-based evaluations, and ethnographic observations.

Collaborative editing for improved usefulness and usability of transcript-enhanced webcasts

One challenge in facilitating skimming or browsing through archives of on-line recordings of webcast lectures is the lack of text transcripts of the recorded lecture. Ideally, transcripts would be obtainable through Automatic Speech Recognition (ASR). However, current ASR systems can only deliver, in realistic lecture conditions, a Word Error Rate of around 45% -- above the accepted threshold of 25%. In this paper, we present the iterative design of a webcast extension that engages users to collaborate in a wiki-like manner on editing the ASR-produced imperfect transcripts, and show that this is a feasible solution for improving the quality of lecture transcripts. We also present the findings of a field study carried out in a real lecture environment investigating how students use and edit the transcripts.

Ethical Encounters in HCI: Research in Sensitive Settings

With HCI researchers conducting studies in increasingly sensitive and difficult settings, ethics is emerging as a key concern for the HCI community. New technologies are now being designed and evaluated in settings that involve vulnerable or marginalized participants and that can be emotionally challenging for researchers. Research in these settings can produce complex ethical dilemmas that are often emergent, diverse, and highly contextualized. In addition, there may be discrepancies between the realities of HCI fieldwork and the formal and often rigid processes of obtaining ethics approval in research institutions. Given these issues, it is important for researchers to communally reflect on ethical encounters in HCI research. This workshop will provide a forum for researchers to share experiences about ethical challenges they have faced. These discussions will be used to develop a handbook of practical lessons representing the breadth and depth of ethical issues emerging in HCI research in sensitive settings.

We need to talk: HCI and the delicate topic of spoken language interaction

Speech and natural language remain our most natural form of interaction; yet the HCI community have been very timid about focusing their attention on designing and developing spoken language interaction techniques. This may be due to a widespread perception that perfect domain-independent speech recognition is an unattainable goal. Progress is continuously being made in the engineering and science of speech and natural language processing, however, and there is also recent research that suggests that many applications of speech require far less than 100% accuracy to be useful in many contexts. Engaging the CHI community now is timely -- many recent commercial applications, especially in the mobile space, are already tapping the increased interest in and need for natural user interfaces (NUIs) by enabling speech interaction in their products. As such, the goal of this panel is to bring together interaction designers, usability researchers, and general HCI practitioners to discuss the opportunities and directions to take in designing more natural interactions based on spoken language, and to look at how we can leverage recent advances in speech processing in order to gain widespread acceptance of speech and natural language interaction.

ALEX: mobile language assistant for low-literacy adults

Basic literacy skills are fundamental building blocks of education, yet for a very large number of adults tasks such as understanding and using everyday items is a challenge. While research, industry, and policy-making is looking at improving access to textual information for low-literacy adults, the literacy-based demands of todays society are continually increasing. Although many community-based organizations offer resources and support to adults with limited literacy skills, current programs have difficulties reaching and retaining those that would benefit most from them. To address these challenges, the National Research Council of Canada is proposing a technological solution to support literacy programs and to assist low-literacy adults in todays information-centric society: ALEX© - Adult Literacy support application for EXperiential learning. ALEX© has been created together with low-literacy adults, following guidelines for inclusive design of mobile assistive tools. It is a mobile language assistant that is designed to be used both in the classroom and in daily life, in order to help low-literacy adults become increasingly literate and independent.

Improving Automatic Speech Recognition for Lectures through Transformation-based Rules Learned from Minimal Data

We demonstrate that transformation-based learning can be used to correct noisy speech recognition transcripts in the lecture domain with an average word error rate reduction of 12.9%. Our method is distinguished from earlier related work by its robustness to small amounts of training data, and its resulting efficiency, in spite of its use of true word error rate computations as a rule scoring function.

Fieldwork with vulnerable populations

Ethical implications for human-computer interaction research

Ethical Encounters in Human-Computer Interaction

In the HCI community, there is growing recognition that a reflective and empathetic approach is needed to conduct ethical research in sensitive settings with people who might be considered vulnerable or marginalized. At our CHI 2015 workshop on ethical encounters, researchers shared personal stories of the challenges and tensions they have faced when conducting HCI research in complex settings such as hospitals, with young mental health patients, in schools for children with disabilities, and with homeless people. These research contexts can present significant challenges for HCI researchers who would not typically receive the training that other professionals working in these environments would normally receive. From our discussions with attendees at the CHI 2015 workshop, we identified a number of ethical issues that researchers are grappling with. In this follow-up workshop we aim to build on the lessons learned and to generate pragmatic but sensitive solutions to manage complex ethical issues for HCI researchers working in challenging settings.

CSCW Research Ethics Town Hall: Working Towards Community Norms

As technologies as well as research practice evolve, new ethical challenges emerge within the CSCW research community. Currently, norms around what constitutes ethical practice in research are scattered and inconsistent. We face a number of open ethical questions around issues such as informed consent, anonymization, Terms of Service, deleted content, and what constitutes public data. The SIGCHI Ethics Committee has been looking towards ways to encourage norm setting and mitigate ethical disagreement during the peer review process. This town hall style panel will be an opportunity to prompt community discussion and collect input into how we can further address these challenges.