Mike Wald

Speech recognition in university classrooms: liberated learning project

The LIBERATED LEARNING PROJECT (LLP) is an applied research project studying two core questions:1) Can speech recognition (SR) technology successfully digitize lectures to display spoken words as text in university classrooms?2) Can speech recognition technology be used successfully as an alternative to traditional classroom notetaking for persons with disabilities?This paper addresses these intriguing questions and explores the underlying complex relationship between speech recognition technology, university educational environments, and disability issues.

Universal access to communication and learning: the role of automatic speech recognition

This communication discusses how automatic speech recognition (ASR) can support universal access to communication and learning through the cost-effective production of text synchronised with speech and describes achievements and planned developments of the Liberated Learning Consortium to: support preferred learning and teaching styles; assist those who for cognitive, physical or sensory reasons find notetaking difficult; assist learners to manage and search online digital multimedia resources; provide automatic captioning of speech for deaf learners or when speech is not available or suitable; assist blind, visually impaired or dyslexic people to read and search material; and, assist speakers to improve their communication skills.

Digital Agility and Digital Decision-Making: Conceptualising Digital Inclusion in the Context of Disabled Learners in Higher Education

Digital inclusion in higher education has tended to be understood solely in terms of accessibility, which does little to further our understanding of the role technology plays in the learning experiences of disabled students. In this article, the authors propose a conceptual framework for exploring digital inclusion in higher education that attempts to broaden the way in which it is understood. The conceptual framework encompasses two strands: one that focuses on technology, personal and contextual factors, and one that focuses on resources and choices. This framework will be used to present and discuss the results of a study which aimed to explore the e‐learning experiences of disabled students at one higher education institution. The discussion will focus particularly on concepts of digital agility and digital decision‐making, and will consider the potential implications for the empowerment of disabled students.

Using Automatic Speech Recognition to Enhance Education for All Students: Turning a Vision into Reality

Legislation requires that educational materials produced by staff should be accessible to disabled students. Speech materials therefore require captioning and the Liberated Learning Initiative has demonstrated that automatic speech recognition provides the potential to make teaching accessible to all and to assist learners to manage and search online digital multimedia resources. This could improve the quality of education as the automatic provision of accessible synchronised lecture notes enables students to concentrate on learning and enables teachers to monitor and review what they said and reflect on it to improve their teaching. Standard automatic speech recognition software lacks certain features that are required to make this vision a reality. The only automatic speech recognition tool that is being developed to specifically overcome these identified problems would appear to be IBM ViaScribe and investigation of its application in educational environments is occurring through the Liberating Learning Consortium. This paper described both achievements and current developments

Creating accessible educational multimedia through editing automatic speech recognition captioning in real time

Lectures can be digitally recorded and replayed to provide multimedia revision material for students who attended the class and a substitute learning experience for students unable to attend. Deaf and hard of hearing people can find it difficult to follow speech through hearing alone or to take notes while they are lip-reading or watching a sign-language interpreter. Notetakers can only summarise what is being said while qualified sign language interpreters with a good understanding of the relevant higher education subject content are in very scarce supply. Synchronising the speech with text captions can ensure deaf students are not disadvantaged and assist all learners to search for relevant specific parts of the multimedia recording by means of the synchronised text. Real time stenography transcription is not normally available in UK higher education because of the shortage of stenographers wishing to work in universities. Captions are time consuming and expensive to create by hand and while Automatic Speech Recognition can be used to provide real time captioning directly from lecturers’ speech in classrooms it has proved difficult to obtain accuracy comparable to stenography. This paper describes the development of a system that enables editors to correct errors in the captions as they are created by Automatic Speech Recognition.

Crowdsourcing correction of speech recognition captioning errors

In this paper, we describe a tool that facilitates crowdsourcing correction of speech recognition captioning errors to provide a sustainable method of making videos accessible to people who find it difficult to understand speech through hearing alone.

Synote: development of a Web-based tool for synchronized annotations

This paper discusses the development of a Web-based media annotation application named Synote, which addresses the important issue that while the whole of a multimedia resource on the Web can be easily bookmarked, searched, linked to and tagged, it is still difficult to search or associate notes or other resources with a certain part of a resource. Synote supports the creation of synchronized notes, bookmarks, tags, links, images and text captions. It is a freely available application that enables any user to make annotations in and search annotations to any fragment of a continuous multimedia resource in the most used browsers and operating systems. In the implementation, Synote categorized different media resources and synchronized them via time line. The presentation of synchronized resources makes full use of Web 2.0 AJAX technology to enrich interoperability for the user experience. Positive evaluation results about the performance, efficiency and effectiveness of Synote were returned when using it with students and teachers for a number of undergraduate courses.

Enriching media fragments with named entities for video classification

With the steady increase of videos published on media sharing platforms such as Dailymotion and YouTube, more and more efforts are spent to automatically annotate and organize these videos. In this paper, we propose a framework for classifying video items using both textual features such as named entities extracted from subtitles, and temporal features such as the duration of the media fragments where particular entities are spotted. We implement four automatic machine learning algorithms for multiclass classification problems, namely Logistic Regression (LG), K-Nearest Neighbour (KNN), Naive Bayes (NB) and Support Vector Machine (SVM). We study the temporal distribution patterns of named entities extracted from 805 Dailymotion videos. The results show that the best performance using the entity distribution is obtained with KNN (overall accuracy of 46.58%) while the best performance using the temporal distribution of named entities for each type is obtained with SVM (overall accuracy of 43.60%). We conclude that this approach is promising for automatically classifying online videos.

Measuring Verbal Communication in Initial Physical Therapy Encounters

Background Communication in clinical encounters is vital in ensuring a positive experience and outcome for both patient and clinician. Objective The purpose of this study was to measure verbal communication between physical therapists and patients with back pain during their initial consultation and trial management of the data using a novel, Web-based application. Design A cross-sectional study was conducted. Methods Nine musculoskeletal physical therapists and 27 patients with back pain participated in this study. Twenty-five initial consultations were observed, audio recorded, and categorized using the Medical Communications Behavior System. Data were managed using Synote, a freely available application enabling synchronization of audio recordings with transcripts and coded notes. Results In this sample, physical therapists spoke for 49.5% of the encounter and patients for 33.1%. Providers and patients spent little time overtly discussing emotions (1.4% and 0.9%, respectively). More-experienced clinicians used more “history/background probes,” more “advice/suggestion,” and less “restatement” than less-experienced staff, although they demonstrated a greater prevalence of talking concurrently and interrupting patients (7.6% compared with 2.6%). Limitations Although studies measuring actual behavior are considered to be the gold standard, audio recordings do not enable nonverbal behaviors to be recorded. Conclusion This study investigated a method for measuring the verbal content of clinical encounters in a physical therapy outpatient setting. The study has directly contributed to developing a research-friendly version of the application (ie, Synote Researcher). Given the pivotal role of communication in ensuring a positive experience and outcome for both patient and provider, investing time in further developing communication skills should be an on-going priority for providers. Further work is needed to explore affective behaviors and the prevalence of interrupting patients, considering differences in sex and provider experience.

GOLIAH: A Gaming Platform for Home-Based Intervention in Autism – Principles and Design

Children with Autism need intensive intervention and this is challenging in terms of manpower, costs, and time. Advances in Information Communication Technology and computer gaming may help in this respect by creating a nomadically deployable closed-loop intervention system involving the child and active participation of parents and therapists. An automated serious gaming platform enabling intensive intervention in nomadic settings has been developed by mapping two pivotal skills in autism spectrum disorder: Imitation and Joint Attention (JA). Eleven games – seven Imitations and four JA – were derived from the Early Start Denver Model. The games involved application of visual and audio stimuli with multiple difficulty levels and a wide variety of tasks and actions pertaining to the Imitation and JA. The platform runs on mobile devices and allows the therapist to (1) characterize the child’s initial difficulties/strengths, ensuring tailored and adapted intervention by choosing appropriate games and (2) investigate and track the temporal evolution of the child’s progress through a set of automatically extracted quantitative performance metrics. The platform allows the therapist to change the game or its difficulty levels during the intervention depending on the child’s progress. Performance of the platform was assessed in a 3-month open trial with 10 children with autism (Trial ID: NCT02560415, Clinicaltrials.gov). The children and the parents participated in 80% of the sessions both at home (77.5%) and at the hospital (90%). All children went through all the games but, given the diversity of the games and the heterogeneity of children profiles and abilities, for a given game the number of sessions dedicated to the game varied and could be tailored through automatic scoring. Parents (N = 10) highlighted enhancement in the child’s concentration, flexibility, and self-esteem in 78, 89, and 44% of the cases, respectively, and 56% observed an enhanced parents–child relationship. This pilot study shows the feasibility of using the developed gaming platform for home-based intensive intervention. However, the overall capability of the platform in delivering intervention needs to be assessed in a bigger open trial.