Stephen Doherty

Eye tracking as an MT evaluation technique

Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc. for some time now. Its application as a technique for measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semi-automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. Ten native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded non-invasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the “bad” sentences, while average fixation duration and pupil dilations were not found to be substantially different for output rated as good and output rated as bad. Comparisons between HTER scores and eye gaze data were also found to correlate well with gaze time and fixation count, but not with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as a semi-automatic MT evaluation technique, which does not require bi-lingual knowledge, and which can potentially tap into the end users’ experience of machine translation output.

Statistical machine translation in the translation curriculum: overcoming obstacles and empowering translators

In this paper we argue that the time is ripe for translator educators to engage with Statistical Machine Translation (SMT) in more profound ways than they have done to date. We explain the basic principles of SMT and reflect on the role of humans in SMT workflows. Against a background of diverging opinions on the latter, we argue for a holistic approach to the integration of SMT into translator training programmes, one that empowers rather than marginalises translators. We discuss potential barriers to the use of SMT by translators generally and in translator training in particular, and propose some solutions to problems thus identified. More specifically, cloud-based services are proposed as a means of overcoming some of the technical and ethical challenges posed by more advanced uses of SMT in the classroom. Ultimately the paper aims to pave the way for the design and implementation of a new translator-oriented SMT syllabus at our own University and elsewhere.

Assessing the Usability of Raw Machine Translated Output: A User-Centered Study Using Eye Tracking

This article reports on the results of a project that aimed to investigate the usability of raw machine translated technical support documentation for a commercial online file storage service. Adopting a user-centered approach, the ISO/TR 16982 definition of usability—goal completion, satisfaction, effectiveness, and efficiency— is utilized and eye-tracking measures that are shown to be reliable indicators of cognitive effort are applied along with a posttask questionnaire. The study investigated these measures for the original user documentation written in English and in four target languages: Spanish, French, German, and Japanese, all of which were translated using a freely available online statistical machine translation engine. Using native speakers for each language, the study found several significant differences between the source and MT output, a finding that indicates a difference in usability between well-formed content and raw machine translated content. One target language in particular, Japanese, was found to have a considerably lower usability level when compared with the original English.

The design and evaluation of a Statistical Machine Translation syllabus for translation students

Despite the acknowledged importance of translation technology in translation studies programmes and the current ascendancy of Statistical Machine Translation (SMT), there has been little reflection to date on how SMT can or should be integrated into the translation studies curriculum. In a companion paper we set out a rationale for including a holistic SMT syllabus in the translation curriculum. In this paper, we show how the priorities and aspirations articulated in that source can be operationalised in the translation technology classroom and lab. We draw on our experience of designing and evaluating an SMT syllabus for a cohort of postgraduate student translators at Dublin City University in 2012. In particular, we report on data derived from a mixed-methods approach that aims to capture the students’ view of the syllabus and their self-assessment of their own learning. Using the construct of self-efficacy, we show significant increases in students’ knowledge of and confidence in using machine translation in general and SMT in particular, after completion of teaching units in SMT. We report on additional insights gleaned from student assignments, and conclude with ideas for future refinements of the syllabus.

A survey of machine translation competences: Insights for translation technology educators and practitioners

This paper describes a large-scale survey of machine translation (MT) competencies conducted by a non-commercial and publicly funded European research project. Firstly, we highlight the increased prevalence of translation technologies in the translation and localisation industry, and develop upon this by reporting on survey data derived from 438 validated respondents, including freelance translators, language service providers, translator trainers, and academics. We then focus on ascertaining the prevalence of translation technology usage on a fine-grained scale to address aspects of MT, quality assessment techniques and post-editing. We report a strong need for an improvement in quality assessment methods, tools, and training, partly due to the large variance in approaches and combinations of methods, and to the lack of knowledge and resources. We note the growing uptake of MT and the perceived increase of its prevalence in future workflows. We find that this adoption of MT has led to significant changes in the human translation process, in which post-editing appears to be exclusively used for high-quality content publication. Lastly, we echo the needs of the translation industry and community in an attempt to provide a more comprehensive snapshot to inform the provision of translation training and the need for increased technical competencies.

A Virtuous Circle: Laundering Translation Memory Data using Statistical Machine Translation

This study compares consistency in target texts produced using translation memory (TM) with that of target texts produced using statistical machine translation (SMT), where the SMT engine is trained on the same texts as are reused in the TM workflow. These comparisons focus specifically on noun and verb inconsistencies, as such inconsistencies appear to be highly prevalent in TM data. The study substitutes inconsistent TM target text nouns and verbs for consistent nouns and verbs from the SMT output to test whether this results in improvements in overall TM consistency and whether an SMT engine trained on the ‘laundered’ TM data performs better than the baseline engine. Improvements were observed in both TM consistency and SMT performance, a finding that indicates the potential of this approach for improving TM/MT integration.

Original Language Subtitles: Their Effects on the Native and Foreign Viewer

Se estudia el impacto de los subtitulos en el mismo idioma de la narrativa audiovisual segun el idioma del receptor (nativo o extranjero). Estudiantes de dos universidades australianas y una espanola fueron asignados al azar a uno de dos grupos experimentales en los que se veia un drama con la banda sonora original en ingles con subtitulos en esa misma lengua (n=81) o sin subtitulos (n=92). La muestra incluia un grupo control de hablantes nativos de ingles, ademas de grupos de hablantes nativos de chino mandarin, coreano y espanol con ingles como lengua extranjera. Como medidas post-hoc, los participantes reportaron, mediante escalas Likert, su percepcion de presencia, transporte, realismo percibido, identificacion con los personajes y disfrute. Los resultados muestran que los subtitulos no reducen las medidas de inmersion. Ademas, que los subtitulos producen mayores puntuaciones de transporte, identificacion con los personajes y percepcion de realismo, cuya varianza se explica, esencialmente, por la primera lengua de los receptores y sus habitos de visionado. Asimismo, los resultados senalan que ni a la presencia y ni al disfrute les afectan la condicion experimental o el idioma del receptor. Finalmente, muestran que el transporte es la medida mas reveladora de la inmersion porque produce las correlaciones mas fuertes y consistentes, aparte de ser un predictor significativo del disfrute de los espectadores.

Assessing Quality in Human- and Machine-Generated Subtitles and Captions

The depth, breadth, and complexity of audiovisual translation (AVT) are growing at a rapid rate. AVT is becoming increasingly merged with language technologies, including computer-assisted translation tools, machine translation, automated subtitling and captioning software, and automatic speech recognition systems. An essential component in this exciting and challenging technological development of current and future applications of AVT is the definition and assessment of quality in a way that is transparent, reliable, consistent, meaningful to all stakeholders, and readily applicable to the growing diversity of AVT. This chapter first provides a critical overview of current and future issues in the assessment of quality in human and machine-generated subtitling and captioning. It builds upon a range of contemporary industry sources and moves into cutting-edge research on the processing and reception of AVT products across a variety of media and languages. We then move to discuss the impact of new media and technologies on best practice, policy, and research. Lastly, we identify numerous challenges and potential solutions for all stakeholders in order to encourage dialogue between disciplines with the aim of articulating and answering questions of quality in AVT in an evolving technological landscape.

On Education and Training in Translation Quality Assessment

In this chapter, we argue that education and training in translation quality assessment (TQA)is being neglected for most, if not all, stakeholders of the translation process, from translators, post-editors, and reviewers to buyers and end-users of translation products and services. Within academia, there is a lack of education and training opportunities to equip translation students, even at postgraduate level, with the knowledge and skills required to understand and use TQA. This has immediate effects on their employability and long-term effects on professional practice. In discussing and building upon previous initiatives to tackle this issue, we provide a range of viewpoints and resources for the provision of such opportunities in collaborative and independent contexts across all modes and academic settings, focusing not just on TQA and machine translation training, but also on the use of assessment strategies in educational contexts that are directly relevant to those used in industry. In closing, we reiterate our argument for the importance of education and training in TQA, on the basis of all the contributions and perspectives presented in the volume.

Telop, Affect, and Media Design: A Multimodal Analysis of Japanese TV Programs:

Japanese and other Asian TV producers have been deploying multi-colored, and highly visible, intra-lingual captions on TV programs to enhance their appeal and to influence their viewers’ interpretations. The practice of adding these captions is far from innocent and is prone to abuse and overuse due to the lack of official guidelines and an evidence base. We conducted a multimodal analysis within the framework of relevance theory to provide an empirically supported insight into the way in which these captions, known as “telop” in Japan, form part of a production’s deliberate and careful media design. Our findings suggest that telop are deployed in conjunction with other communicative resources that are deliberately used to influence viewers’ interpretations, to enhance and make affective values in TV programs more explicit. The increasing use of diegetically integrated captions elsewhere further justifies the need for critical TV and new media research on telop.