Joss Moorkens

Correlations of perceived post-editing effort with measurements of actual effort

Human rating of predicted post-editing effort is a common activity and has been used to train confidence estimation models. However, the correlation between human ratings and actual post-editing effort is under-measured. Moreover, the impact of presenting effort indicators in a post-editing user interface on actual post-editing effort has hardly been researched. In this study, ratings of perceived post-editing effort are tested for correlations with actual temporal, technical and cognitive post-editing effort. In addition, the impact on post-editing effort of the presentation of post-editing effort indicators in the user interface is also tested. The language pair involved in this study is English-Brazilian Portuguese. Our findings, based on a small sample, suggest that there is little agreement between raters for predicted post-editing effort and that the correlations between actual post-editing effort and predicted effort are only moderate, and thus an inefficient basis for MT confidence estimation. Moreover, the presentation of post-editing effort indicators in the user interface appears not to impact on actual post-editing effort.

Is Neural Machine Translation the New State of the Art

Abstract This paper discusses neural machine translation (NMT), a new paradigm in the MT field, comparing the quality of NMT systems with statistical MT by describing three studies using automatic and human evaluation methods. Automatic evaluation results presented for NMT are very promising, however human evaluations show mixed results. We report increases in fluency but inconsistent results for adequacy and post-editing effort. NMT undoubtedly represents a step forward for the MT field, but one that the community should be careful not to oversell.

A Virtuous Circle: Laundering Translation Memory Data using Statistical Machine Translation

This study compares consistency in target texts produced using translation memory (TM) with that of target texts produced using statistical machine translation (SMT), where the SMT engine is trained on the same texts as are reused in the TM workflow. These comparisons focus specifically on noun and verb inconsistencies, as such inconsistencies appear to be highly prevalent in TM data. The study substitutes inconsistent TM target text nouns and verbs for consistent nouns and verbs from the SMT output to test whether this results in improvements in overall TM consistency and whether an SMT engine trained on the ‘laundered’ TM data performs better than the baseline engine. Improvements were observed in both TM consistency and SMT performance, a finding that indicates the potential of this approach for improving TM/MT integration.

Ethical Considerations in NLP Shared Tasks.

Shared tasks are increasingly common in our field, and new challenges are suggested at almost every conference and workshop. However, as this has become an established way of pushing research forward, it is important to discuss how we researchers organise and participate in shared tasks, and make that information available to the community to allow further research improvements. In this paper, we present a number of ethical issues along with other areas of concern that are related to the competitive nature of shared tasks. As such issues could potentially impact on research ethics in the Natural Language Processing community, we also propose the development of a framework for the organisation of and participation in shared tasks that can help mitigate against these issues arising.

On Education and Training in Translation Quality Assessment

In this chapter, we argue that education and training in translation quality assessment (TQA)is being neglected for most, if not all, stakeholders of the translation process, from translators, post-editors, and reviewers to buyers and end-users of translation products and services. Within academia, there is a lack of education and training opportunities to equip translation students, even at postgraduate level, with the knowledge and skills required to understand and use TQA. This has immediate effects on their employability and long-term effects on professional practice. In discussing and building upon previous initiatives to tackle this issue, we provide a range of viewpoints and resources for the provision of such opportunities in collaborative and independent contexts across all modes and academic settings, focusing not just on TQA and machine translation training, but also on the use of assessment strategies in educational contexts that are directly relevant to those used in industry. In closing, we reiterate our argument for the importance of education and training in TQA, on the basis of all the contributions and perspectives presented in the volume.

Approaches to Human and Machine Translation Quality Assessment

In both research and practice, translation quality assessment is a complex task involving a range of linguistic and extra-linguistic factors. This chapter provides a critical overview of the established and developing approaches to the definition and measurement of translation quality in human and machine translation workflows across a range of research, educational, and industry scenarios. We intertwine literature from several interrelated disciplines dealing with contemporary translation quality assessment and, while we acknowledge the need for diversity in these approaches, we argue that there are fundamental and widespread issues that remain to be addressed, if we are to consolidate our knowledge and practice of translation quality assessment in increasingly technologised environments across research, teaching, and professional practice.

Evaluating MT for massive open online courses

This article reports a multifaceted comparison between statistical and neural machine translation (MT) systems that were developed for translation of data from massive open online courses (MOOCs). The study uses four language pairs: English to German, Greek, Portuguese, and Russian. Translation quality is evaluated using automatic metrics and human evaluation, carried out by professional translators. Results show that neural MT is preferred in side-by-side ranking, and is found to contain fewer overall errors. Results are less clear-cut for some error categories, and for temporal and technical post-editing effort. In addition, results are reported based on sentence length, showing advantages and disadvantages depending on the particular language pair and MT paradigm.

Testing interaction with a Mobile MT post-editing app

This is an exploratory inquiry into signed language interpreters’ perceptions of interpreter e-professionalism on social media, specifically Facebook. Given the global pervasiveness of Facebook, this study presents an international perspective, and reports on findings of focus groups held with a total of 12 professional signed language interpreters from the United States of America, the United Kingdom, and Denmark, all of whom are also Facebook users. The findings reveal that Facebook is seen to blur the traditional boundaries between personal and professional realms – an overlap which is perceived to be compounded by the nature of the small community in which signed language interpreters typically work –necessitating boundary management strategies in order to maintain perceptions of professionalism on the site. Facebook is considered a valuable professional resource to leverage for networking, professional development, problem solving and assignment preparation, but it is also perceived as a potential professional liability for both individual interpreters and the profession at large. Maintaining client confidentiality was found to be the most pressing challenge Facebook brings to the profession. Educational measures to raise awareness about e-professionalism were generally viewed favourably.The study probes into translation students’ perception of the value of online peer feedback in improving translation skills. Students enrolled in a translation degree in Australia translated a 250-word text on two separate occasions. On each occasion, the students were given another fellow student’s translation of the same text to mark and provide anonymous peer feedback. The original translations from all the students, together with any peer feedback, were uploaded onto an online forum. The students were encouraged to download their own translation to review the peer feedback in it. They were also encouraged to download and peruse other students’ peer reviewed translations for comparison. Upon completion of the project, the students were surveyed about their perceptions and appreciation of their engagement in the process in the following three capacities: (i) as a feedback provider, (ii) as a feedback recipient, and (iii) as a peruser of other students’ work and the peer feedback therein. Results suggest that translation students appreciate online peer feedback as a valuable activity that facilitates improvement. The students found receiving peer feedback on their own translation especially rewarding, as it offered alternative approaches and perspectives on tackling linguistic/translation issues. In comparing the three capacities, students perceived reviewing feedback on their own work and perusing other students’ work as more beneficial than engaging in giving feedback to others.Title: Tarjamat al-khadamaat al-’aammah ( Community Interpreting and Translation) Author: Dr. Mustapha Taibi (University of Western Sydney) Year of publication: 2011 Publisher: Dar Assalam , Rabat (Morocco) ISBN: 978-9954-22-088-7 191 pagesAccent is known to cause comprehension difficulty, but empirical interpreting studies on its specific impact have been sporadic. According to Mazzetti (1999), an accent is composed of deviated phonemics and prosody, both discussed extensively in the TESL discipline. The current study seeks to examine, in the interpreting setting, the applicability of Anderson-Hsieh, Johnson and Koehlers (1992) finding that deviated prosody hinders comprehension more than problematic phonemics and syllable structure do. Thirty-seven graduate-level interpreting majors, assigned randomly to four groups, rendered four versions of a text read by the same speaker and then filled out a questionnaire while playing back their own renditions. Renditions were later rated for accuracy by two freelance interpreters, whereas the questionnaires analysed qualitatively. Results of analyses indicated that 1) both phonemics and prosody deteriorated comprehension, but prosody had a greater impact; 2) deviated North American English post-vowel /r/, intonation and rhythm were comprehension problem triggers. The finding may be of use to interpreting trainers, trainees and professionals by contributing to their knowledge of accent.The title Conference of the Tongues at first sight raises questions as to the particularities of its pertinence to translation studies, i.e. the range of possible subject matters subsumed, and is somewhat loosely explained in the preface by a short and factual hint to its historical origins (in sixteenth-century Spain in a paratext to a translation of Aesop). There is no further elaboration on the motivation for the choice of this title however.The market for translation services provided by individuals is currently characterized by significant uncertainty because buyers lack clear ways to identify qualified providers from amongst the total pool of translators. Certification and educational diplomas both serve to reduce the resulting information asymmetry, but both suffer from potential drawbacks: translator training programs are currently oversupplying the market with graduates who may lack the specific skills needed in the market and no certification program enjoys universal recognition. In addition, the two may be seen as competing means of establishing qualification. The resulting situation, in which potential clients are uncertain about which signal to trust, is known as a signal jam . In order to overcome this jam and provide more consistent signaling, translator-training programs and professional associations offering certification need to collaborate more closely to harmonize their requirements and deliver continuing professional development (CPD) that help align the outcomes from training and certification.Interpreting is rather like scuba diving. With just a bit of protective equipment, we interpreters plunge for a short time into an often alien world, where a mistake can be very serious, not only for ourselves but for the other divers who are depending on us to understand their surroundings. And as all who dive, we interpreters find this daily foray into a new environment fascinating, exhilarating, but also at times, challenging. One of the high-risk dive sites into which we venture often is the sea of healthcare, where the strange whale-song of medical dialogue, the often incomprehensible behavior of local denizens such as doctors, and the tricky currents of the healthcare system itself require special knowledge and skill to navigate successfully. Did you ever wish for a dive manual for unique world of healthcare? Well, here’s a good one, from linguist, RN and interpreter trainer, Dr. Ineke Crezee of New Zealand.Among all the difficulties inherent in interpreting, numbers stand out as a common and complex problem trigger. This experimental study contributes to research on the causes of errors in the passive simultaneous interpretation (SI) of numbers. Two groups of Italian Master’s degree students (one for English and one for German) were asked to interpret simultaneously a number-dense speech from their respective B language into their mother tongue, Italian. Note-taking was allowed during the test and both the study participants and their lecturers completed a questionnaire afterwards. Data analysis was conducted with statistical and qualitative methods, combining the cognitivist and contextualist approach. The objective was to ascertain whether one main variable may be held responsible for the high error rate related to interpreting numbers and the difficulty perceived by students in the task. The analysis quantifies the relative impact of different causes of difficulties on participants’ delivery of numbers. It stresses the crucial role of the subjective variable represented by interpreters’ skills. Didactic implications and directions for future research are discussed in the conclusion.

Consistency in Translation Memory Corpora: A Mixed Methods Case Study

This study investigates the prevalence and causes of inconsistency in translation memories (TM) using a sequential explanatory mixed methods design. The initial quantitative phase introduces a novel method and typology for measuring and categorizing inconsistencies in TM data. The data are the product of professional computer-aided translation of software documentation. In the follow-on qualitative phase, interviewees compare the quantitative results with their professional experience of TMs. Their confirmation of the quantitative results improves the validity of the study. The interview data also increase the utility of the research, suggesting possible causes and solutions for inconsistency. Results are presented interactively, followed by a short discussion of the findings and their consequences.