Andrei Popescu-Belis

Principles of Context-Based Machine Translation Evaluation

This article defines a Framework for Machine Translation Evaluation( FEMTI) which relates the quality model used to evaluate a machinetranslation system to the purpose and context of the system. Ourproposal attempts to put together, into a coherent picture, previousattempts to structure a domain characterised by overall complexity andlocal difficulties. In this article, we first summarise theseattempts, then present an overview of the ISO/IEC guidelines forsoftware evaluation (ISO/IEC 9126 and ISO/IEC 14598). As anapplication of these guidelines to machine translation software, weintroduce FEMTI, a framework that is made of two interrelatedclassifications or taxonomies. The first classification enablesevaluators to define an intended context of use, while the links tothe second classification generate a relevant quality model (qualitycharacteristics and metrics) for the respective context. The secondclassification provides definitions of various metrics used bythe community. Further on, as part of ongoing, long-term research, weexplain how metrics are analyzed, first from the general pointof view of “meta-evaluation”, then focusing on examples. Finally, weshow how consensus towards the present framework is sought for, andhow feedback from the community is taken into account in the FEMTIlife-cycle.

Using static documents as structured and thematic interfaces to multimedia meeting archives

Static documents play a central role in multimodal applications such as meeting recording and browsing. They provide a variety of structures, in particular thematic, for segmenting meetings, structures that are often hard to extract from audio and video. In this article, we present four steps for creating a strong link between static documents and multimedia meeting archives. First, a document-centric meeting environment is introduced. Then, a document analysis tool is presented, which builds a multi-layered representation of documents and creates indexes that are further on used by document/speech and document/video alignment methods. Finally, a document-based browsing system, integrating the various alignment results, is described along with a preliminary user evaluation.

Shallow dialogue processing using machine learning algorithms (or not)

This paper presents a shallow dialogue analysis model, aimed at human-human dialogues in the context of staff or business meetings. Four components of the model are defined, and several machine learning techniques are used to extract features from dialogue transcripts: maximum entropy classifiers for dialogue acts, latent semantic analysis for topic segmentation, or decision tree classifiers for discourse markers. A rule-based approach is proposed for solving cross-modal references to meeting documents. The methods are trained and evaluated thanks to a common data set and annotation format. The integration of the components into an automated shallow dialogue parser opens the way to multimodal meeting processing and retrieval applications.

Generating Usable Formats for Metadata and Annotations in a Large Meeting Corpus

The AMI Meeting Corpus is now publicly available, including manual annotation files generated in the NXT XML format, but lacking explicit metadata for the 171 meetings of the corpus. To increase the usability of this important resource, a representation format based on relational databases is proposed, which maximizes informativeness, simplicity and reusability of the metadata and annotations. The annotation files are converted to a tabular format using an easily adaptable XSLT-based mechanism, and their consistency is verified in the process. Metadata files are generated directly in the IMDI XML format from implicit information, and converted to tabular format using a similar procedure. The results and tools will be freely available with the AMI Corpus. Sharing the metadata using the Open Archives network will contribute to increase the visibility of the AMI Corpus.

Detection and resolution of references to meeting documents

This article describes a method for document/speech alignment based on explicit verbal references to documents and parts of documents, in the context of multimodal meetings. The article focuses on the two main stages of dialogue processing for alignment: the detection of the expressions referring to documents in transcribed speech, and the recognition of the documents and document elements that they refer to. The detailed evaluation of the implemented modules, first separately and then in a pipeline, shows that results are well above baseline values. The integration of this method with other techniques for document/speech alignment is finally discussed.

Evaluation-driven design of a robust coreference resolution system

In this paper, we describe a system for coreference resolution and emphasize the role of evaluation for its design. The goal of the system is to group referring expressions (identified beforehand in narrative texts) into sets of coreferring expressions that correspond to discourse entities. Several knowledge sources are distinguished, such as referential compatibility between a referring expression and a discourse entity, activation factors for discourse entities, size of working memory, or meta-rules for the creation of discourse entities. For each of them, the theoretical analysis of its relevance is compared to scores obtained through evaluation. After looping through all knowledge sources, an optimal behavior is chosen, then evaluated on test data. The paper also discusses evaluation measures as well as data annotation, and compares the present approach to others in the field.

Apports d'Unicode à l'édition numérique multilingue Une étude de cas au Niger

Cet article met en evidence lapport dUnicode au traitement des documents multilingues en Afrique, a partir dune experience concrete au Niger. Nous analysons dabord les liens entre linformatisation des langues et leur ecriture, puis nous donnons un apercu des difficultes que pose le multilinguisme a ledition assistee par ordinateur, en etudiant le processus editorial dans un institut de documentation nigerien. Nous montrons alors en quoi le standard Unicode peut rendre ce processus plus efficace, avant de decrire plusieurs solutions aux difficultes rencontrees, en fonction des outils informatiques disponibles actuellement et a lavenir.