Hennie Brugman

Automatic Annotation Suggestions for Audiovisual Archives: Evaluation Aspects

Abstract In the context of large and ever growing archives, generating annotation suggestions automatically from textual resources related to the documents to be archived is an interesting option in theory. It could save a lot of work in the time consuming and expensive task of manual annotation and it could help cataloguers attain a higher inter-annotator agreement. However, some questions arise in practice: what is the quality of the automatically produced annotations? How do they compare with manual annotations and with the requirements for annotation that were defined in the archive? If different from the manual annotations, are the automatic annotations wrong? In the CHOICE project, partially hosted at the Netherlands Institute for Sound and Vision, the Dutch public archive for audiovisual broadcasts, we automatically generate annotation suggestions for cataloguers. In this paper, we define three types of evaluation of these annotation suggestions: (1) a classic and strict evaluation measure expressing the overlap between automatically generated keywords and the manual annotations, (2) a loosened evaluation measure for which semantically very similar annotations are also considered as relevant matches, and (3) an in-use evaluation of the usefulness of manual versus automatic annotations in the context of serendipitous browsing. During serendipitous browsing, the annotations (manual or automatic) are used to retrieve and visualize semantically related documents.

OntoELAN: an ontology-based linguistic multimedia annotator

Despite its scientific, political, and practical value, comprehensive information about human languages, in all their variety and complexity, is not readily obtainable and searchable. One reason is that many language data are collected as audio and video recordings which imposes a challenge to document indexing and retrieval. Annotation of multimedia data provides an opportunity for making the semantics explicit and facilitates the searching of multimedia documents. We have developed OntoELAN, an ontology-based linguistic multimedia annotator that features: (1) support for loading and displaying ontologies specified in OWL; (2) creation of a language profile, which allows a user to choose a subset of terms from an ontology and conveniently rename them if needed; (3) creation of ontological tiers, which can be annotated with profile terms and, therefore, corresponding ontological terms; and (4) saving annotations in the XML format as multimedia ontology class instances and, linked to them, class instances of other ontologies used in ontological tiers. To our best knowledge, OntoELAN is the first audio/video annotation tool in linguistic domain that provides support for ontology-based annotation.

Evaluating a thesaurus browser for an audio-visual archive

In this article we report on a user study aimed at evaluating and improving a thesaurus browser. The browser is intended to be used by documentalists of a large public audio-visual archive for finding appropriate indexing terms for TV programs. The subjects involved in the study were documentalists of the Dutch National Audiovisual Archives and of broadcasting corporations. The study provides insight into the value of various thesaurus browsing and searching techniques.

Moderne Informationstechnologie und ihre Auswirkungen auf die korpus-basierte Forschung

Die technologische Entwicklung ermoglicht es den Sozialwissenschaften, distribuierte multi-mediale Korpora mittels Internet- Technologien aufzubauen. Das EUDICO Projekt hat das Ziel, kooperierenden, jedoch an verschiedenen Orten arbeitenden Wissenschaftlern eine einheitliche, format-unabhangige Schnittstelle zu den verschiedenen sprach- und videobasierten Korpora zu bieten. Zur Erreichung dieses Zieles wurde eine moderne Softwarearchitektur implementiert, die die in den Korpora vorhandenen gemeinsamen linguistischen Konzepte abbildet. Die Verwendung von Java- APIs wie dem JavaMediaFramework und modernen Streaming-Protokollen erlaubt die Realisierung einer interaktiven Schnittstelle. Mit EUDICO wurde eine Architektur geschaffen, die einen Schritt hin zu einem universellen Werkzeug fur die korpus-basierte Forschung darstellt, zumal es in einfacher Weise moglich ist, weitere Objektklassen hinzuzufugen.