Christa Womser-Hacker

GeoCLEF 2008: the CLEF 2008 cross-language geographic information retrieval track overview

GeoCLEF is an evaluation task running under the scope of the Cross Language Evaluation Forum (CLEF). The purpose of GeoCLEF is to test and evaluate cross-language geographic information retrieval (GIR). The GeoCLEF 2008 task presented twenty-five geographically challenging search topics for English, German and Portuguese. Eleven participants submitted 131 runs, based on a variety of approaches, including sample documents, named entity extraction and ontology based retrieval. The evaluation methodology and results are presented in the paper.

The effect of named entities on effectiveness in cross-language information retrieval evaluation

The large number of experiments carried out within evaluation initiatives for information retrieval has led to an invaluable source for further research and meta-analysis. In this study, an analysis of the results of the Cross Language Evaluation Forum (CLEF) campaigns for the years 2000 to 2003 is presented. This study considers the performance of the systems for each individual topic. It is dedicated to the influence of named entities on retrieval performance. Named entities in topics lead to significant improvement of the retrieval quality in general and for most systems and tasks. The performance of systems varies for topics without, with one or two and with three or more named entities. This knowledge gained by data mining on the evaluation results can be exploited for the improvement of retrieval systems as well as for the design of topics for future CLEF campaigns.

Linguistic and Statistical Analysis of the CLEF Topics

This paper reports on an analysis of the CLEF 2001 topics. In particular, we investigated potential correlations between features of the topics and the performance of retrieval systems. Although there are some weak relations, we claim that the properties of the CLEF topics do not influence the results of the retrieval systems. We found just one correlation for the English topics. The more linguistic challenges contained in the topic texts, the better the systems performed. However, no correlation for the length of a topic could be found.

Multilingual Topic Generation within the CLEF 2001 Experiments

Topic generation is considered as one of the crucial elements in the information retrieval evaluation process. In the context of CLEF, the main focus lies on evaluating multilingual functions. With respect to topic generation this means that topics have to be created in various languages. Starting with TREC, the important parameters of topic generation are described.

Integration of a Wiki for Collaborative Knowledge Development in an E-Learning Context for University Teaching

E-Learning, which is technology-enhanced learning proved to have potential for adding value to education. Recently, with the shift of the World Wide Web to a more interactive web through the introduction of social software, the term E-Learning 2.0 was coined, indicating that the potential influence of social software on E-Learning has been recognized. This paper investigates the potential of integrating a wiki into an E-Learning module and the impact this integration can have on self-directed, collaborative and responsible learning in an E-Learning setting. Therefore, the focus is set on the integration of usability aspects and the surrounding didactic conditions that need to be considered when social software is to be embedded into learning scenarios. First evaluation results of the wiki-based glossary demonstrating this potential will conclude this paper.

Evaluation of a language identification system for mono- and multilingual text documents

Language identification is an important task for web information retrieval. This paper presents the implementation of a tool for language identification in mono- and multi-lingual documents. The tool implements four algorithms for language identification. Furthermore, we present a n-gram approach for the identification of languages in multi-lingual documents. An evaluation for monolingual texts of varied length is presented. Results for eight languages including Ukrainian and Russian are presented. It could be shown that n-gram-based approaches outperform word-based algorithms for short texts. For longer texts, the performance is comparable. The evaluation for multi-lingual documents is based on real world web documents. Our tool is able to recognize the languages present in a document with reasonable accuracy.

A framework for long‐term learning of topical user preferences in information retrieval

A framework for the long‐term learning of user preferences in information retrieval is presented. The multiple indexing and method‐object relations (MIMOR) model tightly integrates a fusion method and a relevance feedback processor into a learning model. Several black box matching functions can be combined into a linear combination committee machine which reflects the users vague individual cognitive concepts expressed in relevance feedback decisions. An extension based on the soft computing paradigm couples the relevance feedback processor and the matching function into a unified retrieval system.

How do named entities contribute to retrieval effectiveness

The search for features in topics and queries relevant for the performance in information retrieval is an important strategy for system optimization. Named entities in topics are a significant feature contributing to the quality of the retrieval results. In this contribution, we present an analysis on the correlation between the number of named entities present in a topic formulation and the final retrieval quality for these topics by retrieval systems within CLEF. The analysis includes the results of CLEF 2004. We found that a medium positive correlation exists for German, English and Spanish topics. Furthermore, the effect of the document or target language on the retrieval quality is also investigated.

Multilingual Retrieval Experiments with MIMOR at the University of Hildesheim

Fusion and optimization based relevance judgements have proven to be successful strategies in information retrieval. In this year’s CLEF campaign we applied these strategies to multilingual retrieval with four languages. Our fusion experiments were carried out using freely available software. We used the snowball stemmers, internet translation services and the text retrieval tools in Lucene and the new MySQL.

How robust are multilingual information retrieval systems

The results of information retrieval evaluations are often difficult to apply to practical challenges. Recent research interest in the robustness of information systems tries to facilitate the application of research results for practical environments. This paper analyzes a large amount of evaluation experiments from the Cross Language Evaluation Forum (CLEF). Robustness can be interpreted as stressing the importance of difficult topics and is usually measured with the geometric mean of the topic results. Our analysis shows that a small decrease of performance of bi-and multi-lingual retrieval goes along with a tremendous difference between the geometric mean and the average of topics. Consequently, robustness is an important issue especially for cross-language retrieval system evaluation.