Jens Kürsten

The XTRIEVAL Framework at CLEF 2007: Domain-Specific Track

This article describes the architecture and configuration of the XTRIEVAL (eXtensible reTRIeval and EVALuation) framework. A first prototype is described in [1]. For CLEF 2007 a second prototype was implemented which was focused on the cross-language aspect. Runs for all subtasks of the Domain-Specific track were submitted. The performance of our submitted runs was on average compared to other participating groups. Additional experiments on the Multilingual task demonstrated substantial improvement.

Produce. annotate. archive. repurpose --: accelerating the composition and metadata accumulation of tv content

Supporting most aspects of a media providers real workflows such as production, distribution, content description, archiving, and re-use of video items, we developed a holistic framework to solve issues such as lack of human resources, necessity of parallel media distribution, and retrieving previously archived content through editors or consumers.

A tool for comparative IR evaluation on component level

1. MOTIVATION Experimental information retrieval (IR) evaluation is an important instrument to measure the effectiveness of novel methods. Although IR system complexity has grown over years, the general framework for evaluation remained unchanged since its first implementation in the 1960s. Test collections were growing from thousands to millions of documents. Regular reuse resulted in larger topic sets for evaluation. New business models for information access required novel interpretations of effectiveness measures. Nevertheless, most experimental evaluations still rely on an over 50 year old paradigm. Participants of a SIGIR workshop in 2009 [1] discussed the implementation of new methodological standards for evaluation. But at the same time they worried about practicable ways to implement them. A review about recent publications containing experimental evaluations supports this concern [2]. The study also presented a web-based platform for longitudinal evaluation. In a similar way, data from the past decade of CLEF evaluations have been released through the DIRECT system. While the operators of the latter system reported about 50 new users since the release of the data [3], no further contributions were recorded on the web-platform introduced in [2]. In our point of view archiving evaluation data for longitudinal analysis is a first important step. A next step is to develop a methodology that supports researchers in choosing appropriate baselines for comparison. This can be achieved by reporting evaluation results on component level [4] rather than on system level. An exemplary study was presented in [2], where the Indri system was tested with several components switched on or off. Following this idea, an approach to assess novel methods could be to compare to related components only. This would require the community to formally record details of system configurations in connection with experimental results. We suppose that transparent descriptions of system components used in experiments could help researchers in choosing appropriate baselines.

Monolingual retrieval experiments with a domain-specific document corpus at the Chemnitz University of Technology

This article describes the first participation of the Chair Media Informatics of the Chemnitz University of Technology in the Cross Language Evaluation Forum. An experimental prototype is introduced which implements several methods of optimizing search results. The configuration of the prototype is tested with the CLEF training data. The results of the Domain-Specific Monolingual German task suggest that combining the suffix stripping stemming and the decompounding approach is very useful. Also, a local document clustering (LDC) approach used to improve the query expansion (QE) based on pseudorelevance feedback (PRF) seems to be quite beneficial. Nevertheless, the evaluation of the English task using the same configuration suggests that the qualities of the results are highly speech dependent.

A large-scale system evaluation on component-level

This article describes a large-scale empirical evaluation across different types of English text collections. We ran about 140,000 experiments and analyzed the results on system component-level to find out if we can select configurations that perform reliable on specific types of corpora. To our own surprise we observed that a specific set of configuration parameters achieved 95% of the optimal average MAP across all collections. We conclude that this configuration could be used as baseline reference for evaluation of new IR approaches on English text corpora.

Comparing IR system components using beanplots

In this poster we demonstrate an approach to gain a better understanding of the interactions between search tasks, test collections and components and configurations of retrieval systems by testing a large set of experiment configurations against standard ad-hoc test collections.

Video classification as IR task: experiments and observations

This paper describes experiments we conducted in conjunction with the VideoCLEF 2009 classification task. In our second participation in the task we experimented with treating classification as an IR problem and used the Xtrieval framework [1] to run our experiments. We confirmed that the IR approach achieves strong results although the data set was changed. We proposed an automatic threshold to limit the number of labels per document. Query expansion performed better than the corresponding baseline experiments in terms of mean average precision. We also found that combining the ASR transcriptions and the archival metadata improved the classification performance unless query expansion was used.

Putting it all together: the Xtrieval framework at Grid@CLEF 2009

The Xtrieval framework, built at the Chemnitz University of Technology, aims at analyzing the impact of different retrieval models and methods on retrieval effectiveness. For the Grid@CLEF task 2009, the CIRCO framework was integrated into the Xtrieval framework. 15 runs were performed 15 runs in the three languages German, English, and French. For each language two different stemmers and two different retrieval models were used. One run was a fusion run combining the results of the four other experiments. Whereas the different runs demonstrated that the impact of the used retrieval technologies is highly depending on the corpus, the merged approach produced the best results in each language.

CLEF 2008 ad-hoc track: comparing and combining different IR approaches

This article describes post workshop experiments that were conducted after our first participation at the TEL@CLEF task. We used the Xtrieval framework [5], [4] for the preparation and execution of the experiments. We ran 69 experiments in the setting of the CLEF 2008 task, whereof 39 were monolingual and 30 were cross-lingual. We investigated the capabilities of the current version of Xtrieval, which could use the two retrieval cores Lucene and Lemur from now on. Our main goal was to compare and combine the results from those retrieval engines. The translation of the topics for the cross-lingual experiments was realized with a plug-in to access the Google AJAX language API. The performance of our monolingual experiments was better than the best experiments we submitted during the evaluation campaign. Our crosslingual experiments performed very well for all target collections and achieved between 87% and 100% of the monolingual retrieval effectiveness. The combination of the results from the Lucene and the Lemur retrieval core showed very consistent performance.

VideoCLEF 2008: ASR classification with Wikipedia categories

This article describes our participation at the VideoCLEF track. We designed and implemented a prototype for the classification of the Video ASR data. Our approach was to regard the task as text classification problem. We used terms from Wikipedia categories as training data for our text classifiers. For the text classification the Naive-Bayes and kNN classifier from the WEKA toolkit were used. We submitted experiments for classification task 1 and 2. For the translation of the feeds to English (translation task) Googles AJAX language API was used. Although our experiments achieved only low precision of 10 to 15 percent, we assume those results will be useful in a combined setting with the retrieval approach that was widely used. Interestingly, we could not improve the quality of the classification by using the provided metadata.