Víctor Peinado

The Multiple Language Question Answering Track at CLEF 2003

This paper reports on the pilot question answering track that was carried out within the CLEF initiative this year. The track was divided into monolingual and bilingual tasks: monolingual systems were evaluated within the frame of three non-English European languages, Dutch, Italian and Spanish, while in the crosslanguage tasks an English document collection constituted the target corpus for Italian, Spanish, Dutch, French and German queries. Participants were given 200 questions for each task, and were allowed to submit up to two runs per task with up to three responses (either exact answers or 50 bytes long strings) per question. We give here an overview of the track: we report on each task and discuss the creation of the multilingual test sets and the participants’ results.

Creating the DISEQuA corpus: a test set for multilingual question answering

This paper describes the procedure adopted by the three coordinators of the CLEF 2003 question answering track (ITC-irst, UNED and ILLC) to create the question set for the monolingual tasks. Despite the few resources available, the three groups managed to formulate and verify a large pool of original questions in three different languages: Dutch, Italian and Spanish. Part of these queries was translated into English and shared between the three coordinating groups. A second cross-verification was then conducted in order to identify the queries that had an answer in all three monolingual document collections. The result of the joint efforts was the creation of the DISEQuA (Dutch Italian Spanish English Questions and Answers) corpus, a useful and reusable resource that is freely available for the research community. We report on the different stages of the corpus creation, from the monolingual kernels to the multilingual extension.

Overview of iCLEF 2009: exploring search behaviour in a multilingual folksonomy environment

This paper summarises activities from the iCLEF 2009 task. As in 2008, the task was organised based on users participating in an interactive cross-language image search experiment. Organizers provided a default multilingual search system (Flickling) which accessed images from Flickr, with the whole iCLEF experiment run as an online game. Interaction by users with the system was recorded in log files which were shared with participants for further analyses, and provide a future resource for studying various effects on user-orientated cross-language search. In total six groups participated in iCLEF with different approaches, ranging from pure log analysis to specific experiment designs using the Flickling interface.

Interactive question answering: Is Cross-Language harder than monolingual searching?

Is Cross-Language answer finding harder than Monolingual answer finding for users? In this paper we provide initial quantitative and qualitative evidence to answer this question. In our study, which involves 16 users searching questions under four different system conditions, we find that interactive cross-language answer finding is not substantially harder (in terms of accuracy) than its monolingual counterpart, using general purpose Machine Translation systems and standard Information Retrieval machinery, although it takes more time. We have also seen that users need more context to provide accurate answers (full documents) than what is usually considered by systems (paragraphs or passages). Finally, we also discuss the limitations of standard evaluation methodologies for interactive Information Retrieval experiments in the case of cross-language question answering.

UNED at ImageCLEF 2004: detecting named entities and noun phrases for automatic query expansion and structuring

This paper describes UNED experiments at the Image CLEF bilingual ad hoc task. Two different strategies are attempted: i) automatic expansion and translation using noun phrases; ii) automatic detection of named entities in the query for structured search on image caption fields. All our experiments obtain results above the average MAP for the bilingual task. Structured searches using named entities improve performance over a strong baseline (Pirkolas structured query approach), achieving one of the best results for the whole bilingual track. Expansion with noun phrases, however, degrades results, possibly due to the mismatch between train and test collections.

Using syntactic information to extract relevant terms for multi-document summarization

The identification of the key concepts in a set of documents is a useful source of information for several information access applications. We are interested in its application to multi-document summarization, both for the automatic generation of summaries and for interactive summarization systems.In this paper, we study whether the syntactic position of terms in the texts can be used to predict which terms are good candidates as key concepts. Our experiments show that a) distance to the verb is highly correlated with the probability of a term being part of a key concept; b) subject modifiers are the best syntactic locations to find relevant terms; and c) in the task of automatically finding key terms, the combination of statistical term weights with shallow syntactic information gives better results than statistical measures alone.

Interactive cross-language question answering: searching passages versus searching documents

iCLEF 2004 is the first comparative evaluation of interactive Cross-Language Question Answering systems. The UNED group has participated in this task comparing two strategies to help users in the answer finding task: the baseline system is just a standard document retrieval engine searching machine-translated versions of the documents; the contrastive system is identical, but searches passages which contain expressions of the appropriate answer type. Although the users prefer the passage system because searching is faster and simpler, it leads to slightly worse results, because the document context (which is not available in the passage retrieval system) turns out to be useful to verify the correctness of candidate answers; this makes an interesting difference with automatic Q&A systems. In addition, our experiment sets a strong baseline of 69% strict accuracy, showing that Cross-Language Question Answering can be efficiently accomplished by users without using dedicated Q&A technology.