Bernardo Magnini

The Third PASCAL Recognizing Textual Entailment Challenge

This paper presents the Third PASCAL Recognising Textual Entailment Challenge (RTE-3), providing an overview of the dataset creating methodology and the submitted systems. In creating this years dataset, a number of longer texts were introduced to make the challenge more oriented to realistic scenarios. Additionally, a pool of resources was offered so that the participants could share common tools. A pilot task was also set up, aimed at differentiating unknown entailments from identified contradictions and providing justifications for overall system decisions. 26 participants submitted 44 runs, using different approaches and generally presenting new entailment models and achieving higher scores than in the previous challenges.

Multilingual Information Access for Text, Speech and Images

What Happened in CLEF 2004?.- What Happened in CLEF 2004?.- I. Ad Hoc Text Retrieval Tracks.- CLEF 2004: Ad Hoc Track Overview and Results Analysis.- Selection and Merging Strategies for Multilingual Information Retrieval.- Using Surface-Syntactic Parser and Deviation from Randomness.- Cross-Language Retrieval Using HAIRCUT at CLEF 2004.- Experiments on Statistical Approaches to Compensate for Limited Linguistic Resources.- Application of Variable Length N-Gram Vectors to Monolingual and Bilingual Information Retrieval.- Integrating New Languages in a Multilingual Search System Based on a Deep Linguistic Analysis.- IR-n r2: Using Normalized Passages.- Using COTS Search Engines and Custom Query Strategies at CLEF.- Report on Thomson Legal and Regulatory Experiments at CLEF-2004.- Effective Translation, Tokenization and Combination for Cross-Lingual Retrieval.- Two-Stage Refinement of Transitive Query Translation with English Disambiguation for Cross-Language Information Retrieval: An Experiment at CLEF 2004.- Dictionary-Based Amharic - English Information Retrieval.- Dynamic Lexica for Query Translation.- SINAI at CLEF 2004: Using Machine Translation Resources with a Mixed 2-Step RSV Merging Algorithm.- Mono- and Crosslingual Retrieval Experiments at the University of Hildesheim.- University of Chicago at CLEF2004: Cross-Language Text and Spoken Document Retrieval.- UB at CLEF2004: Cross Language Information Retrieval Using Statistical Language Models.- MIRACLEs Hybrid Approach to Bilingual and Monolingual Information Retrieval.- Searching a Russian Document Collection Using English, Chinese and Japanese Queries.- Dublin City University at CLEF 2004: Experiments in Monolingual, Bilingual and Multilingual Retrieval.- Finnish, Portuguese and Russian Retrieval with Hummingbird SearchServerTM at CLEF 2004.- Data Fusion for Effective European Monolingual Information Retrieval.- The XLDB Group at CLEF 2004.- The University of Glasgow at CLEF 2004: French Monolingual Information Retrieval with Terrier.- II. Domain-Specific Document Retrieval.- The Domain-Specific Track in CLEF 2004: Overview of the Results and Remarks on the Assessment Process.- University of Hagen at CLEF 2004: Indexing and Translating Concepts for the GIRT Task.- IRIT at CLEF 2004: The English GIRT Task.- Ricoh at CLEF 2004.- GIRT and the Use of Subject Metadata for Retrieval.- III. Interactive Cross-Language Information Retrieval.- iCLEF 2004 Track Overview: Pilot Experiments in Interactive Cross-Language Question Answering.- Interactive Cross-Language Question Answering: Searching Passages Versus Searching Documents.- Improving Interaction with the User in Cross-Language Question Answering Through Relevant Domains and Syntactic Semantic Patterns.- Cooperation, Bookmarking, and Thesaurus in Interactive Bilingual Question Answering.- Summarization Design for Interactive Cross-Language Question Answering.- Interactive and Bilingual Question Answering Using Term Suggestion and Passage Retrieval.- IV. Multiple Language Question Answering.- Overview of the CLEF 2004 Multilingual Question Answering Track.- A Question Answering System for French.- Cross-Language French-English Question Answering Using the DLT System at CLEF 2004.- Experiments on Robust NL Question Interpretation and Multi-layered Document Annotation for a Cross-Language Question/Answering System.- Making Stone Soup: Evaluating a Recall-Oriented Multi-stream Question Answering System for Dutch.- The DIOGENE Question Answering System at CLEF-2004.- Cross-Lingual Question Answering Using Off-the-Shelf Machine Translation.- Bulgarian-English Question Answering: Adaptation of Language Resources.- Answering French Questions in English by Exploiting Results from Several Sources of Information.- Finnish as Source Language in Bilingual Question Answering.- miraQA: Experiments with Learning Answer Context Patterns from the Web.- Question Answering for Spanish Supported by Lexical Context Annotation.- Question Answering Using Sentence Parsing and Semantic Network Matching.- First Evaluation of Esfinge - A Question Answering System for Portuguese.- University of Evora in QA@CLEF-2004.- COLE Experiments at QA@CLEF 2004 Spanish Monolingual Track.- Does English Help Question Answering in Spanish?.- The TALP-QA System for Spanish at CLEF 2004: Structural and Hierarchical Relaxing of Semantic Constraints.- ILC-UniPI Italian QA.- Question Answering Pilot Task at CLEF 2004.- Evaluation of Complex Temporal Questions in CLEF-QA.- V. Cross-Language Retrieval in Image Collections.- The CLEF 2004 Cross-Language Image Retrieval Track.- Caption and Query Translation for Cross-Language Image Retrieval.- Pattern-Based Image Retrieval with Constraints and Preferences on ImageCLEF 2004.- How to Visually Retrieve Images from the St. Andrews Collection Using GIFT.- UNED at ImageCLEF 2004: Detecting Named Entities and Noun Phrases for Automatic Query Expansion and Structuring.- Dublin City University at CLEF 2004: Experiments with the ImageCLEF St. Andrews Collection.- From Text to Image: Generating Visual Query for Image Retrieval.- Toward Cross-Language and Cross-Media Image Retrieval.- FIRE - Flexible Image Retrieval Engine: ImageCLEF 2004 Evaluation.- MIRACLE Approach to ImageCLEF 2004: Merging Textual and Content-Based Image Retrieval.- Cross-Media Feedback Strategies: Merging Text and Image Information to Improve Image Retrieval.- ImageCLEF 2004: Combining Image and Multi-lingual Search for Medical Image Retrieval.- Multi-modal Information Retrieval Using FINT.- Medical Image Retrieval Using Texture, Locality and Colour.- SMIRE: Similar Medical Image Retrieval Engine.- A Probabilistic Approach to Medical Image Retrieval.- UB at CLEF2004 Cross Language Medical Image Retrieval.- Content-Based Queries on the CasImage Database Within the IRMA Framework.- Comparison and Combination of Textual and Visual Features for Interactive Cross-Language Image Retrieval.- MSU at ImageCLEF: Cross Language and Interactive Image Retrieval.- VI. Cross-Language Spoken Document Retrieval.- CLEF 2004 Cross-Language Spoken Document Retrieval Track.- VII. Issues in CLIR and in Evaluation.- The Key to the First CLEF with Portuguese: Topics, Questions and Answers in CHAVE.- How Do Named Entities Contribute to Retrieval Effectiveness?.

ENSM-SE at CLEF 2006 : Fuzzy Proximity Method with an Adhoc Influence Function in Evaluation of Multilingual and Multi-modal Information Retrieval 7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006, Alicante, Spain

This book constitutes the thoroughly refereed postproceedings of the 7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006, held in Alicante, Spain, September 2006. The revised papers presented together with an introduction were carefully reviewed and selected for inclusion in the book. The papers are organized in topical sections on Multilingual Textual Document Retrieval, Domain-Specifig Information Retrieval, i-CLEF, QA@CLEF, ImageCLEF, CLSR, WebCLEF and GeoCLEF.We experiment a new influence function in our information retrieval method that uses the degree of fuzzy proximity of key terms in a document to compute the relevance of the document to the query. The model is based on the idea that the closer the query terms in a document are to each other the more relevant the document. Our model handles Boolean queries but, contrary to the traditional extensions of the basic Boolean information retrieval model, does not use a proximity operator explicitly. A single parameter makes it possible to control the proximity degree required. To improve our system we use a stemming algorithm before indexing, we take a specific influence function and we merge fuzzy proximity result lists built with different width of influence function. We explain how we construct the queries and report the results of our experiments in the ad-hoc monolingual French task of the CLEF 2006 evaluation campaign.

Is It the Right Answer? Exploiting Web Redundancy for Answer Validation

Answer Validation is an emerging topic in Question Answering, where open domain systems are often required to rank huge amounts of candidate answers. We present a novel approach to answer validation based on the intuition that the amount of implicit knowledge which connects an answer to a question can be quantitatively estimated by exploiting the redundancy of Web information. Experiments carried out on the TREC-2001 judged-answer collection show that the approach achieves a high level of performance (i.e. 81% success rate). The simplicity and the efficiency of this approach make it suitable to be used as a module in Question Answering systems.

Recognizing textual entailment: Rational, evaluation and approaches

The goal of identifying textual entailment – whether one piece of text can be plausibly inferred from another – has emerged in recent years as a generic core problem in natural language understanding. Work in this area has been largely driven by the PASCAL Recognizing Textual Entailment (RTE) challenges, which are a series of annual competitive meetings. The current work exhibits strong ties to some earlier lines of research, particularly automatic acquisition of paraphrases and lexical semantic relationships and unsupervised inference in applications such as question answering, information extraction and summarization. It has also opened the way to newer lines of research on more involved inference methods, on knowledge representations needed to support this natural language understanding challenge and on the use of learning methods in this context. RTE has fostered an active and growing community of researchers focused on the problem of applied entailment. This special issue of the JNLE provides an opportunity to showcase some of the most important work in this emerging area.

The role of domain information in Word Sense Disambiguation

This paper explores the role of domain information in word sense disambiguation. The underlying hypothesis is that domain labels, such as MEDICINE, ARCHITECTURE and SPORT, provide a useful way to establish semantic relations among word senses, which can be profitably used during the disambiguation process. Results obtained at the SENSEVAL-2 initiative confirm that for a significant subset of words domain information can be used to disambiguate with a very high level of precision.

Revising the wordnet domains hierarchy: semantics, coverage and balancing

The continuous expansion of the multilingual information society has led in recent years to a pressing demand for multilingual linguistic resources suitable to be used for different applications. In this paper we present the WordNet Domains Hierarchy (WDH), a language-independent resource composed of 164, hierarchically organized, domain labels (e.g. Architecture, Sport, Medicine). Although WDH has been successfully applied to various Natural Language Processing tasks, the first available version presented some problems, mostly related to the lack of a clear semantics of the domain labels. Other correlated issues were the coverage and the balancing of the domains. We illustrate a new version of WDH addressing these problems by an explicit and systematic reference to the Dewey Decimal Classification. The new version of WDH has a better defined semantics and is applicable to a wider range of tasks.

The Multiple Language Question Answering Track at CLEF 2003

This paper reports on the pilot question answering track that was carried out within the CLEF initiative this year. The track was divided into monolingual and bilingual tasks: monolingual systems were evaluated within the frame of three non-English European languages, Dutch, Italian and Spanish, while in the crosslanguage tasks an English document collection constituted the target corpus for Italian, Spanish, Dutch, French and German queries. Participants were given 200 questions for each task, and were allowed to submit up to two runs per task with up to three responses (either exact answers or 50 bytes long strings) per question. We give here an overview of the track: we report on each task and discuss the creation of the multilingual test sets and the participants’ results.

A SAT-based algorithm for context matching

The development of more and more complex distributed applications over large networks of computers has raised the problem of semantic interoperability across applications based on local and autonomous semantic schemas (e.g., concept hierarchies, taxonomies, ontologies). In this paper we propose to view each semantic schema as a context (in the sense defined in [1]), and propose an algorithm for automatically discovering relations across contexts (where relations are defined in the sense of [7]). The main feature of the algorithm is that the problem of finding relationships between contexts is encoded as a problem of logical satisfiability, and so the discovered mappings have a well-defined semantic. The algorithm we describe has been implemented as part of a peer-to-peer system for Distributed Knowledge Management, and tested on significant cases.

Experiments in Word Domain Disambiguation for Parallel Texts

This paper describes some preliminary results about Word Domain Disambiguation, a variant of Word Sense Disambiguation where words in a text are tagged with a domain label in place of a sense label. The English WORDNET and its aligned Italian version, MULTIWORDNET, both augmented with domain labels, are used as the main information repositories. A baseline algorithm for Word Domain Disambiguation is presented and then compared with a mutual help disambiguation strategy, which takes advantages of the shared senses of parallel bilingual texts.