Jesús Herrera

Textual entailment recognition based on dependency analysis and wordnet

The Recognizing Textual Entailment System shown here is based on the use of a broad-coverage parser to extract dependency relationships; in addition, WordNet relations are used to recognize entailment at the lexical level. The work investigates whether the mapping of dependency trees from text and hypothesis give better evidence of entailment than the matching of plain text alone. While the use of WordNet seems to improve systems performance, the notion of mapping between trees here explored (inclusion) shows no improvement, suggesting that other notions of tree mappings should be explored such as tree edit distances or tree alignment distances.

The Multiple Language Question Answering Track at CLEF 2003

This paper reports on the pilot question answering track that was carried out within the CLEF initiative this year. The track was divided into monolingual and bilingual tasks: monolingual systems were evaluated within the frame of three non-English European languages, Dutch, Italian and Spanish, while in the crosslanguage tasks an English document collection constituted the target corpus for Italian, Spanish, Dutch, French and German queries. Participants were given 200 questions for each task, and were allowed to submit up to two runs per task with up to three responses (either exact answers or 50 bytes long strings) per question. We give here an overview of the track: we report on each task and discuss the creation of the multilingual test sets and the participants’ results.

Creating the DISEQuA corpus: a test set for multilingual question answering

This paper describes the procedure adopted by the three coordinators of the CLEF 2003 question answering track (ITC-irst, UNED and ILLC) to create the question set for the monolingual tasks. Despite the few resources available, the three groups managed to formulate and verify a large pool of original questions in three different languages: Dutch, Italian and Spanish. Part of these queries was translated into English and shared between the three coordinating groups. A second cross-verification was then conducted in order to identify the queries that had an answer in all three monolingual document collections. The result of the joint efforts was the creation of the DISEQuA (Dutch Italian Spanish English Questions and Answers) corpus, a useful and reusable resource that is freely available for the research community. We report on the different stages of the corpus creation, from the monolingual kernels to the multilingual extension.

The effect of entity recognition on answer validation

The Answer Validation Exercise (AVE) 2006 is aimed at evaluating systems able to decide whether the responses of a Question Answering (QA) system are correct or not. Since most of the questions and answers contain entities, the use of a textual entailment relation between entities is studied here for the task of Answer Validation. We present some experiments concluding that the entity entailment relation is a feature that improves a SVM based classifier close to the best result in AVE 2006.

Inferring the scope of negation in biomedical documents

In the last few years negation detection systems for biomedical texts have been developed successfully. In this paper we present a system that finds and annotates the scope of negation in English sentences. It infers which words are affected by negations by browsing dependency syntactic structures. Thus, firstly a greedy algorithm detects negation cues, like no or not. And secondly the scope of these negation cues is computed. We tested the system over the Bioscope corpus, annotated with negation, obtaining competitive results. The system presented in this paper can be accessed via web.

A feasibility study on low level techniques for improving parsing accuracy for spanish using maltparser

In the last years dependency parsing has been accomplished by machine learning–based systems showing great accuracy but usually under 90% for Labelled Attachment Score (LAS) Maltparser is one of such systems Machine learning allows to obtain parsers for every language having an adequate training corpus Since generally such systems can not be modified the following question arises: Can we beat this 90% LAS by using better training corpora? Some previous work points that high level techniques are not sufficient for building more accurate training corpora Thus, by analyzing the words that are more frequently incorrectly attached or labelled, we study the feasibility of some low level techniques, based on n–version parsing models, in order to obtain better parsing accuracy.

Experiments of UNED at the Third Recognising Textual Entailment Challenge

This paper describes the experiments developed and the results obtained in the participation of UNED in the Third Recognising Textual Entailment (RTE) Challenge. The experiments are focused on the study of the effect of named entities in the recognition of textual entailment. While Named Entity Recognition (NER) provides remarkable results (accuracy over 70%) for RTE on QA task, IE task requires more sophisticated treatment of named entities such as the identification of relations between them.

Spanish question answering evaluation

This paper reports the most significant issues related to the launching of a Monolingual Spanish Question Answering evaluation track at the Cross Language Evaluation Forum (CLEF 2003). It introduces some questions about multilingualism and describes the methodology for test suite production, task, judgment of answers as well as the results obtained by the participant systems.

The effect of answer validation on the performance of Question-Answering systems

Abstract Question Answering (QA) systems help users to find relevant information. However, current QA systems still return several incorrect answers, which decreases the confidence of users in such systems. Some researchers have suggested the inclusion of validation in QA with the purpose of reducing wrong responses and, therefore, improving results. Nevertheless, there has not been a precise study about how validation contributes to QA results. In this paper, we include a new analysis of the impact of validation on QA results. We reuse data from real QA systems and show that validation enables QA systems to reduce incorrect answers and obtain better results. We think this study will allow researchers to develop better QA systems.