Richard F. E. Sutcliffe

Overview of the CLEF 2005 multilingual question answering track

The general aim of the third CLEF Multilingual Question Answering Track was to set up a common and replicable evaluation framework to test both monolingual and cross-language Question Answering (QA) systems that process queries and documents in several European languages. Nine target languages and ten source languages were exploited to enact 8 monolingual and 73 cross-language tasks. Twenty-four groups participated in the exercise. Overall results showed a general increase in performance in comparison to last year. The best performing monolingual system irrespective of target language answered 64.5% of the questions correctly (in the monolingual Portuguese task), while the average of the best performances for each target language was 42.6%. The cross-language step instead entailed a considerable drop in performance. In addition to accuracy, the organisers also measured the relation between the correctness of an answer and a system’s stated confidence in it, showing that the best systems did not always provide the most reliable confidence score. We provide an overview of the 2005 QA track, detail the procedure followed to build the test sets and present a general analysis of the results.

Overview of ResPubliQA 2009: question answering evaluation over European legislation

This paper describes the first round of ResPubliQA, a Question Answering (QA) evaluation task over European legislation, proposed at the Cross Language Evaluation Forum (CLEF) 2009. The exercise consists of extracting a relevant paragraph of text that satisfies completely the information need expressed by a natural language question. The general goals of this exercise are (i) to study if the current QA technologies tuned for newswire collections and Wikipedia can be adapted to a new domain (law in this case); (ii) to move to a more realistic scenario, considering people close to law as users, and paragraphs as system output; (iii) to compare current QA technologies with pure Information Retrieval (IR) approaches; and (iv) to introduce in QA systems the Answer Validation technologies developed in the past three years. The paper describes the task in more detail, presenting the different types of questions, the methodology for the creation of the test sets and the new evaluation measure, and analyzing the results obtained by systems and the more successful approaches. Eleven groups participated with 28 runs. In addition, we evaluated 16 baseline runs (2 per language) based only in pure IR approach, for comparison purposes. Considering accuracy, scores were generally higher than in previous QA campaigns.

Artificial Intelligence and Cognitive Science

In recent years, there has been a considerable amount of interest in using Natural Language Processing in Information Retrieval research, with specific implementations varying from the word-level morphological analysis to syntactic parsing to conceptual-level semantic analysis. In particular, different degrees of phrase-level syntactic information have been incorporated in information retrieval systems working on English or Germanic languages such as Dutch. In this paper we study the impact of using such information, in the form of syntactic dependency pairs, in the performance of a text retrieval system for a Romance language, Spanish.

Cross-language french-english question answering using the DLT system at CLEF 2005

This paper describes the main components of the system built by the DLT Group at Limerick for participation in the QA Task at CLEF. The document indexing we used was again sentence-by-sentence but this year the Lucene Engine was adopted. We also experimented with retrieval query expansion using Local Context Analysis. Results were broadly similar to last year.

QA4MRE 2011-2013: Overview of Question Answering for Machine Reading Evaluation

This paper describes the methodology for testing the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. This was the attempt of the QA4MRE challenge which was run as a Lab at CLEF 2011---2013. The traditional QA task was replaced by a new Machine Reading task, whose intention was to ask questions that required a deep knowledge of individual short texts and in which systems were required to choose one answer, by analysing the corresponding test document in conjunction with background text collections provided by the organization. Four different tasks have been organized during these years: Main Task, Processing Modality and Negation for Machine Reading, Machine Reading of Biomedical Texts about Alzheimers disease, and Entrance Exams. This paper describes their motivation, their goals, their methodology for preparing the data sets, their background collections, their metrics used for the evaluation, and the lessons learned along these three years.

Distributed representations in a text based information retrieval system: a new way of using the vector space model

In this paper we discuss how the Vector Space Model of Information Retrieval can be used in a new way by combining connectionist ideas about distributed representations with the concept of propositional structure (semantic case structure) derived from mainstream Natural Language Understanding research. We show how distributed representations may be used to capture both amorphous concept representations and propositional structures and we discuss a prototype Information Retrieval system, PELICAN, which has been constructed in order to experiment with these ideaa.

Constructing Distributed Semantic Lexical Representations using a Machine Readable Dictionary

We present an experiment in which distributed semantic representations are automatically constructed for 4263 nouns taken from the Merriam Webster Compact Electronic Dictionary. The algorithm for defining a concept involves extracting taxonomic links from dictionary definitions and using these to infer weighted features to add to that concept’s representation. Initial analysis of the results is promising: word representations tend to cluster in a semantically intuitive fashion. The approach can be generalised to the rest of the dictionary by improving the parser.

Disambiguation by association as a practical method: Experiments and findings*

Abstract Word sense disambiguation is vital to any form of text analysis which involves word meanings. However it is a very difficult task to solve accurately. We have replicated two well known methods due to Lesk (1986) and Ide and Veronis (1990), and have conducted trials using both methods on a corpus of 100 sentences. We also carried out experiments to determine whether the use of syntactic tagging would improve results. There are three principal findings of this work. Firstly, syntactic tagging improves the performance of all the disambiguation algorithms. Secondly, the Ide and Veronis method of depth 2 performs slightly better than the Lesk method. Thirdly, the performance of a particular algorithm is heavily dependent on the way in which it is measured.

Distributed Subsymbolic Representations for Natural Language: How many Features Do You Need?

In a Natural Language Understanding system, be it connectionist or otherwise, it is often desirable for representations to be as compact as possible. In this paper we present a simple algorithm for thinning down an existing set of distributed concept representations which form the lexicon in a prototype story paraphrase system which exploits both conventional and connectionist approaches to Artificial Intelligence (AI). We also present some performance measures for evaluating a lexicon’s performance. The main result is that the algorithm appears to work well — we can use it to balance the level of detail in a lexicon against the amount of space it requires. There are also interesting ramifications concerning meaning in natural language.

Creation of a semantic lexicon by Traversal of a machine tractable concept taxonomy

Abstract In order to develop computer systems which can process the content of natural language texts it is necessary to capture the meaning of words in a way which is robust and domain independent. In this article we discuss a particular kind of word representation based on sets of weighted semantic features. We describe how a lexicon of such patterns can be created automatically from a hierarchical concept ontology and evaluated using a psychometric method. Finally we summarise our findings in this research, giving the strengths and weaknesses of the advocated approach.