Mary Elaine Califf

Bottom-up relational learning of pattern matching rules for information extraction

Information extraction is a form of shallow text processing that locates a specified set of relevant items in a natural-language document. Systems for this task require significant domain-specific knowledge and are time-consuming and difficult to build by hand, making them a good application for machine learning. We present an algorithm, RAPIER, that uses pairs of sample documents and filled templates to induce pattern-match rules that directly extract fillers for the slots in the template. RAPIER is a bottom-up learning algorithm that incorporates techniques from several inductive logic programming systems. We have implemented the algorithm in a system that allows patterns to have constraints on the words, part-of-speech tags, and semantic classes present in the filler and the surrounding text. We present encouraging experimental results on two domains.

Induction of first-order decision lists: results on learning the past tense of English verbs

This paper presents a method for inducing logic programs from examples that learns a new class of concepts called first-order decision lists, defined as ordered lists of clauses each ending in a cut. The method, called Foidl, is based on Foil (Quinlan, 1990) but employs intensional background knowledge and avoids the need for explicit negative examples. It is particularly useful for problems that involve rules with specific exceptions, such as learning the past-tense of English verbs, a task widely studied in the context of the symbolic/connectionist debate. Foidl is able to learn concise, accurate programs for this problem from significantly fewer examples than previous methods (both connectionist and symbolic).

Evaluating machine learning for information extraction

Comparative evaluation of Machine Learning (ML) systems used for Information Extraction (IE) has suffered from various inconsistencies in experimental procedures. This paper reports on the results of the Pascal Challenge on Evaluating Machine Learning for Information Extraction, which provides a standardised corpus, set of tasks, and evaluation methodology. The challenge is described and the systems submitted by the ten participants are briefly introduced and their performance is analysed.

Evaluation of machine learning-based information extraction algorithms: criticisms and recommendations

We survey the evaluation methodology adopted in information extraction (IE), as defined in a few different efforts applying machine learning (ML) to IE. We identify a number of critical issues that hamper comparison of the results obtained by different researchers. Some of these issues are common to other NLP-related tasks: e.g., the difficulty of exactly identifying the effects on performance of the data (sample selection and sample size), of the domain theory (features selected), and of algorithm parameter settings. Some issues are specific to IE: how leniently to assess inexact identification of filler boundaries, the possibility of multiple fillers for a slot, and how the counting is performed. We argue that, when specifying an IE task, these issues should be explicitly addressed, and a number of methodological characteristics should be clearly defined. To empirically verify the practical impact of the issues mentioned above, we perform a survey of the results of different algorithms when applied to a few standard datasets. The survey shows a serious lack of consensus on these issues, which makes it difficult to draw firm conclusions on a comparative evaluation of the algorithms. Our aim is to elaborate a clear and detailed experimental methodology and propose it to the IE community. Widespread agreement on this proposal should lead to future IE comparative evaluations that are fair and reliable. To demonstrate the way the methodology is to be applied we have organized and run a comparative evaluation of ML-based IE systems (the Pascal Challenge on ML-based IE) where the principles described in this article are put into practice. In this article we describe the proposed methodology and its motivations. The Pascal evaluation is then described and its results presented.

Learning the past tense of English verbs using inductive logic programming

This paper presents results on using a new inductive logic programming method called Foidl to learn the past tense of English verbs. The past tense task has been widely studied in the context of the symbolic/connectionist debate. Previous papers have presented results using various neural-network and decision-tree learning methods. We have developed a technique for learning a special type of Prolog program called a first-order decision list, defined as an ordered list of clauses each ending in a cut. Foidl is based on Foil [19] but employs intensional background knowledge and avoids the need for explicit negative examples. It is particularly useful for problems that involve rules with specific exceptions, such as the past-tense task. We present results showing that Foidl learns a more accurate past-tense generator from significantly fewer examples than all previous methods.

Advantages of decision lists and implicit negatives in inductive logic programming

This paper demonstrates the capabilities offoidl, an inductive logic programming (ILP) system whose distinguishing characteristics are the ability to produce first-order decision lists, the use of an output completeness assumption as a substitute for negative examples, and the use originally motivated by the problem of learning to generate the past tense of English verbs; however, this paper demonstrates its superior performance on two different sets of benchmark ILP problems. Tests on the finite element mesh design problem show thatfoidl’s decision lists enable it to produce generally more accurate results than a range of methods previously applied to this problem. Tests with a selection of list-processing problems from Bratko’s introductory Prolog text demonstrate that the combination of implicit negatives and intensionality allowfoidl to learn correct programs from far fewer examples thanfoil.

Helping him see: guiding a visually impaired student through the computer science curriculum

In this paper, we describe some of the challenges of teaching computer science to a visually impaired student along with suggested solutions for these challenges. We include perspectives of both professors and a severely visually impaired student.

Effective incorporation of ethics into courses that focus on programming

This paper discusses some of the issues involved in incorporating ethics material into programming courses. Incorporating ethics into such courses raises particular challenges because of the time-intensive nature of the courses and because of the difficulty of finding material that is both relevant to the course and comprehensible to the students. The paper presents four case studies that we have used successfully when incorporating ethics material into our own programming courses.

Efficient and effective induction of first order decision lists

We present BUFOIDL, a new bottom-up algorithm for learning first order decision lists. Although first order decision lists have potential as a representation for learning concepts that include exceptions, such as language constructs, previous systems suffered from limitations that we seek to overcome in BUFOIDL. We present experiments comparing BUFOIDL to previous work in the area, demonstrating the systems potential.

Improving learning by choosing examples intelligently in two natural language tasks

In this chapter, we present relational learning algorithms for two natural language processing tasks, semantic parsing and information extraction. We describe the algorithms and present experimental results showing their effectiveness. We also describe our application of active learning techniques to these learning systems.We applied certainty-based selective sampling to each system, using fairly simple notions of certainty. We show that these selective sampling techniques greatly reduce the number of annotated examples required for the systems to achieve good generalization performance.