Udo Kruschwitz

Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation

We are witnessing a paradigm shift in Human Language Technology (HLT) that may well have an impact on the field comparable to the statistical revolution: acquiring large-scale resources by exploiting collective intelligence. An illustration of this new approach is Phrase Detectives, an interactive online game with a purpose for creating anaphorically annotated resources that makes use of a highly distributed population of contributors with different levels of expertise. The purpose of this article is to first of all give an overview of all aspects of Phrase Detectives, from the design of the game and the HLT methods we used to the results we have obtained so far. It furthermore summarizes the lessons that we have learned in developing this game which should help other researchers to design and implement similar games.

Automatically structuring domain knowledge from text: An overview of current research

This paper presents an overview of automatic methods for building domain knowledge structures (domain models) from text collections. Applications of domain models have a long history within knowledge engineering and artificial intelligence. In the last couple of decades they have surfaced noticeably as a useful tool within natural language processing, information retrieval and semantic web technology. Inspired by the ubiquitous propagation of domain model structures that are emerging in several research disciplines, we give an overview of the current research landscape and some techniques and approaches. We will also discuss trade-offs between different approaches and point to some recent trends.

Using Games to Create Language Resources: Successes and Limitations of the Approach

One of the more novel approaches to collaboratively creating language resources in recent years is to use online games to collect and validate data. The most significant challenges collaborative systems face are how to train users with the necessary expertise and how to encourage participation on a scale required to produce high quality data comparable with data produced by “traditional” experts. In this chapter we provide a brief overview of collaborative creation and the different approaches that have been used to create language resources, before analysing games used for this purpose. We discuss some key issues in using a gaming approach, including task design, player motivation and data quality, and compare the costs of each approach in terms of development, distribution and ongoing administration. In conclusion, we summarise the benefits and limitations of using a gaming approach to resource creation and suggest key considerations for evaluating its utility in different research scenarios.

Exploiting structure for intelligent Web search

Together with the rapidly growing amount of online data, we register an immense need for intelligent search engines that access a restricted amount of data as found in intranets or other limited domains. These sorts of search engine must go beyond simple keyword indexing/matching, but they also have to be easily adaptable to new domains without huge costs. The paper presents a mechanism that addresses both of these points: first of all, the internal document structure is being used to extract concepts which impose a directory-like structure on the documents, similar to those found in classified directories. Furthermore, this is done in an efficient way which is largely language independent and does not make assumptions about the document structure.

An adaptable search system for collections of partially structured documents

To help users search domain-specific document collections or those limited in size, the author created a search system based on a generic framework. The system incorporates a simple domain-independent dialogue manager and an automatically created model of the domain.

Constructing an Anaphorically Annotated Corpus with Non-Experts: Assessing the Quality of Collaborative Annotations

This paper reports on the ongoing work of Phrase Detectives, an attempt to create a very large anaphorically annotated text corpus. Annotated corpora of the size needed for modern computational linguistics research cannot be created by small groups of hand-annotators however the ESP game and similar games with a purpose have demonstrated how it might be possible to do this through Web collaboration. We show that this approach could be used to create large, high-quality natural language resources.

Users want more sophisticated search assistants: results of a task-based evaluation

The Web provides a massive knowledge source. The same is true for intranets and other electronic document collections. However, much of that knowledge is encoded implicitly and cannot be applied directly without processing it into some more appropriate structures. Searching, browsing, question answering for example could all benefit from domain specific knowledge contained in the do cuments; and in applications such as simple search we do not actually need very “deep” knowledge structures such as ontologies but we can get a long way with a model of the domain that consists of term hierarchies. We combine domain knowledge automatically acquired by exploiting the documents’ markup structure with knowled ge extracted on the fly to assist a user with ad hoc search requests. Such a search system can suggest query modification options derived from the actual data and thus guide a user through the space of documents. This paper gives a detailed account of a task-based evaluation that compares a search system which utilizes the outlined domain knowledge against a standard search system. We found that users do use the query modification suggestions proposed by the system. The main conclusion we can draw from this evaluation however is that users prefer a system that can suggest query modifications ov er a standard search engine which simply presents a ranked list of documents. Most interestingly, we observe this user preference despite the fact that the baseline system even performs slightly better under certain criteri a.

A demonstration of human computation using the Phrase Detectives annotation game

The goal of the ANAWIKI project is to experiment with Web collaboration and human computation to create largescale linguistically annotated corpora. We will present ongoing work and initial results of Phrase Detectives, a game designed to collect judgments about anaphoric annotations.

YPA — an Intelligent Directory Enquiry Assistant

The YPA project is building a system to make the information in classified directories more accessible. BTs Yellow Pages®1 provides an example of classified database with which this work would be useful.There are two reasons for doing this: (i) directories like Yellow Pages contain much useful but hard-to-access information, especially in the free text in semi-display advertisements; (ii) more generally, the project is a demonstrator for exploitation of semi-structured data — data that is less systematic than database entries or logical clauses, but more systematic than free text because it has been marked up, for display or some other purpose.Accessing the directory source data file requires both natural language processing (for softening the interface to the system, and separately for analysis of natural-language-like constructs in the data) and information retrieval techniques, which are assisted by shallow knowledge. Deep world knowledge is impractical.The project seeks to get maximum effect from conveniently simplified approximations of standard natural language processing and knowledge representation. The paper gives an overview of the system, and illustrates its style with points about how the source data file is analysed. The YPA requires further development, but already demonstrates the effectiveness of shallow processing applied to semi-structured data.

MultiLing 2015: Multilingual Summarization of Single and Multi-Documents, On-line Fora, and Call-center Conversations

In this paper we present an overview of MultiLing 2015, a special session at SIGdial 2015. MultiLing is a communitydriven initiative that pushes the state-ofthe-art in Automatic Summarization by providing data sets and fostering further research and development of summarization systems. There were in total 23 participants this year submitting their system outputs to one or more of the four tasks of MultiLing: MSS, MMS, OnForumS and CCCS. We provide a brief overview of each task and its participation and evaluation.