Ben Hachey

Cheap and easy entity evaluation

The AIDA-YAGO dataset is a popular target for whole-document entity recognition and disambiguation, despite lacking a shared evaluation tool. We review evaluation regimens in the literature while comparing the output of three approaches, and identify research opportunities. This utilises our open, accessible evaluation tool. We exemplify a new paradigm of distributed, shared evaluation, in which evaluation software and standardised, versioned system outputs are provided online.

CLPsych 2016 Shared Task: Triaging content in online peer-support forums.

This paper introduces a new shared task for the text mining community. It aims to directly support the moderators of a youth mental health forum by asking participants to automatically triage posts into one of four severity labels: green, amber, red or crisis. The task attracted 60 submissions from 15 different teams, the best of whom achieve scores well above baselines. Their approaches and results provide valuable insights to enable moderators of peer support forums to react quickly to the most urgent, concerning content.

Hugo: Entity-based News Search and Summarisation

We describe Hugo -- a service initially available on iOS that solicits a structured, semantic query and returns entity-specific news articles. Retrieval is powered by a semantic annotation pipeline that includes named entity linking and automatic summarisation. Search and entity linking use an in-house knowledge base initialised with Wikipedia data and continually curated to include new entities. Hugo delivers timely knowledge about a users professional network, in particular new people they want to know more about.

English Event Detection With Translated Language Features.

We propose novel radical features from automatic translation for event extraction. Event detection is a complex language processing task for which it is expensive to collect training data, making generalisation challenging. We derive meaningful subword features from automatic translations into target language. Results suggest this method is particularly useful when using languages with writing systems that facilitate easy decomposition into subword features, e.g., logograms and Cangjie. The best result combines logogram features from Chinese and Japanese with syllable features from Korean, providing an additional 3.0 points f-score when added to state-of-the-art generalisation features on the TAC KBP 2015 Event Nugget task.

An Interface for Assisted Curation of Knowledge Bases from Unstructured Text

We present an interactive graphical tool for assisted curation of knowledge bases from unstructured text data. Given text input, the user can create a knowledge base from scratch, including sub-tasks of entity mention annotation, matching mentions that refer to the same entity, and extracting relations between entities. The interface is designed to enable organizations to extract valuable knowledge from text data that may otherwise remain unexploited. We evaluate the interface through a formative user study. The study results suggest several key directions for refinement. Results also highlight the efficacy of the interface: all participants were able to create a knowledge base from scratch.

The Computable News project: Research in the Newsroom

We report on a four year academic research project to build a natural language processing platform in support of a large media company. The Computable News platform processes news stories, producing a layer of structured data that can be used to build rich applications. We describe the underlying platform and the research tasks that we explored building it. The platform supports a wide range of prototype applications designed to support different newsroom functions. We hope that this qualitative review provides some insight into the challenges involved in this type of project.

Newton: Building an Authority-Driven Company Tagging and Resolution System

Abstract We describe an entity detection and resolution system called Newton that is being used to identify company names in Reuters news articles and ground the mention text to a company authority database. The system is required to be fast and precise on arbitrary web news sources. We introduce an infrastructure for authority-driven lookup-tagging followed by joint mention and disambiguation classification using a support vector machine. Performance on a corpus of 70k automatically annotated documents from the Reuters News Archive is 0.89/0.76 precision/recall for mention detection and 0.94/0.95 precision/recall for resolution resulting in an 0.84/0.72 precision/recall for detection and resolution combined.