Network


Latest external collaboration on country level. Dive into details by clicking on the dots.

Hotspot


Dive into the research topics where Jon Chamberlain is active.

Publication


Featured researches published by Jon Chamberlain.


Ksii Transactions on Internet and Information Systems | 2013

Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation

Massimo Poesio; Jon Chamberlain; Udo Kruschwitz; Livio Robaldo; Luca Ducceschi

We are witnessing a paradigm shift in Human Language Technology (HLT) that may well have an impact on the field comparable to the statistical revolution: acquiring large-scale resources by exploiting collective intelligence. An illustration of this new approach is Phrase Detectives, an interactive online game with a purpose for creating anaphorically annotated resources that makes use of a highly distributed population of contributors with different levels of expertise. The purpose of this article is to first of all give an overview of all aspects of Phrase Detectives, from the design of the game and the HLT methods we used to the results we have obtained so far. It furthermore summarizes the lessons that we have learned in developing this game which should help other researchers to design and implement similar games.


The People's Web Meets NLP | 2013

Using Games to Create Language Resources: Successes and Limitations of the Approach

Jon Chamberlain; Karën Fort; Udo Kruschwitz; Mathieu Lafourcade; Massimo Poesio

One of the more novel approaches to collaboratively creating language resources in recent years is to use online games to collect and validate data. The most significant challenges collaborative systems face are how to train users with the necessary expertise and how to encourage participation on a scale required to produce high quality data comparable with data produced by “traditional” experts. In this chapter we provide a brief overview of collaborative creation and the different approaches that have been used to create language resources, before analysing games used for this purpose. We discuss some key issues in using a gaming approach, including task design, player motivation and data quality, and compare the costs of each approach in terms of development, distribution and ongoing administration. In conclusion, we summarise the benefits and limitations of using a gaming approach to resource creation and suggest key considerations for evaluating its utility in different research scenarios.


Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources | 2009

Constructing an Anaphorically Annotated Corpus with Non-Experts: Assessing the Quality of Collaborative Annotations

Jon Chamberlain; Udo Kruschwitz; Massimo Poesio

This paper reports on the ongoing work of Phrase Detectives, an attempt to create a very large anaphorically annotated text corpus. Annotated corpora of the size needed for modern computational linguistics research cannot be created by small groups of hand-annotators however the ESP game and similar games with a purpose have demonstrated how it might be possible to do this through Web collaboration. We show that this approach could be used to create large, high-quality natural language resources.


knowledge discovery and data mining | 2009

A demonstration of human computation using the Phrase Detectives annotation game

Jon Chamberlain; Massimo Poesio; Udo Kruschwitz

The goal of the ANAWIKI project is to experiment with Web collaboration and human computation to create largescale linguistically annotated corpora. We will present ongoing work and initial results of Phrase Detectives, a game designed to collect judgments about anaphoric annotations.


Modeling, Learning, and Processing of Text Technological Data Structures | 2011

Markup Infrastructure for the Anaphoric Bank: Supporting Web Collaboration

Massimo Poesio; Nils Diewald; Maik Stührenberg; Jon Chamberlain; Daniel Jettka; Daniela Goecke; Udo Kruschwitz

Modern NLP systems rely either on unsupervised methods, or on data created as part of governmental initiatives such as MUC, ACE, or GALE. The data created in these efforts tend to be annotated according to task-specific schemes. The Anaphoric Bank is an attempt to create large quantities of data annotated with anaphoric information according to a general purpose and linguistically motivated scheme. We do this by pooling smaller amounts of data annotated according to rich schemes that are by and large compatible, and by taking advantage of Web collaboration. In this chapter we discuss the markup infrastructure that underpins the two modalities of Web collaboration in the project: expert annotation and game-based annotation.


Handbook of Human Computation | 2013

Methods for Engaging and Evaluating Users of Human Computation Systems

Jon Chamberlain; Udo Kruschwitz; Massimo Poesio

One of the most significant challenges facing some Human Computation Systems is how to motivate participation on a scale required to produce high quality data. This chapter discusses methods that can be used to design the task interface, motivate users and evaluate the system, using as an example Phrase Detectives, a game-with-a-purpose to collect data on anaphoric co-reference in text.


JMIR medical informatics | 2017

Expert Search Strategies: The Information Retrieval Practices of Healthcare Information Professionals

Tony Russell-Rose; Jon Chamberlain

Background Healthcare information professionals play a key role in closing the knowledge gap between medical research and clinical practice. Their work involves meticulous searching of literature databases using complex search strategies that can consist of hundreds of keywords, operators, and ontology terms. This process is prone to error and can lead to inefficiency and bias if performed incorrectly. Objective The aim of this study was to investigate the search behavior of healthcare information professionals, uncovering their needs, goals, and requirements for information retrieval systems. Methods A survey was distributed to healthcare information professionals via professional association email discussion lists. It investigated the search tasks they undertake, their techniques for search strategy formulation, their approaches to evaluating search results, and their preferred functionality for searching library-style databases. The popular literature search system PubMed was then evaluated to determine the extent to which their needs were met. Results The 107 respondents indicated that their information retrieval process relied on the use of complex, repeatable, and transparent search strategies. On average it took 60 minutes to formulate a search strategy, with a search task taking 4 hours and consisting of 15 strategy lines. Respondents reviewed a median of 175 results per search task, far more than they would ideally like (100). The most desired features of a search system were merging search queries and combining search results. Conclusions Healthcare information professionals routinely address some of the most challenging information retrieval problems of any profession. However, their needs are not fully supported by current literature search systems and there is demand for improved functionality, in particular regarding the development and management of search strategies.


Proceedings of the First International Workshop on Gamification for Information Retrieval | 2014

The annotation-validation (AV) model: rewarding contribution using retrospective agreement

Jon Chamberlain

Evaluating contributions from users of systems with large datasets is a challenge across many domains, from task assessment in crowdsourcing to document relevance in information retrieval. This paper introduces a model for rewarding and evaluating users using retrospective validation, with only a small gold standard required to initiate the system. A simulation of the model shows that users are rewarded appropriately for high quality responses however analysis of data from an implementation of the model in a text annotation game indicates it may not be sophisticated enough to predict user performance.


european conference on information retrieval | 2016

Real-World Expertise Retrieval: The Information Seeking Behaviour of Recruitment Professionals

Tony Russell-Rose; Jon Chamberlain

Recruitment professionals perform complex search tasks in order to find candidates that match client job briefs. In completing these tasks, they have to contend with many core Information Retrieval (IR) challenges such as query formulation and refinement and results evaluation. However, despite these and other similarities with more established information professions such as patent lawyers and healthcare librarians, this community has been largely overlooked in IR research. This paper presents results of a survey of recruitment professionals, investigating their information seeking behaviour and needs regarding IR systems and applications.


Information Technology | 2018

Optimising crowdsourcing efficiency: Amplifying human computation with validation

Jon Chamberlain; Udo Kruschwitz; Massimo Poesio

Abstract Crowdsourcing has revolutionised the way tasks can be completed but the process is frequently inefficient, costing practitioners time and money. This research investigates whether crowdsourcing can be optimised with a validation process, as measured by four criteria: quality; cost; noise; and speed. A validation model is described, simulated and tested on real data from an online crowdsourcing game to collect data about human language. Results show that by adding an agreement validation (or a like/upvote) step fewer annotations are required, noise and collection time are reduced and quality may be improved.

Collaboration


Dive into the Jon Chamberlain's collaboration.

Top Co-Authors

Avatar
Top Co-Authors

Avatar
Top Co-Authors

Avatar

Livio Robaldo

University of Luxembourg

View shared research outputs
Top Co-Authors

Avatar
Top Co-Authors

Avatar
Top Co-Authors

Avatar

Leif Azzopardi

University of Strathclyde

View shared research outputs
Top Co-Authors

Avatar
Top Co-Authors

Avatar

Karën Fort

Centre national de la recherche scientifique

View shared research outputs
Top Co-Authors

Avatar
Top Co-Authors

Avatar
Researchain Logo
Decentralizing Knowledge