Neil Ireson

Evaluating machine learning for information extraction

Comparative evaluation of Machine Learning (ML) systems used for Information Extraction (IE) has suffered from various inconsistencies in experimental procedures. This paper reports on the results of the Pascal Challenge on Evaluating Machine Learning for Information Extraction, which provides a standardised corpus, set of tasks, and evaluation methodology. The challenge is described and the systems submitted by the ten participants are briefly introduced and their performance is analysed.

Real-Time Detection, Tracking, and Monitoring of Automatically Discovered Events in Social Media

We introduce ReDites, a system for realtime event detection, tracking, monitoring and visualisation. It is designed to assist Information Analysts in understanding and exploring complex events as they unfold in the world. Events are automatically detected from the Twitter stream. Then those that are categorised as being security-relevant are tracked, geolocated, summarised and visualised for the end-user. Furthermore, the system tracks changes in emotions over events, signalling possible flashpoints or abatement. We demonstrate the capabilities of ReDites using an extended use case from the September 2013 Westgate shooting incident. Through an evaluation of system latencies, we also show that enriched events are made available for users to explore within seconds of that event occurring.

Evaluation of machine learning-based information extraction algorithms: criticisms and recommendations

We survey the evaluation methodology adopted in information extraction (IE), as defined in a few different efforts applying machine learning (ML) to IE. We identify a number of critical issues that hamper comparison of the results obtained by different researchers. Some of these issues are common to other NLP-related tasks: e.g., the difficulty of exactly identifying the effects on performance of the data (sample selection and sample size), of the domain theory (features selected), and of algorithm parameter settings. Some issues are specific to IE: how leniently to assess inexact identification of filler boundaries, the possibility of multiple fillers for a slot, and how the counting is performed. We argue that, when specifying an IE task, these issues should be explicitly addressed, and a number of methodological characteristics should be clearly defined. To empirically verify the practical impact of the issues mentioned above, we perform a survey of the results of different algorithms when applied to a few standard datasets. The survey shows a serious lack of consensus on these issues, which makes it difficult to draw firm conclusions on a comparative evaluation of the algorithms. Our aim is to elaborate a clear and detailed experimental methodology and propose it to the IE community. Widespread agreement on this proposal should lead to future IE comparative evaluations that are fair and reliable. To demonstrate the way the methodology is to be applied we have organized and run a comparative evaluation of ML-based IE systems (the Pascal Challenge on ML-based IE) where the principles described in this article are put into practice. In this article we describe the proposed methodology and its motivations. The Pascal evaluation is then described and its results presented.

Local community situational awareness during an emergency

With the recent exponential growth in the use of public forums for communication and sharing of information, Emergency Response organisations are realising that there is a potential to exploit these forums to extract information to improve the overall awareness of the events that occur during an emergency. This paper describes an approach using information extraction, topic and event identification and its application in a case study. The results indicate that despite the ill-formed, variable quality and conversational nature of forum posts a degree of coherent event information can be acquired.

Design of a Traffic Junction Controller Using Classifier Systems and Fuzzy Logic

Traffic control in large cities is a difficult and non-trivial optimization problem. Most of the automated urban traffic control systems are based on deterministic algorithms and have a multi-level architecture; to achieve global optimality, hierarchical control algorithms are generally employed. However, these algorithms are often slow to react to varying conditions, and it has been recognized that incorporating computational intelligence into the lower levels can remove some burdens of algorithm calculation and decision making from higher levels. An alternative approach is to use a fully distributed architecture in which there is effectively only one (low) level of control. Such systems are aimed at increasing the response time of the controller and, again, these often incorporate computational intelligence techniques. This paper presents preliminary work into designing an intelligent local controller primarily for distributed traffic control systems. The idea is to use a classifier system with a fuzzy rule representation to determine useful junction control rules within the dynamic environment.

Toponym resolution in social media

Increasingly user-generated content is being utilised as a source of information, however each individual piece of content tends to contain low levels of information. In addition, such information tends to be informal and imperfect in nature; containing imprecise, subjective, ambiguous expressions. However the content does not have to be interpreted in isolation as it is linked, either explicitly or implicitly, to a network of interrelated content; it may be grouped or tagged with similar content, comments may be added by other users or it may be related to other content posted at the same time or by the same author or members of the authors social network. This paper generally examines how ambiguous concepts within user-generated content can be assigned a specific/formal meaning by considering the expanding context of the information, i.e. other information contained within directly or indirectly related content, and specifically considers the issue of toponym resolution of locations.

Human-centred decision support: The IDIOMS system

The requirement for anthropocentric, or human-centred decision support is outlined, and the IDIOMS management information tool, which implements several human-centred principles, is described. IDIOMS provides a flexible decision support environment in which applications can be modelled using both ‘objective’ database information, and user-centred ‘subjective’ and contextual information. The system has been tested on several real applications, demonstrating its power and flexibility.

Distributed Learning Control of Traffic Signals

This paper presents a distributed learning control strategy for traffic signals. The strategy uses a fully distributed architecture in which there is effectively only one (low) level of control. Such strategy is aimed at incorporating computational intelligence techniques into the control system to increase the response time of the controller. The idea is implemented by employing learning classifier systems and TCP/IP based communication server, which supports the communication service in the control system. Simulation results in a simplified traffic network show that the control strategy can determine useful control rules within the dynamic traffic environment, and thus improve the traffic conditions.

Knowledge sharing in E-collaboration

For eCollaboration to be effective, especially where it attempts to promote true collective decision-making, it is necessary to consider how knowledge is shared. The paper examines the knowledge sharing literature from the perspective of eCollaboration and discusses the critical challenges, principally the motivation of knowledge sources and maintenance of semantics, and describes how techniques and technologies can be employed to alleviate the difficulties. The paper concludes with an example of how such technologies are being applied for Emergency Response, to facilitate knowledge sharing both amongst the citizens and between the citizens and organisations.

A Communication Architecture for Multi-agent Learning Systems

This paper presents a simple communication architecture for Multi-Agent Learning Systems. The service provided by the communication architecture allows each agent to connect to the user interface, the application and the other agents. The communication architecture is implemented using TCP/IP. An application example in a simplified traffic environment shows that the communication architecture can provide reliable and efficient communication services for Multi-Agent Learning Systems.