Peter C. R. Lane

Incremental syntactic parsing of natural language corpora with simple synchrony networks

The article explores the use of Simple Synchrony Networks (SSNs) for learning to parse English sentences drawn from a corpus of naturally occurring text. Parsing natural language sentences requires taking a sequence of words and outputting a hierarchical structure representing how those words fit together to form constituents. Feedforward and simple recurrent networks have had great difficulty with this task, in part because the number of relationships required to specify a structure is too large for the number of unit outputs they have available. SSNs have the representational power to output the necessary O(n/sup 2/) possible structural relationships because SSNs extend the O(n) incremental outputs of simple recurrent networks with the O(n) entity outputs provided by temporal synchrony variable binding. The article presents an incremental representation of constituent structures which allows SSNs to make effective use of both these dimensions. Experiments on learning to parse naturally occurring text show that this output format supports both effective representation and effective generalization in SSNs. To emphasize the importance of this generalization ability, the article also proposes a short-term memory mechanism for retaining a bounded number of constituents during parsing. This mechanism improves the O(n/sup 2/) speed of the basic SSN architecture to linear time, but experiments confirm that the generalization ability of SSN networks is maintained.

On developing robust models for favourability analysis: Model choice, feature sets and imbalanced data

Locating documents carrying positive or negative favourability is an important application within media analysis. This article presents some empirical results on the challenges facing a machine-learning approach to this kind of opinion mining. Some of the challenges include the often considerable imbalance in the distribution of positive and negative samples, changes in the documents over time, and effective training and evaluation procedures for the models. This article presents results on three data sets generated by a media-analysis company, classifying documents in two ways: detecting the presence of favourability, and assessing negative vs. positive favourability. We describe our experiments in developing a machine-learning approach to automate the classification process. We explore the effect of using five different types of features, the robustness of the models when tested on data taken from a later time period, and the effect of balancing the input data by undersampling. We find varying choices for the optimum classifier, feature set and training strategy depending on the task and data set.

A Connectionist Architecture for Learning to Parse

We present a connectionist architecture and demonstrate that it can learn syntactic parsing from a corpus of parsed text. The architecture can represent syntactic constituents, and can learn generalizations over syntactic constituents, thereby addressing the sparse data problems of previous connectionist architectures. We apply these Simple Synchrony Networks to mapping sequences of word tags to parse trees. After training on parsed samples of the Brown Corpus, the networks achieve precision and recall on constituents that approaches that of statistical methods for this task.

Copy detection in Chinese documents using Ferret

The Ferret copy detector has been used since 2001 to find plagiarism in large collections of students’ coursework in English. This article reports on extending its application to Chinese, with experiments on corpora of coursework collected from two Chinese universities. Our experiments show that Ferret can find both artificially constructed plagiarism and actually occurring, previously undetected plagiarism. We discuss issues of representation, focus on the effectiveness of a sub-symbolic approach, and show that Ferret does not need to find word boundaries first.

Discovering predictive variables when evolving cognitive models

A non-dominated sorting genetic algorithm is used to evolve models of learning from different theories for multiple tasks. Correlation analysis is performed to identify parameters which affect performance on specific tasks; these are the predictive variables. Mutation is biased so that changes to parameter values tend to preserve values within the populations current range. Experimental results show that optimal models are evolved, and also that uncovering predictive variables is beneficial in improving the rate of convergence.

Developing reproducible and comprehensible computational models

Quantitative predictions for complex scientific theories are often obtained by running simulations on computational models. In order for a theory to meet with wide-spread acceptance, it is important that the model be reproducible and comprehensible by independent researchers. However, the complexity of computational models can make the task of replication all but impossible. Previous authors have suggested that computer models should be developed using high-level specification languages or large amounts of documentation. We argue that neither suggestion is sufficient, as each deals with the prescriptive definition of the model, and does not aid in generalising the use of the model to new contexts. Instead, we argue that a computational model should be released as three components: (a) a well-documented implementation; (b) a set of tests illustrating each of the key processes within the model; and (c) a set of canonical results, for reproducing the models predictions in important experiments. The included tests and experiments would provide the concrete exemplars required for easier comprehension of the model, as well as a confirmation that independent implementations and later versions reproduce the theorys canonical results.

A theory-driven testing methodology for developing scientific software

Computer implementations of theoretical concepts play an ever-increasing role in the development and application of scientific ideas. As the scale of such implementations increases from relatively small models and empirical setups to overarching frameworks from which many kinds of results may be obtained, it is important to consider the methodology by which these implementations are developed. Using cognitive architectures as an example, we discuss the relation between an implementation of an architecture and its underlying theory, a relation between a computer program and its description. We argue for the use of an agile development methodology, based around a three-layer scientific test harness and continuous refactoring, as most suitable for developing scientific software. The ideas are illustrated with extended examples of implementing unified theories of human learning, taken from the chunking and template theories.

Learning Perceptual Schemas to Avoid the Utility Problem

This paper describes principles for representing and organising planning knowledge in a machine learning architecture. One of the difficulties with learning about tasks requiring planning is the utility problem: as more knowledge is acquired by the learner, the utilisation of that knowledge takes on a complexity which overwhelms the mechanisms of the original task. This problem does not, however, occur with human learners: on the contrary, it is usually the case that, the more knowledgeable the learner, the greater the efficiency and accuracy in locating a solution. The reason for this lies in the types of knowledge acquired by the human learner and its organisation. We describe the basic representations which underlie the superior abilities of human experts, and describe algorithms for using equivalent representations in a machine learning architecture.

Attention Mechanisms in the CHREST Cognitive Architecture

In this paper, we describe the attention mechanisms in CHREST, a computational architecture of human visual expertise. CHREST organises information acquired by direct experience from the world in the form of chunks . These chunks are searched for, and verified, by a unique set of heuristics, comprising the attention mechanism. We explain how the attention mechanism combines bottom-up and top-down heuristics from internal and external sources of information. We describe some experimental evidence demonstrating the correspondence of CHRESTs perceptual mechanisms with those of human subjects. Finally, we discuss how visual attention can play an important role in actions carried out by human experts in domains such as chess.

Simple Synchrony Networks : Learning to Parse Natural Language with Temporal Synchrony Variable Binding

The Simple Synchrony Network (SSN) is a new connectionist architecture, incorporating the insights of Temporal Synchrony Variable Binding (TSVB) into Simple Recurrent Networks. The use of TSVB means SSNs can output representations of structures, and can learn generalisations over the constituents of these structures (as required by systematicity). This paper describes the SSN and an associated training algorithm, and demonstrates SSNs’ generalisation abilities through results from training SSNs to parse real natural language sentences.