Stefan Wermter

Connectionist, Statistical, and Symbolic Approaches to Learning for Natural Language Processing

Learning approaches for natural language processing.- Separating learning and representation.- Natural language grammatical inference: A comparison of recurrent neural networks and machine learning methods.- Extracting rules for grammar recognition from Cascade-2 networks.- Generating English plural determiners from semantic representations: A neural network learning approach.- Knowledge acquisition in concept and document spaces by using self-organizing neural networks.- Using hybrid connectionist learning for speech/language analysis.- SKOPE: A connectionist/symbolic architecture of spoken Korean processing.- Integrating different learning approaches into a multilingual spoken language translation system.- Learning language using genetic algorithms.- A statistical syntactic disambiguation program and what it learns.- Training stochastic grammars on semantical categories.- Learning restricted probabilistic link grammars.- Learning PP attachment from corpus statistics.- A minimum description length approach to grammar inference.- Automatic classification of dialog acts with Semantic Classification Trees and Polygrams.- Sample selection in natural language learning.- Learning information extraction patterns from examples.- Implications of an automatic lexical acquisition system.- Using learned extraction patterns for text classification.- Issues in inductive learning of domain-specific text extraction rules.- Applying machine learning to anaphora resolution.- Embedded machine learning systems for natural language processing: A general framework.- Acquiring and updating hierarchical knowledge for machine translation based on a clustering technique.- Applying an existing machine learning algorithm to text categorization.- Comparative results on using inductive logic programming for corpus-based parser construction.- Learning the past tense of English verbs using inductive logic programming.- A dynamic approach to paradigm-driven analogy.- Can punctuation help learning?.- Using parsed corpora for circumventing parsing.- A symbolic and surgical acquisition of terms through variation.- A revision learner to acquire verb selection rules from human-made rules and examples.- Learning from texts - A terminological metareasoning perspective.

Emergent Neural Computational Architectures based on Neuroscience

This book is the result of a series of International Workshops organised bythe EmerNet project on Emergent Neural Computational Architectures basedon Neuroscience sponsored by the Engineering and Physical Sciences ResearchCouncil (EPSRC). The overall aim of the book is to present a broad spectrum ofcurrent research into biologically inspired computational systems and hence en-courage the emergence of new computational approaches based on neuroscience.It is generally understood that the present approaches for computing do not havethe performance, exibility and reliability of biological information processingsystems. Although there is a massive body of knowledge regarding how process-ing occurs in the brain and central nervous system this has had little impact onmainstream computing so far.The process of developing biologically inspired computerised systems involvesthe examination of the functionality and architecture of the brain with an empha-sis on the information processing activities. Biologically inspired computerisedsystems address neural computation from the position of both neuroscience,and computing by using experimental evidence to create general neuroscience-inspired systems.The book focuses on the main research areas of modular organisation androbustness, timing and synchronisation, and learning and memory storage. Theissues considered as part of these include: How can the modularity in the brainbe used to produce large scale computational architectures? How does the hu-man memory manage to continue to operate despite failure of its components?How does the brain synchronise its processing? How does the brain computewith relatively slow computing elements but still achieve rapid and real-timeperformance? How can we build computational models of these processes andarchitectures? How can we design incremental learning algorithms and dynamicmemory architectures? How can the natural information processing systems beexploited for arti cial computational methods?We hope that this book stimulates and encourages new research in this area.We would like to thank all contributors to this book and the few hundred partici-pants of the various workshops. Especially we would like to express our thanks toMark Elshaw, network assistant in the EmerNet network who put in tremendouse ort during the process of publishing this book.Finally, we would like to thank EPSRC and James Fleming for their supportand Alfred Hofmann and his sta at Springer for their continuing assistance.March 2001Stefan WermterJim AustinDavid Willshaw

Data mining using rule extraction from Kohonen self-organising maps

The Kohonen self-organising feature map (SOM) has several important properties that can be used within the data mining/knowledge discovery and exploratory data analysis process. A key characteristic of the SOM is its topology preserving ability to map a multi-dimensional input into a two-dimensional form. This feature is used for classification and clustering of data. However, a great deal of effort is still required to interpret the cluster boundaries. In this paper we present a technique which can be used to extract propositional IF..THEN type rules from the SOM network’s internal parameters. Such extracted rules can provide a human understandable description of the discovered clusters.

Neural Network Agents for Learning Semantic Text Classification

The research project AgNeT develops Agents for Neural Text routing in the internet. Unrestricted potentially faulty text messages arrive at a certain delivery point (e.g. email address or world wide web address). These text messages are scanned and then distributed to one of several expert agents according to a certain task criterium. Possible specific scenarios within this framework include the learning of the routing of publication titles or news titles. In this paper we describe extensive experiments for semantic text routing based on classified library titles and newswire titles. This task is challenging since incoming messages may contain constructions which have not been anticipated. Therefore, the contributions of this research are in learning and generalizing neural architectures for the robust interpretation of potentially noisy unrestricted messages. Neural networks were developed and examined for this topic since they support robustness and learning in noisy unrestricted real-world texts. We describe and compare different sets of experiments. The first set of experiments tests a recurrent neural network for the task of library title classification. Then we describe a larger more difficult newswire classification task from information retrieval. The comparison of the examined models demonstrates that techniques from information retrieval integrated into recurrent plausibility networks performed well even under noise and for different corpora.

Robot Docking with Neural Vision and Reinforcement

We present a solution for robotic docking, i.e. the approach of a robot toward a table so that it can grasp an object. One constraint is that our PeopleBot robot has a short non-extendable gripper and wide “shoulders”. Therefore it must approach the table at a perpendicular angle so that the gripper can reach over it. Another constraint is the use of vision to locate the object. Only the angle is supplied as additional input.

Self-organizing neural integration of pose-motion features for human action recognition

The visual recognition of complex, articulated human movements is fundamental for a wide range of artificial systems oriented toward human-robot communication, action classification, and action-driven perception. These challenging tasks may generally involve the processing of a huge amount of visual information and learning-based mechanisms for generalizing a set of training actions and classifying new samples. To operate in natural environments, a crucial property is the efficient and robust recognition of actions, also under noisy conditions caused by, for instance, systematic sensor errors and temporarily occluded persons. Studies of the mammalian visual system and its outperforming ability to process biological motion information suggest separate neural pathways for the distinct processing of pose and motion features at multiple levels and the subsequent integration of these visual cues for action perception. We present a neurobiologically-motivated approach to achieve noise-tolerant action recognition in real time. Our model consists of self-organizing Growing When Required (GWR) networks that obtain progressively generalized representations of sensory inputs and learn inherent spatio-temporal dependencies. During the training, the GWR networks dynamically change their topological structure to better match the input space. We first extract pose and motion features from video sequences and then cluster actions in terms of prototypical pose-motion trajectories. Multi-cue trajectories from matching action frames are subsequently combined to provide action dynamics in the joint feature space. Reported experiments show that our approach outperforms previous results on a dataset of full-body actions captured with a depth sensor, and ranks among the best results for a public benchmark of domestic daily actions.

Biomimetic Neural Learning for Intelligent Robots

Towards Biomimetic Neural Learning for Intelligent Robots.- Towards Biomimetic Neural Learning for Intelligent Robots.- I: Biomimetic Multimodal Learning in Neuron-Based Robots.- The Intentional Attunement Hypothesis The Mirror Neuron System and Its Role in Interpersonal Relations.- Sequence Detector Networks and Associative Learning of Grammatical Categories.- A Distributed Model of Spatial Visual Attention.- A Hybrid Architecture Using Cross-Correlation and Recurrent Neural Networks for Acoustic Tracking in Robots.- Image Invariant Robot Navigation Based on Self Organising Neural Place Codes.- Detecting Sequences and Understanding Language with Neural Associative Memories and Cell Assemblies.- Combining Visual Attention, Object Recognition and Associative Information Processing in a NeuroBotic System.- Towards Word Semantics from Multi-modal Acoustico-Motor Integration: Application of the Bijama Model to the Setting of Action-Dependant Phonetic Representations.- Grounding Neural Robot Language in Action.- A Spiking Neural Network Model of Multi-modal Language Processing of Robot Instructions.- II: Biomimetic Cognitive Behaviour in Robots.- A Virtual Reality Platform for Modeling Cognitive Development.- Learning to Interpret Pointing Gestures: Experiments with Four-Legged Autonomous Robots.- Reinforcement Learning Using a Grid Based Function Approximator.- Spatial Representation and Navigation in a Bio-inspired Robot.- Representations for a Complex World: Combining Distributed and Localist Representations for Learning and Planning.- MaximumOne: An Anthropomorphic Arm with Bio-inspired Control System.- LARP, Biped Robotics Conceived as Human Modelling.- Novelty and Habituation: The Driving Forces in Early Stage Learning for Developmental Robotics.- Modular Learning Schemes for Visual Robot Control.- Neural Robot Detection in RoboCup.- A Scale Invariant Local Image Descriptor for Visual Homing.

Hybrid neural document clustering using guided self-organization and WordNet

Document clustering is text processing that groups documents with similar concepts. Its usually considered an unsupervised learning approach because theres no teacher to guide the training process, and topical information is often assumed to be unavailable. A guided approach to document clustering that integrates linguistic top-down knowledge from WordNet into text vector representations based on the extended significance vector weighting technique improves both classification accuracy and average quantization error. In our guided self-organization approach we integrate topical and semantic information from WordNet. Because a document-training set with preclassified information implies relationships between a word and its preference class, we propose a novel document vector representation approach to extract these relationships for document clustering. Furthermore, merging statistical methods, competitive neural models, and semantic relationships from symbolic Word-Net, our hybrid learning approach is robust and scales up to a real-world task of clustering 100,000 news documents.

Knowledge extraction from radial basis function networks and multilayer perceptrons

This paper deals with an evaluation and comparison of the accuracy and complexity of symbolic rules extracted from radial basis function networks and multilayer perceptrons. Here we examine the ability of rule extraction algorithms to extract meaningful rules that describe the overall performance of a particular network. In addition, the paper also highlights the suitability of a specific neural network architecture for particular classification problems. The study carried out on the extracted rule quality and complexity also has a direct bearing on the use of rule extraction algorithms for data mining and knowledge discovery.

SCREEN: learning a flat syntactic and semantic spoken language analysis using artificial neural networks

Previous approaches of analyzing spontaneously spoken language often have been based on encoding syntactic and semantic knowledge manually and symbolically. While there has been some progress using statistical or connectionist language models, many current spoken-language systems still use a relatively brittle, hand-coded symbolic grammar or symbolic semantic component. In contrast, we describe a so-called screening approach for learning robust processing of spontaneously spoken language. A screening approach is a flat analysis which uses shallow sequences of category representations for analyzing an utterance at various syntactic, semantic and dialog levels. Rather than using a deeply structured symbolic analysis, we use a flat connectionist analysis. This screening approach aims at supporting speech and language processing by using (1) data-driven learning and (2) robustness of connectionist networks. In order to test this approach, we have developed the screen system which is based on this new robust, learned and flat analysis. In this paper, we focus on a detailed description of screens architecture, the flat syntactic and semantic analysis, the interaction with a speech recognizer, and a detailed evaluation analysis of the robustness under the influence of noisy or incomplete input. The main result of this paper is that flat representations allow more robust processing of spontaneous spoken language than deeply structured representations. In particular, we show how the fault-tolerance and learning capability of connectionist networks can support a flat analysis for providing more robust spoken-language processing within an overall hybrid symbolic/connectionist framework.