Josep M. Sopena

Feature selection forcing overtraining may help to improve performance

One of the main drawbacks of machine learning systems is the negative effect caused by overtraining. If the points in the dataset are perfectly fitted, the generalization performance is usually bad. We propose to take profit of overtraining, together with Feature Selection, to improve the performance of a learning system. The main idea lies in the hypothesis that when the dataset is as fitted as possible, the system is forced to use all the available variables as much as possible. Noisy and useless variables can be detected if generalization improves when the system is not allowed to use them. Forcing overtraining, noisy and useless variables should be more outstanding. In order to test this hypothesis, we performed several Feature Selection experiments using Feedforward Neural Networks. The particular Feature Selection procedure used was Sequential Backward Selection. Experimental results with several real-world problems suggest that our hypothesis seems to be well-founded. Ironically, forcing overtraining may help to achieve good performance.

A Connectionist Approach to Prepositional Phrase Attachment for Real World Tuts

In this paper we describe a neural network-based approach to prepositional phrase attachment disam biguation for real world texts. Although the use of semantic classes in this task seems intuitively to be adequate, methods employed to date have not used them very effectively. Causes of their poor results are discussed. Our model, which uses only classes, scores appreciably better than the other class-based methods which have been tested on the Wall Street Journal corpus. To date, the best result obtained using only classes was a score of 79.1%; we obtained an accuracy score of 86.8%. This score is among the best reported in the literature using this corpus.

Composicionalidad, cómputo de estructura y redes neuronales

Resumen Los problemas que presentan los modelos neuronales de procesamiento del lenguajey la representación del significado derivan de dos problemas principales: el problema del ‘binding’ y el problema de la composicionalidad. A su vez estos dos problemas derivan del problema de cómo representar estímulos complejos con estructura interna, como son las oraciones, mediante vectores de activación. En este artículo presentamos un modelo neuronal de procesamiento de lenguaje (ANNLP) que resuelve estos dos problemas y cuyo rendimiento es muy eficiente procesando textos reales para un amplio abanico de tareas que incluyen la desambiguación del sentido en contextos proposicionales, la desambiguación sintáctica y el parsing. Para conseguir este nivel de eficiencia hemos tenido que dotar al modelo de un conjunto de características cuyo elemento central reside en la propuesta de cómo computar estructura (estructura sintáctica y de significado) mediante vectores de activación. Se aporta evidencia a favor de que el modelo y en concreto la propuesta de cómo representar estructura no es una simple solución ingenierila los problemas del binding y la composicionalidad, sino que es plausible psicológicamente.

Variable Binding Using Serial Order in Recurrent Neural Networks

The scope of this paper is the variable binding problem in connectionist reasoning. We present a new approach to the problem, which differs in several aspects from former solutions. Mainly, an inductive reasoning stance, which seems more suitable for neural networks, is taken into account. The proposed solution deals with some of the limitations that are often present in earlier models, such as cross-talk, hardware limitations and static knowledge. A modified Elman architecture (Sopena, 1991) is used to store and retrieve items sequentially, so that temporal order allows us to bind different predicates which are represented by distributed patterns. This process makes it possible to represent conditional rules with a good degree of generalization. Distributed representations and temporal order offer a simpler treatment of variable binding than localist and spatial models do.