Alessandro Sperduti

Speed up learning and network optimization with extended back propagation

Abstract Methods to speed up learning in back propagation and to optimize the network architecture have been recently studied. This paper shows how adaptation of the steepness of the sigmoids during learning treats these two topics in a common framework. The adaptation of the steepness of the sigmoids is obtained by gradient descent. The resulting learning dynamics can be simulated by a standard network with fixed sigmoids and a learning rule whose main component is a gradient descent with adaptive learning parameters. A law linking variation on the weights to variation on the steepness of the sigmoids is discovered. Optimization of units is obtained by introducing a tendency to decay to zero in the steepness values. This decay corresponds to a decay of the sensitivity of the units. Units with low final sensitivity can be removed after a given transformation of the biases of the network. A decreasing initial distribution of the steepness values is suggested to obtain a good compromise between speed of learning and network optimization. Simulation of the proposed procedure has shown an improvement of the mean convergence rate with respect to the standard back propagation and good optimization performance. Several 4-3-1 networks for the four bits parity problem were discovered.

An improved boosting algorithm and its application to text categorization

We describe an improved boosting algorithm, called {\sc AdaBoost.MH

Application of Cascade Correlation Networks for Structures toChemistry

^{KR}

Logo Recognition by Recursive Neural Networks

}, and its application to text categorization. Boosting is a method for supervised learning which has successfully been applied to many different domains, and that has proven one of the best performers in text categorization exercises so far. Boosting is based on the idea of relying on the collective judgment of a committee of classifiers that are trained sequentially. In training the

On the computational power of recurrent neural networks for structures

i

A Novel Approach to QSPR/QSAR Based on Neural Networks for Structures

-th classifier special emphasis is placed on the correct categorization of the training documents which have proven harder for the previously trained classifiers. {\sc AdaBoost.MH

Extended Cascade-Correlation for Syntactic and Structural Pattern Recognition

^{KR}

Neural Networks for Adaptive Processing of Structured Data

} is based on the idea to build, at every iteration of the learning phase, not a single classifier but a sub-committee of the

A Supervised Self-Organizing Map for Structured Data

K

Discriminant Pattern Recognition Using Transformation-Invariant Neurons

classifiers which, at that iteration, look the most promising. We report the results of systematic experimentation of this method performed on the standard {\sf Reuters-21578} benchmark. These experiments have shown that {\sc AdaBoost.MH