Marco Gori

On the problem of local minima in backpropagation

The authors propose a theoretical framework for backpropagation (BP) in order to identify some of its limitations as a general learning procedure and the reasons for its success in several experiments on pattern recognition. The first important conclusion is that examples can be found in which BP gets stuck in local minima. A simple example in which BP can get stuck during gradient descent without having learned the entire training set is presented. This example guarantees the existence of a solution with null cost. Some conditions on the network architecture and the learning environment that ensure the convergence of the BP algorithm are proposed. It is proven in particular that the convergence holds if the classes are linearly separable. In this case, the experience gained in several experiments shows that multilayered neural networks (MLNs) exceed perceptrons in generalization to new examples. >

A survey of hybrid ANN/HMM models for automatic speech recognition

Abstract In spite of the advances accomplished throughout the last decades, automatic speech recognition (ASR) is still a challenging and difficult task. In particular, recognition systems based on hidden Markov models (HMMs) are effective under many circumstances, but do suffer from some major limitations that limit applicability of ASR technology in real-world environments. Attempts were made to overcome these limitations with the adoption of artificial neural networks (ANN) as an alternative paradigm for ASR, but ANN were unsuccessful in dealing with long time-sequences of speech signals. Between the end of the 1980s and the beginning of the 1990s, some researchers began exploring a new research area, by combining HMMs and ANNs within a single, hybrid architecture. The goal in hybrid systems for ASR is to take advantage from the properties of both HMMs and ANNs, improving flexibility and recognition performance. A variety of different architectures and novel training algorithms have been proposed in literature. This paper reviews a number of significant hybrid models for ASR, putting together approaches and techniques from a highly specialistic and non-homogeneous literature. Efforts concentrate on describing and referencing architectures and algorithms, their advantages and limitations, as well as on categorizing them into broad classes. Early attempts to emulate HMMs by ANNs are first described. Then we focus on ANNs to estimate posterior probabilities of the states of an HMM and on “global” optimization, where a single, overall training criterion is defined over the HMM and the ANNs. Connectionist vector quantization for discrete HMMs, and other more recent approaches are also reviewed. It is pointed out that, in addition to their theoretical interest, hybrid systems have been allowing for tangible improvements in recognition performance over the standard HMMs in difficult and significant benchmark tasks.

Representation of finite state automata in recurrent radial basis function networks

In this paper, we propose some techniques for injecting finite state automata into Recurrent Radial Basis Function networks (R2BF). When providing proper hints and constraining the weight space properly, we show that these networks behave as automata. A technique is suggested for forcing the learning process to develop automata representations that is based on adding a proper penalty function to the ordinary cost. Successful experimental results are shown for inductive inference of regular grammars.

On the problem of local minima in recurrent neural networks

Many researchers have recently focused their efforts on devising efficient algorithms, mainly based on optimization schemes, for learning the weights of recurrent neural networks. As in the case of feedforward networks, however, these learning algorithms may get stuck in local minima during gradient descent, thus discovering sub-optimal solutions. This paper analyses the problem of optimal learning in recurrent networks by proposing conditions that guarantee local minima free error surfaces. An example is given that also shows the constructive role of the proposed theory in designing networks suitable for solving a given task. Moreover, a formal relationship between recurrent and static feedforward networks is established such that the examples of local minima for feedforward networks already known in the literature can be associated with analogous ones in recurrent networks.

Optimal learning in artificial neural networks: A review of theoretical results

Abstract The effectiveness of connectionist models in emulating intelligent behaviour and solving significant practical problems is strictly related to the capability of the learning algorithms to find optimal or near-optimal solutions and to generalize to new examples. This paper reviews some theoretical contributions to optimal learning in the attempt to provide a unified view and give the state of the art in the field. The focus of the review is on the problem of local minima in the cost function that is likely to affect more or less any learning algorithm. Starting from this analysis, we briefly review proposals for discovering optimal solutions and suggest conditions for designing architectures tailored to a given task.

Learning the dynamic nature of speech with back-propagation for sequences

Abstract A novel learning algorithm is proposed, called Back-Propagation for Sequences (BPS), for a particular class of dynamic neural networks in which some units have local feedback. These networks can be trained to respond to sequences of input patterns and seem particularly suited for phoneme recognition. They exhibit a forgetting behavior and consequently only recently past information is taken into account for classification purposes. BPS permits online weight updating and it has the same time complexity and space requirements as back-propagation (BP) applied to feedforward networks. We present experimental results for problems connected with Automatic Speech Recognition.

Autoassociator-based models for speaker verification

In this paper, we propose an autoassociator-based connectionist model that turns out to be very useful for problems of pattern verification. The model is based on feedforward networks acting as autoassociators trained to reproduce patterns presented at the input to the output layer. The verification is established on the basis of the distance between the input and the output vectors. We give experimental results for assessing the effectiveness of the model for problems of speech verification. The performances were evaluated on DARPA-TIMIT database in continuous speech, using different thresholds and preprocessing schemes, with very promising results.

Terminal attractor algorithms: A critical analysis

Abstract One of the fundamental drawbacks of learning by gradient descent techniques is the susceptibility to local minima during training. Recently, some authors have independently introduced new learning algorithms that are based on the properties of terminal attractors and repellers. These algorithms were claimed to perform global optimization of the cost in finite time, provided that a null solution exists. In this paper, we prove that, in the case of local minima free error functions, terminal attractor algorithms guarantee that the optimal solution is reached in a number of steps that is independent of the cost function. Moreover, in the case of multimodal functions, we prove that, unfortunately, there are no theoretical guarantees that a global solution can be reached or that the algorithms perform satisfactorily from an operational point of view, unless particular favourable conditions are satisfied. On the other hand, the ideas behind these innovative methods are very interesting and deserve further investigations.

A two level knowledge approach for understanding documents of a multi-class domain

In this paper an architecture for understanding documents of a domain that can be grouped into classes is shown. Documents are grouped with respect to the physical structure. The architecture is based on two knowledge descriptions of the domain: one is independent from the classes and one related to the classes. Such knowledge levels are used to understand the documents of the domain. The understanding phase is described in relation with the phases of analysis and classification of such documents.

Links between LVQ and Backpropagation

Abstract In this paper we show that there are some intriguing links between the Backpropagation and LVQ algorithms. We show that Backpropagation used for training the weights of radial basis function networks exhibits an increasing competitive nature as the dispersion parameters decrease. In particular, we prove that LVQ can be regarded as a competitive learning scheme taking place in radial basis function networks.