Monica Bianchini

Learning without local minima in radial basis function networks

Learning from examples plays a central role in artificial neural networks. The success of many learning schemes is not guaranteed, however, since algorithms like backpropagation may get stuck in local minima, thus providing suboptimal solutions. For feedforward networks, optimal learning can be achieved provided that certain conditions on the network and the learning environment are met. This principle is investigated for the case of networks using radial basis functions (RBF). It is assumed that the patterns of the learning environment are separable by hyperspheres. In that case, we prove that the attached cost function is local minima free with respect to all the weights. This provides us with some theoretical foundations for a massive application of RBF in pattern recognition.

On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures

Recently, researchers in the artificial neural network field have focused their attention on connectionist models composed by several hidden layers. In fact, experimental results and heuristic considerations suggest that deep architectures are more suitable than shallow ones for modern applications, facing very complex problems, e.g., vision and human language understanding. However, the actual theoretical results supporting such a claim are still few and incomplete. In this paper, we propose a new approach to study how the depth of feedforward neural networks impacts on their ability in implementing high complexity functions. First, a new measure based on topological concepts is introduced, aimed at evaluating the complexity of the function implemented by a neural network, used for classification purposes. Then, deep and shallow neural architectures with common sigmoidal activation functions are compared, by deriving upper and lower bounds on their complexity, and studying how the complexity depends on the number of hidden units and the used activation function. The obtained results seem to support the idea that deep networks actually implements functions of higher complexity, so that they are able, with the same number of resources, to address more difficult problems.

Learning in multilayered networks used as autoassociators

Gradient descent learning algorithms may get stuck in local minima, thus making the learning suboptimal. In this paper, we focus attention on multilayered networks used as autoassociators and show some relationships with classical linear autoassociators. In addition, by using the theoretical framework of our previous research, we derive a condition which is met at the end of the learning process and show that this condition has a very intriguing geometrical meaning in the pattern space.

On the problem of local minima in recurrent neural networks

Many researchers have recently focused their efforts on devising efficient algorithms, mainly based on optimization schemes, for learning the weights of recurrent neural networks. As in the case of feedforward networks, however, these learning algorithms may get stuck in local minima during gradient descent, thus discovering sub-optimal solutions. This paper analyses the problem of optimal learning in recurrent networks by proposing conditions that guarantee local minima free error surfaces. An example is given that also shows the constructive role of the proposed theory in designing networks suitable for solving a given task. Moreover, a formal relationship between recurrent and static feedforward networks is established such that the examples of local minima for feedforward networks already known in the literature can be associated with analogous ones in recurrent networks.

2005 Special Issue: Recursive neural networks for processing graphs with labelled edges: theory and applications

In this paper, we introduce a new recursive neural network model able to process directed acyclic graphs with labelled edges. The model uses a state transition function which considers the edge labels and is independent both from the number and the order of the children of each node. The computational capabilities of the new recursive architecture are assessed. Moreover, in order to test the proposed architecture on a practical challenging application, the problem of object detection in images is also addressed. In fact, the localization of target objects is a preliminary step in any recognition system. The proposed technique is general and can be applied in different detection systems, since it does not exploit any a priori knowledge on the particular problem. Some experiments on face detection, carried out on scenes acquired by an indoor camera, are reported, showing very promising results.

Optimal learning in artificial neural networks: A review of theoretical results

Abstract The effectiveness of connectionist models in emulating intelligent behaviour and solving significant practical problems is strictly related to the capability of the learning algorithms to find optimal or near-optimal solutions and to generalize to new examples. This paper reviews some theoretical contributions to optimal learning in the attempt to provide a unified view and give the state of the art in the field. The focus of the review is on the problem of local minima in the cost function that is likely to affect more or less any learning algorithm. Starting from this analysis, we briefly review proposals for discovering optimal solutions and suggest conditions for designing architectures tailored to a given task.

Recursive neural networks learn to localize faces

Localizing faces in images is a difficult task, and represents the first step towards the solution of the face recognition problem. Moreover, devising an effective face detection method can provide some suggestions to solve similar object and pattern detection problems. This paper presents a novel approach to the solution of the face localization problem using Recursive neural networks (RNNs). The proposed method assumes a graph-based representation of images that combines structural and symbolic visual features. Such graphs are then processed by RNNs, in order to establish the possible presence and the position of faces inside the image. A novel RNN model that can deal with graphs with labeled edges has been also exploited. Some experiments on snapshots from video sequences are reported, showing very promising results.

Recursive processing of cyclic graphs

Recursive neural networks are a powerful tool for processing structured data. According to the recursive learning paradigm, the input information consists of directed positional acyclic graphs (DPAGs). In fact, recursive networks are fed following the partial order defined by the links of the graph. Unfortunately, the hypothesis of processing DPAGs is sometimes too restrictive, being the nature of some real-world problems intrinsically cyclic. In this paper, a methodology is proposed, which allows us to process any cyclic directed graph. Therefore, the computational power of recursive networks is definitely established, also clarifying the underlying limitations of the model.

Processing directed acyclic graphs with recursive neural networks

Recursive neural networks are conceived for processing graphs and extend the well-known recurrent model for processing sequences. In Frasconi et al. (1998), recursive neural networks can deal only with directed ordered acyclic graphs (DOAGs), in which the children of any given node are ordered. While this assumption is reasonable in some applications, it introduces unnecessary constraints in others. In this paper, it is shown that the constraint on the ordering can be relaxed by using an appropriate weight sharing, that guarantees the independence of the network output with respect to the permutations of the arcs leaving from each node. The method can be used with graphs having low connectivity and, in particular, few outcoming arcs. Some theoretical properties of the proposed architecture are given. They guarantee that the approximation capabilities are maintained, despite the weight sharing.

Terminal attractor algorithms: A critical analysis

Abstract One of the fundamental drawbacks of learning by gradient descent techniques is the susceptibility to local minima during training. Recently, some authors have independently introduced new learning algorithms that are based on the properties of terminal attractors and repellers. These algorithms were claimed to perform global optimization of the cost in finite time, provided that a null solution exists. In this paper, we prove that, in the case of local minima free error functions, terminal attractor algorithms guarantee that the optimal solution is reached in a number of steps that is independent of the cost function. Moreover, in the case of multimodal functions, we prove that, unfortunately, there are no theoretical guarantees that a global solution can be reached or that the algorithms perform satisfactorily from an operational point of view, unless particular favourable conditions are satisfied. On the other hand, the ideas behind these innovative methods are very interesting and deserve further investigations.