Zhengxue Li

Deterministic convergence of an online gradient method for BP neural networks

Online gradient methods are widely used for training feedforward neural networks. We prove in this paper a convergence theorem for an online gradient method with variable step size for backward propagation (BP) neural networks with a hidden layer. Unlike most of the convergence results that are of probabilistic and nonmonotone nature, the convergence result that we establish here has a deterministic and monotone nature.

Approximation to a Compact Set of Functions by Feedforward Neural Networks

This paper is concerned with the approximation capability of feedforward neural networks to a compact set of functions. We follow a general approach that covers all the existing results and gives some new results in this respect. To elaborate, we have proved the following: If a family of feedforward neural networks is dense in H, a complete linear metric space of functions, then given a compact set V sub H and an error bound epsiv, one can fix the quantity of the hidden neurons and the weights between the input and hidden layers, such that in order to approximate any function f isin V with accuracy epsiv, one only has to further choose suitable weights between the hidden and output layers.

Convergence of gradient method for a fully recurrent neural network

Recurrent neural networks have been successfully used for analysis and prediction of temporal sequences. This paper is concerned with the convergence of a gradient-descent learning algorithm for training a fully recurrent neural network. In literature, stochastic process theory has been used to establish some convergence results of probability nature for the on-line gradient training algorithm, based on the assumption that a very large number of (or infinitely many in theory) training samples of the temporal sequences are available. In this paper, we consider the case that only a limited number of training samples of the temporal sequences are available such that the stochastic treatment of the problem is no longer appropriate. Instead, we use an off-line gradient training algorithm for the fully recurrent neural network, and we accordingly prove some convergence results of deterministic nature. The monotonicity of the error function in the iteration is also guaranteed. A numerical example is given to support the theoretical findings.

Convergence of batch gradient learning algorithm with smoothing L1/2 regularization for Sigma–Pi–Sigma neural networks ☆

Abstract Sigma–Pi–Sigma neural networks are known to provide more powerful mapping capability than traditional feed-forward neural networks. The L1/2 regularizer is very useful and efficient, and can be taken as a representative of all the L q ( 0 q 1 ) regularizers. However, the nonsmoothness of L1/2 regularization may lead to oscillation phenomenon. The aim of this paper is to develop a novel batch gradient method with smoothing L1/2 regularization for Sigma–Pi–Sigma neural networks. Compared with conventional gradient learning algorithm, this method produces sparser weights and simpler structure, and it improves the learning efficiency. A comprehensive study on the weak and strong convergence results for this algorithm are also presented, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed value, respectively.

Extreme learning machine for interval neural networks

Interval data offer a valuable way of representing the available information in complex problems where uncertainty, inaccuracy, or variability must be taken into account. Considered in this paper is the learning of interval neural networks, of which the input and output are vectors with interval components, and the weights are real numbers. The back-propagation (BP) learning algorithm is very slow for interval neural networks, just as for usual real-valued neural networks. Extreme learning machine (ELM) has faster learning speed than the BP algorithm. In this paper, ELM is applied for learning of interval neural networks, resulting in an interval extreme learning machine (IELM). There are two steps in the ELM for usual feedforward neural networks. The first step is to randomly generate the weights connecting the input and the hidden layers, and the second step is to use the Moore–Penrose generalized inversely to determine the weights connecting the hidden and output layers. The first step can be directly applied for interval neural networks. But the second step cannot, due to the involvement of nonlinear constraint conditions for IELM. Instead, we use the same idea as that of the BP algorithm to form a nonlinear optimization problem to determine the weights connecting the hidden and output layers of IELM. Numerical experiments show that IELM is much faster than the usual BP algorithm. And the generalization performance of IELM is much better than that of BP, while the training error of IELM is a little bit worse than that of BP, implying that there might be an over-fitting for BP.

A Modified Learning Algorithm for Interval Perceptrons with Interval Weights

In many applications, it is natural to use interval data to describe various kinds of uncertainties. This paper is concerned with a one-layer interval perceptron with the weights and the outputs being intervals and the inputs being real numbers. In the original learning method for this interval perceptron, an absolute value function is applied for newly learned radii of the interval weights, so as to force the radii to be positive. This approach seems unnatural somehow, and might cause oscillation in the learning procedure as indicated in our numerical experiments. In this paper, a modified learning method is proposed for this one-layer interval perceptron. We do not use the function of the absolute value, and instead, we replace, in the error function, the radius of each interval weight by a quadratic term. This simple trick does not cause any additional computational work for the learning procedure, but it brings about the following three advantages: First, the radii of the intervals of the weights are guaranteed to be positive during the learning procedure without the help of the absolute value function. Secondly, the oscillation mentioned above is eliminated and the convergence of the learning procedure is improved, as indicated by our numerical experiments. Finally, a by-product is that the convergence analysis of the learning procedure is now an easy job, while the analysis for the original learning method is at least difficult, if not impossible, due to the non-smoothness of the absolute value function involved.

Convergence of Gradient Descent Algorithm for a Recurrent Neuron

Probabilistic convergence results of online gradient descent algorithm have been obtained by many authors for the training of recurrent neural networks with innitely many training samples. This paper proves deterministic convergence of o2ine gradient descent algorithm for a recurrent neural network with nite number of training samples. Our results can be hopefully extended to more complicated recurrent neural networks, and serve as a complementary result to the existing probability convergence results.

Solving Path Planning of UAV Based on Modified Multi-Population Differential Evolution Algorithm

In this paper we solve the path planning of Unmanned Aerial Vehicle (UAV) using differential evolution algorithm (DE). Based on traditional DE, we proposed a modified multi-population differential evolution algorithm (MMPDE) which adopts the multi-population framework and two new operators: chemical adsorption mutation operator and selection mutation operator. The simulation experiments show that the new algorithm has good performance.

The binary output units of neural network

When solving a multi-classification problem with k kinds of samples, if we use a multiple linear perceptron, k output nodes will be widely-used. In this paper, we introduce binary output units of multiple linear perceptron by analyzing the classification problems of vertices of the regular hexahedron in the Three-dimensional Euclidean Space. And we define Binary Approach and One-for-Each Approach to the problem. Then we obtain a theorem with the help of which we can find a Binary Approach that requires more less classification planes than the One-for-Each Approach when solving a One-for-Each Separable Classification Problem. When we apply the Binary Approach to the design of output units of multiple linear perceptron, the output units required will decrease greatly and more problems could be solved.

Convergence analysis for feng's MCA neural network learning algorithm

The minor component analysis is widely used in many fields, such as signal processing and data analysis, so it has very important theoretical significance and practical values for the convergence analysis of these algorithms. In this paper we seek the convergence condition for Fengs MCA learning algorithm in deterministic discrete time system. Finally numerical experiments show the correctness of our theory.