Haikun Wei

Dynamics of learning near singularities in layered networks

We explicitly analyze the trajectories of learning near singularities in hierarchical networks, such as multilayer perceptrons and radial basis function networks, which include permutation symmetry of hidden nodes, and show their general properties. Such symmetry induces singularities in their parameter space, where the Fisher information matrix degenerates and odd learning behaviors, especially the existence of plateaus in gradient descent learning, arise due to the geometric structure of singularity. We plot dynamic vector fields to demonstrate the universal trajectories of learning near singularities. The singularity induces two types of plateaus, the on-singularity plateau and the near-singularity plateau, depending on the stability of the singularity and the initial parameters of learning. The results presented in this letter are universally applicable to a wide class of hierarchical models. Detailed stability analysis of the dynamics of learning in radial basis function networks and multilayer perceptrons will be presented in separate work.

Direct interval forecasting of wind speed using radial basis function neural networks in a multi-objective optimization framework

Point predictions of wind speed can hardly be reliable and accurate when the uncertainty level increases in data. Prediction intervals (PIs) provide a solution to quantify the uncertainty associated with point predictions. In this paper, we adopt radial basis function (RBF) neural networks to perform interval forecasting of the future wind speed. A two-step method is proposed to determine the RBF connection weights in a multi-objective optimization framework. In the first step, the centers of the RBF are determined using the K-means clustering algorithm and the hidden-output weights of the RBF are pre-trained using the least squares algorithm. In the second step, the hidden-output weights are further adjusted by the non-dominated sorting genetic algorithm-II (NSGA-II), which aims at concurrently minimizing the width and maximizing the coverage probability of the constructed intervals. We test the performance of the proposed method on three real data sets, which are collected from different wind farms in China. The experimental results indicate that the proposed method can provide higher quality PIs than the conventional multi-layer perceptron (MLP) based methods.

Dynamics of learning near singularities in radial basis function networks

The radial basis function (RBF) networks are one of the most widely used models for function approximation in the regression problem. In the learning paradigm, the best approximation is recursively or iteratively searched for based on observed data (teacher signals). One encounters difficulties in such a process when two component basis functions become identical, or when the magnitude of one component becomes null. In this case, the number of the components reduces by one, and then the reduced component recovers as the learning process proceeds further, provided such a component is necessary for the best approximation. Strange behaviors, especially the plateau phenomena, have been observed in dynamics of learning when such reduction occurs. There exist singularities in the space of parameters, and the above reduction takes place at the singular regions. This paper focuses on a detailed analysis of the dynamical behaviors of learning near the overlap and elimination singularities in RBF networks, based on the averaged learning equation that is applicable to both on-line and batch mode learning. We analyze the stability on the overlap singularity by solving the eigenvalues of the Hessian explicitly. Based on the stability analysis, we plot the analytical dynamic vector fields near the singularity, which are then compared to those real trajectories obtained by a numeric method. We also confirm the existence of the plateaus in both batch and on-line learning by simulation.

Natural gradient learning algorithms for rbf networks

Radial basis function (RBF) networks are one of the most widely used models for function approximation and classification. There are many strange behaviors in the learning process of RBF networks, such as slow learning speed and the existence of the plateaus. The natural gradient learning method can overcome these disadvantages effectively. It can accelerate the dynamics of learning and avoid plateaus. In this letter, we assume that the probability density function (pdf) of the input and the activation function are gaussian. First, we introduce natural gradient learning to the RBF networks and give the explicit forms of the Fisher information matrix and its inverse. Second, since it is difficult to calculate the Fisher information matrix and its inverse when the numbers of the hidden units and the dimensions of the input are large, we introduce the adaptive method to the natural gradient learning algorithms. Finally, we give an explicit form of the adaptive natural gradient learning algorithm and compare it to the conventional gradient descent method. Simulations show that the proposed adaptive natural gradient method, which can avoid the plateaus effectively, has a good performance when RBF networks are used for nonlinear functions approximation.

A novel graph-based optimization framework for salient object detection

In traditional graph-based optimization framework for salient object detection, an image is over-segmented into superpixels and mapped to one single graph. The saliency value of each superpixel is then computed based on the similarity between connected nodes and the saliency related queries. When applying the traditional graph-based optimization framework to the salient object detection problem in natural scene images, we observe at least two limitations: only one graph is employed to describe the information contained in an image and no cognitive property about visual saliency is explicitly modeled in the optimization framework. In this work, we propose a novel graph-based optimization framework for salient object detection. Firstly, we employ multiple graphs in our optimization framework. A natural scene image is usually complex, employing multiple graphs from different image properties can better describe the complex information contained in the image. Secondly, we model one popular cognitive property about visual saliency (visual rarity) in our graph-based optimization framework, making this framework more suitable for saliency detection problem. Specifically, we add a regularization term to constrain the saliency value of each superpixel according to visual rarity in our optimization framework. Our experimental results on four benchmark databases with comparisons to fifteen representative methods demonstrate that our graph-based optimization framework is effective and computationally efficient. HighlightsA novel graph-based optimization framework for salient object detection is proposed in the paper.Multiple graphs are employed in our optimization framework to better describe a natural scene image.Visual rarity is modeled as a regularization term in our framework to better detect saliency.Experimental results on four datasets with fifteen methods prove the effectiveness of our method.

Joint Structured Sparsity Regularized Multiview Dimension Reduction for Video-Based Facial Expression Recognition

Video-based facial expression recognition (FER) has recently received increased attention as a result of its widespread application. Using only one type of feature to describe facial expression in video sequences is often inadequate, because the information available is very complex. With the emergence of different features to represent different properties of facial expressions in videos, an appropriate combination of these features becomes an important, yet challenging, problem. Considering that the dimensionality of these features is usually high, we thus introduce multiview dimension reduction (MVDR) into video-based FER. In MVDR, it is critical to explore the relationships between and within different feature views. To achieve this goal, we propose a novel framework of MVDR by enforcing joint structured sparsity at both inter- and intraview levels. In this way, correlations on and between the feature spaces of different views tend to be well-exploited. In addition, a transformation matrix is learned for each view to discover the patterns contained in the original features, so that the different views are comparable in finding a common representation. The model can be not only performed in an unsupervised manner, but also easily extended to a semisupervised setting by incorporating some domain knowledge. An alternating algorithm is developed for problem optimization, and each subproblem can be efficiently solved. Experiments on two challenging video-based FER datasets demonstrate the effectiveness of the proposed framework.

Using a Model Structure Selection Technique to Forecast Short-Term Wind Speed for a Wind Power Plant in North China

AbstractModel structure selection with respect to short-term wind speed forecasting is relatively difficult due to the stochastic and intermittent nature of the wind speed distribution. In order to overcome the disadvantages in traditional approaches such as computing burden and low accuracy, a novel model structure selection technique about short-term wind speed forecasting is proposed in order to improve the computational efficiency and forecasting accuracy using the model variable selection, variable order estimation, model structure optimization techniques, and so on. The detailed and complete process flow associated to the theoretical analysis of the proposed model structure selection technique is described. Moreover, both the so-called overkill in the data filtering and so-called overfitting in the learning processing are avoided by a proper technique in the design of proposed approach. In order to verify the effectiveness of proposed strategy in a practical application, all the experimental results...

Dynamics of Learning In Hierarchical Models – Singularity and Milnor Attractor

We study the dynamics of learning in a hierarchical model such as multilayer perceptron. Such a model includes singularities, which affects its dynamics seriously. The Milnor type attractors appear, because of the singularity. We will show its trajectories explicitly, and present the topological nature of the singularities.

Theoretical and numerical analysis of learning dynamics near singularity in multilayer perceptrons

Abstract The multilayer perceptron is one of the most widely used neural networks in applications, however, its learning behavior often becomes very slow, which is due to the singularities in the parameter space. In this paper, we analyze the learning dynamics near singularities in multilayer perceptrons by using traditional methods. We obtain the explicit expressions of the averaged learning equations which play a significant role in theoretical and numerical analysis. After obtaining the best approximation on overlap singularity, the stability of overlap singularity is analyzed. Then we take the numerical analysis on singular regions. Real averaged dynamics near the singularities are obtained in comparison with the theoretical learning trajectories near singularity. In the simulation we analyze the averaged learning dynamics, batch mode learning dynamics and on-line learning dynamics, respectively.

Averaged learning equations of error-function-based multilayer perceptrons

The multilayer perceptrons (MLPs) have strange behaviors in the learning process caused by the existing singularities in the parameter space. A detailed theoretical or numerical analysis of the MLPs is difficult due to the non-integrability of the traditional log-sigmoid activation function which leads to difficulties in obtaining the averaged learning equations (ALEs). In this paper, the error function is suggested as the activation function of the MLPs. By solving the explicit expressions of two important expectations, we obtain the averaged learning equations which make it possible for further analysis of the learning dynamics in MLPs. The simulation results also indicate that the ALEs play a significant role in investigating the singular behaviors of MLPs.