Kwok-Wo Wong

A unified sequential method for PCA

We propose a strictly local unified sequential method for principal component analysis. Any principal component analysis algorithm for linear feedforward neural networks can be used as the weight updating equation in our method. When principal components are extracted one by one sequentially, we suggest that the initial weight vector for the next component extraction should be orthogonal to the eigen-subspace already extracted. Simulation results show that both the convergence and the precision of the extraction are improved. Our method is also capable of extracting full eigenspace by using the neural network approach.

Handwritten digit recognition using multilayer feedforward neural networks with periodic and monotonic activation functions

The problem of handwritten digit recognition is dealt with by multilayer feedforward neural networks with different types of neuronal activation functions. Three types of activation functions are adopted in the network, namely, the traditional sigmoid function, sinusoidal function and a periodic function that can be considered as a combination of the first two functions. To speed up the learning, as well as to reduce the network size, an extended Kalman filter algorithm with the pruning method is used to train the network. Simulation results show that periodic activation functions perform better than monotonic ones in solving multi-cluster classification problems such as handwritten digit recognition.

A Practical Sequential Method for Principal Component Analysis

When increasing numbers of principalcomponents are extracted by using the sequentialmethod proposed in [1] by Banour and Azimi-Sadjadi, the accumulated extractionerror will become dominant and affect the extractionsof the remaining principal components. To improvethis, we suggest that the initial weight vector forthe extraction of the next component should beorthogonal to the eigensubspace spanned by the alreadyextracted weight vectors. Simulation results showthat both the convergence and the accuracy of theextraction are improved. Our improved method is alsocapable of extracting full eigenspace accurately.

A LOCAL TRAINING AND PRUNING APPROACH FOR NEURAL NETWORKS

The training of neural networks using the extended Kalman filter (EKF) algorithm is plagued by the drawback of high computational complexity and storage requirement that may become prohibitive even for networks of moderate size. In this paper, we present a local EKF training and pruning approach that can solve this problem. In particular, the by-products obtained along with the local EKF training can be utilized to measure the importance of the network weights. Comparing with the original global approach, the proposed local EKF training and pruning approach results in a much lower computational complexity and storage requirement. Hence, it is more practical in solving real world problems. The performance of the proposed algorithm is demonstrated on one medium- and one large-scale problems, namely, sunspot data prediction and handwritten digit recognition.

Combining DEKF algorithm and trace rule for fast on-line invariance extraction and recognition

Abstract A self-organizing network is used to perform invariance extraction and recognition of handwritten digits. To extract the invariance effectively, we propose to combine the trace learning rule and the on-line dual extended Kalman filter (DEKF) algorithm. Furthermore, a new activation function is suggested to replace the traditional sigmoid activation function so as to reduce the sensitivity of the extracted features to samples with large variance. Computer simulations show that both the learning speed and the recognition rate are improved using a compact network.

Combining PCA and MCA by using recursive least square learning method

By using the fact that the derivatives of the ith network output with respect to the weights connected to the jth output neuron (i/spl ne/j) are zero, a modified RLS method is proposed for principal and minor components analysis. After the extraction of significant components of the input vectors, the error covariance matrix obtained in the learning process is used to perform minor components analysis. The minor components found are then pruned so as to achieve a higher compression ratio. Simulation results show that both the convergent speed and the compression ratio are improved. These indicate that our method combines the extraction of principal components and the pruning of minor components effectively.

Using recursive least square learning method for principal and minor components analysis

In combining principal and minor components analysis, a parallel extraction method based on the recursive least square algorithm is suggested to extract the principal components of the input vectors. After the extraction, the error covariance matrix obtained in the learning process is used to perform minor components analysis. The minor components found are then pruned so as to achieve a higher compression ratio. Simulation results show that both the convergent speed and the compression ratio are improved, which in turn indicate that our method effectively combines the extraction of the principal components and the pruning of the minor components.