Daniel S. Yeung

Localized Generalization Error Model and Its Application to Architecture Selection for Radial Basis Function Neural Network

The generalization error bounds found by current error models using the number of effective parameters of a classifier and the number of training samples are usually very loose. These bounds are intended for the entire input space. However, support vector machine (SVM), radial basis function neural network (RBFNN), and multilayer perceptron neural network (MLPNN) are local learning machines for solving problems and treat unseen samples near the training samples to be more important. In this paper, we propose a localized generalization error model which bounds from above the generalization error within a neighborhood of the training samples using stochastic sensitivity measure. It is then used to develop an architecture selection technique for a classifier with maximal coverage of unseen samples by specifying a generalization error threshold. Experiments using 17 University of California at Irvine (UCI) data sets show that, in comparison with cross validation (CV), sequential learning, and two other ad hoc methods, our technique consistently yields the best testing classification accuracy with fewer hidden neurons and less training time.

Fuzzy support vector machines for solving two-class problems

A support vector machine (SVM) was originally developed to solve two-class non-fuzzy problems. An SVM can act as a linear learning machine when handling data in a high dimensional feature space for non-linear separable and non-separable problems. A few methods have been proposed to solve two-class and multi-class classification problems by including fuzzy concepts. In this paper, we propose a new fuzzy support vector machine which improves the traditional SVM by adding fuzzy memberships to each training sample to indicate degree of membership of this sample to different classes. This fuzzy SVM is more complete and meaningful, and could generalize the traditional non-fuzzy SVM to a fuzzy one, i.e., the traditional non-fuzzy SVM is an extreme case of our fuzzy SVM when the degrees of membership of a sample to two different classes are the same.

Localized generalization error based active learning for image annotation

Content-based image auto-annotation becomes a hot research topic owing to the development of image retrieval system and the storing technology of multimedia information. It is a key step in most of those image processing applications. In this work, we adopt active learning to image annotation for reducing the number of labeled images required for supervised learning procedure. Localized Generalization Error Model (L-GEM) based active learning uses localized generalization error bound as the sample selection criterion. In each turn, the most informative sample from a set of unlabeled samples is selected by the L-GEM based active learning will be labeled and added to the training dataset. A heuristic and a Q value selection improvement methods are introduced in this paper. The experimental results show that the proposed active learning efficiently reduces the number of labeled training samples. Moreover, the improvement method improve the performances in both testing accuracy and training time which are both essential in image annotation applications.

Statistical output sensitivity to input and weight perturbations of radial basis function neural networks

This paper presents a statistical sensitivity measure of the outputs to input and weight perturbations, for an ensemble of Radial Basis Function Neural Networks (RBFNM with the popular Gaussian kernel functbn A stochastic model of RBFNNs is assumed in which all inputs, weights and their perturbations are assumed to be independendy identically distributed (Lid) random variables The variance of the direrence of the outputs before and afier the input and weight perturbations is used to defne the sensithi@ measure. A bottom up approach is used in that the sensitivity measure of a single neuron is derivedjirst, then for the entire network The derived ana&tical formulae for the network sensitivity measure are validated by showing that the theoretical result and the simuhtion result differ by an average of 2%

Sphere Classification for Ambiguous Data

In some cases, an ambiguous pattern may belong to more than one class, however it is forcibly classified to one of these classes in conventional support vector machine. Handling those ambiguous patterns in this way may loss the uncertainty information of the patterns. Therefore, we prefer to keep the uncertainty information in the ambiguous patterns. In this work, instead of two-class classification, we propose to classify samples into four classes: namely positive, negative, ambiguous and outlier classes

MPEG-7 descriptor selection using Localized Generalization Error Model with mutual information

MPEG-7 provides a set of descriptors to describe the content of an image. However, how to select or combine descriptors for a specific image classification problem is still an open problem. Currently, descriptors are usually selected by human experts. Moreover, selecting the same set of descriptors for different classes of images may not be reasonable. In this work we propose a MPEG-7 descriptor selection method which selects different MPEG-7 descriptors for different image class in an image classification problem. The proposed method L-GEMIM combines Localized Generalization Error Model (L-GEM) and Mutual Information (MI) to assess the relevance of MPEG-7 descriptors for a particular image class. The L-GEMIM model assesses the relevance based on the generalization capability of a MPEG-7 descriptor using L-GEM and prevents redundant descriptors being selected by MI. Experimental results using 4,000 images in 4 classes show that L-GEMIM selects better set of MPEG-7 descriptors yielding a higher testing accuracy of image classification.

Quantitative study on candlestick pattern for Shenzhen Stock Market

Shenzhen stock market grows rapidly yet is still a young market when compared with Hong Kong, New York and London markets. Its daily turnover reaches billions US dollars. A good prediction of stock price will bring us substantial pecuniary reward. Technical analysis is a widely adopted financial prediction tool in worldwide stock markets. Candlestick pattern is one of the most efficient methods in technical analysis. However, does candlestick pattern prediction works for Shenzhen stocks? Candlestick patterns are always defined by fuzzy terms, could we have a quantitative definition of these patterns? We perform a quantitative study on these two major research problems in this paper. We study the morning star pattern in this work and the method in this paper could be extended to other patterns easily. So, we propose adopting the radial basis function neural networks trained with localized generalization error model to predict whether or not the stock price will increase after the appearance of it. Then, we extract the patterns from the neural network to provide a quantitative definition of the morning star pattern for a particular stock. Experimental results show that our modification to the morning star pattern prediction prevents up to 69% of false prediction of the morning star pattern. We also provide a quantitative measure of the morning star patterns for two of the Shenzhen stocks.

L-gem based co-training for CBIR with relevance feedback

Relevance feedback has been developed for several years and becomes an effective method for capturing userpsilas concepts to improve the performance of content-based image retrieval (CBIR). In contrast to fully labeled training dataset in supervised learning, semi-supervised learning and active learning deal with training dataset with only a small portion of labeled samples. This is more realistic because one could easily find thousands of unlabeled images from the Internet. How to make use of such unlabeled resources on the Internet is an important research topic. Co-training method is to expand the number of labeled samples in semi-supervised learning by swapping training samples between two classifiers. In this work, we propose to apply the localized generalization error model (L-GEM) to co-training. Two radial basis function neural networks (RBFNN) with different features split is adopted in the co-training and the unlabeled samples with lowest L-GEM value is added to the co-training in next iteration. In the CBIR system, we output those positive images with lowest L-GEM value as the highest confident image and output those images with highest L-GEM to ask for user labeling. Higher the L-GEM value of a sample is, the less confident is the classifier to predict its image class. Experimental results show that the proposed method could effectively improve the image retrieval results.

Refinement of rule-based intrusion detection system for denial of service attacks by support vector machine

With the tremendous increase in connectivity and accessibility to the Internet, information security has become a serious global issue. Denial of service (DoS), one of the attacks evolved in recent years, has devastating effect to the commercial activities. We propose a hybrid intrusion detection system (HIDS) which incorporates the benefits of both rule-based and SVM techniques. In brief, the SVM is used to select important features and generate rules, while the rule-based system is then applied to detect the DoS attacks. The rule set generated by the HIDS is more accurate and compact. Experimental results show that the HIDS has a better performance than the rule-based system with rules extracted only from human experts.

Denial of service detection by support vector machines and radial-basis function neural network

Denial of service (DoS) problem is one of serious attacks in the Internet. The attackers attempt to exhaust the resource of the service provider in order to prevent legitimate users from using the system. Most of the detecting DoS tools, such as rule-based and threshold detection approaches, rely on the objective opinion of the domain experts. This work aims to apply machine learning techniques, such as radial-basis function neural network (RBFNN) and support vector machines (SVM), to solve the DoS problem and compare which technique, is better to detect DoS. The main advantage of this detection method is that it has the ability to detect or predict new attacks when some patterns are similar to the attack patterns learnt in the past. Thus it can detect novel attacks for which signatures have not been defined.