Francisco Fernández-Navarro

A dynamic over-sampling procedure based on sensitivity for multi-class problems

Classification with imbalanced datasets supposes a new challenge for researches in the framework of machine learning. This problem appears when the number of patterns that represents one of the classes of the dataset (usually the concept of interest) is much lower than in the remaining classes. Thus, the learning model must be adapted to this situation, which is very common in real applications. In this paper, a dynamic over-sampling procedure is proposed for improving the classification of imbalanced datasets with more than two classes. This procedure is incorporated into a memetic algorithm (MA) that optimizes radial basis functions neural networks (RBFNNs). To handle class imbalance, the training data are resampled in two stages. In the first stage, an over-sampling procedure is applied to the minority class to balance in part the size of the classes. Then, the MA is run and the data are over-sampled in different generations of the evolution, generating new patterns of the minimum sensitivity class (the class with the worst accuracy for the best RBFNN of the population). The methodology proposed is tested using 13 imbalanced benchmark classification datasets from well-known machine learning problems and one complex problem of microbial growth. It is compared to other neural network methods specifically designed for handling imbalanced data. These methods include different over-sampling procedures in the preprocessing stage, a threshold-moving method where the output threshold is moved toward inexpensive classes and ensembles approaches combining the models obtained with these techniques. The results show that our proposal is able to improve the sensitivity in the generalization set and obtains both a high accuracy level and a good classification level for each class.

MELM-GRBF: A modified version of the extreme learning machine for generalized radial basis function neural networks

Abstract In this paper, we propose a methodology for training a new model of artificial neural network called the generalized radial basis function (GRBF) neural network. This model is based on generalized Gaussian distribution, which parametrizes the Gaussian distribution by adding a new parameter τ . The generalized radial basis function allows different radial basis functions to be represented by updating the new parameter τ . For example, when GRBF takes a value of τ = 2 , it represents the standard Gaussian radial basis function. The model parameters are optimized through a modified version of the extreme learning machine (ELM) algorithm. In the methodology proposed (MELM-GRBF), the centers of each GRBF were taken randomly from the patterns of the training set and the radius and τ values were determined analytically, taking into account that the model must fulfil two constraints: locality and coverage. An thorough experimental study is presented to test its overall performance. Fifteen datasets were considered, including binary and multi-class problems, all of them taken from the UCI repository. The MELM-GRBF was compared to ELM with sigmoidal, hard-limit, triangular basis and radial basis functions in the hidden layer and to the ELM-RBF methodology proposed by Huang et al. (2004) [1] . The MELM-GRBF obtained better results in accuracy than the corresponding sigmoidal, hard-limit, triangular basis and radial basis functions for almost all datasets, producing the highest mean accuracy rank when compared with these other basis functions for all datasets.

Evolutionary Generalized Radial Basis Function neural networks for improving prediction accuracy in gene classification using feature selection

Radial Basis Function Neural Networks (RBFNNs) have been successfully employed in several function approximation and pattern recognition problems. The use of different RBFs in RBFNN has been reported in the literature and here the study centres on the use of the Generalized Radial Basis Function Neural Networks (GRBFNNs). An interesting property of the GRBF is that it can continuously and smoothly reproduce different RBFs by changing a real parameter @t. In addition, the mixed use of different RBF shapes in only one RBFNN is allowed. Generalized Radial Basis Function (GRBF) is based on Generalized Gaussian Distribution (GGD), which adds a shape parameter, @t, to standard Gaussian Distribution. Moreover, this paper describes a hybrid approach, Hybrid Algorithm (HA), which combines evolutionary and gradient-based learning methods to estimate the architecture, weights and node topology of GRBFNN classifiers. The feasibility and benefits of the approach are demonstrated by means of six gene microarray classification problems taken from bioinformatic and biomedical domains. Three filters were applied: Fast Correlation-Based Filter (FCBF), Best Incremental Ranked Subset (BIRS), and Best Agglomerative Ranked Subset (BARS); this was done in order to identify salient expression genes from among the thousands of genes in microarray data that can directly contribute to determining the class membership of each pattern. After different gene subsets were obtained, the proposed methodology was performed using the selected gene subsets as new input variables. The results confirm that the GRBFNN classifier leads to a promising improvement in accuracy.

Cost-sensitive AdaBoost algorithm for ordinal regression based on extreme learning machine.

In this paper, the well known stagewise additive modeling using a multiclass exponential (SAMME) boosting algorithm is extended to address problems where there exists a natural order in the targets using a cost-sensitive approach. The proposed ensemble model uses an extreme learning machine (ELM) model as a base classifier (with the Gaussian kernel and the additional regularization parameter). The closed form of the derived weighted least squares problem is provided, and it is employed to estimate analytically the parameters connecting the hidden layer to the output layer at each iteration of the boosting algorithm. Compared to the state-of-the-art boosting algorithms, in particular those using ELM as base classifier, the suggested technique does not require the generation of a new training dataset at each iteration. The adoption of the weighted least squares formulation of the problem has been presented as an unbiased and alternative approach to the already existing ELM boosting techniques. Moreover, the addition of a cost model for weighting the patterns, according to the order of the targets, enables the classifier to tackle ordinal regression problems further. The proposed method has been validated by an experimental study by comparing it with already existing ensemble methods and ELM techniques for ordinal regression, showing competitive results.

An experimental study of different ordinal regression methods and measures

In this paper, an experimental study of different ordinal regression methods and measures is presented. The first objective is to gather the results of a considerably high number of methods, datasets and measures, since there are not many previous comparative studies of this kind in the literature. The second objective is to detect the redundancy between the evaluation measures used for ordinal regression. The results obtained present the maximum MAE (maximum of the mean absolute error of the difference between the true and the predicted ranks of the worst classified class) as a very interesting alternative for ordinal regression, being the less uncorrelated with respect to the rest of measures. Additionally, SVOREX and SVORIM are found to yield very good performance when the objective is to minimize this maximum MAE.

Parameter estimation of q-Gaussian Radial Basis Functions Neural Networks with a Hybrid Algorithm for binary classification

A classification problem is a decision-making task that many researchers have studied. A number of techniques have been proposed to perform binary classification. Neural networks are one of the artificial intelligence techniques that has had the most successful results when applied to this problem. Our proposal is the use of q-Gaussian Radial Basis Function Neural Networks (q-Gaussian RBFNNs). This basis function includes a supplementary degree of freedom in order to adapt the model to the distribution of data. A Hybrid Algorithm (HA) is used to search for a suitable architecture for the q-Gaussian RBFNN. The use of this type of more flexible kernel could greatly improve the discriminative power of RBFNNs. In order to test performance, the RBFNN with the q-Gaussian basis functions is compared to RBFNNs with Gaussian, Cauchy and Inverse Multiquadratic RBFs, and to other recent neural networks approaches. An experimental study is presented on 11 binary-classification datasets taken from the UCI repository. Moreover, aerial imagery taken in mid-May, mid-June and mid-July was used to evaluate the potential of the methodology proposed for discriminating Ridolfia segetum patches (one of the most dominant and harmful weeds in sunflower crops) in two naturally infested fields in southern Spain.

Evolutionary q -Gaussian radial basis function neural networks for multiclassification

This paper proposes a radial basis function neural network (RBFNN), called the q-Gaussian RBFNN, that reproduces different radial basis functions (RBFs) by means of a real parameter q. The architecture, weights and node topology are learnt through a hybrid algorithm (HA). In order to test the overall performance, an experimental study with sixteen data sets taken from the UCI repository is presented. The q-Gaussian RBFNN was compared to RBFNNs with Gaussian, Cauchy and inverse multiquadratic RBFs in the hidden layer and to other probabilistic classifiers, including different RBFNN design methods, support vector machines (SVMs), a sparse classifier (sparse multinomial logistic regression, SMLR) and a non-sparse classifier (regularized multinomial logistic regression, RMLR). The results show that the q-Gaussian model can be considered very competitive with the other classification methods.

Memetic pareto differential evolutionary artificial neural networks to determine growth multi-classes in predictive microbiology

The main objective of this research is to automatically design Artificial Neural Network models with sigmoid basis units for multiclassification tasks in predictive microbiology. The classifiers obtained achieve a double objective: a high classification level in the dataset and high classification levels for each class. The Memetic Pareto Differential Evolution Neural Network chosen to learn the structure and weights of the Neural Networks is a Differential Evolutionary approach based on the Pareto Differential Evolution multiobjective evolutionary algorithm. The Pareto Differential Evolution algorithm is augmented with a local search using the improved Resilient Backpropagation with backtracking–iRprop+ algorithm. To analyze the robustness of this methodology, it has been applied to two complex classification problems in predictive microbiology (Staphylococcus aureus and Shigella flexneri). The results obtained show that the generalization ability and the classification rate in each class can be more efficiently improved within this multiobjective algorithm.

Development of a multi-classification neural network model to determine the microbial growth/no growth interface

Boundary models have been recognized as useful tools to predict the ability of microorganisms to grow at limiting conditions. However, at these conditions, microbial behaviour can vary, being difficult to distinguish between growth or no growth. In this paper, the data from the study of Valero et al. [Valero, A., Pérez-Rodríguez, F., Carrasco, E., Fuentes-Alventosa, J.M., García-Gimeno, R.M., Zurera, G., 2009. Modelling the growth boundaries of Staphylococcus aureus: Effect of temperature, pH and water activity. International Journal of Food Microbiology 133 (1-2), 186-194] belonging to growth/no growth conditions of Staphylococcus aureus against temperature, pH and a(w) were divided into three categorical classes: growth (G), growth transition (GT) and no growth (NG). Subsequently, they were modelled by using a Radial Basis Function Neural Network (RBFNN) in order to create a multi-classification model that was able to predict the probability of belonging at one of the three mentioned classes. The model was developed through an over sampling procedure using a memetic algorithm (MA) in order to balance in part the size of the classes and to improve the accuracy of the classifier. The multi-classification model, named Smote Memetic Radial Basis Function (SMRBF) provided a quite good adjustment to data observed, being able to correctly classify the 86.30% of training data and the 82.26% of generalization data for the three observed classes in the best model. Besides, the high number of replicates per condition tested (n=30) produced a smooth transition between growth and no growth. At the most stringent conditions, the probability of belonging to class GT was higher, thus justifying the inclusion of the class in the new model. The SMRBF model presented in this study can be used to better define microbial growth/no growth interface and the variability associated to these conditions so as to apply this knowledge to a food safety in a decision-making process.

Ordinal Neural Networks Without Iterative Tuning

Ordinal regression (OR) is an important branch of supervised learning in between the multiclass classification and regression. In this paper, the traditional classification scheme of neural network is adapted to learn ordinal ranks. The model proposed imposes monotonicity constraints on the weights connecting the hidden layer with the output layer. To do so, the weights are transcribed using padding variables. This reformulation leads to the so-called inequality constrained least squares (ICLS) problem. Its numerical solution can be obtained by several iterative methods, for example, trust region or line search algorithms. In this proposal, the optimum is determined analytically according to the closed-form solution of the ICLS problem estimated from the Karush-Kuhn-Tucker conditions. Furthermore, following the guidelines of the extreme learning machine framework, the weights connecting the input and the hidden layers are randomly generated, so the final model estimates all its parameters without iterative tuning. The model proposed achieves competitive performance compared with the state-of-the-art neural networks methods for OR.