Francisco J. Martínez-Estudillo

Evolutionary product unit based neural networks for regression

This paper presents a new method for regression based on the evolution of a type of feed-forward neural networks whose basis function units are products of the inputs raised to real number power. These nodes are usually called product units. The main advantage of product units is their capacity for implementing higher order functions. Nevertheless, the training of product unit based networks poses several problems, since local learning algorithms are not suitable for these networks due to the existence of many local minima on the error surface. Moreover, it is unclear how to establish the structure of the network since, hitherto, all learning methods described in the literature deal only with parameter adjustment. In this paper, we propose a model of evolution of product unit based networks to overcome these difficulties. The proposed model evolves both the weights and the structure of these networks by means of an evolutionary programming algorithm. The performance of the model is evaluated in five widely used benchmark functions and a hard real-world problem of microbial growth modeling. Our evolutionary model is compared to a multistart technique combined with a Levenberg-Marquardt algorithm and shows better overall performance in the benchmark functions as well as the real-world problem.

Hybridization of evolutionary algorithms and local search by means of a clustering method

This paper presents a hybrid evolutionary algorithm (EA) to solve nonlinear-regression problems. Although EAs have proven their ability to explore large search spaces, they are comparatively inefficient in fine tuning the solution. This drawback is usually avoided by means of local optimization algorithms that are applied to the individuals of the population. The algorithms that use local optimization procedures are usually called hybrid algorithms. On the other hand, it is well known that the clustering process enables the creation of groups (clusters) with mutually close points that hopefully correspond to relevant regions of attraction. Local-search procedures can then be started once in every such region. This paper proposes the combination of an EA, a clustering process, and a local-search procedure to the evolutionary design of product-units neural networks. In the methodology presented, only a few individuals are subject to local optimization. Moreover, the local optimization algorithm is only applied at specific stages of the evolutionary process. Our results show a favorable performance when the regression method proposed is compared to other standard methods.

Evolutionary product-unit neural networks classifiers

This paper proposes a classification method based on a special class of feed-forward neural network, namely product-unit neural networks. Product-units are based on multiplicative nodes instead of additive ones, where the nonlinear basis functions express the possible strong interactions between variables. We apply an evolutionary algorithm to determine the basic structure of the product-unit model and to estimate the coefficients of the model. We use softmax transformation as the decision rule and the cross-entropy error function because of its probabilistic interpretation. The approach can be seen as nonlinear multinomial logistic regression where the parameters are estimated using evolutionary computation. The empirical and specific multiple comparison statistical test results, carried out over several benchmark data sets and a complex real microbial Listeria growth/no growth problem, show that the proposed model is promising in terms of its classification accuracy and the number of the model coefficients, yielding a state-of-the-art performance.

Logistic Regression by Means of Evolutionary Radial Basis Function Neural Networks

This paper proposes a hybrid multilogistic methodology, named logistic regression using initial and radial basis function (RBF) covariates. The process for obtaining the coefficients is carried out in three steps. First, an evolutionary programming (EP) algorithm is applied, in order to produce an RBF neural network (RBFNN) with a reduced number of RBF transformations and the simplest structure possible. Then, the initial attribute space (or, as commonly known as in logistic regression literature, the covariate space) is transformed by adding the nonlinear transformations of the input variables given by the RBFs of the best individual in the final generation. Finally, a maximum likelihood optimization method determines the coefficients associated with a multilogistic regression model built in this augmented covariate space. In this final step, two different multilogistic regression algorithms are applied: one considers all initial and RBF covariates (multilogistic initial-RBF regression) and the other one incrementally constructs the model and applies cross validation, resulting in an automatic covariate selection [simplelogistic initial-RBF regression (SLIRBF)]. Both methods include a regularization parameter, which has been also optimized. The methodology proposed is tested using 18 benchmark classification problems from well-known machine learning problems and two real agronomical problems. The results are compared with the corresponding multilogistic regression methods applied to the initial covariate space, to the RBFNNs obtained by the EP algorithm, and to other probabilistic classifiers, including different RBFNN design methods [e.g., relaxed variable kernel density estimation, support vector machines, a sparse classifier (sparse multinomial logistic regression)] and a procedure similar to SLIRBF but using product unit basis functions. The SLIRBF models are found to be competitive when compared with the corresponding multilogistic regression methods and the RBFEP method. A measure of statistical significance is used, which indicates that SLIRBF reaches the state of the art.

Logistic regression using covariates obtained by product-unit neural network models

We propose a logistic regression method based on the hybridation of a linear model and product-unit neural network models for binary classification. In a first step we use an evolutionary algorithm to determine the basic structure of the product-unit model and afterwards we apply logistic regression in the new space of the derived features. This hybrid model has been applied to seven benchmark data sets and a new microbiological problem. The hybrid model outperforms the linear part and the nonlinear part obtaining a good compromise between them and they perform well compared to several other learning classification techniques. We obtain a binary classifier with very promising results in terms of classification accuracy and the complexity of the classifier.

Memetic Pareto Evolutionary Artificial Neural Networks to determine growth/no-growth in predictive microbiology

The main objective of this work is to automatically design neural network models with sigmoid basis units for binary classification tasks. The classifiers that are obtained achieve a double objective: a high classification level in the dataset and a high classification level for each class. We present MPENSGA2, a Memetic Pareto Evolutionary approach based on the NSGA2 multiobjective evolutionary algorithm which has been adapted to design Artificial Neural Network models, where the NSGA2 algorithm is augmented with a local search that uses the improved Resilient Backpropagation with backtracking-IRprop+ algorithm. To analyze the robustness of this methodology, it was applied to four complex classification problems in predictive microbiology to describe the growth/no-growth interface of food-borne microorganisms such as Listeria monocytogenes, Escherichia coli R31, Staphylococcus aureus and Shigella flexneri. The results obtained in Correct Classification Rate (CCR), Sensitivity (S) as the minimum of sensitivities for each class, Area Under the receiver operating characteristic Curve (AUC), and Root Mean Squared Error (RMSE), show that the generalization ability and the classification rate in each class can be more efficiently improved within a multiobjective framework than within a single-objective framework.

Evolutionary learning by a sensitivity-accuracy approach for multi-class problems

Performance evaluation is decisive when improving classifiers. Accuracy alone is insufficient because it cannot capture the myriad of contributing factors differentiating the performances of two different classifiers and approaches based on a multi-objective perspective are hindered by the growing of the Pareto optimal front as the number of classes increases. This paper proposes a new approach to deal with multi-class problems based on the accuracy (C) and minimum sensitivity (S) given by the lowest percentage of examples correctly predicted to belong to each class. From this perspective, we compare different fitness functions (accuracy, C , entropy, E , sensitivity, S , and area, A ) in an evolutionary scheme. We also present a two stage evolutionary algorithm with two sequential fitness functions, the entropy for the first step and the area for the second step. This methodology is applied to solve six benchmark classification problems. The two-stage approach obtains promising results and achieves a high classification rate level in the global dataset with an acceptable level of accuracy for each class.

A two-stage evolutionary algorithm based on sensitivity and accuracy for multi-class problems

The machine learning community has traditionally used correct classification rates or accuracy (C) values to measure classifier performance and has generally avoided presenting classification levels of each class in the results, especially for problems with more than two classes. C values alone are insufficient because they cannot capture the myriad of contributing factors that differentiate the performance of two different classifiers. Receiver Operating Characteristic (ROC) analysis is an alternative to solve these difficulties, but it can only be used for two-class problems. For this reason, this paper proposes a new approach for analysing classifiers based on two measures: C and sensitivity (S) (i.e., the minimum of accuracies obtained for each class). These measures are optimised through a two-stage evolutionary process. It was conducted by applying two sequential fitness functions in the evolutionary process, including entropy (E) for the first stage and a new fitness function, area (A), for the second stage. By using these fitness functions, the C level was optimised in the first stage, and the S value of the classifier was generally improved without significantly reducing C in the second stage. This two-stage approach improved S values in the generalisation set (whereas an evolutionary algorithm (EA) based only on the S measure obtains worse S levels) and obtained both high C values and good classification levels for each class. The methodology was applied to solve 16 benchmark classification problems and two complex real-world problems in analytical chemistry and predictive microbiology. It obtained promising results when compared to other competitive multi-class classification algorithms and a multi-objective alternative based on E and S.

Evolutionary learning using a sensitivity-accuracy approach for classification

Accuracy alone is insufficient to evaluate the performance of a classifier especially when the number of classes increases This paper proposes an approach to deal with multi-class problems based on Accuracy (C) and Sensitivity (S) We use the differential evolution algorithm and the ELM-algorithm (Extreme Learning Machine) to obtain multi-classifiers with a high classification rate level in the global dataset with an acceptable level of accuracy for each class This methodology is applied to solve four benchmark classification problems and obtains promising results.

Time Series Representation by a Novel Hybrid Segmentation Algorithm

Time series representation can be approached by segmentation genetic algorithms (GAs) with the purpose of automatically finding segments approximating the time series with the lowest possible error. Although this is an interesting data mining field, obtaining the optimal segmentation of time series in different scopes is a very challenging task. In this way, very accurate algorithms are needed. On the other hand, it is well-known that GAs are relatively poor when finding the precise optimum solution in the region where they converge. Thus, this paper presents a hybrid GA algorithm including a local search method, aimed to improve the quality of the final solution. The local search algorithm is based on two well-known algorithms: Bottom-Up and Top-Down. A real-world time series in the Spanish Stock Market field (IBEX35) and a synthetic database (Donoho-Johnstone) used in other researches were used to test the proposed methodology.