Pablo Fabián Verdes

Neural network ensembles: evaluation of aggregation algorithms

Ensembles of artificial neural networks show improved generalization capabilities that outperform those of single networks. However, for aggregation to be effective, the individual networks must be as accurate and diverse as possible. An important problem is, then, how to tune the aggregate members in order to have an optimal compromise between these two conflicting conditions. We present here an extensive evaluation of several algorithms for ensemble construction, including new proposals and comparing them with standard methods in the literature. We also discuss a potential problem with sequential aggregation algorithms: the non-frequent but damaging selection through their heuristics of particularly bad ensemble members. We introduce modified algorithms that cope with this problem by allowing individual weighting of aggregate members. Our algorithms and their weighted modifications are favorably tested against other methods in the literature, producing a sensible improvement in performance on most of the standard statistical databases used as benchmarks.

Weed seeds identification by machine vision

Abstract The implementation of new methods for reliable and fast identification and classification of seeds is of major technical and economical importance in the agricultural industry. As in ocular inspection, the automatic classification of seeds should be based on knowledge of seed size, shape, color and texture. In this work, we assess the discriminating power of these characteristics for the unique identification of seeds of 57 weed species. Using the performance of a naive Bayes classifier as selection criterion, we identified a nearly optimal set of 12 (six morphological+four color+two textural) seed characteristics to be used as classification parameters. We found that, as expected, size and shape characteristics have larger discriminating power than color and textural ones. However, all these features are required to reach an identification performance acceptable for practical applications. In spite of its simplicity, the naive Bayes classifier reveals itself surprisingly good for the identification of seed species. This might be due to the careful selection of the feature set, leading to nearly independent parameters as assumed by this method. We also found that, using the same feature set, a more sophisticated classifier based on an artificial neural network committee performs only slightly better than this simple Bayesian approach.

Aggregation algorithms for neural network ensemble construction

How to generate and aggregate base learners to have optimal ensemble generalization capabilities is an important questions in building composite regression/classification machines. We present here an evaluation of several algorithms for artificial neural networks aggregation in the regression settings, including new proposals and comparing them with standard methods in the literature. We also discuss a potential problem with sequential algorithms: the non frequent but damaging selection through their heuristics of particularly bad ensemble members. We show that one can cope with this problem by allowing individual weighting of aggregate members. Our algorithms and their weighted modifications are favorably tested against other methods in the literature, producing a performance improvement on the standard statistical databases used as benchmarks.

Selecting diverse members of neural network ensembles

Ensembles of artificial neural networks have been used as classification/regression machines, showing improved generalization capabilities that outperform those of single networks. However, it has been recognized that for aggregation to be effective the individual network must be as accurate and diverse as possible. An important problem is, then, how to choose the aggregate members in order to have an optimal compromise between these two conflicting conditions. We propose here a new method for selecting members of regression/classification ensembles that leads to small aggregates with few but very diverse individual predictors. Using artificial neural networks as individual learners, the algorithm is favorably tested against other methods in the literature, producing a remarkable performance improvement on the standard statistical databases used as benchmarks. In addition, and as a concrete application, we study the sunspot time series and predict the remaining of the current cycle 23 of solar activity.

Predictions of the maximum amplitude for solar cycle 23 and its subsequent behavior using nonlinear methods

Two nonlinear methods are employed for the prediction of the maximum amplitude for solar cycle 23 and its declining behavior. First, a new heuristic method based on the second derivative of the (conveniently smoothed) sunspot data is proposed. The curvature of the smoothed sunspot data at cycle minimum appears to correlate (R ≃ 0.92) with the cycles later-occurring maximum amplitude. Secondly, in order to predict the near-maximum and declining activity of solar cycle 23, a neural network analysis of the annual mean sunspot time series is also performed. The results of the present study are then compared with some other recent predictions.

ARTIFICIAL NEURAL NETWORK LEARNING OF NONSTATIONARY BEHAVIOR IN TIME SERIES

We refine and complement a previously-proposed artificial neural network method for learning hidden signals forcing nonstationary behavior in time series. The method adds an extra input unit to the network and feeds it with the proposed profile for the unknown perturbing signal. The correct time evolution of this new input parameter is learned simultaneously with the intrinsic stationary dynamics underlying the series, which is accomplished by minimizing a suitably-defined error function for the training process. We incorporate here the use of validation data, held out from the training set, to accurately determine the optimal value of a hyperparameter required by the method. Furthermore, we evaluate this algorithm in a controlled situation and show that it outperforms other existing methods in the literature. Finally, we discuss a preliminary application to the real-world sunspot time series and link the obtained hidden perturbing signal to the secular evolution of the solar magnetic field.

A late-stopping method for optimal aggregation of neural networks.

Ensembles of artificial neural networks have been used in the last years as classification/regression machines, showing improved generalization capabilities that outperform those of single networks. However, it has been recognized that for aggregation to be effective the individual networks must be as accurate and diverse as possible. An important problem is, then, how to tune the aggregate members in order to have an optimal compromise between these two conflicting conditions. We propose here a simple method for constructing regression/classification ensembles of neural networks that leads to overtrained aggregate members with an adequate balance between accuracy and diversity. The algorithm is favorably tested against other methods recently proposed in the literature, producing an improvement in performance on the standard statistical databases used as benchmarks. In addition, and as a concrete application, we apply our method to the sunspot time series and predict the remainder of the current cycle 23 of solar activity.

Nonstationary regression with support vector machines

Abstract In this work, we introduce a method for data analysis in nonstationary environments: time-adaptive support vector regression (TA-SVR). The proposed approach extends a previous development that was limited to classification problems. Focusing our study on time series applications, we show that TA-SVR can improve the accuracy of several aspects of nonstationary data analysis, namely the tasks of modelling and prediction, input relevance estimation, and reconstruction of a hidden forcing profile.

Improved multiclass feature selection via list combination

Abstract Feature selection is a crucial machine learning technique aimed at reducing the dimensionality of the input space. By discarding useless or redundant variables, not only it improves model performance but also facilitates its interpretability. The well-known Support Vector Machines–Recursive Feature Elimination (SVM-RFE) algorithm provides good performance with moderate computational efforts, in particular for wide datasets. When using SVM-RFE on a multiclass classification problem, the usual strategy is to decompose it into a series of binary ones, and to generate an importance statistics for each feature on each binary problem. These importances are then averaged over the set of binary problems to synthesize a single value for feature ranking. In some cases, however, this procedure can lead to poor selection. In this paper we discuss six new strategies, based on list combination, designed to yield improved selections starting from the importances given by the binary problems. We evaluate them on artificial and real-world datasets, using both One–Vs–One (OVO) and One–Vs–All (OVA) strategies. Our results suggest that the OVO decomposition is most effective for feature selection on multiclass problems. We also find that in most situations the new K-First strategy can find better subsets of features than the traditional weight average approach.

Extracting driving signals from non-stationary time series

We propose a simple method for the reconstruction of slow dynamics perturbations from non-stationary time series records. The method traces the evolution of the perturbing signal by simultaneously learning the intrinsic stationary dynamics and the time dependency of the changing parameter. For this purpose, an extra input unit is added to a feedforward artificial neural network and a suitable error function minimized in the training process. Testing of our algorithm on synthetic data shows its efficacy and allows extracting general criteria for applications on real-world problems. Finally, a preliminary study of the well-known sunspot time series recovers particular features of this series, including recently reported changes in solar activity during last century.