Péricles B. C. de Miranda

Combining a multi-objective optimization approach with meta-learning for SVM parameter selection

Support Vector Machine (SVM) is a supervised technique, which achieves good performance on different learning problems. However, adjustments on its model are essentials to the SVM work well. Optimization techniques have been used to automatize this process finding suitable configurations of parameters which attends some learning problems. This work utilizes Particle Swarm Optimization (PSO) applied to the SVM parameter selection problem. As the learning systems are essentially a multi-objective problem, a multi-objective PSO (MOPSO) was used to maximize the success rate and minimize the number of support vectors of the model. Nevertheless, we propose the combination of Meta-Learning (ML) with a modified MOPSO which uses the crowding distance mechanism (MOPSO-CDR). In this combination, solutions provided by ML are possibly located in good regions in the search space. Hence, using a reduced number of successful candidates, the search process would converge faster and be less expensive. In our work, we implemented a prototype in which MOPSO-CDR was used to select the values of two SVM parameters for classification problems. In the performed experiments, the proposed solution (MOPSO-CDR using ML) was compared to the MOPSO-CDR with random initialization, obtaining pareto fronts with higher quality on a set of 40 classification problems.

A hybrid meta-learning architecture for multi-objective optimization of SVM parameters

Support Vector Machines (SVMs) have achieved a considerable attention due to their theoretical foundations and good empirical performance when compared to other learning algorithms in different applications. However, the SVM performance strongly depends on the adequate calibration of its parameters. In this work we proposed a hybrid multi-objective architecture which combines meta-learning (ML) with multi-objective particle swarm optimization algorithms for the SVM parameter selection problem. Given an input problem, the proposed architecture uses a ML technique to suggest an initial Pareto front of SVM configurations based on previous similar learning problems; the suggested Pareto front is then refined by a multi-objective optimization algorithm. In this combination, solutions provided by ML are possibly located in good regions in the search space. Hence, using a reduced number of successful candidates, the search process would converge faster and be less expensive. In the performed experiments, the proposed solution was compared to traditional multi-objective algorithms with random initialization, obtaining Pareto fronts with higher quality on a set of 100 classification problems.

Multi-ring Particle Swarm Optimization

Particle swarm optimization (PSO) has been used to solve many different types of optimization problems. By applying PSO to problems where the feasible solutions are too much difficult to find, new ways of solving the problems are required, mainly for hyper dimensional spaces. Many variations on the basic PSO form have been explored, targeting the velocity update equation. Other approaches attempt to change the structure of the swarm. In this paper a novel PSO topology based on multiples rings is proposed for improving the results achieved focusing on the diversity provided by the ring rotations. A comparison with star and ring topologies was performed. Our simulation results have shown that the proposed topology achieves better results than the well known star and ring topologies.

Multi-objective optimization and Meta-learning for SVM parameter selection

Support Vector Machines (SVMs) have become a well succeed technique due to the good performance it achieves on different learning problems. However, the performance depends on adjustments on its model. The automatic SVM parameter selection is a way to deal with this. This approach is considered an optimization problem whose goal is to find suitable configuration of parameters which attends some learning problem. This work proposes the use of Particle Swarm Optimization (PSO) to treat the SVM parameter selection problem. As the design of learning systems is inherently a multi-objective optimization problem, a multi-objective PSO (MOPSO) was used to maximize the success rate and minimize the number of support vectors of the model. Moreover, we propose the combination of Meta-Learning (ML) with MOPSO to the cited problem. ML is used to recommend SVM parameters, to a given input problem, based on well-succeeded parameters adopted in previous similar problems. In this combination, initial solutions provided by ML are possibly located in good regions in the search space. Hence, using a reduced number of candidate search points, the search process, to find an adequate solution, would be less expensive. We highlight that, the combination of search algorithms with ML was just studied in the single objective field and the use of MOPSO in this context has not been investigated. In our work, we implemented a prototype in which MOPSO was used to select the values of two SVM parameters for classification problems. In the performed experiments, the proposed solution (MOPSO using ML or Hybrid MOPSO) was compared to a MOPSO with random initialization, obtaining paretos with higher quality on a set of 40 classification problems.

An experimental study of the combination of meta-learning with particle swarm algorithms for SVM parameter selection

Support Vector Machines (SVMs) have become a well succeed algorithm due to the good performance it achieves on different learning problems. However, to perform well the SVM formulation requires adjustments on its model. Avoiding the trial and error procedure, the automatic SVM parameter selection is a way to deal with this. The automatic parameter selection is commonly considered an optimization problem whose goal is to find suitable configuration of parameters which attends some learning problem. In the current work, we propose a study of the combination of Meta-learning (ML) with Particle Swarm Optimization (PSO) algorithms to optimize the SVM model, seeking for combinations of parameters which maximize the success rate of SVM. ML is used to recommend SVM parameters, to a given input problem, based on well-succeeded parameters adopted in previous similar problems. In this combination, initial solutions provided by ML are possibly located in good regions in the search space. Hence, using a reduced number of candidate search points, in the search process, to find an adequate solution, would be less expensive. In our work, we implemented five benchmarks PSO approaches applied to select two SVM parameters for classification. The experiments consist in comparing the performance of the search algorithms using a traditional random initialization and using ML suggestions as initial population. This research analysed the influence of meta-learning on convergence of the optimization algorithms, verifying that the combination of PSO techniques with ML obtained solutions with higher quality on a set of 40 classification problems.

Multi-Objective Particle Swarm Optimization using speciation

Particle Swarm Optimization (PSO) has been successfully extended to solve Multi-Objective Problems. These approaches are known as Multi-Objective Particle Swarm Optimizers (MOPSO). Most of the MOPSO proposes a different scheme to select the leaders used to update the velocity by using non-dominated solutions stored on an External Archive. MOPSO-CDR is one of these approaches and selects the social and the cognitive leaders based on the crowding distance. In this paper we propose a MOPSO with two distinct operation modes. The two modes are the basic mode, which is the same mode used in the MOPSO-CDR, and the speciation mode, where the swarm is divided in sub-swarms. In the latter, each swarm has a different target. The algorithm changes the operation mode based on the evaluation of the External Archive. We used well know metrics to evaluate the evolution of the Pareto Fronts, such as spacing and maximum spread. These metrics are used to determine the switching rules between the operation modes. We demonstrated that our proposal outperformed five other algorithms in five well know benchmark functions.

Generation of Particle Swarm Optimization algorithms: An experimental study using Grammar-Guided Genetic Programming

Abstract Particle Swarm Optimization (PSO) is largely used to solve optimization problems effectively. Nonetheless, the PSO performance depends on the fine tuning of different parameters. To make the algorithm design process more independent from human intervention, some researchers have treated this task as an optimization problem. Grammar-Guided Genetic Programming (GGGP) algorithms, in particular, have been widely studied and applied in the context of algorithm optimization. GGGP algorithms produce customized designs based on a set of production rules defined in the grammar, differently from methods that simply select designs in a pre-defined limited search space. Although GGGP algorithms have been largely used in other contexts, they have not been deeply investigated in the generation of PSO algorithms. Thus, this work applies GGGP algorithms in the context of PSO algorithm design problem. Herein, we performed an experimental study comparing different GGGP approaches for the generation of PSO algorithms. The main goal is to perform a deep investigation aiming to identify pros and cons of each approach in the current task. In the experiments, a comparison between a tree-based GGGP approach and commonly used linear GGGP approaches for the generation of PSO algorithms was performed. The results showed that the tree-based GGGP produced better algorithms than the counterparts. We also compared the algorithms generated by the tree-based technique to state-of-the-art optimization algorithms, and it achieved competitive results.

A Novel Context-Free Grammar to Guide the Construction of Particle Swarm Optimization Algorithms

Particle Swarm Optimization algorithm (PSO) has been largely studied over the years due to its flexibility and competitive results in different applications. Nevertheless, its performance depends on different aspects of design (e.g., inertia factor, velocity equation, topology). The task of deciding which is the best algorithm design to solve a particular problem is challenging due to the great number of possible variations and parameters to take into account. This work proposes a novel context-free grammar for Grammar-Guided Genetic Programming (GGGP) algorithms to guide the construction of Particle Swarm Optimizers. The proposed grammar addresses four aspects of the PSO algorithm that may strongly influence on its convergence: swarm initialization, neighborhood topology, velocity update equation and mutation operator. To evaluate this approach, a GGGP algorithm was set with the proposed grammar and applied to optimize the PSO algorithm in 32 unconstrained continuous optimization problems. In the experiments, we compared the designs generated considering the proposed grammar with the designs produced by other grammars proposed in the literature to automate PSO designs. The results obtained by the proposed grammar were better than the counterparts. Besides, we also compared the generated algorithms to state-of-art algorithms. The results have shown that the algorithms produced from the grammar achieved competitive results.

Tree-Based Grammar Genetic Programming to Evolve Particle Swarm Algorithms

Particle Swarm Optimization (PSO) is largely used to solve optimization problems effectively. Nonetheless, the PSO performance depends on the fine tuning of different parameters. To make the algorithm design process more independent from human intervention, some researchers have treated this task as an optimization problem. Grammar-guided Genetic Programming algorithms (GGGP), in special, have been widely studied and applied in the context of algorithm optimization. GGGP algorithms produce customized designs based on a set of production rules defined in the grammar, differently from methods that simply select designs in a pre-defined limited search space. In this work, we proposed a tree-based GGGP technique for the generation of PSO algorithms. This paper intends to investigate whether this approach can improve the production of PSO algorithms when compared to other GGGP techniques already used to solve the current problem. In the experiments, a comparison between the tree-based and the commonly used linearized GGGP approach for the generation of PSO algorithms was performed. The results showed that the tree-based GGGP produced better algorithms than the counterparts. We also compared the algorithms generated by the tree-based technique to state-of-art optimization algorithms, and the results showed that the algorithms produced by the tree-based GGGP achieved competitive results.

Active testing for SVM parameter selection

The Support Vector Machine algorithm is sensitive to the choice of parameter settings. If these are not set correctly, the algorithm may have a substandard performance. It has been shown that meta-learning can be used to support the selection of SVM parameters. However, it is very dependent on the quality of the dataset and the meta-features used to characterize the dataset. As alternative for this problem, a recent technique called Active Testing characterized a dataset based on the pairwise performance differences between possible solutions. This approach selects the most useful cross-validation tests. Each new cross-validation test will contribute information to a better estimate of dataset similarity, and thus better predict which algorithms are most promising on the new dataset. In this paper we propose the application of Active Testing for the SVM parameter problem. We test it on the problem of setting the RBF kernel parameters for classification problems and we compare its similarity strategy with based on data characteristics. The results showed the variants of Active Testing that rely on cross-validation tests to estimate dataset similarity provides better solutions than those that rely on data characteristics.