Roberto Battiti

First- and second-order methods for learning: between steepest descent and Newton's method

On-line first-order backpropagation is sufficiently fast and effective for many large-scale classification problems but for very high precision mappings, batch processing may be the method of choice. This paper reviews first- and second-order optimization methods for learning in feedforward neural networks. The viewpoint is that of optimization: many methods can be cast in the language of optimization techniques, allowing the transfer to neural nets of detailed results about computational complexity and safety procedures to ensure convergence and to avoid numerical problems. The review is not intended to deliver detailed prescriptions for the most appropriate methods in specific applications, but to illustrate the main characteristics of the different methods and their mutual relations.

The Reactive Tabu Search

We propose an algorithm for combinatorial optimization where an explicit check for the repetition of configurations is added to the basic scheme of Tabu search. In our Tabu scheme the appropriate size of the list is learned in an automated way by reacting to the occurrence of cycles. In addition, if the search appears to be repeating an excessive number of solutions excessively often, then the search is diversified by making a number of random moves proportional to a moving average of the cycle length. The reactive scheme is compared to a “strict” Tabu scheme that forbids the repetition of configurations and to schemes with a fixed or randomly varying list size. From the implementation point of view we show that the Hashing or Digital Tree techniques can be used in order to search for repetitions in a time that is approximately constant. We present the results obtained for a series of computational tests on a benchmark function, on the 0-1 Knapsack Problem, and on the Quadratic Assignment Problem. INFORMS Journal on Computing , ISSN 1091-9856, was published as ORSA Journal on Computing from 1989 to 1995 under ISSN 0899-1499.

Statistical learning theory for location fingerprinting in wireless LANs

In this paper, techniques and algorithms developed in the framework of Statistical Learning Theory are applied to the problem of determining the location of a wireless device by measuring the signal strength values from a set of access points (location fingerprinting). Statistical Learning Theory provides a rich theoretical basis for the development of models starting from a set of examples. Signal strength measurement is part of the normal operating mode of wireless equipment, in particular Wi-Fi, so that no special-purpose hardware is required.The proposed techniques, based on the Support Vector Machine paradigm, have been implemented and compared, on the same data set, with other approaches considered in scientific literature. Tests performed in a real-world environment show that results are comparable, with the advantage of a low algorithmic complexity in the normal operating phase. Moreover, the algorithm is particularly suitable for classification, where it outperforms the other techniques.

Democracy in neural nets: voting schemes for classification

Abstract In this paper we discuss some possible ways to combine the outputs of a set of neural network classifiers to reach a combined decision with a higher performance, in terms of lower rejection rates and/or better accuracy rates. The methods considered range from the requirement of a complete agreement among the individual classifications to election schemes based on the distribution of votes collected by the different classes. In addition, the rejection rules based on the different output classes can be complemented by rules that also consider the information in the individual output vectors, with the possibility of using threshold requirements and that of averaging the different vectors. Although the Bayesian framework and some probabilistic assumptions provide useful indications about the potential advantage of different combination schemes, the combined performance ultimately depends on the joint probability distribution of the outputs, and it can be estimated by joining the results of different nets on the same test set. The combination methods are very flexible, they permit a straightforward cooperation of neural and traditional recognizers, and they are appropriate in a development environment where experiments are performed with different kinds of nets and features for a selected application. From our experiments in the field of handwritten digit recognition (up to a total of more than 50,000 characters), we found that the use of a small number of nets (two to three) with a sufficiently large uncorrelation in their mistakes reaches a combined performance that is significantly higher than the best obtainable from the individual nets, with a negligible effort after starting from a pool of networks produced in the development phase of an application. In particular, for a real-world OCR application, the best accuracy increase is about half the increase in the rejection rate, so that accuracies of the order of 99.5% can be reached by rejecting less than 5% of the patterns. This performance is significant for real applications.

Reactive Search and Intelligent Optimization

Reactive Search integrates sub-symbolic machine learning techniques into search heuristics for solving complex optimization problems. By automatically adjusting the working parameters, a reactive search self-tunes and adapts, effectively learning by doing until a solution is found. Intelligent Optimization, a superset of Reactive Search, concerns online and off-line schemes based on the use of memory, adaptation, incremental development of models, experimental algorithms applied to optimization, intelligent tuning and design of heuristics. Reactive Search and Intelligent Optimization is an excellent introduction to the main principles of reactive search, as well as an attempt to develop some fresh intuition for the approaches. The book looks at different optimization possibilities with an emphasis on opportunities for learning and self-tuning strategies. While focusing more on methods than on problems, problems are introduced wherever they help make the discussion more concrete, or when a specific problem has been widely studied by reactive search and intelligent optimization heuristics. Individual chapters cover reacting on the neighborhood; reacting on the annealing schedule; reactive prohibitions; model-based search; reacting on the objective function; relationships between reactive search and reinforcement learning; and much more. Each chapter is structured to show basic issues and algorithms; the parameters critical for the success of the different methods discussed; and opportunities and schemes for the automated tuning of these parameters. Anyone working in decision making in business, engineering, economics or science will find a wealth of information here.

MOEA/D-ACO: A Multiobjective Evolutionary Algorithm Using Decomposition and AntColony

Combining ant colony optimization (ACO) and the multiobjective evolutionary algorithm (EA) based on decomposition (MOEA/D), this paper proposes a multiobjective EA, i.e., MOEA/D-ACO. Following other MOEA/D-like algorithms, MOEA/D-ACO decomposes a multiobjective optimization problem into a number of single-objective optimization problems. Each ant (i.e., agent) is responsible for solving one subproblem. All the ants are divided into a few groups, and each ant has several neighboring ants. An ant group maintains a pheromone matrix, and an individual ant has a heuristic information matrix. During the search, each ant also records the best solution found so far for its subproblem. To construct a new solution, an ant combines information from its groups pheromone matrix, its own heuristic information matrix, and its current solution. An ant checks the new solutions constructed by itself and its neighbors, and updates its current solution if it has found a better one in terms of its own objective. Extensive experiments have been conducted in this paper to study and compare MOEA/D-ACO with other algorithms on two sets of test problems. On the multiobjective 0-1 knapsack problem, MOEA/D-ACO outperforms the MOEA/D with conventional genetic operators and local search on all the nine test instances. We also demonstrate that the heuristic information matrices in MOEA/D-ACO are crucial to the good performance of MOEA/D-ACO for the knapsack problem. On the biobjective traveling salesman problem, MOEA/D-ACO performs much better than the BicriterionAnt on all the 12 test instances. We also evaluate the effects of grouping, neighborhood, and the location information of current solutions on the performance of MOEA/D-ACO. The work in this paper shows that reactive search optimization scheme, i.e., the “learning while optimizing” principle, is effective in improving multiobjective optimization algorithms.

PILGRIM: A location broker and mobility-aware recommendation system

Mobile computing adds a mostly unexplored dimension to data mining: users position is a relevant piece of information, and recommendation systems, selecting and ranking links of interest to the user, have the opportunity to take location into account. In this paper a mobility-aware recommendation system that considers the location of the user to filter recommended links is proposed. To avoid the potential problems and costs of insertion by hand, a new middleware layer, the location broker, maintains a historic database of locations and corresponding links used in the past and develops models relating resources to their spatial usage pattern. These models are used to calculate a preference metric when the current user is asking for resources of interest. Mobility scenarios are described and analyzed in terms of possible user requirements. The features of the PILGRIM mobile recommendation system are outlined together with a preliminary experimental evaluation of different metrics.

The continuous reactive tabu search: Blending combinatorial optimization and stochastic search for global optimization

A novel algorithm for the global optimization of functions (C-RTS) is presented, in which a combinatorial optimization method cooperates with a stochastic local minimizer. The combinatorial optimization component, based on the Reactive Tabu Search recently proposed by the authors, locates the most promising “boxes”, in which starting points for the local minimizer are generated. In order to cover a wide spectrum of possible applications without user intervention, the method is designed with adaptive mechanisms: the box size is adapted to the local structure of the function to be optimized, the search parameters are adapted to obtain a proper balance of diversification and intensification. The algorithm is compared with some existing algorithms, and the experimental results are presented for a variety of benchmark tasks.

Parallel biased search for combinatorial optimization: genetic algorithms and TABU

Combinatorial optimization problems arise in different fields and require computing resources that grow very rapidly with the problem dimension. Therefore the use of massively parallel architectures represents an opportunity to be considered, characterized by very large speed-ups for significant applications. In this paper we consider some relatively general techniques that are paradigmatic of different parallel approaches, ranging from the concurrent execution of independent searches to a fully interacting ‘population’ of candidate solutions. In particular, we briefly summarize the TABU and GA algorithms, discuss their parallel implementation and present some experimental results on two benchmark problems: QAP and the N-k model. A new ‘reactive’ TABU scheme based on the ‘open hashing’ technique is also presented.

Reactive search, a history-sensitive heuristic for MAX-SAT

The Reactive Search (RS) method proposes the integration of a simple history-sensitive (machine learning) scheme into local search for the on-line determination of free parameters. In this paper a new RS algorithm is proposed for the approximated solution of the Maximum Satisfiability problem: a component based on local search with temporary prohibitions (Tabu Search) is complemented with a reactive scheme that determines the appropriate value of the prohibition parameter by monitoring the Hamming distance along the search trajectory. The proposed algorithm (H-RTS) can therefore be characterized as a dynamic version of Tabu Search. In addition, the non-oblivious functions recently introduced in the framework of approximation algorithms are used to discover a better local optimum in the initial part of the search The algorithm is developed in two phases. First the bias-diversification properties of individual candidate components are analyzed by extensive empirical evaluation, then a reactive scheme is added to the winning component, based on Tabu Search. The final tests on a benchmark of random MAX-3-SAT and MAX-4-SAT problems demonstrate the superiority of H-RTS with respect to alternative heuristics.