Amauri Holanda de Souza Júnior

Ensemble of Minimal Learning Machines for Pattern Classification

The use of ensemble methods for pattern classification have gained attention in recent years mainly due to its improvements on classification rates. This paper evaluates ensemble learning methods using the Minimal Learning Machines (MLM), a recently proposed supervised learning algorithm. Additionally, we introduce an alternative output estimation procedure to reduce the complexity of the standard MLM. The proposed methods are evaluated on real datasets and compared to several state-of-the-art classification algorithms.

Minimal learning machine: a new distance-based method for supervised learning

In this work, a novel supervised learning method, the Minimal Learning Machine (MLM), is proposed. Learning a MLM consists in reconstructing the mapping existing between input and output distance matrices and then estimating the response from the geometrical configuration of the output points. Given its general formulation, the Minimal Learning Machine is inherently capable to operate on nonlinear regression problems as well as on multidimensional response spaces. In addition, an intuitive extension of the MLM is proposed to deal with classification problems. On the basis of our experiments, the Minimal Learning Machine is able to achieve accuracies that are comparable to many de facto standard methods for regression and it offers a computationally valid alternative to such approaches.

Regional models: A new approach for nonlinear system identification via clustering of the self-organizing map

Abstract Global modelling consists in fitting a single regression model to the available data, using the whole set of input and output observations. On the other side of the spectrum stands the local modelling approach, in which the input space is segmented into several small partitions and a specialized regression model is fit to each partition. In this paper, we propose a novel approach, called Regional Models (RM), that stands in between the global and local modelling ones. The proposal extends the two-level clustering approach by Vesanto and Alhoniemi (2000 [1] ) to regression problems, more specifically, to system identification. In this regard, we first partition the input space using the Self-Organizing Map (SOM), and then perform clustering over the prototypes of the trained SOM. Finally, regional regression models are built over the clusters (i.e. over the regions) of SOM prototypes, not over each SOM prototype as in local modelling. Under the proposed framework, we build regional linear and nonlinear regression models. For the linear case, we use autoregressive models with eXogenous (ARX) whose parameters are estimated using the ordinary least-squares (OLS) method. Regional nonlinear ARX (NARX) models are built using the Extreme Learning Machine network. Additionally, we develop robust variants of the proposed regional models through the use of M-estimation, a statistical framework for handling outliers, since the OLS is highly sensitive to them. Comprehensive performance evaluation of the proposed models on synthetic and real-world datasets is carried out and the results compared to those achieved by standard global and local models.

A Speech Recognition System for Embedded Applications Using the SOM and TS-SOM Networks

The self-organizing map (SOM) (Kohonen, 1982) is one of the most important neural network architecture. Since its invention it has been applied to so many areas of Science and Engineering that it is virtually impossible to list all the applications available to date (van Hulle, 2010; Yin, 2008). In most of these applications, such as image compression (Amerijckx et al., 1998), time series prediction (Guillen et al., 2010; Lendasse et al., 2002), control systems (Cho et al., 2006; Barreto & Araujo, 2004), novelty detection (Frota et al., 2007), speech recognition and modeling (Gas et al., 2005), robotics (Barreto et al., 2003) and bioinformatics (Martin et al., 2008), the SOM is designed to be used by systems whose computational resources (e.g. memory space and CPU speed) are fully available. However, in applications where such resources are limited (e.g. embedded software systems, such as mobile phones), the SOM is rarely used, especially due to the cost of the best-matching unit (BMU) search (Sagheer et al., 2006). Essentially, the process of developing automatic speech recognition (ASR) systems is a challenging tasks due to many factors, such as variability of speaker accents, level of background noise, and large quantity of phonemes or words to deal with, voice coding and parameterization, among others. Concerning the development of ASR applications to mobile phones, to all the aforementioned problems, others are added, such as battery consumption requirements and low microphone quality. Despite those difficulties, with the significant growth of the information processing capacity of mobile phones, they are being used to perform tasks previously carried out only on personal computers. However, the standard user interface still limits their usability, since conventional keyboards are becoming smaller and smaller. A natural way to handle this new demand of embedded applications is through speech/voice commands. Since the neural phonetic typewriter (Kohonen, 1988), the SOM has been used in a standalone fashion for speech coding and recognition (see Kohonen, 2001, pp. 360-362). Hybrid architectures, such as SOM with MultiLayer Perceptrons (SOM-MLP) and SOM with Hidden Markov Models (SOM-HMM), have also been proposed (Gas et al., 2005; Somervuo, 2000). More specifically, studies involving speech recognition in mobile devices systems include those by Olsen et al. (2008); Alhonen et al. (2007) and Varga & Kiss (2008). It is worth noticing that Portuguese is the eighth, perhaps, the seventh most spoken language worldwide and the third among the Western countries, after English and Spanish. Despite 6

Ensemble of Efficient Minimal Learning Machines for Classification and Regression

Minimal Learning Machine (MLM) is a recently proposed supervised learning algorithm with performance comparable to most state-of-the-art machine learning methods. In this work, we propose ensemble methods for classification and regression using MLMs. The goal of ensemble strategies is to produce more robust and accurate models when compared to a single classifier or regression model. Despite its successful application, MLM employs a computationally intensive optimization problem as part of its test procedure (out-of-sample data estimation). This becomes even more noticeable in the context of ensemble learning, where multiple models are used. Aiming to provide fast alternatives to the standard MLM, we also propose the Nearest Neighbor Minimal Learning Machine and the Cubic Equation Minimal Learning Machine to cope with classification and single-output regression problems, respectively. The experimental assessment conducted on real-world datasets reports that ensemble of fast MLMs perform comparably or superiorly to reference machine learning algorithms.

Robust Regional Modeling for Nonlinear System Identification Using Self-Organizing Maps

Global modeling is a common approach to the problem of learning dynamical input-output mappings. It consists in fitting a single regression model, starting from the whole set of input and output measurements. On the other side of the spectrum, the local modeling approach segments the input space into several localized partitions (usually, Voronoi cells) and a number of specialized regression models are fit over each partition. Regional modeling stands in between the global and local approach. Firstly, the input space is indeed divided into partitions (as in local modeling), then partitions are merged into larger regions over which the regression models are built. In this paper, we extend the regional modeling approach through the use of robust regression, a statistical framework that better handles outliers and deviation of residuals from gaussianity. The approach is validated using two benchmark problems in system identification and its performance compared to those achieved by standard global and local models.

Euclidean distance estimation in incomplete datasets

This paper proposes a method to estimate the expected value of the Euclidean distance between two possibly incomplete feature vectors. Under the Missing at Random assumption, we show that the Euclidean distance can be modeled by a Nakagami distribution, for which the parameters we express as a function of the moments of the unknown data distribution. In our formulation the data distribution is modeled using a mixture of Gaussians. The proposed method, named Expected Euclidean Distance (EED), is validated through a series of experiments using synthetic and real-world data. Additionally, we show the application of EED to the Minimal Learning Machine (MLM), a distance-based supervised learning method. Experimental results show that EED outperforms existing methods that estimate Euclidean distances in an indirect manner. We also observe that the application of EED to the MLM provides promising results.

Spectroscopic monitoring of diesel fuels using Supervised Distance Preserving Projections

Abstract In this work, we discuss a recently proposed approach for supervised dimensionality reduction, the Supervised Distance Preserving Projection and, we investigate its applicability to monitoring materials properties from spectroscopic observations. Motivated by continuity preservation, the SDPP is a linear projection method where the local geometry of the points in the low-dimensional subspace mimics the geometry of the points in the response space. Such a mapping facilitates an efficient regressor design and it may also uncover useful information for visualisation. An experimental evaluation is conducted to show the performance of the SDPP and compare it with a number of state-of-the-art approaches for unsupervised and supervised dimensionality reduction. For the task, the results obtained on a benchmark problem consisting of a set of NIR spectra of diesel fuels and six different chemico-physical properties of those fuels is discussed. Based on the experimental results, the SDPP leads to accurate and parsimonious projections that can be used in the design of efficient regression models.

Extending the Minimal Learning Machine for Pattern Classification

The Minimal Learning Machine (MLM) has been recently proposed as a novel supervised learning method for regression problems aiming at reconstructing the mapping between input and output distance matrices. Estimation of the response is then achieved from the geometrical configuration of the output points. Thanks to its comprehensive formulation, the MLM is inherently capable of dealing with nonlinear problems and multidimensional output spaces. In this paper, we introduce an extension of the MLM to classification tasks, thus providing a unified framework for multiresponse regression and classification problems. On the basis of our experiments, the MLM achieves results that are comparable to many de facto standard methods for classification with the advantage of offering a computationally lighter alternative to such approaches.

Regional models for nonlinear system identification using the self-organizing map

Global modelling is a common approach to the problem of learning nonlinear dynamical input-output mappings. It consists in training a single multilayer neural network model using the whole dataset. On the other side of the spectrum stands the local modelling approach, in which the input space is divided into very small partitions and simpler (e.g. linear) models are trained, one per partition. In this paper, we propose a novel approach, called Regional Models (RM), that stands in between the global and local modelling ones. By following the approach by Vesanto and Alhoniemi [11], we first partition the input-output space using the Self-Organizing map (SOM), and then perform clustering over the prototypes of the trained SOM in order to find clusters of prototypes. Finally, a regional model is built for each cluster using the data vectors mapped to that cluster. The proposed approach is evaluated on two benchmarking problems and its performance is compared to those achieved by standard global and local models.