Levy Boccato

2012 Special Issue: An extended echo state network using Volterra filtering and principal component analysis

Echo state networks (ESNs) can be interpreted as promoting an encouraging compromise between two seemingly conflicting objectives: (i) simplicity of the resulting mathematical model and (ii) capability to express a wide range of nonlinear dynamics. By imposing fixed weights to the recurrent connections, the echo state approach avoids the well-known difficulties faced by recurrent neural network training strategies, but still preserves, to a certain extent, the potential of the underlying structure due to the existence of feedback loops within the dynamical reservoir. Moreover, the overall training process is relatively simple, as it amounts essentially to adapting the readout, which usually corresponds to a linear combiner. However, the linear nature of the output layer may limit the capability of exploring the available information, since higher-order statistics of the signals are not taken into account. In this work, we present a novel architecture for an ESN in which the linear combiner is replaced by a Volterra filter structure. Additionally, the principal component analysis technique is used to reduce the number of effective signals transmitted to the output layer. This idea not only improves the processing capability of the network, but also preserves the simplicity of the training process. The proposed architecture is then analyzed in the context of a set of representative information extraction problems, more specifically supervised and unsupervised channel equalization, and blind separation of convolutive mixtures. The obtained results, when compared to those produced by already proposed ESN versions, highlight the benefits brought by the novel network proposal and characterize it as a promising tool to deal with challenging signal processing tasks.

An echo state network architecture based on volterra filtering and PCA with application to the channel equalization problem

Echo state networks represent a promising alternative to the classical approaches involving recurrent neural networks, as they ally processing capability, due to the existence of feedback loops within the dynamical reservoir, with a simplified training process. However, the existing networks cannot fully explore the potential of the underlying structure, since the outputs are computed via linear combinations of the internal states. In this work, we propose a novel architecture for an echo state network that employs the Volterra filter structure in the output layer together with the Principal Component Analysis technique. This idea not only improves the processing capability of the network, but also preserves the simplicity of the training process. The proposed architecture has been analyzed in the context of the channel equalization problem, and the obtained results highlight the adequacy and the advantages of the novel network, which achieved a convincing performance, overcoming the other echo state networks, especially in the most challenging scenarios.

Application of natural computing algorithms to maximum likelihood estimation of direction of arrival

This work presents a study of the performance of populational meta-heuristics belonging to the field of natural computing when applied to the problem of direction of arrival (DOA) estimation, as well as an overview of the literature about the use of such techniques in this problem. These heuristics offer a promising alternative to the conventional approaches in DOA estimation, as they search for the global optima of the maximum likelihood (ML) function in a framework characterized by an elegant balance between global exploration and local improvement, which are interesting features in the context of multimodal optimization, to which the ML-DOA estimation problem belongs. Thus, we shall analyze whether these algorithms are capable of implementing the ML estimator, i.e., finding the global optima of the ML function. In this work, we selected three representative natural computing algorithms to perform DOA estimation: differential evolution, clonal selection algorithm, and the particle swarm. Simulation results involving different scenarios confirm that these methods can reach the performance of the ML estimator, regardless of the number of sources and/or their nature. Moreover, the number of points evaluated by such methods is quite inferior to that associated with a grid search, which gives support to their application.

Self-organization and lateral interaction in echo state network reservoirs

Abstract Echo state networks (ESNs) are recurrent structures that give rise to an interesting trade-off between achievable performance and tractability. This is a consequence of the fact that the key element of these networks – the recurrent intermediate layer known as dynamical reservoir – is not, as a rule, subject to supervised training, which is restricted to the linear output layer, also termed as readout. This trade-off, aside from being of theoretical significance, establishes ESNs as most attractive tools for both online and offline information processing. There are two key aspects to be taken into account in the ESN design: (i) the unsupervised definition of the synaptic weights of the reservoir and (ii) the definition of the structure and of the training strategy associated with the readout. This work is concerned with the first of these aspects: it proposes novel strategies for ESN reservoir design based on the theoretical framework built by Kohonen׳s classical works on self-organization – which includes the notions of short-range positive feedback and lateral inhibition – and also on the related and more recent notion of neural gas. It is shown, with the aid of a representative set of simulation results, that the proposed methodologies are capable of leading to significant performance improvements in the context of relevant information processing tasks – channel equalization and chaotic time series prediction – particularly when the input data suits well a cluster-based profile.

UNORGANIZED MACHINES FOR SEASONAL STREAMFLOW SERIES FORECASTING

Modern unorganized machines--extreme learning machines and echo state networks--provide an elegant balance between processing capability and mathematical simplicity, circumventing the difficulties associated with the conventional training approaches of feedforward/recurrent neural networks (FNNs/RNNs). This work performs a detailed investigation of the applicability of unorganized architectures to the problem of seasonal streamflow series forecasting, considering scenarios associated with four Brazilian hydroelectric plants and four distinct prediction horizons. Experimental results indicate the pertinence of these models to the focused task.

Automatic feature selection for BCI: An analysis using the davies-bouldin index and extreme learning machines

In this work, we present a novel framework for automatic feature selection in brain-computer interfaces (BCIs). The proposal, which manipulates features generated in the frequency domain by an estimate of the power spectral density of the EEG signals, is based on feature optimization (with both binary and real coding) using a state-of-the-art artificial immune network, the cob-aiNet. In order to analyze the performance of the proposed framework, two approaches are adopted: a direct use of the Davies-Bouldin index and the use of metrics associated with the operation of an extreme learning machine (ELM) in the role of a classifier. The results reveal that the proposal has the potential of improving the performance of a BCI system, and also provide elements for an analysis of the spectral content of EEG signals and of the performance of ELMs in motor imagery paradigms.

Echo state networks and extreme learning machines: a comparative study on seasonal streamflow series prediction

Extreme Learning Machines (ELMs) and Echo State Networks (ESNs) represent promising alternatives in time series forecasting in view of their intrinsic trade-off between performance and mathematical tractability. Both approaches share a key feature: their supervised parameter adaptation is restricted to the output layer, the remaining synaptic weights being chosen according to a priori unsupervised schemes. This work performs a comparative investigation regarding the performances of a classic ELM and ESNs in the context of the prediction of monthly seasonal streamflow series associated with Brazilian hydroelectric plants. With respect to the ESN, two possible reservoir design approaches are tested, as well as the novel architecture of Boccato et al., which is characterized by the use a Volterra filter and PCA in the readout. Additionally, a MLP is included to establish a base for comparison. Results show the relevance of these architectures in modeling seasonal streamflow series.

Performance analysis of nonlinear echo state network readouts in signal processing tasks

Echo state networks (ESNs) characterize an attractive alternative to conventional recurrent neural network (RNN) approaches as they offer the possibility of preserving, to a certain extent, the processing capability of a recurrent architecture and, at the same time, of simplifying the training process. However, the original ESN architecture cannot fully explore the potential of the RNN, given that only the second-order statistics of the signals are effectively used. In order to overcome this constraint, distinct proposals promote the use of a nonlinear readout aiming to explore higher-order available information though still maintaining a closed-form solution in the least-squares sense. In this work, we review two proposals of nonlinear readouts - a Volterra filter structure and an extreme learning machine - and analyze the performance of these architectures in the context of two relevant signal processing tasks: supervised channel equalization and chaotic time series prediction. The obtained results reveal that the nonlinear readout can be decisive in the process of aproximating the desired signal. Additionally, we discuss the possibility of combining both ideas of nonlinear readouts and preliminary results indicate that a performance improvement can be attained.

Unorganized Machines: From Turing’s Ideas to Modern Connectionist Approaches

This work presents a discussion about the relationship between the contributions of Alan Turing – the centenary of whose birth is celebrated in 2012 – to the field of artificial neural networks and modern unorganized machines: reservoir computing (RC) approaches and extreme learning machines (ELMs). Firstly, the authors review Turing’s connectionist proposals and also expose the fundamentals of the main RC paradigms – echo state networks and liquid state machines, - as well as of the design and training of ELMs. Throughout this exposition, the main points of contact between Turing’s ideas and these modern perspectives are outlined, being, then, duly summarized in the second and final part of the work. This paper is useful in offering a distinct appreciation of Turing’s pioneering contributions to the field of neural networks and also in indicating some perspectives for the future development of the field that may arise from the synergy between these views.

Multivariate PDF matching via kernel density estimation

In this work, a measure of similarity based on the matching of multivariate probability density functions (PDFs) is proposed. In consonance with the information theoretic learning (ITL) framework, the affinity comparison between the joint PDFs is performed using a quadratic distance, estimated with the aid of the Parzen window method with Gaussian kernels. The motivation underlying this proposal is to introduce a criterion capable of quantifying, to a significant extent, the statistical dependence present on information sources endowed with temporal and/or spatial structure, like audio, images and coded data. The measure is analyzed and compared with the canonical ITL-based approach - correntropy - for a set of blind equalization scenarios. The comparison includes elements like surface analysis, performance comparison in terms of bit error rate and a qualitative discussion concerning image processing. It is also important to remark that the study includes the application of two computational intelligence paradigms: extreme learning machines and differential evolution. The results indicate that the proposal can be, in some scenarios, a more informative formulation than correntropy.