Richard M. Everson

Karhunen–Loève procedure for gappy data

The problem of using the Karhunen–Loeve transform with partial data is addressed. Given a set of empirical eigenfunctions, we show how to recover the modal coefficients for each gappy snapshot by a least-squares procedure. This method gives an unbiased estimate of the data that lie in the gaps and permits gaps to be filled in a reasonable manner. In addition, a scheme is advanced for finding empirical eigenfunctions from gappy data. It is shown numerically that this procedure obtains spectra and eigenfunctions that are close to those obtained from unmarred data.

Independent Component Analysis: Principles and Practice

1. Introduction Stephen Roberts and Richard Everson 2. Fast ICA by a fixed-point algorithm that maximizes non-Gaussianity Aapo Hyvarinen 3. ICA, graphical models and variational methods Hagai Attias 4. Nonlinear independent component analysis Juha Karhunen 5. Separation of non-stationary natural signals Lucas Parra and Clay Spence 6. Separation of non-stationary sources: algorithms and performance Jean-Francois Cardoso and Dinh-Tuan Pham 7. Blind source separation by sparse decomposition in a signal dictionary Michael Zibulevsky, Barak Pearlmutter, Pau Bofill and Pavel Kisilev 8. Ensemble learning for blind source separation James Miskin and David MacKay 9. Image processing methods using ICA mixture models Te-Won Lee and Michael S. Lewicki 10. Latent class and trait models for data classification and visualisation Mark Girolami 11. Particle filters for non-stationary ICA Richard Everson and Stephen Roberts 12. ICA: model order selection and dynamic source models William Penny, Stephen Roberts and Richard Everson.

A MOPSO algorithm based exclusively on pareto dominance concepts

In extending the Particle Swarm Optimisation methodology to multi-objective problems it is unclear how global guides for particles should be selected. Previous work has relied on metric information in objective space, although this is at variance with the notion of dominance which is used to assess the quality of solutions. Here we propose methods based exclusively on dominance for selecting guides from a non-dominated archive. The methods are evaluated on standard test problems and we find that probabilistic selection favouring archival particles that dominate few particles provides good convergence towards and coverage of the Pareto front. We demonstrate that the scheme is robust to changes in objective scaling. We propose and evaluate methods for confining particles to the feasible region, and find that allowing particles to explore regions close to the constraint boundaries is important to ensure convergence to the Pareto front.

Using unconstrained elite archives for multiobjective optimization

Multiobjective evolutionary algorithms (MOEAs) have been the subject of numerous studies over the past 20 years. Recent work has highlighted the use of an active archive of elite, nondominated solutions to improve the optimization speed of these algorithms. However, preserving all elite individuals is costly in time (due to the linear comparison with all archived solutions needed before a new solution can be inserted into the archive). Maintaining an elite population of a fixed maximum size (by clustering or other means) alleviates this problem, but can cause retreating (or oscillitory) and shrinking estimated Pareto fronts - which can affect the efficiency of the search process. New data structures are introduced to facilitate the use of an unconstrained elite archive, without the need for a linear comparison to the elite set for every new individual inserted. The general applicability of these data structures is shown by their use in an evolution-strategy-based MOEA and a genetic-algorithm-based MOEA. It is demonstrated that MOEAs using the new data structures run significantly faster than standard, unconstrained archive MOEAs, and result in estimated Pareto fronts significantly ahead of MOEAs using a constrained archive. It is also shown that the use of an unconstrained elite archive permits robust criteria for algorithm termination to be used, and that the use of the data structure can also be used to increase the speed of algorithms using /spl epsi/-dominance methods.

Weakly Supervised Joint Sentiment-Topic Detection from Text

Sentiment analysis or opinion mining aims to use automated tools to detect subjective information such as opinions, attitudes, and feelings expressed in text. This paper proposes a novel probabilistic modeling framework called joint sentiment-topic (JST) model based on latent Dirichlet allocation (LDA), which detects sentiment and topic simultaneously from text. A reparameterized version of the JST model called Reverse-JST, obtained by reversing the sequence of sentiment and topic generation in the modeling process, is also studied. Although JST is equivalent to Reverse-JST without a hierarchical prior, extensive experiments show that when sentiment priors are added, JST performs consistently better than Reverse-JST. Besides, unlike supervised approaches to sentiment classification which often fail to produce satisfactory performance when shifting to other domains, the weakly supervised nature of JST makes it highly portable to other domains. This is verified by the experimental results on data sets from five different domains where the JST model even outperforms existing semi-supervised approaches in some of the data sets despite using no labeled documents. Moreover, the topics and topic sentiment detected by JST are indeed coherent and informative. We hypothesize that the JST model can readily meet the demand of large-scale sentiment analysis from the web in an open-ended fashion.

Dominance-Based Multiobjective Simulated Annealing

Simulated annealing is a provably convergent optimizer for single-objective problems. Previously proposed multiobjective extensions have mostly taken the form of a single-objective simulated annealer optimizing a composite function of the objectives. We propose a multiobjective simulated annealer utilizing the relative dominance of a solution as the system energy for optimization, eliminating problems associated with composite objective functions. We also propose a method for choosing perturbation scalings promoting search both towards and across the Pareto front. We illustrate the simulated annealers performance on a suite of standard test problems and provide comparisons with another multiobjective simulated annealer and the NSGA-II genetic algorithm. The new simulated annealer is shown to promote rapid convergence to the true Pareto front with a good coverage of solutions across it comparing favorably with the other algorithms. An application of the simulated annealer to an industrial problem, the optimization of a code-division-multiple access (CDMA) mobile telecommunications networks air interface, is presented and the simulated annealer is shown to generate nondominated solutions with an even and dense coverage that outperforms single objective genetic algorithm optimizers.

Management and Analysis of Large Scientific Datasets

The method of empirical eigenfunctions (Karhunen-Loève procedure) is developed within a framework suitable for dealing with large scientific datasets. It is shown that this furnishes an intrinsic representation of any given database which is always, in a well-defined mathematical sense, the optimal description. The methodology is illustrated by a variety of examples, arising out of current research and taken from pattern recognition, turbulent flow, physiology, and oceanographic flow. In each instance examples of the empirical eigenfunctions are presented.

Multi-class ROC analysis from a multi-objective optimisation perspective

The receiver operating characteristic (ROC) has become a standard tool for the analysis and comparison of classifiers when the costs of misclassification are unknown. There has been relatively little work, however, examining ROC for more than two classes. Here we discuss and present an extension to the standard two-class ROC for multi-class problems. We define the ROC surface for the Q-class problem in terms of a multi-objective optimisation problem in which the goal is to simultaneously minimise the Q(Q-1) misclassification rates, when the misclassification costs and parameters governing the classifiers behaviour are unknown. We present an evolutionary algorithm to locate the Pareto front-the optimal trade-off surface between misclassifications of different types. The use of the Pareto optimal surface to compare classifiers is discussed and we present a straightforward multi-class analogue of the Gini coefficient. The performance of the evolutionary algorithm is illustrated on a synthetic three class problem, for both k-nearest neighbour and multi-layer perceptron classifiers.

Chaotic dynamics of a bouncing ball

Abstract A detailed study of a mapping on a two-dimensional manifold is made. The mapping describes a system subject to periodic forcing, in particular an imperfectly elastic ball bouncing on a vibrating platform. Quasiperiodic motion on a one-dimensional manifold is proven, and observed numerically, at low forcing, while at higher forcing Smale horseshoes are present. We examine the evolution of the attracting set with changing parameter. Spatial structure is oganised by fixed points of the mapping and sudden changes occur by crises. A new type of chaos, in which a trajectory alternates between two distinct chaotic regions, is described and explained in terms of manifold collisions. Throughout we are concerned to examine the behaviour of Lyapunov exponents. Typical behaviour of Lyapunov exponents in the quasiperiodic regime under the influence of external noise is discussed. At higher forcing a certain symmetry of the attractor allows an analytic expression for the exponents to be given.

Inferring the eigenvalues of covariance matrices from limited, noisy data

The eigenvalue spectrum of covariance matrices is of central importance to a number of data analysis techniques. Usually, the sample covariance matrix is constructed from a limited number of noisy samples. We describe a method of inferring the true eigenvalue spectrum from the sample spectrum. Results of Silverstein (1986), which characterize the eigenvalue spectrum of the noise covariance matrix, and inequalities between the eigenvalues of Hermitian matrices are used to infer probability densities for the eigenvalues of the noise-free covariance matrix, using Bayesian inference. Posterior densities for each eigenvalue are obtained, which yield error estimates. The evidence framework gives estimates of the noise variance and permits model order selection by estimating the rank of the covariance matrix. The method is illustrated with numerical examples.