Ernest Fokoué

Mixtures of factor analysers. Bayesian estimation and inference by stochastic simulation

Factor Analysis (FA) is a well established probabilistic approach to unsupervised learning for complex systems involving correlated variables in high-dimensional spaces. FA aims principally to reduce the dimensionality of the data by projecting high-dimensional vectors on to lower-dimensional spaces. However, because of its inherent linearity, the generic FA model is essentially unable to capture data complexity when the input space is nonhomogeneous. A finite Mixture of Factor Analysers (MFA) is a globally nonlinear and therefore more flexible extension of the basic FA model that overcomes the above limitation by combining the local factor analysers of each cluster of the heterogeneous input space. The structure of the MFA model offers the potential to model the density of high-dimensional observations adequately while also allowing both clustering and local dimensionality reduction. Many aspects of the MFA model have recently come under close scrutiny, from both the likelihood-based and the Bayesian perspectives. In this paper, we adopt a Bayesian approach, and more specifically a treatment that bases estimation and inference on the stochastic simulation of the posterior distributions of interest. We first treat the case where the number of mixture components and the number of common factors are known and fixed, and we derive an efficient Markov Chain Monte Carlo (MCMC) algorithm based on Data Augmentation to perform inference and estimation. We also consider the more general setting where there is uncertainty about the dimensionalities of the latent spaces (number of mixture components and number of common factors unknown), and we estimate the complexity of the model by using the sample paths of an ergodic Markov chain obtained through the simulation of a continuous-time stochastic birth-and-death point process. The main strengths of our algorithms are that they are both efficient (our algorithms are all based on familiar and standard distributions that are easy to sample from, and many characteristics of interest are by-products of the same process) and easy to interpret. Moreover, they are straightforward to implement and offer the possibility of assessing the goodness of the results obtained. Experimental results on both artificial and real data reveal that our approach performs well, and can therefore be envisaged as an alternative to the other approaches used for this model.

A Mathematical Formalization of Hierarchical Temporal Memory's Spatial Pooler

Hierarchical temporal memory (HTM) is an emerging machine learning algorithm, with the potential to provide a means to perform predictions on spatiotemporal data. The algorithm, inspired by the neocortex, currently does not have a comprehensive mathematical framework. This work brings together all aspects of the spatial pooler (SP), a critical learning component in HTM, under a single unifying framework. The primary learning mechanism is explored, where a maximum likelihood estimator for determining the degree of permanence update is proposed. The boosting mechanisms are studied and found to be only relevant during the initial few iterations of the network. Observations are made relating HTM to well-known algorithms such as competitive learning and attribute bagging. Methods are provided for using the SP for classification as well as dimensionality reduction. Empirical evidence verifies that given the proper parameterizations, the SP may be used for feature learning.

Frequentist Approximation of the Bayesian Posterior Inclusion Probability by Stochastic Subsampling

This paper uses stochastic subsampling of the dataset to provide a frequentist approximation of what is known in the Bsyesian framework as the posterior inclusion probability. The unique merit of this contribution lies in the fact that it makes it easier for typically frequentist-minded practitioners, of which there are very many, to relate to the way the Bayesian paradigm allows the computation of the nicely interpretable variable importance. Despite its computationally intensive nature due to the need to fit a very large number of models, the proposed approach is readily applicable to both classification and regression tasks and can be done in comparatively comparative computational times thanks to the availability of parallel computing facilities through cloud and cluster computing. Finally, the scheme proposed is very general and can therefore be easily adapted to all kinds of statistical prediction tasks. Application of the proposed method to some very famous benchmark datasets shows that it mimics the Bayesian counterpart quite well in the important context of variable selection.

Prediction Error Reduction Function as a Variable Importance Score

This paper introduces and develops a novel variable importance score function in the context of ensemble learning and demonstrates its appeal both theoretically and empirically. Our proposed score function is simple and more straightforward than its counterpart proposed in the context of random forest, and by avoiding permutations, it is by design computationally more efficient than the random forest variable importance function. Just like the random forest variable importance function, our score handles both regression and classification seamlessly. One of the distinct advantage of our proposed score is the fact that it offers a natural cut off at zero, with all the positive scores indicating importance and significance, while the negative scores are deemed indications of insignificance. An extra advantage of our proposed score lies in the fact it works very well beyond ensemble of trees and can seamlessly be used with any base learners in the random subspace learning context. Our examples, both simulated and real, demonstrate that our proposed score does compete mostly favorably with the random forest score.

An Optimal Experimental Design Perspective on Radial Basis Function Regression

This article provides a new look at radial basis function regression that reveals striking similarities with the traditional optimal experimental design framework. We show theoretically and computationally that the so-called relevant vectors derived through the relevance vector machine (RVM) and corresponding to the centers of the radial basis function network, are very similar and often identical to the support points obtained through various optimal experimental design criteria like D-optimality. This allows us to provide a statistical meaning to the relevant centers in the context of radial basis function regression, but also opens the door to a variety of ways of approach optimal experimental design in multivariate settings.

Random Subspace Learning (RASSEL) with data driven weighting schemes

We present a novel adaptation of the random subspace learning approach to regression analysis and classification of high dimension low sample size data, in which the use of the individual strength of each explanatory variable is harnessed to achieve a consistent selection of a predictively optimal collection of base learners. In the context of random subspace learning, random forest (RF) occupies a prominent place as can be seen by the vast number of extensions of the random forest idea and the multiplicity of machine learning applications of random forest. The adaptation of random subspace learning presented in this paper differs from random forest in the following ways: (a) instead of using trees as RF does, we use multiple linear regression (MLR) as our regression base learner and the generalized linear model (GLM) as our classification base learner and (b) rather than selecting the subset of variables uniformly as RF does, we present the new concept of sampling variables based on a multinomial distribution with weights (success ’probabilities’) driven through p independent one-way analysis of variance (ANOVA) tests on the predictor variables. The proposed framework achieves two substantial benefits, namely, (1) the avoidance of the extra computational burden brought by the permutations needed by RF to de-correlate the predictor variables, and (2) the substantial reduction in the average test error gained with the base learners used.

Nonnegative Matrix Factorization with Toeplitz Penalty

Nonnegative Matrix Factorization (NMF) is an unsupervised learning algorithm that produces a linear, parts-based approximation of a data matrix. NMF constructs a nonnegative low rank basis matrix and a nonnegative low rank matrix of weights which, when multiplied together, approximate the data matrix of interest using some cost function. The NMF algorithm can be modified to include auxiliary constraints which impose task-specific penalties or restrictions on the cost function of the matrix factorization. In this paper we propose a new NMF algorithm that makes use of non-datadependent auxiliary constraints which incorporate a Toeplitz matrix into the multiplicative updating of the basis and weight matrices. We compare the facial recognition performance of our new Toeplitz Nonnegative Matrix Factorization (TNMF) algorithm to the performance of the Zellner Nonnegative Matrix Factorization (ZNMF) algorithm which makes use of data-dependent auxiliary constraints. We also compare the facial recognition performance of the two aforementioned algorithms with the performance of several preexisting constrained NMF algorithms that have non-data-dependent penalties. The facial recognition performances are evaluated using the Cambridge ORL Database of Faces and the Yale Database of Faces.

A penalized maximum likelihood approach to the adaptive learning of the spatial pooler permanence

Hierarchical Temporal Memory is a machine learning algorithm for spatio-temporal information processing. One of the key functional units in this algorithm is the spatial pooler, which has been demonstrated to be efficient in classification, dimensionality reduction and for preprocessing non-spatial inputs. Formalization of the spatial pooler is proposed in recent literature. In this work, we present a principled theoretical formulation of the spatial poolers underlying learning scheme. Constraints from the active connected and emmeshed columns in a spatial pooler are included in the analysis. It has been observed that locally adaptive learning enhanced the performance of the spatial pooler as a feature selector.

Data Properties Underlying Human Monitoring Performance

The application of phasor measurement unit (PMU) data in the power industry is currently an area of intense interest. The key driver for PMU technology is to use the precise time sources provided by Global Positioning System (GPS) satellites to accurately measure the relative voltage and current phase angles at buses across an interconnect at a very high sampling rate. Presenting PMU data to power system operators in a format that is truly useful for them and that affords improved situation awareness (SA) and fast and accurate decision making is a particular challenge to display design. This paper described development of prototype displays of PMU data where the design criteria were derived from characteristics of the data as well as the situation awareness requirements of power system dispatchers. A prototype display that meets all these design criteria is described.

An Information-Theoretical Alternative to the Cronbach's Alpha Coefficient of Item Reliability

We propose an information-theoretic alternative to the popular Cronbach alpha coefficient of reliability. Particularly suitable for contexts in which instruments are scored on a strictly nonnumeric scale, our proposed index is based on functions of the entropy of the distributions of defined on the sample space of responses. Our reliability index tracks the Cronbach alpha coefficient uniformly while offering several other advantages discussed in great details in this paper.