Ryan P. Browne

Mixtures of Shifted AsymmetricLaplace Distributions

A mixture of shifted asymmetric Laplace distributions is introduced and used for clustering and classification. A variant of the EM algorithm is developed for parameter estimation by exploiting the relationship with the generalized inverse Gaussian distribution. This approach is mathematically elegant and relatively computationally straightforward. Our novel mixture modelling approach is demonstrated on both simulated and real data to illustrate clustering and classification applications. In these analyses, our mixture of shifted asymmetric Laplace distributions performs favourably when compared to the popular Gaussian approach. This work, which marks an important step in the non-Gaussian model-based clustering and classification direction, concludes with discussion as well as suggestions for future work.

Mixtures of skew-t factor analyzers

A mixture of skew-t factor analyzers is introduced as well as a family of mixture models based thereon. The particular formulation of the skew-t distribution used arises as a special case of the generalized hyperbolic distribution. Like their Gaussian and t-distribution analogues, mixtures of skew-t factor analyzers are very well-suited for model-based clustering of high-dimensional data. The alternating expectation–conditional maximization algorithm is used for model parameter estimation and the Bayesian information criterion is used for model selection. The models are applied to both real and simulated data, giving superior clustering results when compared to a well-established family of Gaussian mixture models.

A mixture of generalized hyperbolic distributions

We introduce a mixture of generalized hyperbolic distributions as an alternative to the ubiquitous mixture of Gaussian distributions as well as their near relatives of which the mixture of multivariate t and skew-t distributions are predominant. The mathematical development of our mixture of generalized hyperbolic distributions model relies on its relationship with the generalized inverse Gaussian distribution. The latter is reviewed before our mixture models are presented along with details of the aforesaid reliance. Parameter estimation is outlined within the expectation-maximization framework before the clustering performance of our mixture models is illustrated via applications on simulated and real data. In particular, the ability of our models to recover parameters for data from underlying Gaussian and skew-t distributions is demonstrated. Finally, the role of Generalized hyperbolic mixtures within the wider model-based clustering, classification, and density estimation literature is discussed.

Model-Based Learning Using a Mixture of Mixtures of Gaussian and Uniform Distributions

We introduce a mixture model whereby each mixture component is itself a mixture of a multivariate Gaussian distribution and a multivariate uniform distribution. Although this model could be used for model-based clustering (model-based unsupervised learning) or model-based classification (model-based semi-supervised learning), we focus on the more general model-based classification framework. In this setting, we fit our mixture models to data where some of the observations have known group memberships and the goal is to predict the memberships of observations with unknown labels. We also present a density estimation example. A generalized expectation-maximization algorithm is used to estimate the parameters and thereby give classifications in this mixture of mixtures model. To simplify the model and the associated parameter estimation, we suggest holding some parameters fixed-this leads to the introduction of more parsimonious models. A simulation study is performed to illustrate how the model allows for bursts of probability and locally higher tails. Two further simulation studies illustrate how the model performs on data simulated from multivariate Gaussian distributions and on data from multivariate t-distributions. This novel approach is also applied to real data and the performance of our approach under the various restrictions is discussed.

Estimating common principal components in high dimensions

We consider the problem of minimizing an objective function that depends on an orthonormal matrix. This situation is encountered, for example, when looking for common principal components. The Flury method is a popular approach but is not effective for higher dimensional problems. We obtain several simple majorization–minimization (MM) algorithms that provide solutions to this problem and are effective in higher dimensions. We use mixture model-based clustering applications to illustrate our MM algorithms. We then use simulated data to compare them with other approaches, with comparisons drawn with respect to convergence and computational time.

A mixture of common skew-t factor analysers

A mixture of common skew-t factor analyzers model is introduced for model-based clustering of high-dimensional data. By assuming common component factor loadings, this model allows clustering to be performed in the presence of a large number of mixture components or when the number of dimensions is too large to be well-modelled by the mixtures of factor analyzers model or a variant thereof. Furthermore, assuming that the component densities follow a skew-t distribution allows robust clustering of skewed data. The alternating expectation-conditional maximization algorithm is employed for parameter estimation. We demonstrate excellent clustering performance when our model is applied to real and simulated data.This paper marks the first time that skewed common factors have been used.

A mixture of generalized hyperbolic factor analyzers

The mixture of factor analyzers model, which has been used successfully for the model-based clustering of high-dimensional data, is extended to generalized hyperbolic mixtures. The development of a mixture of generalized hyperbolic factor analyzers is outlined, drawing upon the relationship with the generalized inverse Gaussian distribution. An alternating expectation-conditional maximization algorithm is used for parameter estimation, and the Bayesian information criterion is used to select the number of factors as well as the number of components. The performance of our generalized hyperbolic factor analyzers model is illustrated on real and simulated data, where it performs favourably compared to its Gaussian analogue and other approaches.

Orthogonal Stiefel manifold optimization for eigen-decomposed covariance parameter estimation in mixture models

Within the mixture model-based clustering literature, parsimonious models with eigen-decomposed component covariance matrices have dominated for over a decade. Although originally introduced as a fourteen-member family of models, the current state-of-the-art is to utilize just ten of these models; the rationale for not using the other four models usually centers around parameter estimation difficulties. Following close examination of these four models, we find that two are actually easily implemented using existing algorithms but that two benefit from a novel approach. We present and implement algorithms that use an accelerated line search for optimization on the orthogonal Stiefel manifold. Furthermore, we show that the ‘extra’ models that these decompositions facilitate outperform the current state-of-the art when applied to two benchmark data sets.

Mixtures of multivariate power exponential distributions

An expanded family of mixtures of multivariate power exponential distributions is introduced. While fitting heavy-tails and skewness have received much attention in the model-based clustering literature recently, we investigate the use of a distribution that can deal with both varying tail-weight and peakedness of data. A family of parsimonious models is proposed using an eigen-decomposition of the scale matrix. A generalized expectation-maximization algorithm is presented that combines convex optimization via a minorization-maximization approach and optimization based on accelerated line search algorithms on the Stiefel manifold. Lastly, the utility of this family of models is illustrated using both toy and benchmark data.

Hypothesis Testing for Mixture Model Selection

ABSTRACT Gaussian mixture models with eigen-decomposed covariance structures, i.e. the Gaussian parsimonious clustering models (GPCM), make up the most popular family of mixture models for clustering and classification. Although the GPCM family has been used for almost 20 years, selecting the best member of the family in a given situation remains a troublesome problem. Likelihood ratio (LR) tests are developed to tackle this problem; given a number of mixture components, these LR tests compare each member of the family to the heteroscedastic model under the alternative hypothesis. Along the way, a novel maximum likelihood estimation procedure is developed for two members of the GPCM family. Simulations show that the reference distribution provides a reasonable approximation for the LR statistics when the sample size is not too small and when the mixture components are separate enough; accordingly, in the remaining configurations, a parametric bootstrap approach is also discussed and evaluated. Furthermore, a closed testing procedure, having the defined LR tests as local tests, is considered to assess, in a straightforward way, a unique model in the general family. In contrast with the information criteria that are often employed in the literature as ‘black boxes’, it is only based on one subjective element, the significance level, whose meaning is clear to everyone. Simulation results are presented to investigate the performance of the procedure in situations with gradual departure from the homoscedastic model and its robustness with respect to elliptical departures from normality in each mixture component. Finally, the advantages of the procedure are illustrated via applications to some well-known data sets.