Nickolay T. Trendafilov

A Modified Principal Component Technique Based on the LASSO

In many multivariate statistical techniques, a set of linear functions of the original p variables is produced. One of the more difficult aspects of these techniques is the interpretation of the linear functions, as these functions usually have nonzero coefficients on all p variables. A common approach is to effectively ignore (treat as zero) any coefficients less than some threshold value, so that the function becomes simple and the interpretation becomes easier for the users. Such a procedure can be misleading. There are alternatives to principal component analysis which restrict the coefficients to a smaller number of possible values in the derivation of the linear functions, or replace the principal components by “principal variables.” This article introduces a new technique, borrowing an idea proposed by Tibshirani in the context of multiple regression where similar problems arise in interpreting regression equations. This approach is the so-called LASSO, the “least absolute shrinkage and selection operator,” in which a bound is introduced on the sum of the absolute values of the coefficients, and in which some coefficients consequently become zero. We explore some of the properties of the new technique, both theoretically and using simulation studies, and apply it to an example.

DALASS: Variable selection in discriminant analysis via the LASSO

The objective of DALASS is to simplify the interpretation of Fishers discriminant function coefficients. The DALASS problem-discriminant analysis (DA) modified so that the canonical variates satisfy the LASSO constraint-is formulated as a dynamical system on the unit sphere. Both standard and orthogonal canonical variates are considered. The globally convergent continuous-time algorithms are illustrated numerically and applied to some well-known data sets.

Projected gradient approach to the numerical solution of the SCoTLASS

The SCoTLASS problem-principal component analysis modified so that the components satisfy the Least Absolute Shrinkage and Selection Operator (LASSO) constraint-is reformulated as a dynamical system on the unit sphere. The LASSO inequality constraint is tackled by exterior penalty function. A globally convergent algorithm is developed based on the projected gradient approach. The algorithm is illustrated numerically and discussed on a well-known data set.

The Orthogonally Constrained Regression Revisited

The Penrose regression problem, including the orthonormal Procrustes problem and rotation problem to a partially specified target, is an important class of data matching problems arising frequently in multivariate analysis, yet its optimality conditions have never been clearly understood. This work offers a way to calculate the projected gradient and the projected Hessian explicitly. One consequence of this calculation is the complete characterization of the first order and the second order necessary and sufficient optimality conditions for this problem. Another application is the natural formulation of a continuous steepest descent ow that can serve as a globally convergent numerical method. Applications to the orthonormal Procrustes problem and Penrose regression problem with partially specified target are demonstrated in this article. Finally, some numerical results are reported and commented.

On a differential equation approach to the weighted orthogonal Procrustes problem

The weighted orthogonal Procrustes problem, an important class of data matching problems in multivariate data analysis, is reconsidered in this paper. It is shown that a steepest descent flow on the manifold of orthogonal matrices can naturally be formulated. This formulation has two important implications: that the weighted orthogonal Procrustes problem can be solved as an initial value problem by any available numerical integrator and that the first order and the second order optimality conditions can also be derived. The proposed approach is illustrated by numerical examples.

Independent Component Analysis of Climate Data: A New Look at EOF Rotation

The complexity inherent in climate data makes it necessary to introduce more than one statistical tool to the researcher to gain insight into the climate system. Empirical orthogonal function (EOF) analysis is one of the most widely used methods to analyze weather/climate modes of variability and to reduce the dimensionality of the system. Simple structure rotation of EOFs can enhance interpretability of the obtained patterns but cannot provide anything more than temporal uncorrelatedness. In this paper, an alternative rotation method based on independent component analysis (ICA) is considered. The ICA is viewed here as a method of EOF rotation. Starting from an initial EOF solution rather than rotating the loadings toward simplicity, ICA seeks a rotation matrix that maximizes the independence between the components in the time domain. If the underlying climate signals have an independent forcing, one can expect to find loadings with interpretable patterns whose time coefficients have properties that go beyond simple noncorrelation observed in EOFs. The methodology is presented and an application to monthly means sea level pressure (SLP) field is discussed. Among the rotated (to independence) EOFs, the North Atlantic Oscillation (NAO) pattern, an Arctic Oscillation–like pattern, and a Scandinavian-like pattern have been identified. There is the suggestion that the NAO is an intrinsic mode of variability independent of the Pacific.

Exploratory Factor Analysis of Data Matrices With More Variables Than Observations

A new approach for exploratory factor analysis (EFA) of data matrices with more variables p than observations n is presented. First, the classic EFA model (n > p) is considered as a specific data matrix decomposition with fixed unknown matrix parameters. Then, it is generalized to a new model, called for short GEFA, which covers both cases of data, with either n > p or p ≥ n. An alternating least squares algorithm GEFALS is proposed for simultaneous estimation of all GEFA model parameters. Like principal component analysis (PCA), GEFALS is based on singular value decomposition, which makes GEFA an attractive alternative to PCA for descriptive data analysis and dimensionality reduction. The existence and uniqueness of the GEFA parameter estimation is studied and the convergence properties of GEFALS are established. Finally, the new approach is applied to both artificial (Thurstone’s 26-variable box data) and real high-dimensional data, while the performance of GEFALS is illustrated with simulation experiment. Some codes and data are available online as supplemental materials.

The multimode Procrustes problem

In this paper, we consider a generalization of the well-known Procrustes problem relevant to principal component analysis of multidimensional data arrays. This multimode Procrustes problem is a complex constrained minimization problem which involves the simultaneous least-squares fitting of several matrices. We propose two solutions of the problem: the projected gradient approach which leads to solving ordinary differential equations on matrix manifolds, and differential-geometric approach for optimization on products of matrix manifolds. A numerical example concerning the three-mode Procrustes illustrates the developed algorithms.

Stepwise estimation of common principal components

The standard common principal components (CPCs) may not always be useful for simultaneous dimensionality reduction in k groups. Moreover, the original FG algorithm finds the CPCs in arbitrary order, which does not reflect their importance with respect to the explained variance. A possible alternative is to find an approximate common subspace for all k groups. A new stepwise estimation procedure for obtaining CPCs is proposed, which imitates standard PCA. The stepwise CPCs facilitate simultaneous dimensionality reduction, as their variances are decreasing at least approximately in all k groups. Thus, they can be a better alternative for dimensionality reduction than the standard CPCs. The stepwise CPCs are found sequentially by a very simple algorithm, based on the well-known power method for a single covariance/correlation matrix. Numerical illustrations on well-known data are considered.

A majorization algorithm for simultaneous parameter estimation in robust exploratory factor analysis

A new approach for fitting the exploratory factor analysis (EFA) model is considered. The EFA model is fitted directly to the data matrix by minimizing a weighted least squares (WLS) goodness-of-fit measure. The WLS fitting problem is solved by iteratively performing unweighted least squares fitting of the same model. A convergent reweighted least squares algorithm based on iterative majorization is developed. The influence of large residuals in the loss function is curbed using Hubers criterion. This procedure leads to robust EFA that can resist the effect of outliers in the data. Applications to real and simulated data illustrate the performance of the proposed approach.