Agnar Höskuldsson

Variable and subset selection in PLS regression

The purpose of this paper is to present some useful methods for introductory analysis of variables and subsets in relation to PLS regression. We present here methods that are efficient in finding the appropriate variables or subset to use in the PLS regression. The general conclusion is that variable selection is important for successful analysis of chemometric data. An important aspect of the results presented is that lack of variable selection can spoil the PLS regression, and that cross-validation measures using a test set can show larger variation, when we use different subsets of X, than obtained by different methods. We also present an approach to orthogonal scatter correction. The procedures and comparisons are applied to industrial data.

The H-principle in modelling with applications to chemometrics

Abstract Hoskuldsson, A., 1992. The H-principle in modelling with applications to chemometrics. Chemometrics and Intelligent Laboratory Systems, 14: 139–153. Heisenberg formulated certain rules and principles for describing and predicting physical systems. The H-principle is a mathematical formulation of these principles, when modelling data. One can show that it has two major benefits compared to other principles of modelling data: (a) it determines a proper balance between fit (how well the model fits the data) and the variance of a predictor derived from the model; (b) it optimizes the mean square error of prediction with respect to bias and prediction variance associated with the model. Application of the H-principle to modelling generally gives more stable predictions than other models, because it eliminates variables/components with low predictive abilities. In the special case of partial least squares (PLS) regression it gives the criteria of PLS. The H-principle is here applied in the principal components analysis context to the selection of variables. In the context of regression it is applied to stepwise regression and nonlinear PLS. It is also used to determine the number of variables / components to be used in the model.

The H-principle: New ideas, algorithms and methods in applied mathematics and statistics

Abstract The H-principle, or the Heisenberg principle of mathematical modelling, is a new principle of carrying out mathematical analysis of data. It has its conceptual basis in the philosophical discussions of the 1920s concerning description of physical systems. It is a mathematical formulation of concepts given in the Heisenberg uncertainty relation. The main idea is to include the model uncertainties in the modelling procedure. The principle suggests that the modelling is carried out in steps, such that at each step we determine the improvement and the associated precison. The improvement and the associated precision are then balanced in a way prescribed by the Heisenberg uncertainty principle. This principle thus prescribes how the modelling procedure should be carried out. We have applied this to different fields of science. It has then generated new ideas, algorithms and methods. Here we shall present some results arrived at, when this principle was applied to some important areas of science. The algorithms are illustrated by chemometric examples.

Dimension of linear models

Abstract Determination of the proper dimension of a given linear model is one of the most important tasks in the applied modeling work. We consider here eight criteria that can be used to determine the dimension of the model, or equivalently, the number of components to use in the model. Four of these criteria are widely used ones, while the remaining four are ones derived from the H-principle of mathematical modeling. Many examples from practice show that the criteria derived from the H-principle function better than the known and popular criteria for the number of components. We shall briefly review the basic problems in determining the dimension of linear models. Then each of the eight measures are treated. The results are illustrated by examples.

Causal and path modelling

Abstract A general methodology that carries out causal and path modelling by the same tools as known by linear regression is presented. Data can be one block (like in PCA), two blocks (like in regression analysis), several blocks, e.g., derived from multi-way data, or a network of data blocks. Causality questions that we typically ask in PCA can be carried out for each block of data. The data blocks can make up a path, where each node contains two adjoining blocks. The two neighbouring data blocks have either the same number of variables or the same number of samples. The methods are based on the H-principle of mathematical modelling of data. A very general path or network of data blocks can be analysed. An important aspect of this approach is that most methods of linear regression analysis can be carried out within this framework. The procedures are based on projections of one latent structure onto the following one. These methods can therefore be used to detect (differential) changes in the latent structure (e.g., in loadings or scores) from one block to another.

THE HEISENBERG MODELLING PROCEDURE AND APPLICATION TO NONLINEAR MODELLING

Abstract The main features of the Heisenberg modelling procedure are presented. These include weighing schemes, the simultaneous expansion of X and X + , orthogonality requirements, stopping rules and objectives of modelling. These are applied to linear modelling in order to show how these tools work. We show how the criteria of linear PLS regression can be extended to include nonlinear regression. It is shown how the Heisenberg modelling procedure applies to modelling general nonlinear models. An important aspect of the Heisenberg modelling procedure is that it provides tools for outlier detection, sensitivity analysis, dimension analysis and variable (data) selection for each of its procedures.

Experimental design and priority PLS regression

In this paper some fundamental issues of experimental design are treated. The rules, ideas and algorithms of the H‐principle are used to analyse models that are derived from experimental design. A new approach to analysing models from experimental design is proposed that is called priority PLS regression. The idea is to study the models sequentially. where we focus on a part of the design matrix X at each step. The basic ideas of experimental design are reviewed and ‘matched’ with the methodology of the H‐principle.

Path regression models and process control optimisation

New regression methods to analyse multi‐block and path models are presented. The multi‐block and path data are assumed to be organised in a forward‐oriented network of data blocks. This means that there are input data blocks, where modelling starts, and output data blocks that are at the end of the network. Input and output data blocks are connected by intermediate data blocks. It is shown how the method of partial least squares (PLS) regression can be extended to the data blocks that are connected in the path. A simple optimisation procedure in score space is presented, which determines optimal scores at normal operating conditions. It is shown how the optimisation procedure applies to any data block in the path. The advantage of the presented methods is due to that similar method as in PLS regression can be applied to any two connected data blocks. It is indicated that present methods are more efficient to carry out regressions than path methods presented in the literature. The results are illustrated by process data. Copyright

Combined principal component preprocessing and n-tuple neural networks for improved classification

We present a combined principal component analysis/neural network scheme for classification. The data used to illustrate the method consist of spectral fluorescence recordings from seven different production facilities, and the task is to relate an unknown sample to one of these seven factories. The data are first preprocessed by performing an individual principal component analysis on each of the seven groups of data. The components found are then used for classifying the data, but instead of making a single multiclass classifier, we follow the ideas of turning a multiclass problem into a number of two‐class problems. For each possible pair of classes we further apply a transformation to the calculated principal components in order to increase the separation between the classes. Finally we apply the so‐called n‐tuple neural network to the transformed data in order to give the classification system non‐linear capabilities, and all derived two‐class models are combined to facilitate multiclass classification. Validation results show that the combined scheme is superior to the individual methods. Copyright

Stable solutions of linear dynamic models

Here we present a new approach to analyse dynamic models. It is based on the H‐principle of mathematical modelling. The key issue is to identify the covariance matrices involved and solve the set of equations by steps, where at each step we optimize the selection of the covariance. The advantage of the procedure is that updating of model estimates in linear least squares, biased estimation and Kalman filtering can be achieved in a stable and secure manner. It implies that e.g. Kalman filtering can be carried out for hundreds or thousands of variables. The solution obtained at each time step provides the optimal balance between the prediction and the estimation aspect of the model. The developed procedures have several advantages over traditional methods: we do not have initial conditions that are difficult to estimate; it is easy to find influential variables; and it is easy to carry out sensitivity tests and others. Copyright