John W. Van Ness

On the Effects of Dimension in Discriminant Analysis

Given fixed numbers of labeled objects on which training data can be obtained, how many variables should be used for a particular discriminant algorithm? This, of course, cannot be answeredin general since it depends on the characteristics of the populations, the sample sizes, and the algorithm. Some insight is gained in this article by studying Gaussian populations and five algorithms: linear discrimination with urlknown means and known covariance, linear discrimination with unknown means and unknown covariances, quadratic discrimination with unknown covariances and two nonparametric Bayes-type algorithms having density estimates using different, kernels (Gaussian and Cauchy).

On the dominance of non-parametric Bayes rule discriminant algorithms in high dimensions

Considerable attention has been given to the relative performance of the various commonly used discriminant analysis algorithms. This performance has been studied under varying conditions. This author and others have been particularly interested in the behavior of the algorithms as dimension is varied. Here we consider three basic questions: which algorithms perform better in high dimensions, when does it pay to add or delete a dimension, and how discriminant algorithms are best implemented in high dimensions. One of the more interesting results has been the relatively good performance of non-parametric Bayes theorem type algorithms compared to parametric (linear and quadratic) algorithms. Surprisingly this effect occurs even when the underlying distributions are “ideal” for the parametric algorithms, provided, at least, that the true covariance matrices are not too close to singular. Monte Carlo results presented here further confirm this unexpected behavior and add to the existing literature (particularly Van Ness(9) and Van Ness et al.(11) by studying a different class of underlying Gaussian distributions. These and earlier results point out certain procedures, discussed here, which should be used in the selection of the density estimation windows for non-parametric algorithms to improve their performance. Measures of the effect on the various algorithms of adding dimensions are given graphically. A summary of some of the conclusions about several of the more common algorithms is included.

On the Effects of Dimension in Discriminant Analysis for Unequal Covariance Populations

This paper is a continuation of earlier work (Van Ness and Simpson [9]) studying the high dimensionality problem in discriminant analysis. Frequently one has potentially many possible variables (dimensions) to be measured on each object but is limited to a fixed training data size. For particular populations, we give here the change in probability of correct classilication caused by adding dimensions. This gives insight into how many variables one should use for fixed training data sizes, especially when dealing with the populations of these studies. We consider six basic discriminant analysis algorithms. Graphs are provided which compare the relative performance of the algorithms in high dimensions.

The Use of Shrinkage Estimators in Linear Discriminant Analysis

Probably the most common single discriminant algorithm in use today is the linear algorithm. Unfortunately, this algorithm has been shown to frequently behave poorly in high dimensions relative to other algorithms, even on suitable Gaussian data. This is because the algorithm uses sample estimates of the means and covariance matrix which are of poor quality in high dimensions. It seems reasonable that if these unbiased estimates were replaced by estimates which are more stable in high dimensions, then the resultant modified linear algorithm should be an improvement. This paper studies using a shrinkage estimate for the covariance matrix in the linear algorithm. We chose the linear algorithm, not because we particularly advocate its use, but because its simple structure allows one to more easily ascertain the effects of the use of shrinkage estimates. A simulation study assuming two underlying Gaussian populations with common covariance matrix found the shrinkage algorithm to significantly outperform the standard linear algorithm in most cases. Several different means, covariance matrices, and shrinkage rules were studied. A nonparametric algorithm, which previously had been shown to usually outperform the linear algorithm in high dimensions, was included in the simulation study for comparison.

Approximate Confidence Intervals for the Number of Clusters

Abstract We consider clustering for the purpose of data reduction. Similar objects are grouped together in clusters so that one can then work with the few cluster descriptors instead of the many data points. The quality of any given clustering is measured by a loss function that takes into account both the parsimony of the clustering and the loss of information due to clustering. An optimal clustering can be obtained by minimizing the theoretical loss function. It is shown that a sample version of the loss function and optimal clustering converge strongly to their theoretical counterparts as the sample size tends to infinity. We then develop a bootstrap-based procedure for obtaining approximate confidence bounds on the number of clusters in the “best” clustering. The effectiveness of this procedure is evaluated in a simulation study. An application is presented.

A routine for converting regression algorithms into corresponding orthogonal regression algorithms

The routine converts any standard regression algorithm (that calculates both the coefficients and residuals) into a corresponding <italic>orthogonal</italic> regression algorithm. Thus, a standard, or robust, or <italic>L</italic><subscrpt>1</subscrpt> regression algorithm is converted into the corresponding standard, or robust, or <italic>L</italic><subscrpt>1</subscrpt> <italic>orthogonal</italic> algorithm. Such orthogonal procedures are important for three basic reasons. First, they solve the classical errors-in-variables (EV) regression problem. Standard <italic>L</italic><subscrpt>2</subscrpt> orthogonal regression, obtained by converting ordinary least squares regression, is the maximum likelihood solution of the EV problem under Gaussian assumptions. However, this <italic>L</italic><subscrpt>2</subscrpt> solution is known to be unstable under even slight deviations from the model. Thus this routines ability to create <italic>robust</italic> orthogonal regression algorithms from robust ordinary regression algorithms will also be very useful in practice. Second, orthogonal regression is intimately related to principal components procedures. Therefore, this routine can also be used to create corresponding <italic>L</italic><subscrpt>1</subscrpt>, robust, etc., principal components algorithms. And third, orthogonal regression treats the <italic>x</italic> and <italic>y</italic> variables symmetrically. This is very important in many science and engineering modeling problems. Monte Carlo studies, which test the effectiveness of the routine under a variety of types of data, are given.

Space-conserving agglomerative algorithms

This paper evaluates a general, infinite family of clustering algorithms, called the Lance and Williams algorithms, with respect to the space-conserving criterion. An admissible clustering criterion is defined using the space conserving idea. Necessary and sufficient conditions for Lance and Williams clustering algorithms to satisfy space-conserving admissibility are provided. Space-dilating, space-contracting, and well-structured clustering algorithms are also discussed.

Standard and robust orthogonal regression

A fast routine for converting regression algorithms into corresponding orthogonal regression (OR) algorithms was introduced in Ammann and Van Ness (1988). The present paper discusses the properties of various ordinary and robust OR procedures created using this routine. OR minimizes the sum of the orthogonal distances from the regression plane to the data points. OR has three types of applications. First, L 2 OR is the maximum likelihood solution of the Gaussian errors-in-variables (EV) regression problem. This L 2 solution is unstable, thus the robust OR algorithms created from robust regression algorithms should prove very useful. Secondly, OR is intimately related to principal components analysis. Therefore, the routine can also be used to create L 1, robust, etc. principal components algorithms. Thirdly, OR treats the x and y variables symmetrically which is important in many modeling problems. Using Monte Carlo studies this paper compares the performance of standard regression, robust regression, OR,...

Admissible Discriminant Analysis

Abstract Since it is usually impossible to determine a “best” discriminant analysis, admissible discriminant analyses are suggested. Let A denote some property which should be satisfied by any reasonable procedure either in general or when used in a special application. Any procedure which satisifies A is called A-admissible. Seven admissibility conditions are defined and ten discriminant algorithms are compared with them.

Robust discriminant analysis: Training data breakdown point

This paper discusses the robustness of discriminant analysis against contamination in the training data, the test data are assumed uncontaminated. The concept of training data breakdown point for discriminant analysis is introduced. It is quite different from the usual breakdown point in robust statistics. In the robust location parameter estimation problem, outliers are the main concern, but in discriminant analysis, not only are outliers a concern, but also inliers.