Sanford Weisberg

Least angle regression

DISCUSSION OF “LEAST ANGLE REGRESSION” BY EFRONET AL.By Jean-Michel Loubes and Pascal MassartUniversit´e Paris-SudThe issue of model selection has drawn the attention of both applied andtheoretical statisticians for a long time. Indeed, there has been an enor-mous range of contribution in model selection proposals, including work byAkaike (1973), Mallows (1973), Foster and George (1994), Birg´e and Mas-sart (2001a) and Abramovich, Benjamini, Donoho and Johnstone (2000).Over the last decade, modern computer-driven methods have been devel-oped such as All Subsets, Forward Selection, Forward Stagewise or Lasso.Such methods are useful in the setting of the standard linear model, wherewe observe noisy data and wish to predict the response variable using onlya few covariates, since they provide automatically linear models that fit thedata. The procedure described in this paper is, on the one hand, numeri-cally very efficient and, on the other hand, very general, since, with slightmodifications, it enables us to recover the estimates given by the Lasso andStagewise.1. Estimation procedure. The “LARS” method is based on a recursiveprocedure selecting, at each step, the covariates having largest absolute cor-relation with the response y. In the case of an orthogonal design, the esti-mates can then be viewed as an lDISCUSSION OF “LEAST ANGLE REGRESSION” BY EFRONET AL.By Berwin A. TurlachUniversity of Western AustraliaI would like to begin by congratulating the authors (referred to belowas EHJT) for their interesting paper in which they propose a new variableselection method (LARS) for building linear models and show how their newmethod relates to other methods that have been proposed recently. I foundthe paper to be very stimulating and found the additional insight that itprovides about the Lasso technique to be of particular interest.My comments center around the question of how we can select linearmodels that conform with the marginality principle [Nelder (1977, 1994)and McCullagh and Nelder (1989)]; that is, the response surface is invariantunder scaling and translation of the explanatory variables in the model.Recently one of my interests was to explore whether the Lasso techniqueor the nonnegative garrote [Breiman (1995)] could be modified such that itincorporates the marginality principle. However, it does not seem to be atrivial matter to change the criteria that these techniques minimize in such away that the marginality principle is incorporated in a satisfactory manner.On the other hand, it seems to be straightforward to modify the LARStechnique to incorporate this principle. In their paper, EHJT address thisissue somewhat in passing when they suggest toward the end of Section 3that one first fit main effects only and interactions in a second step to controlthe order in which variables are allowed to enter the model. However, sucha two-step procedure may have a somewhat less than optimal behavior asthe following, admittedly artificial, example shows.Assume we have a vector of explanatory variables X =(XThe purpose of model selection algorithms such as All Subsets, Forward Selection and Backward Elimination is to choose a linear model on the basis of the same set of data to which the model will be applied. Typically we have available a large collection of possible covariates from which we hope to select a parsimonious set for the efficient prediction of a response variable. Least Angle Regression (LARS), a new model selection algorithm, is a useful and less greedy version of traditional forward selection methods. Three main properties are derived: (1) A simple modification of the LARS algorithm implements the Lasso, an attractive version of ordinary least squares that constrains the sum of the absolute regression coefficients; the LARS modification calculates all possible Lasso estimates for a given problem, using an order of magnitude less computer time than previous methods. (2) A different LARS modification efficiently implements Forward Stagewise linear regression, another promising new model selection method; this connection explains the similar numerical results previously observed for the Lasso and Stagewise, and helps us understand the properties of both methods, which are seen as constrained versions of the simpler LARS algorithm. (3) A simple approximation for the degrees of freedom of a LARS estimate is available, from which we derive a Cp estimate of prediction error; this allows a principled choice among the range of possible LARS estimates. LARS and its variants are computationally efficient: the paper describes a publicly available algorithm that requires only the same order of magnitude of computational effort as ordinary least squares applied to the full set of covariates.

An Introduction to Regression Graphics.

Getting Started. Simple Regression Plots. Two--Dimensional Plots. Scatterplot Matrices. Three--Dimensional Plots. Visualizing Linear Regression with Two Predictors. Visualizing Regression Without Linearity. Finding Dimension. Predictor Transformations. Response Transformations. Checking Models. Assessing Predictors. Influence and Outliers. Confidence Regions. Appendices. References. Index.

Characterizations of an Empirical Influence Function for Detecting Influential Cases in Regression

Traditionally, most of the effort in fitting full rank linear regression models has centered on the study of the presence, strength and form of relationships between the measured variables. As is now well known, least squares regression computations can be strongly influenced by a few cases, and a fitted model may more accurately reflect unusual features of those cases than the overall relationships between the variables. It is of interest, therefore, for an analyst to be able to find influential cases and, based on them, make decisions concerning their usefulness in a problem at hand. Based on an empirical influence function, we discuss methodologies for assessing the influence of individual or groups of cases on a regression problem. We conclude with an example using data from the Florida Area Cumulus Experiments (FACE) on cloud seeding.

Directions in robust statistics and diagnostics

In robust statistics, new procedures which have been derived from theoretical considerations are beginning to find their way into applications. Diagnostics have been designed to supplement standard methodology with both graphical and non-graphical procedures. Many diagnostics, particularly graphical ones, have been generally included in common computing packages. A theoretical basis for some diagnostics methods, however, has been a recent development and is the topic of a large part of this volume.

An Approximate Analysis of Variance Test for Non-Normality Suitable for Machine Calculation

Replacement by a simple approximation of expected values of order statistics in the Shapiro-Francis (1972) W′ statistics, yield a statistics that is more srlitable for machine computation. We show that is equivalent to W′ and discuss its percentage points.

Mixed effects models for fish growth

Fish growth in a particular year has both intrinsic and environmental components. Intrinsic growth can depend on both the age and size of the fish and on particular characteristics of the individual fish. The environmental component is the influence of external conditions such as food supply on the growth increment. In this article, we present mixed-effects models as an alternative to fixed-effects linear models for incremental fish growth used previously in the literature and show how these models overcome many of the shortcomings of the fixed-effects approach. In addition, widely available software allows for fitting these models and for elaboration of them to learn about the effects of additional factors such as temperature, species interactions, management practices, the introduction of an invasive species, or other known environmental variables. Finally, we provide a connection with the more usual modeling of size-attained data through the use of growth functions such as the von Bertalanffy.

Graphics for Assessing the Adequacy of Regression Models

Abstract Graphical paradigms for assessing the adequacy of nearly any regression model are discussed. The fundamental idea is to examine the fit of a model using a sequence of marginal model plots. On each plot, nonparametric estimates of fit derived from the model are compared to nonparametric fits based on the observed data. Several examples are given.

Regression diagnostics with dynamic graphics

We develop uses for two recently proposed types of dynamic displays—rotation and animation—in regression diagnostics. Some of the general issues that we address by using these displays include checking for interactions and normality, assessing the need to transform the data, and adding predictors to a model. Animation is used in probability plotting and as an aid to understanding the effects of adding variables to a model. Rotation is used for threedimensional added-variable and residual plots, each of which may be effective for diagnosing the presence of an interaction.

Natural Variation in RPS2-Mediated Resistance among Arabidopsis Accessions: Correlation between Gene Expression Profiles and Phenotypic Responses

Natural variation in gene expression (expression traits or e-traits) is increasingly used for the discovery of genes controlling traits. An important question is whether a particular e-trait is correlated with a phenotypic trait. Here, we examined the correlations between phenotypic traits and e-traits among 10 Arabidopsis thaliana accessions. We studied defense against Pseudomonas syringae pv tomato DC3000 (Pst), with a focus on resistance gene–mediated resistance triggered by the type III effector protein AvrRpt2. As phenotypic traits, we measured growth of the bacteria and extent of the hypersensitive response (HR) as measured by electrolyte leakage. Genetic variation among accessions affected growth of Pst both with (Pst avrRpt2) and without (Pst) the AvrRpt2 effector. Variation in HR was not correlated with variation in bacterial growth. We also collected gene expression profiles 6 h after mock and Pst avrRpt2 inoculation using a custom microarray. Clusters of genes whose expression levels are correlated with bacterial growth or electrolyte leakage were identified. Thus, we demonstrated that variation in gene expression profiles of Arabidopsis accessions collected at one time point under one experimental condition has the power to explain variation in phenotypic responses to pathogen attack.

Comparison of Model Misspecification Diagnostics Using Residuals from Least Mean of Squares and Least Median of Squares Fits

Abstract This article explores model misspecification diagnostics based on least squares and least median of squares fits. It shows that in some circumstances, least median of squares methods (or any other estimator with the exact fit property) fail to reveal an incorrectly specified mean function, but least squares methods succeed.