Iain M. Johnstone

Least angle regression

DISCUSSION OF “LEAST ANGLE REGRESSION” BY EFRONET AL.By Jean-Michel Loubes and Pascal MassartUniversit´e Paris-SudThe issue of model selection has drawn the attention of both applied andtheoretical statisticians for a long time. Indeed, there has been an enor-mous range of contribution in model selection proposals, including work byAkaike (1973), Mallows (1973), Foster and George (1994), Birg´e and Mas-sart (2001a) and Abramovich, Benjamini, Donoho and Johnstone (2000).Over the last decade, modern computer-driven methods have been devel-oped such as All Subsets, Forward Selection, Forward Stagewise or Lasso.Such methods are useful in the setting of the standard linear model, wherewe observe noisy data and wish to predict the response variable using onlya few covariates, since they provide automatically linear models that fit thedata. The procedure described in this paper is, on the one hand, numeri-cally very efficient and, on the other hand, very general, since, with slightmodifications, it enables us to recover the estimates given by the Lasso andStagewise.1. Estimation procedure. The “LARS” method is based on a recursiveprocedure selecting, at each step, the covariates having largest absolute cor-relation with the response y. In the case of an orthogonal design, the esti-mates can then be viewed as an lDISCUSSION OF “LEAST ANGLE REGRESSION” BY EFRONET AL.By Berwin A. TurlachUniversity of Western AustraliaI would like to begin by congratulating the authors (referred to belowas EHJT) for their interesting paper in which they propose a new variableselection method (LARS) for building linear models and show how their newmethod relates to other methods that have been proposed recently. I foundthe paper to be very stimulating and found the additional insight that itprovides about the Lasso technique to be of particular interest.My comments center around the question of how we can select linearmodels that conform with the marginality principle [Nelder (1977, 1994)and McCullagh and Nelder (1989)]; that is, the response surface is invariantunder scaling and translation of the explanatory variables in the model.Recently one of my interests was to explore whether the Lasso techniqueor the nonnegative garrote [Breiman (1995)] could be modified such that itincorporates the marginality principle. However, it does not seem to be atrivial matter to change the criteria that these techniques minimize in such away that the marginality principle is incorporated in a satisfactory manner.On the other hand, it seems to be straightforward to modify the LARStechnique to incorporate this principle. In their paper, EHJT address thisissue somewhat in passing when they suggest toward the end of Section 3that one first fit main effects only and interactions in a second step to controlthe order in which variables are allowed to enter the model. However, sucha two-step procedure may have a somewhat less than optimal behavior asthe following, admittedly artificial, example shows.Assume we have a vector of explanatory variables X =(XThe purpose of model selection algorithms such as All Subsets, Forward Selection and Backward Elimination is to choose a linear model on the basis of the same set of data to which the model will be applied. Typically we have available a large collection of possible covariates from which we hope to select a parsimonious set for the efficient prediction of a response variable. Least Angle Regression (LARS), a new model selection algorithm, is a useful and less greedy version of traditional forward selection methods. Three main properties are derived: (1) A simple modification of the LARS algorithm implements the Lasso, an attractive version of ordinary least squares that constrains the sum of the absolute regression coefficients; the LARS modification calculates all possible Lasso estimates for a given problem, using an order of magnitude less computer time than previous methods. (2) A different LARS modification efficiently implements Forward Stagewise linear regression, another promising new model selection method; this connection explains the similar numerical results previously observed for the Lasso and Stagewise, and helps us understand the properties of both methods, which are seen as constrained versions of the simpler LARS algorithm. (3) A simple approximation for the degrees of freedom of a LARS estimate is available, from which we derive a Cp estimate of prediction error; this allows a principled choice among the range of possible LARS estimates. LARS and its variants are computationally efficient: the paper describes a publicly available algorithm that requires only the same order of magnitude of computational effort as ordinary least squares applied to the full set of covariates.

Adapting to Unknown Smoothness via Wavelet Shrinkage

Abstract We attempt to recover a function of unknown smoothness from noisy sampled data. We introduce a procedure, SureShrink, that suppresses noise by thresholding the empirical wavelet coefficients. The thresholding is adaptive: A threshold level is assigned to each dyadic resolution level by the principle of minimizing the Stein unbiased estimate of risk (Sure) for threshold estimates. The computational effort of the overall procedure is order N · log(N) as a function of the sample size N. SureShrink is smoothness adaptive: If the unknown function contains jumps, then the reconstruction (essentially) does also; if the unknown function has a smooth piece, then the reconstruction is (essentially) as smooth as the mother wavelet will allow. The procedure is in a sense optimally smoothness adaptive: It is near minimax simultaneously over a whole interval of the Besov scale; the size of this interval depends on the choice of mother wavelet. We know from a previous paper by the authors that traditional smoot...

Prostate Specific Antigen in the Diagnosis and Treatment of Adenocarcinoma of the Prostate. II. Radical Prostatectomy Treated Patients

Serum prostate specific antigen was determined (Yang polyclonal radioimmunoassay) in 102 men before hospitalization for radical prostatectomy. Prostate specimens were subjected to detailed histological and morphometric analysis. Levels of prostate specific antigen were significantly different between patients with and without a Gleason score of 7 or greater (p less than 0.001), capsular penetration greater than 1 cm. in linear extent (p less than 0.001), seminal vesicle invasion (p less than 0.001) and pelvic lymph node metastasis (p less than 0.005). Prostate specific antigen was strongly correlated with volume of prostate cancer (r equals 0.70). Bivariate and multivariate analyses indicate that cancer volume is the primary determinant of serum prostate specific antigen levels. Prostate specific antigen was elevated 3.5 ng. per ml. for every cc of cancer, a level at least 10 times that observed for benign prostatic hyperplasia. Prostate specific antigen is useful as a preoperative marker because no patient with lymph node metastasis had serum levels of less than 10 ng. per ml. (4 times the upper limit of normal range). Of the patients with greater than 50 ng. per ml. two-thirds had microscopic lymph node metastasis and 90 per cent had seminal vesicle invasion. Serum prostatic acid phosphatase levels showed a significantly weaker correlation with cancer volume (r equals 0.51) and every other pathological parameter. Of the patients 73 per cent had serum prostatic acid phosphatase levels in the normal range (0 to 2.1 ng. per ml.), including 7 per cent who had pelvic lymph node metastasis. Postoperative prostate specific antigen values were available in 97 of 102 patients, with a mean and maximum followup of 12 and 38 months. No patient with pelvic lymph node metastasis achieved an undetectable prostate specific antigen level without adjunctive therapy (hormonal or radiation). No difference in preoperative or postoperative prostate specific antigen levels, cancer volume, seminal vesicle invasion or incidence of pelvic lymph node metastasis was seen between patients with no capsular penetration and those with minimal capsular penetration (1 cm. or less total linear extent of full thickness penetration), providing the first quantitative evidence that small amounts of capsular penetration may not be of biological or prognostic significance.

On Consistency and Sparsity for Principal Components Analysis in High Dimensions.

Principal components analysis (PCA) is a classic method for the reduction of dimensionality of data in the form of n observations (or cases) of a vector with p variables. Contemporary datasets often have p comparable with or even much larger than n. Our main assertions, in such settings, are (a) that some initial reduction in dimensionality is desirable before applying any PCA-type search for principal modes, and (b) the initial reduction in dimensionality is best achieved by working in a basis in which the signals have a sparse representation. We describe a simple asymptotic model in which the estimate of the leading principal component vector via standard PCA is consistent if and only if p(n)/n → 0. We provide a simple algorithm for selecting a subset of coordinates with largest sample variances, and show that if PCA is done on the selected subset, then consistency is recovered, even if p(n) ≫ n.

Adapting to unknown sparsity by controlling the false discovery rate

We attempt to recover an n-dimensional vector observed in white noise, where n is large and the vector is known to be sparse, but the degree of sparsity is unknown. We consider three different ways of defining sparsity of a vector: using the fraction of nonzero terms; imposing power-law decay bounds on the ordered entries; and controlling the lp norm for p small. We obtain a procedure which is asymptotically minimax for l r loss, simultaneously throughout a range of such sparsity classes. The optimal procedure is a data-adaptive thresholding scheme, driven by control of the False Discovery Rate (FDR). FDR control is a relatively recent innovation in simultaneous testing, ensuring that at most a certain fraction of the rejected null hypotheses will correspond to false rejections. In our treatment, the FDR control parameter qn also plays a determining role in asymptotic minimaxity. If q = lim qn ∈ [0,1/2] and also qn > γ/log(n) we get sharp asymptotic minimaxity, simultaneously, over a wide range of sparse parameter spaces and loss functions. On the other hand, q = lim qn ∈ (1/2,1], forces the risk to exceed the minimax risk by a factor growing with q. To our knowledge, this relation between ideas in simultaneous inference and asymptotic decision theory is new. Our work provides a new perspective on a class of model selection rules which has been introduced recently by several authors. These new rules impose complexity penalization of the form 2 � log( potential model size / actual model size ). We exhibit a close connection with FDR-controlling procedures under stringent control of the false discovery rate.

Medical Care Costs and Quality of Life after Randomization to Coronary Angioplasty or Coronary Bypass Surgery

BACKGROUND Randomized trials comparing coronary angioplasty with bypass surgery in patients with multivessel coronary disease have shown no significant differences in overall rates of death and myocardial infarction. We compared quality of life, employment, and medical care costs during five years of follow-up among patients treated with angioplasty or bypass surgery. METHODS A total of 934 of the 1829 patients enrolled in the randomized Bypass Angioplasty Revascularization Investigation participated in this study. Detailed data on quality of life were collected annually, and economic data were collected quarterly. RESULTS During the first three years of follow-up, functional-status scores on the Duke Activity Status Index, which measures the ability to perform common activities of daily living, improved more in patients assigned to surgery than in those assigned to angioplasty (P<0.05). Other measures of quality of life improved equally in both groups throughout the follow-up period. Patients in the angioplasty group returned to work five weeks sooner than did patients in the surgery group (P<0.001). The initial mean cost of angioplasty was 65 percent that of surgery (

The anatomy of the posterior communicating artery as a risk factor for ischemic cerebral infarction.

21,113 vs.

A Preliminary Study of Diltiazem in the Prevention of Coronary Artery Disease in Heart-Transplant Recipients

32,347, P<0.001), but after five years the total medical cost of angioplasty was 95 percent that of surgery (

Empirical Bayes selection of wavelet thresholds

56,225 vs.

Needles and straw in haystacks: Empirical bayes estimates of possibly sparse sequences

58,889), a difference of