Roy E. Welsch

Robust regression using iteratively reweighted least-squares

The rapid development of the theory of robust estimation (Huber, 1973) has created a need for computational procedures to produce robust estimates. We will review a number of different computational approaches for robust linear regression but focus on one—iteratively reweighted least-squares (IRLS). The weight functions that we discuss are a part of a semi-portable subroutine library called ROSEPACK (RObust Statistical Estimation PACKage) that has been developed by the authors and Virginia Klema at the Computer Research Center of the National Bureau of Economic Research, Inc. in Cambridge, Mass. with the support of the National Science Foundation. This library (Klema, 1976) makes it relatively simple to implement an IRLS regression package.

The Hat Matrix in Regression and ANOVA

Abstract In least-squares fitting it is important to understand the influence which a data y value will have on each fitted y value. A projection matrix known as the hat matrix contains this information and, together with the Studentized residuals, provides a means of identifying exceptional data points. This approach also simplifies the calculations involved in removing a data point, and it requires only simple modifications in the preferred numerical least-squares algorithms.

Algorithm 573: NL2SOL—An Adaptive Nonlinear Least-Squares Algorithm [E4]

Reference [ 1] explains the algorithm realized by NL2SOL in detail. The algorithm amounts to a variation on Newtons method in which part of the Hessian matrix is computed exactly and part is approximated by a secant (quasi-Newton) updating method. Once the iterates come sufficiently close to a local solution, they usually converge quite rapidly. To promote convergence from poor starting guesses, NL2SOL uses a model/trust-region technique along with an adaptive

Efficient Bounded-Influence Regression Estimation

Abstract The least squares estimator for β in the classical linear regression model is strongly efficient under certain conditions. However, in the presence of heavy-tailed errors and/or anomalous data, the least squares efficiency can be markedly reduced. In this article we propose an estimator that limits the influence of any small subset of the data and show that it satisfies a first-order condition for strong efficiency subject to the constraint. We then show that the estimator is asymptotically normal. The article concludes with an outline of an algorithm for computing a bounded-influence regression estimator and with an example comparing least squares, robust regression as developed by Huber, and the estimator proposed in this article.

Efficient Computing of Regression Diagnostics

Abstract Multiple regression diagnostic methods have recently been developed to help data analysts identify failures of data to adhere to the assumptions that customarily accompany regression models. However, the mathematical development of regression diagnostics has not generally led to efficient computing formulas. Conflicting terminology and the use of closely related but subtly different statistics has caused confusion. This article attempts to make regression diagnostics more readily available to those who compute regressions with packaged statistics programs. We review regression diagnostic methodology, highlighting ambiguities of terminology and relationships among similar methods. We present new formulas for efficient computing of regression diagnostics. Finally, we offer specific advice on obtaining regression diagnostics from existing statistics programs, with examples drawn from Minitab and SAS.

Regression Sensitivity Analysis and Bounded-Influence Estimation

Publisher Summary Economists and others have been building and using econometric models for many years. A subset of these builders and users has always been concerned about model reliability, sensitivity, and validity. The energy crisis put certain aspects of modeling into the public and political spotlight. Many questions have been raised about the integrity of the modeling process and in 1975, the National Science Foundation sponsored a conference at Vail, Colorado on model formulation, validation, and improvement. This conference caused a number of statisticians to pay more attention to the statistical questions raised in connection with model reliability, sensitivity, and validity. This chapter describes Huber robust estimation procedure and some of the progress that has been made by examining a particular model and set of data. It presents partial-regression plots and some of the standard output from least-squares regression.

INFLUENCE FUNCTIONS AND REGRESSION DIAGNOSTICS

Publisher Summary This chapter focuses on influence functions and regression diagnostics. Influential-data diagnostics are becoming an accepted part of data analysis. The chapter discusses asymptotic influence functions and the identification of influential subsets of data points. Asymptotic influence functions can be used to identify influential observations by finding bounded-influence regression estimates and a weight for each observation. Low weights indicate influential observations. One of the reasons for measuring influence is to see if there is severe imbalance in the influence of the individual observations. Ideally, the influence of each observation would be about the same with some allowance for stochastic variation. The chapter presents the exploratory approach that considers an influence measure as a batch of n numbers and used the techniques of exploratory data analysis including stem-and-leaf plots, box plots, and transformations to symmetry to identify unusual observations. The features most noticed are gaps among groups of observations with influence of approximately the same magnitude. Determining cut-offs for influential subsets is a complex matter. For single observations, computational costs are not significant and choices among diagnostics may be made on the basis of utility and experience. When subsets are involved, the best diagnostics may be too expensive to obtain and compromises are often needed.

Estimation for dirty data and flawed models

Publisher Summary This chapter focuses on resistant estimation procedures and methods for evaluating the impact of particular data elements on regression estimates. Model builders using macroeconomic time series are often plagued by occasional unusual events, leading them to decrease the weights to be attached to these data in the spirit of resistant estimation. Even when there are good data and theory that correspond reasonably well to the process being modeled, there are episodic model failures. The chapter discusses some model failures that can arise in practice. It describes recent developments in methods for the detection of influential data in regression and discusses several issues about inference in the resistant case and the main theoretical foundations of robust and bounded-influence (BIF) estimation. The chapter presents an example of BIF applied to the Harrison–Rubinfeld large cross-section hedonic price index. The chapter also presents some recent results on instrumental-variables bounded-influence estimation, and discusses resistant estimation for time-series models.

Techniques for nonlinear least squares and robust regression

Recently, the authors and others have made considerable progress in developing algorithms for solving certain large-residual nonlinear least-squares problems where Gauss-Newton (GN) methods can be expected to perform poorly. These methods take account of the term in the Hessian ignored by the GN methods and use quasi-Newton procedures to update this term explicitly. This paper reviews these new approaches and discusses how they can be modified to give good performance on nonlinear models with robust loss functions where lack of scale invariance causes several new problems to arise.

Algorithm 717: Subroutines for maximum likelihood and quasi-likelihood estimation of parameters in nonlinear regression models

We present FORTRAN 77 subroutines that solve statistical parameter estimation problems for general nonlinear models, e.g., nonlinear least-squares, maximum likelihood, maximum quasi-likelihood, generalized nonlinear least-squares, and some robust fitting problems. The accompanying test examples include members of the generalized linear model family, extensions using nonlinear predictors (“nonlinear GLIM”), and probabilistic choice models, such as linear-in-parameter multinomial probit models. The basic method, a generalization of the NL2SOL algorithm for nonlinear least-squares, employs a model/trust-region scheme for computing trial steps, exploits special structure by maintaining a secant approximation to the second-order part of the Hessian, and adaptively switches between a Gauss-Newton and an augmented Hessian approximation. Gauss-Newton steps are computed using a corrected seminormal equations approach. The subroutines include variants that handle simple bounds on the parameters, and that compute approximate regression diagnostics.