Gerda Claeskens

Model Selection and Model Averaging

Given a data set, you can fit thousands of models at the push of a button, but how do you choose the best? With so many candidate models, overfitting is a real danger. Is the monkey who typed Hamlet actually a good writer? Choosing a model is central to all statistical work with data. We have seen rapid advances in model fitting and in the theoretical understanding of model selection, yet this book is the first to synthesize research and practice from this active field. Model choice criteria are explained, discussed and compared, including the AIC, BIC, DIC and FIC. The uncertainties involved with model selection are tackled, with discussions of frequentist and Bayesian methods; model averaging schemes are presented. Real-data examples are complemented by derivations providing deeper insight into the methodology, and instructive exercises build familiarity with the methods. The companion website features Data sets and R code.

Frequentist Model Average Estimators

The traditional use of model selection methods in practice is to proceed as if the final selected model had been chosen in advance, without acknowledging the additional uncertainty introduced by model selection. This often means underreporting of variability and too optimistic confidence intervals. We build a general large-sample likelihood apparatus in which limiting distributions and risk properties of estimators post-selection as well as of model average estimators are precisely described, also explicitly taking modeling bias into account. This allows a drastic reduction in complexity, as competing model averaging schemes may be developed, discussed, and compared inside a statistical prototype experiment where only a few crucial quantities matter. In particular, we offer a frequentist view on Bayesian model averaging methods and give a link to generalized ridge estimators. Our work also leads to new model selection criteria. The methods are illustrated with real data applications.

The Focused Information Criterion

A variety of model selection criteria have been developed, of general and specific types. Most of these aim at selecting a single model with good overall properties, for example, formulated via average prediction quality or shortest estimated overall distance to the true model. The Akaike, the Bayesian, and the deviance information criteria, along with many suitable variations, are examples of such methods. These methods are not concerned, however, with the actual use of the selected model, which varies with context and application. The present article takes the view that the model selector should instead focus on the parameter singled out for interest; in particular, a model that gives good precision for one estimand may be worse when used for inference for another estimand. We develop a method that, for a given focus parameter, estimates the precision of any submodel-based estimator. The framework is that of large-sample likelihood inference. Using an unbiased estimate of limiting risk, we propose a focused information criterion for model selection. We investigate and discuss properties of the method, establish some connections to Akaikes information criterion, and illustrate its use in a variety of situations.

Non‐parametric small area estimation using penalized spline regression

We propose a new small area estimation approach that combines small area random effects with a smooth, nonparametrically specified trend. By using penalized splines as the representation for the nonparametric trend, it is possible to express the small area estimation problem as a mixed effect model regression. This model is readily fitted using existing model fitting approaches such as restricted maximum likelihood. We develop a corresponding bootstrap approach for model inference and estimation of the small area prediction mean squared error. The applicability of the method is demonstrated on a survey of lakes in the Northeastern US.

On model selection and model misspecification in causal inference

Standard variable selection procedures, primarily developed for the construction of outcome prediction models, are routinely applied when assessing exposure effects in observational studies. We argue that this tradition is sub-optimal and prone to yield bias in exposure effect estimators as well as their corresponding uncertainty estimators. We weigh the pros and cons of confounder-selection procedures and propose a procedure directly targeting the quality of the exposure effect estimator. We further demonstrate that certain strategies for inferring causal effects have the desirable features (a) of producing (approximately) valid confidence intervals, even when the confounder-selection process is ignored, and (b) of being robust against certain forms of misspecification of the association of confounders with both exposure and outcome.

Testing the Fit of a Parametric Function

Abstract General methods for testing the fit of a parametric function are proposed. The idea underlying each method is to “accept” the prescribed parametric model if and only if it is chosen by a model selection criterion. Several different selection criteria are considered, including one based on a modified version of the Akaike information criterion and others based on various score statistics. The tests have a connection with nonparametric smoothing because they use orthogonal series estimators to detect departures from a parametric model. An important aspect of the tests is that they can be applied in a wide variety of settings, including generalized linear models, spectral analysis, the goodness-of-fit problem, and longitudinal data analysis. Implementation using standard statistical software is straightforward. Asymptotic distribution theory for several test statistics is described, and the tests are shown to be consistent against essentially any alternative hypothesis. Simulations and a data exampl...

Simultaneous Confidence Bands for Penalized Spline Estimators

In this article we construct simultaneous confidence bands for a smooth curve using penalized spline estimators. We consider three types of estimation methods: (a) as a standard (fixed effect) nonparametric model, (b) using the mixed-model framework with the spline coefficients as random effects, and (c) a full Bayesian approach. The volume-of-tube formula is applied for the first two methods and compared with Bayesian simultaneous confidence bands from a frequentist perspective. We show that the mixed-model formulation of penalized splines can help obtain, at least approximately, confidence bands with either Bayesian or frequentist properties. Simulations and data analysis support the proposed methods. The R package ConfBands accompanies the article.

Model-assisted estimation for complex surveys using penalised splines

Estimation of finite population totals in the presence of auxiliary information is considered. A class of estimators based on penalised spline regression is proposed. These estimators are weighted linear combinations of sample observations, with weights calibrated to known control totals. They allow straightforward extensions to multiple auxiliary variables and to complex designs. Under standard design conditions, the estimators are design consistent and asymptotically normal, and they admit consistent variance estimation using familiar design-based methods. Data-driven penalty selection is considered in the context of unequal probability sampling designs. Simulation experiments show that the estimators are more efficient than parametric regression estimators when the parametric model is incorrectly specified, while being approximately as efficient when the parametric specification is correct. An example using Forest Health Monitoring survey data from the U.S. Forest Service demonstrates the applicability of the methodology in the context of a two-phase survey with multiple auxiliary variables. Copyright 2005, Oxford University Press.

Multivariate functional halfspace depth

This article defines and studies a depth for multivariate functional data. By the multivariate nature and by including a weight function, it acknowledges important characteristics of functional data, namely differences in the amount of local amplitude, shape, and phase variation. We study both population and finite sample versions. The multivariate sample of curves may include warping functions, derivatives, and integrals of the original curves for a better overall representation of the functional data via the depth. We present a simulation study and data examples that confirm the good performance of this depth function. Supplementary materials for this article are available online.

Some theory for penalized spline generalized additive models

Generalized additive models have become one of the most widely used modern statistical tools. Traditionally, they are fit through scatterplot smoothing and the backfitting algorithm. However, a more recent development is the direct fitting through the use of low-rank smoothers (Hastie, J. Roy. Statist. Soc. Ser. B 58 (1996) 379). A particularly attractive example of this is through use of penalized splines (Marx and Eilers, Comput. Statist. Data Anal. 28 (1998) 193). Such an approach has a number of advantages, particularly regarding computation. In this paper, we exploit the explicitness of penalized spline additive models to derive some useful and revealing theoretical approximations.