Gareth M. James

An Introduction to Statistical Learning

Statistics An Intduction to Stistical Lerning with Applications in R An Introduction to Statistical Learning provides an accessible overview of the fi eld of statistical learning, an essential toolset for making sense of the vast and complex data sets that have emerged in fi elds ranging from biology to fi nance to marketing to astrophysics in the past twenty years. Th is book presents some of the most important modeling and prediction techniques, along with relevant applications. Topics include linear regression, classifi cation, resampling methods, shrinkage approaches, tree-based methods, support vector machines, clustering, and more. Color graphics and real-world examples are used to illustrate the methods presented. Since the goal of this textbook is to facilitate the use of these statistical learning techniques by practitioners in science, industry, and other fi elds, each chapter contains a tutorial on implementing the analyses and methods presented in R, an extremely popular open source statistical soft ware platform. Two of the authors co-wrote Th e Elements of Statistical Learning (Hastie, Tibshirani and Friedman, 2nd edition 2009), a popular reference book for statistics and machine learning researchers. An Introduction to Statistical Learning covers many of the same topics, but at a level accessible to a much broader audience. Th is book is targeted at statisticians and non-statisticians alike who wish to use cutting-edge statistical learning techniques to analyze their data. Th e text assumes only a previous course in linear regression and no knowledge of matrix algebra.

Finding the Number of Clusters in a Dataset: An Information-Theoretic Approach

One of the most difficult problems in cluster analysis is identifying the number of groups in a dataset. Most previously suggested approaches to this problem are either somewhat ad hoc or require parametric assumptions and complicated calculations. In this article we develop a simple, yet powerful nonparametric method for choosing the number of clusters based on distortion, a quantity that measures the average distance, per dimension, between each observation and its closest cluster center. Our technique is computationally efficient and straightforward to implement. We demonstrate empirically its effectiveness, not only for choosing the number of clusters, but also for identifying underlying structure, on a wide range of simulated and real world datasets. In addition, we give a rigorous theoretical justification for the method based on information-theoretic ideas. Specifically, results from the subfield of electrical engineering known as rate distortion theory allow us to describe the behavior of the distortion in both the presence and absence of clustering. Finally, we note that these ideas potentially can be extended to a wide range of other statistical model selection problems.

Clustering for Sparsely Sampled Functional Data

We develop a flexible model-based procedure for clustering functional data. The technique can be applied to all types of curve data but is particularly useful when individuals are observed at a sparse set of time points. In addition to producing final cluster assignments, the procedure generates predictions and confidence intervals for missing portions of curves. Our approach also provides many useful tools for evaluating the resulting models. Clustering can be assessed visually via low-dimensional representations of the curves, and the regions of greatest separation between clusters can be determined using a discriminant function. Finally, we extend the model to handle multiple functional and finite-dimensional covariates and show how it can be applied to standard finite-dimensional clustering problems involving missing data.

Generalized linear models with functional predictors

Summary. We present a technique for extending generalized linear models to the situation where some of the predictor variables are observations from a curve or function. The technique is particularly useful when only fragments of each curve have been observed. We demonstrate, on both simulated and real data sets, how this approach can be used to perform linear, logistic and censored regression with functional predictors. In addition, we show how functional principal components can be used to gain insight into the relationship between the response and functional predictors. Finally, we extend the methodology to apply generalized linear models and principal components to standard missing data problems.

Functional linear discriminant analysis for irregularly sampled curves

We introduce a technique for extending the classical method of linear discriminant analysis (LDA) to data sets where the predictor variables are curves or functions. This procedure, which we call functional linear discriminant analysis (FLDA), is particularly useful when only fragments of the curves are observed. All the techniques associated with LDA can be extended for use with FLDA. In particular FLDA can be used to produce classifications on new (test) curves, give an estimate of the discriminant function between classes and provide a one- or two-dimensional pictorial representation of a set of curves. We also extend this procedure to provide generalizations of quadratic and regularized discriminant analysis.

Variance and Bias for General Loss Functions

When using squared error loss, bias and variance and their decomposition of prediction error are well understood and widely used concepts. However, there is no universally accepted definition for other loss functions. Numerous attempts have been made to extend these concepts beyond squared error loss. Most approaches have focused solely on 0-1 loss functions and have produced significantly different definitions. These differences stem from disagreement as to the essential characteristics that variance and bias should display. This paper suggests an explicit list of rules that we feel any “reasonable” set of definitions should satisfy. Using this framework, bias and variance definitions are produced which generalize to any symmetric loss function. We illustrate these statistics on several loss functions with particular emphasis on 0-1 loss. We conclude with a discussion of the various definitions that have been proposed in the past as well as a method for estimating these quantities on real data sets.

Functional Regression: A New Model for Predicting Market Penetration of New Products

The Bass model has been a standard for analyzing and predicting the market penetration of new products. We demonstrate the insights to be gained and predictive performance of functional data analysis (FDA), a new class of nonparametric techniques that has shown impressive results within the statistics community, on the market penetration of 760 categories drawn from 21 products and 70 countries. We propose a new model called Functional Regression and compare its performance to several models, including the Classic Bass model, Estimated Means, Last Observation Projection, a Meta-Bass model, and an Augmented Meta-Bass model for predicting eight aspects of market penetration. Results (a) validate the logic of FDA in integrating information across categories, (b) show that Augmented Functional Regression is superior to the above models, and (c) product-specific effects are more important than country-specific effects when predicting penetration of an evolving new product.

Bayesian sparse hidden components analysis for transcription regulation networks

MOTIVATION In systems like Escherichia Coli, the abundance of sequence information, gene expression array studies and small scale experiments allows one to reconstruct the regulatory network and to quantify the effects of transcription factors on gene expression. However, this goal can only be achieved if all information sources are used in concert. RESULTS Our method integrates literature information, DNA sequences and expression arrays. A set of relevant transcription factors is defined on the basis of literature. Sequence data are used to identify potential target genes and the results are used to define a prior distribution on the topology of the regulatory network. A Bayesian hidden component model for the expression array data allows us to identify which of the potential binding sites are actually used by the regulatory proteins in the studied cell conditions, the strength of their control, and their activation profile in a series of experiments. We apply our methodology to 35 expression studies in E.Coli with convincing results. AVAILABILITY www.genetics.ucla.edu/labs/sabatti/software.html SUPPLEMENTARY INFORMATION The supplementary material are available at Bioinformatics online.

Functional Adaptive Model Estimation

In this article we are interested in modeling the relationship between a scalar, Y, and a functional predictor, X(t). We introduce a highly flexible approach called functional adaptive model estimation (FAME), which extends generalized linear models (GLMs), generalized additive models (GAMs), and projection pursuit regression (PPR) to handle functional predictors. The FAME approach can model any of the standard exponential family of response distributions that are assumed for GLM or GAM while maintaining the flexibility of PPR. For example, standard linear or logistic regression with functional predictors, as well as far more complicated models, can easily be applied using this approach. We use a functional principal components decomposition of the predictor functions to aid visualization of the relationship between X(t) and Y. We also show how the FAME procedure can be extended to deal with multiple functional and standard finite-dimensional predictors, possibly with missing data. We illustrate the FAME approach on simulated data, as well as on the prediction of arthritis based on bone shape. We end with a discussion of the relationships between standard regression approaches, their extensions to functional data, and FAME.

Hidden markov models for longitudinal comparisons

Medical researchers interested in temporal, multivariate measurements of complex diseases have recently begun developing health state models, which divide the space of patient characteristics into medically distinct clusters. The current state of the art in health services research uses k-means clustering to form the health states and a first-order Markov chain to describe transitions between the states. This fitting procedure ignores information from temporally adjacent observations and prevents uncertainty from parameter estimation and cluster assignments from being incorporated into the analysis. A natural way to address these issues is to combine clustering and longitudinal analyses using a hidden Markov model. We fit hidden Markov models to longitudinal data using Bayesian methods that account for all of the uncertainty in the parameters, conditional only on the underlying correctness of the model. Potential lack of time homogeneity in the Markov chain is accounted for by embedding transition probabilities into a hierarchical model that provides Bayesian shrinkage across time. We illustrate this approach by developing a hidden Markov health state model for comparing the effectiveness of clozapine and haloperidol, two antipsychotic medications for schizophrenia. We find that clozapine outperforms haloperidol and identify the types of patients in whom clozapines advantage is greatest and weakest. Finally, we discuss the advantages and disadvantages of hidden Markov models in comparison with the current methodology.