John M. Chambers

Graphical methods for data analysis

Introduction. Portraying the distribution of a set of data. Comparing data distributions. Studying two-dimensional data. Studying multi-dimensional data. Plotting multivariate data. Assessing distributional assumptions data. Developing and assessing regression models. General principles and techniques. References. Appendix: tables of data sets. Index.

Statistical Models in S

The interactive data analysis and graphics language S (Becker, Chambers and Wilks, 1988) has become a popular environment for both data analysts and research statisticians. A common complaint, however, has concerned the lack of statistical modeling tools, such as those provided by GLIM© or GENSTAT©.

The new S language: a programming environment for data analysis and graphics

This book provides documentation for a new version of the S system released in 1988. The New S Language enhances the features that have made S popular: interactive computing, flexible graphics, data management and a large collection of functions. The New S language features make possible new applications and higher-level programming, including a single unified language, user-defined functions as first-class objects, symbolic computations, more accurate numerical calculations and a new approach to graphics. S now provides direct interfaces to the powerful tool of the UNIX operating system and to algorithms implemented in Fortran and C.

Software for data analysis

* The only advanced programming book on R * Begins with simple interactive use and progresses by gradual stages * Written by the award winning author of the S language from which R evolved John Chambers has been the principal designer of the S language since its beginning, and in 1999 received the ACM System Software award for S, the only statistical software to receive this award. He is author or coauthor of the landmark books on S. Now he turns to R, the enormously successful open-source system based on the S language. Rs international support and the thousands of packages and other contributions have made it the standard for statistical computing in research and teaching. This book guides the reader through programming with R, beginning with simple interactive use and progressing by gradual stages, starting with simple functions. More advanced programming techniques can be added as needed, allowing users to grow into software contributors, benefiting their careers and the community. R packages provide a powerful mechanism for contributions to be organized and communicated. The techniques covered include such modern programming enhancements as classes and methods, namespaces, and interfaces to spreadsheets or data bases, as well as computations for data visualization, numerical methods, and the use of text data.

Greater or lesser statistics: a choice for future research

This paper contrasts two views of statistics, greater and lesser statistics. Greater statistics can be defined simply, if loosely, as everything related to learning from data, from the first planning or collection to the last presentation or report. Lesser statistics is the body of specifically statistical methodology that has evolved within the profession roughly, statistics as defined by texts, journals, and doctoral dissertations. Greater statistics tend to be inclusive, eclectic with respect to methodology, closely associated with other disciplines, and practiced by many outside of academia and often outside professional statistics. Lesser statistics tends to be exclusive, oriented to mathematical techniques, less frequently collaborative with other disciplines, and primarily practiced by members of university departments of statistics. Future directions for statistics research will reflect the tendency of statisticians to define their work consistently with one or the other of these views. A general drift towards lesser statistics has been underway for several generations, reflecting a natural desire to give statistics its own theoretical basis, but limiting both the influence of statistics and the benefits the field has provided to society, particularly in recent years. Statistical computing had the potential to widen our view. To be valuable, statistical software must be integrated into the whole process of learning from data. Observation of current activity suggests, however, that most research in statistical computing remains work in service of lesser statistics, such as support for classical probability-based theory. This may, in its context, be valuable and should not be denigrated. Certainly, it is relatively safe and likely to be appreciated by ones professional colleagues. The need to learn from difficult but important sources of data provides a strong impetus to research in greater statistics outside of traditional topics. The impetus is strong enough that much of the research advances needed will be carried out, by someone. If statisticians remain aloof, others will act. Statistics will lose; in addition, I believe science and society will lose also, because the 0960-3174 , 9 1993 Chapman & Hall statisticians mental attitude at its best provides qualities likely to be missing otherwise.

Auditing of Data Analyses

We describe concepts and software for the auditing of data analyses. Auditing begins with the record of a data analysis session. The record tells what statements were executed and what objects were accessed or changed, and can be processed to recreate chosen statements in the analysis for purposes of verification. It can also be the starting point for asking a variety of questions about the analysis, through an interactive, exploratory interface, as a data analysts assistant.We have constructed such an auditing facility and examined some actual data analyses with it. The record of the data analysis is converted into a special data structure; the data structure in turn is used to examine and display the interconnections among statements in the record. Our facility demonstrates that the verification process is possible and computationally reasonable, even for quite large analyses. At the same time, interactive exploration of the audited analyses presents some interesting and extremely challenging problems.

Computing with Data: Concepts and Challenges

Abstract This article examines work in “computing with data”—in computing support for scientific and other activities to which statisticians can contribute. Relevant computing techniques, besides traditional statistical computing, include data management, visualization, interactive languages, and user-interface design. The article emphasizes the concepts underlying computing with data, with emphasis on how those concepts can help in practical work. We look at past, present, and future: Some concepts as they arose in the past and as they have proved valuable in current software; applications in the present, with one example in particular, to illustrate the challenges these present; and new directions for future research, including one exciting joint project.

Monitoring Networked Applications With Incremental Quantile Estimation

Networked applications have software components that reside on different computers. Email, for example, has database, processing, and user interface components that can be distributed across a network and shared by users in different locations or work groups. End-to-end performance and reliability metrics describe the software quality experienced by these groups of users, taking into account all the software components in the pipeline. Each user produces only some of the data needed to understand the quality of the application for the group, so group performance metrics are obtained by combining summary statistics that each end computer periodically (and automatically) sends to a central server. The group quality metrics usually focus on medians and tail quantiles rather than on averages. Distributed quantile estimation is challenging, though, especially when passing large amounts of data around the network solely to compute quality metrics is undesirable. This paper describes an Incremental Quantile (IQ) estimation method that is designed for performance monitoring at arbitrary levels of network aggregation and time resolution when only a limited amount of data can be transferred. Applications to both real and simulated data are provided.

Design of the S system for data analysis

S is a language and system for interactive data analysis and graphics. It emphasizes interactive analysis and graphics, ease of use, flexibility, and extensibility. While sharing many characteristics with other statistical systems, S differs significantly in its design goals, its implementation, and the way it is used. This paper presents some of the design concepts and implementation techniques in S and relates these general ideas in computing to the specific design goals for S and to other statistical systems.

Some Thoughts on Expert Software

Current successes in making computers available for data analysis will intensify the challenge to make them more useful. Cheaper, smaller and more reliable hardware will make computers available to many new users with data to analyse: few of them will have much statistical training or routine access to professional statisticians. We have a moral obligation to guide such users to good, helpful and defensible analysis. The best of current statistical systems allow users to produce many statistical summaries, graphical displays and other aids to data analysis. We have done relatively well in providing the mechanics of the analysis. Can we now go on to the strategy?