Cliff Spiegelman

Repeatability and Reproducibility in Proteomic Identifications by Liquid Chromatography−Tandem Mass Spectrometry

The complexity of proteomic instrumentation for LC-MS/MS introduces many possible sources of variability. Data-dependent sampling of peptides constitutes a stochastic element at the heart of discovery proteomics. Although this variation impacts the identification of peptides, proteomic identifications are far from completely random. In this study, we analyzed interlaboratory data sets from the NCI Clinical Proteomic Technology Assessment for Cancer to examine repeatability and reproducibility in peptide and protein identifications. Included data spanned 144 LC-MS/MS experiments on four Thermo LTQ and four Orbitrap instruments. Samples included yeast lysate, the NCI-20 defined dynamic range protein mix, and the Sigma UPS 1 defined equimolar protein mix. Some of our findings reinforced conventional wisdom, such as repeatability and reproducibility being higher for proteins than for peptides. Most lessons from the data, however, were more subtle. Orbitraps proved capable of higher repeatability and reproducibility, but aberrant performance occasionally erased these gains. Even the simplest protein digestions yielded more peptide ions than LC-MS/MS could identify during a single experiment. We observed that peptide lists from pairs of technical replicates overlapped by 35-60%, giving a range for peptide-level repeatability in these experiments. Sample complexity did not appear to affect peptide identification repeatability, even as numbers of identified spectra changed by an order of magnitude. Statistical analysis of protein spectral counts revealed greater stability across technical replicates for Orbitraps, making them superior to LTQ instruments for biomarker candidate discovery. The most repeatable peptides were those corresponding to conventional tryptic cleavage sites, those that produced intense MS signals, and those that resulted from proteins generating many distinct peptides. Reproducibility among different instruments of the same type lagged behind repeatability of technical replicates on a single instrument by several percent. These findings reinforce the importance of evaluating repeatability as a fundamental characteristic of analytical technologies.

Performance Metrics for Liquid Chromatography-Tandem Mass Spectrometry Systems in Proteomics Analyses

A major unmet need in LC-MS/MS-based proteomics analyses is a set of tools for quantitative assessment of system performance and evaluation of technical variability. Here we describe 46 system performance metrics for monitoring chromatographic performance, electrospray source stability, MS1 and MS2 signals, dynamic sampling of ions for MS/MS, and peptide identification. Applied to data sets from replicate LC-MS/MS analyses, these metrics displayed consistent, reasonable responses to controlled perturbations. The metrics typically displayed variations less than 10% and thus can reveal even subtle differences in performance of system components. Analyses of data from interlaboratory studies conducted under a common standard operating procedure identified outlier data and provided clues to specific causes. Moreover, interlaboratory variation reflected by the metrics indicates which system components vary the most between laboratories. Application of these metrics enables rational, quantitative quality assessment for proteomics and other LC-MS/MS analytical applications.

Interlaboratory Study Characterizing a Yeast Performance Standard for Benchmarking LC-MS Platform Performance

Optimal performance of LC-MS/MS platforms is critical to generating high quality proteomics data. Although individual laboratories have developed quality control samples, there is no widely available performance standard of biological complexity (and associated reference data sets) for benchmarking of platform performance for analysis of complex biological proteomes across different laboratories in the community. Individual preparations of the yeast Saccharomyces cerevisiae proteome have been used extensively by laboratories in the proteomics community to characterize LC-MS platform performance. The yeast proteome is uniquely attractive as a performance standard because it is the most extensively characterized complex biological proteome and the only one associated with several large scale studies estimating the abundance of all detectable proteins. In this study, we describe a standard operating protocol for large scale production of the yeast performance standard and offer aliquots to the community through the National Institute of Standards and Technology where the yeast proteome is under development as a certified reference material to meet the long term needs of the community. Using a series of metrics that characterize LC-MS performance, we provide a reference data set demonstrating typical performance of commonly used ion trap instrument platforms in expert laboratories; the results provide a basis for laboratories to benchmark their own performance, to improve upon current methods, and to evaluate new technologies. Additionally, we demonstrate how the yeast reference, spiked with human proteins, can be used to benchmark the power of proteomics platforms for detection of differentially expressed proteins at different levels of concentration in a complex matrix, thereby providing a metric to evaluate and minimize preanalytical and analytical variation in comparative proteomics experiments.

Analysis of Aggregate Shape Characteristics and its Relationship to Hot Mix Asphalt Performance

ABSTRACT This paper presents the development of a methodology for the classification of aggregates based on their shape, angularity, and texture characteristics. This methodology utilizes the Aggregate Imaging System (AIMS) to measure aggregate characteristics, and the clustering statistical method to analyze the measurements. The outcome of this analysis method is the percentage of aggregate particles that belong to groups or clusters that have statistically different characteristics. The “Categorical Counts” method is employed in this study for comparing the characteristics of different aggregate samples. This method detects not only the statistical difference between aggregate samples, but it is also capable of identifying differences in each of the clusters. The quality of AIMS measurements is evaluated through the analysis of repeatability and reproducibility. Finally, statistical analysis was conducted to determine whether aggregate properties are significant enough to influence the measured mechanical properties of a wide range hot mix asphalt (HMA) mixtures or not. Although the results showed that aggregate shape characteristics had strong correlations with measured mechanical properties, the data was not comprehensive enough to develop predictive equations of mechanical properties as functions of aggregate and other mixture properties. Discussion is provided on the laboratory experimental factors that influence the correlations between aggregate characteristics and measured mixture mechanical properties.

A Bayesian Approach to Calibration Intervals and Properly Calibrated Tolerance Intervals

In this article we consider a Bayesian approach to inference in which there is a calibration relationship between measured and true quantities of interest. One situation in which this approach is useful is for unknowns in which calibration intervals are obtained. The other situation is when inference about a population is desired in which tolerance intervals are produced. The Bayesian approach easily handles a general calibration relationship, say nonlinear, with nonnormal errors. The population may also be general, say lognormal, for quantities which are nonnegative. The Bayesian approach is illustrated with three examples and implemented with the freely available WinBUGS software.

A simple method for linearizing nonlinear spectra for calibration

Abstract The development and acceptance of spectral calibration methods has been an important success story for the field of chemometrics. This paper contains a new study of a very old calibration method ( K -matrix calibration, parallel calibration, or generalized inverse prediction) and partial least squares (PLS), the mainstay of modern chemometrics. We show that with some modest amount of modification, the old method of calibration is comparable, in terms of prediction, to PLS for spectroscopy involving nonlinear spectral responses.

Partial least squares and classification and regression trees

Abstract High-dimensional data are found in scientific fields. Difficulties arise when one applies classical classification methods to these high-dimensional data sets, because of multicollinearity. Problems with high-dimensional data sets can be overcome by reducing the dimensions of data sets. The partial least squares (PLS) method is a new method used for dimension reduction. The classification and regression trees method is applied to the reduced data for solving classification problems. A new stopping criterion for the PLS procedure is introduced. Yeh, C.H. and Spiegelman, C.H., 1994. Partial least squares and classification and regression trees. Chemometrics and Intelligent Laboratory Systems , 22: 17–23.

Chapter Fiveteen Identification, Resolution and Apportionment of Contamination Sources

Abstract Current approaches and recent developments in methods and software associated with multivariate factor analysis and related methods in the analysis of environmental data for the identification, resolution and apportionment of contamination sources are discussed and compared. The chapter first focuses on techniques to be applied in the analysis of the various factors contributing to contamination of the environment, among which we list Principal Component Analysis, and alternative methods and tools such as: Unmix, Positive Matrix Factorization (PMF) and the Multilinear Engine (ME), and Multivariate Curve Resolution Alternating Least Squares (MCR-ALS). In cases where uncertainties were not experimentally available, the use of jackknife uncertainty estimations methods for receptor modelling is described and recommended. Time series extension of multivariate receptor modelling has been developed to account for temporal dependence in air pollution data into estimation of source compositions and uncertainty estimations. In the case of air pollution it is also of interest to estimate the average concentration of a given pollutant at the monitoring site after the air masses have travelled over a certain point (source) on the map. The use of non-parametric regression (kernel smoothing) methods proves to be useful. A method is given for source apportionment of local sources of air pollution by non-parametric regression of the concentration of a pollutant on wind speed and direction. Non-parametric methods have shown that nearby sources (such as freeways) are not always important contributors to high pollutant concentrations. Finally, a number of applications of receptor modelling techniques are presented, including source identification by the new CATT tool (Combined Aerosol Trajectory Tools). Ensemble backward trajectory techniques have been employed to identify regional origins of air pollutants subject to synoptic-scale atmospheric transport. In another application, one detailed study is reported with the analysis of the results obtained from the application of PMF and from PCA-MLRA (Principal Component Analysis with Multilinear Regression Receptor Modelling) to one dataset containing compositional PM 10 data at an industrial site in Northern Spain. Even though similar results were obtained with both models, PMF achieved a higher level of detail in the apportionment of sources than PCA-MLRA. However, it was also noted that the application of PMF is more time consuming (at least 50% more time) than PCA-MLRA.

The Nonlinear Approach to Linear Calibration Intervals

We show that a nonlinear approach to single use calibration curves gives reasonable intervals. The nonlinear approach produces intervals even when the classical approach fails to do so. The intervals obtained with the nonlinear approach are always slightly shorter, but the coverage rate remains reasonable, as demonstrated in our simulations. The advantages of the nonlinear approach are that intervals are centered at the maximum likelihood estimator (MLE) and that they can be obtained using any standard statistical package which contains nonlinear regression methods.

Gap bootstrap methods for massive data sets with an application to transportation engineering

In this paper we describe two bootstrap methods for massive data sets. Naive applications of common resampling methodology are often impractical for massive data sets due to computational burden and due to complex patterns of inhomogeneity. In contrast, the proposed methods exploit certain structural properties of a large class of massive data sets to break up the original problem into a set of simpler subproblems, solve each subproblem separately where the data exhibit approximate uniformity and where computational complexity can be reduced to a manageable level, and then combine the results through certain analytical considerations. The validity of the proposed methods is proved and their finite sample properties are studied through a moderately large simulation study. The methodology is illustrated with a real data example from Transportation Engineering, which motivated the development of the proposed methods.