Walter Liggett

Measurement Reproducibility in the Early Stages of Biomarker Development

Biomarker discovery and development requires measurement reproducibility studies in addition to case-control studies. Parallel pursuit of reproducibility studies is especially important for emerging technologies such as protein biomarkers based on time-of-flight mass spectrometry, the case considered in this paper. For parallel studies, a way to improve reproducibility prior to identification of protein species is necessary. One approach is use of functional principal components analysis (PCA) as the basis for assessing measurement reproducibility. Reproducibility studies involve repeated measurement of a reference material such as a human serum standard. Measurement in our example is by SELDI-TOF (surface-enhanced laser desorption and ionization time-of-flight) mass spectrometry. Reproducibility is defined in reference to a source of variation, which in our example is associated with a type of commercially available protein biochip. We obtained spectra for 8 spots on each 11 chips. Two spectra are generally more alike when obtained from the same chip rather than different chips. Thus, our experiment indicates potential improvements from reducing variation in chip manufacture and chip handling during measurement. Our analysis involves careful registration of the spectra and characterization of the spectral differences. As shown by our example, a metrological analysis may enhance case-control studies by guiding optimization of the measurements underlying the biomarker.

The National Bureau of Standards Rocky Flats Soil standard reference material

Abstract The National Bureau of Standards (NBS) in collaboration with a number of environmental laboratories of the ICRM has recently issued a soil standard certified for radionuclidic concentrations of activation and fission products and natural radionuclides. Initial disagreements between laboratories of measured concentrations have led to a careful examination of the characteristics of the sample and the radiochemical procedures employed by the participants. A number of assay problems were identified and are discussed. The sample was found to contain an average of approximately 1.8 “hot” (239+240Pu) particles per bottle of 90 g of soil. A statistical analysis of over seventy measurements was used to determine that the homogeneity of the material excluding hot particles is satisfactory. A sampling method for using this material for quality control of plutonium measurements which minimizes the effect of hot particles is described.

Low-level radioactivity ocean sediment standard reference material

Abstract The National Institute of Standards and Technology (NIST), in cooperation with experienced international laboratories, will issue a low-level radioactivity Ocean Sediiment Standard Reference Material (SRM) 4357 in 1996. The Ocean Sediment joins the NIST suite of six other ‘natural matrix’ environmental radioactivity Standard Reference Materials. This family of natural matrix materials have already been used: (1) to develop radiochemical procedures; (2) to test radiochemical procedures already in use for environmental and biokinetic evaluations; (3) to calibrate instruments; (4) to intercompare and evaluate radiochemical methods; (5) to test competency of technicians to do radiochemical assays; and (6) to demonstrate that data output is reliable. The participating laboratoires provided data for the radionuclides are of sufficient quality to establish certified values and reanonable uncertainty limits given the relatively low concentrations of the radionuclides. The uncertainties for a number of certified radionuclides are non-symmetrical and large because of the non-normal distribution of reported values.

Statistical assessment of subsampling procedures

Abstract As shown by the examples in this paper, the concentrations in subsamples are not necessarily independently and normally distributed despite vigorous grinding and mixing of the original sample. Studies of the statistical properties of subsample concentrations should test for deviations from independence and normality and, if deviations are found, should model the observed distribution. The tests include an analysis of variance to check for less variation among nearby subsamples than among widely spaced subsamples, as well as the computation of the probability plot correlation coefficient to check for nonnormality. These tests are illustrated with 239 Pu + 240 Pu measurements on subsamples prepared for use as standard reference materials. These materials are used in quality assurance for environmental radioactivity measurements. The material consisting of Human Liver (SRM 4352) exhibits dependence, which may be due to inadequate mixing or to later handling that caused segregation. The material consisting of Human Lung (SRM 4351) and the material consisting of soil from Rocky Flats (SRM 4353) exhibit nonnormality due to the particulate nature of some plutonium-bearing fractions of the material. The lung measurements are modeled with the gamma distribution. The soil measurements are modeled with a combination of the normal distribution and a distribution that models the occurrence of outliers. The use of these models to describe the subsamples involves assessment of the contribution of the measurement error.

Capability in Rockwell C Scale Hardness

A measurement system is capable if it produces measurements with uncertainties small enough for demonstration of compliance with product specifications. To establish the capability of a system for Rock-well C scale hardness, one must assess measurement uncertainty and, when hardness is only an indicator, quantify the relation between hardness and the product property of real interest. The uncertainty involves several components, which we designate as lack of repeatability, lack of reproducibility, machine error, and indenter error. Component-by-component assessment leads to understanding of mechanisms and thus to guidance on system upgrades if these are necessary. Assessment of some components calls only for good-quality test blocks, and assessment of others requires test blocks that NIST issues as Standard Reference Materials (SRMs). The important innovation introduced in this paper is improved handling of the hardness variation across test-block surfaces. In addition to hardness itself, the methods in this paper might be applicable to other local measurement of a surface.

Normalization and Technical Variation in Gene Expression Measurements

Using data from the Microarray Quality Control (MAQC) project, we demonstrate two data-analysis methods that shed light on the normalization of gene expression measurements and thereby on their technical variation. One is an improved method for normalization of multiple assays with mRNA concentrations related by a parametric model. The other is a method for characterizing limitations on the effectiveness of normalization in reducing technical variation. We apply our improved normalization to the four project materials as part of testing the linearity of the probe responses. We find that the lack of linearity is statistically significant but small enough that its sources cannot be easily identified. Applying our characterization method to assays of the same material, we show that there is a source of variation that cannot be eliminated by normalization and therefore must be dealt with by other means. Four high-density, single probe, one-color microarray platforms underlie our demonstration.

System Performance and Natural Language Expression of Information Needs

Consider information retrieval systems that respond to a query (a natural language statement of a topic, an information need) with an ordered list of 1000 documents from the document collection. From the responses to queries that all express the same topic, one can discern how the words associated with a topic result in particular system behavior. From what is discerned from different topics, one can hypothesize abstract topic factors that influence system performance. An example of such a factor is the specificity of the topics primary key word. This paper shows that statements about the effect of abstract topic factors on system performance can be supported empirically. A combination of statistical methods is applied to system responses from NISTs Text REtrieval Conference. We analyze each topic using a measure of irrelevant-document exclusion computed for each response and a measure of dissimilarity between relevant-document return orders computed for each pair of responses. We formulate topic factors through graphical comparison of measurements for different topics. Finally, we propose for each topic a four-dimensional summarization that we use to select topic comparisons likely to depict topic factors clearly.

Calibration for measurements with background correction applied to uranium-235 enrichment

Abstract In enrichment measurement by gamma-ray counting, two spectral regions are observed: In one, the response is due to the enrichment and the background, and in the other, to the background alone. Calibration consists of determining not only the relation between the response and the enrichment but also the relation between the two background levels. A calibration procedure with this property is developed under the assumption that the random errors have constant variance and the assumption that the two background levels are proportional. This procedure provides a consistent estimator for the calibration curve and interval estimates for the unknowns measured after calibration. These intervals have stated percent coverage of the true values when large numbers of measurements are made on the basis of the same calibration experiment. The two assumptions may not adequately fit some enrichment measurements. The first assumption is never strictly valid since the random error depends in part on the level. The second assumption is valid for some but not all sources of background radiation. The calibration procedure is applied to enrichment measurements made with the SAM-2 enrichment meter. With these measurements as illustrations, techniques for judging the validity of the assumptions are presented.