Federico Goodsaid

Evaluation of DNA microarray results with quantitative gene expression platforms

We have evaluated the performance characteristics of three quantitative gene expression technologies and correlated their expression measurements to those of five commercial microarray platforms, based on the MicroArray Quality Control (MAQC) data set. The limit of detection, assay range, precision, accuracy and fold-change correlations were assessed for 997 TaqMan Gene Expression Assays, 205 Standardized RT (Sta)RT-PCR assays and 244 QuantiGene assays. TaqMan is a registered trademark of Roche Molecular Systems, Inc. We observed high correlation between quantitative gene expression values and microarray platform results and found few discordant measurements among all platforms. The main cause of variability was differences in probe sequence and thus target location. A second source of variability was the limited and variable sensitivity of the different microarray platforms for detecting weakly expressed genes, which affected interplatform and intersite reproducibility of differentially expressed genes. From this analysis, we conclude that the MAQC microarray data set has been validated by alternative quantitative gene expression platforms thus supporting the use of microarray platforms for the quantitative characterization of gene expression.

Rat toxicogenomic study reveals analytical consistency across microarray platforms

To validate and extend the findings of the MicroArray Quality Control (MAQC) project, a biologically relevant toxicogenomics data set was generated using 36 RNA samples from rats treated with three chemicals (aristolochic acid, riddelliine and comfrey) and each sample was hybridized to four microarray platforms. The MAQC project assessed concordance in intersite and cross-platform comparisons and the impact of gene selection methods on the reproducibility of profiling data in terms of differentially expressed genes using distinct reference RNA samples. The real-world toxicogenomic data set reported here showed high concordance in intersite and cross-platform comparisons. Further, gene lists generated by fold-change ranking were more reproducible than those obtained by t-test P value or Significance Analysis of Microarrays. Finally, gene lists generated by fold-change ranking with a nonstringent P-value cutoff showed increased consistency in Gene Ontology terms and pathways, and hence the biological impact of chemical exposure could be reliably deduced from all platforms analyzed.

Renal biomarker qualification submission: a dialog between the FDA-EMEA and Predictive Safety Testing Consortium

The first formal qualification of safety biomarkers for regulatory decision making marks a milestone in the application of biomarkers to drug development. Following submission of drug toxicity studies and analyses of biomarker performance to the Food and Drug Administration (FDA) and European Medicines Agency (EMEA) by the Predictive Safety Testing Consortiums (PSTC) Nephrotoxicity Working Group, seven renal safety biomarkers have been qualified for limited use in nonclinical and clinical drug development to help guide safety assessments. This was a pilot process, and the experience gained will both facilitate better understanding of how the qualification process will probably evolve and clarify the minimal requirements necessary to evaluate the performance of biomarkers of organ injury within specific contexts.

The External RNA Controls Consortium: a progress report

Standard controls and best practice guidelines advance acceptance of data from research, preclinical and clinical laboratories by providing a means for evaluating data quality. The External RNA Controls Consortium (ERCC) is developing commonly agreed-upon and tested controls for use in expression assays, a true industry-wide standard control.Standard controls and best practice guidelines advance acceptance of data from research, preclinical and clinical laboratories by providing a means for evaluating data quality. The External RNA Controls Consortium (ERCC) is developing commonly agreed-upon and tested controls for use in expression assays, a true industry-wide standard control.

Cross-platform comparability of microarray technology: Intra-platform consistency and appropriate data analysis procedures are essential

BackgroundThe acceptance of microarray technology in regulatory decision-making is being challenged by the existence of various platforms and data analysis methods. A recent report (E. Marshall, Science, 306, 630–631, 2004), by extensively citing the study of Tan et al. (Nucleic Acids Res., 31, 5676–5684, 2003), portrays a disturbingly negative picture of the cross-platform comparability, and, hence, the reliability of microarray technology.ResultsWe reanalyzed Tans dataset and found that the intra-platform consistency was low, indicating a problem in experimental procedures from which the dataset was generated. Furthermore, by using three gene selection methods (i.e., p-value ranking, fold-change ranking, and Significance Analysis of Microarrays (SAM)) on the same dataset we found that p-value ranking (the method emphasized by Tan et al.) results in much lower cross-platform concordance compared to fold-change ranking or SAM. Therefore, the low cross-platform concordance reported in Tans study appears to be mainly due to a combination of low intra-platform consistency and a poor choice of data analysis procedures, instead of inherent technical differences among different platforms, as suggested by Tan et al. and Marshall.ConclusionOur results illustrate the importance of establishing calibrated RNA samples and reference datasets to objectively assess the performance of different microarray platforms and the proficiency of individual laboratories as well as the merits of various data analysis procedures. Thus, we are progressively coordinating the MAQC project, a community-wide effort for microarray quality control.

Biomarker qualification pilot process at the US Food and Drug Administration

New biomarkers of safety and efficacy are becoming powerful tools in drug development. Their application can be accelerated if a consensus can be reached about their qualification for regulatory applications. This consensus requires a review structure within the US Food and Drug Administration (FDA) that can evaluate qualification data for these biomarkers and determine whether these biomarkers can be qualified. A pilot process and corresponding Biomarker Qualification Review Team have been developed to test how the FDA can work on biomarker qualification.

Microarray scanner calibration curves: characteristics and implications

BackgroundMicroarray-based measurement of mRNA abundance assumes a linear relationship between the fluorescence intensity and the dye concentration. In reality, however, the calibration curve can be nonlinear.ResultsBy scanning a microarray scanner calibration slide containing known concentrations of fluorescent dyes under 18 PMT gains, we were able to evaluate the differences in calibration characteristics of Cy5 and Cy3. First, the calibration curve for the same dye under the same PMT gain is nonlinear at both the high and low intensity ends. Second, the degree of nonlinearity of the calibration curve depends on the PMT gain. Third, the two PMTs (for Cy5 and Cy3) behave differently even under the same gain. Fourth, the background intensity for the Cy3 channel is higher than that for the Cy5 channel. The impact of such characteristics on the accuracy and reproducibility of measured mRNA abundance and the calculated ratios was demonstrated. Combined with simulation results, we provided explanations to the existence of ratio underestimation, intensity-dependence of ratio bias, and anti-correlation of ratios in dye-swap replicates. We further demonstrated that although Lowess normalization effectively eliminates the intensity-dependence of ratio bias, the systematic deviation from true ratios largely remained. A method of calculating ratios based on concentrations estimated from the calibration curves was proposed for correcting ratio bias.ConclusionIt is preferable to scan microarray slides at fixed, optimal gain settings under which the linearity between concentration and intensity is maximized. Although normalization methods improve reproducibility of microarray measurements, they appear less effective in improving accuracy.

Strategic paths for biomarker qualification

Biomarkers may be qualified using different qualification processes. A passive approach for qualification has been to accept the end of discussions in the scientific literature as an indication that a biomarker has been accepted. An active approach to qualification requires development of a comprehensive process by which a consensus may be reached about the qualification of a biomarker. Active strategies for qualification include those associated with context-independent as well as context-dependent qualifications.

QA/QC: challenges and pitfalls facing the microarray community and regulatory agencies

The scientific community has been enthusiastic about DNA microarray technology for pharmacogenomic and toxicogenomic studies in the hope of advancing personalized medicine and drug development. The US Food and Drug Administration has been proactive in promoting the use of pharmacogenomic data in drug development and has issued a draft guidance for the pharmaceutical industry on data submissions. However, many challenges and pitfalls are facing the microarray community and regulatory agencies before microarray data can be reliably applied to support regulatory decision making. Four types of factors (i.e., technical, instrumental, computational and interpretative) affect the outcome of a microarray study, and a major concern about microarray studies has been the lack of reproducibility and accuracy. Intralaboratory data consistency is the foundation of reliable knowledge extraction and meaningful crosslaboratory or crossplatform comparisons; unfortunately, it has not been seriously evaluated and demonstrated in every study. Profound problems in data quality have been observed from analyzing published data sets, and many laboratories have been struggling with technical troubleshooting rather than generating reliable data of scientific significance. The microarray community and regulatory agencies must work together to establish a set of consensus quality assurance and quality control criteria for assessing and ensuring data quality, to identify critical factors affecting data quality, and to optimize and standardize microarray procedures so that biologic interpretation and decision-making are not based on unreliable data. These fundamental issues must be adequately addressed before microarray technology can be transformed from a research tool to clinical practices.

Voluntary exploratory data submissions to the US FDA and the EMA: experience and impact

Heterogeneity in the underlying mechanisms of disease processes and inter-patient variability in drug responses are major challenges in drug development. To address these challenges, biomarker strategies based on a range of platforms, such as microarray gene-expression technologies, are increasingly being applied to elucidate these sources of variability and thereby potentially increase drug development success rates. With the aim of enhancing understanding of the regulatory significance of such biomarker data by regulators and sponsors, the US Food and Drug Administration initiated a programme in 2004 to allow sponsors to submit exploratory genomic data voluntarily, without immediate regulatory impact. In this article, a selection of case studies from the first 5 years of this programme — which is now known as the voluntary exploratory data submission programme, and also involves collaboration with the European Medicines Agency — are discussed, and general lessons are highlighted.