Armin Graber

Sample size and statistical power considerations in high-dimensionality data settings: a comparative study of classification algorithms

BackgroundData generated using omics technologies are characterized by high dimensionality, where the number of features measured per subject vastly exceeds the number of subjects in the study. In this paper, we consider issues relevant in the design of biomedical studies in which the goal is the discovery of a subset of features and an associated algorithm that can predict a binary outcome, such as disease status. We compare the performance of four commonly used classifiers (K-Nearest Neighbors, Prediction Analysis for Microarrays, Random Forests and Support Vector Machines) in high-dimensionality data settings. We evaluate the effects of varying levels of signal-to-noise ratio in the dataset, imbalance in class distribution and choice of metric for quantifying performance of the classifier. To guide study design, we present a summary of the key characteristics of omics data profiled in several human or animal model experiments utilizing high-content mass spectrometry and multiplexed immunoassay based techniques.ResultsThe analysis of data from seven omics studies revealed that the average magnitude of effect size observed in human studies was markedly lower when compared to that in animal studies. The data measured in human studies were characterized by higher biological variation and the presence of outliers. The results from simulation studies indicated that the classifier Prediction Analysis for Microarrays (PAM) had the highest power when the class conditional feature distributions were Gaussian and outcome distributions were balanced. Random Forests was optimal when feature distributions were skewed and when class distributions were unbalanced. We provide a free open-source R statistical software library (MVpower) that implements the simulation strategy proposed in this paper.ConclusionNo single classifier had optimal performance under all settings. Simulation studies provide useful guidance for the design of biomedical studies involving high-dimensionality data.

The Liver Toxicity Biomarker Study Phase I: Markers for the Effects of Tolcapone or Entacapone

The Liver Toxicity Biomarker Study is a systems toxicology approach to discover biomarkers that are indicative of a drug’s potential to cause human idiosyncratic drug-induced liver injury. In phase I, the molecular effects in rat liver and blood plasma induced by tolcapone (a “toxic” drug) were compared with the molecular effects in the same tissues by dosing with entacapone (a “clean” drug, similar to tolcapone in chemical structure and primary pharmacological mechanism). Two durations of drug exposure, 3 and 28 days, were employed. Comprehensive molecular analysis of rat liver and plasma samples yielded marker analytes for various drug–vehicle or drug–drug comparisons. An important finding was that the marker analytes associated with tolcapone only partially overlapped with marker analytes associated with entacapone, despite the fact that both drugs have similar chemical structures and the same primary pharmacological mechanism of action. This result indicates that the molecular analyses employed in the study are detecting substantial “off-target” markers for the two drugs. An additional interesting finding was the modest overlap of the marker data sets for 3-day exposure and 28-day exposure, indicating that the molecular changes in liver and plasma caused by short- and long-term drug treatments do not share common characteristics.

MUMAL: Multivariate analysis in shotgun proteomics using machine learning techniques

BackgroundThe shotgun strategy (liquid chromatography coupled with tandem mass spectrometry) is widely applied for identification of proteins in complex mixtures. This method gives rise to thousands of spectra in a single run, which are interpreted by computational tools. Such tools normally use a protein database from which peptide sequences are extracted for matching with experimentally derived mass spectral data. After the database search, the correctness of obtained peptide-spectrum matches (PSMs) needs to be evaluated also by algorithms, as a manual curation of these huge datasets would be impractical. The target-decoy database strategy is largely used to perform spectrum evaluation. Nonetheless, this method has been applied without considering sensitivity, i.e., only error estimation is taken into account. A recently proposed method termed MUDE treats the target-decoy analysis as an optimization problem, where sensitivity is maximized. This method demonstrates a significant increase in the retrieved number of PSMs for a fixed error rate. However, the MUDE model is constructed in such a way that linear decision boundaries are established to separate correct from incorrect PSMs. Besides, the described heuristic for solving the optimization problem has to be executed many times to achieve a significant augmentation in sensitivity.ResultsHere, we propose a new method, termed MUMAL, for PSM assessment that is based on machine learning techniques. Our method can establish nonlinear decision boundaries, leading to a higher chance to retrieve more true positives. Furthermore, we need few iterations to achieve high sensitivities, strikingly shortening the running time of the whole process. Experiments show that our method achieves a considerably higher number of PSMs compared with standard tools such as MUDE, PeptideProphet, and typical target-decoy approaches.ConclusionOur approach not only enhances the computational performance, and thus the turn around time of MS-based experiments in proteomics, but also improves the information content with benefits of a higher proteome coverage. This improvement, for instance, increases the chance to identify important drug targets or biomarkers for drug development or molecular diagnostics.

MUMAL2: Improving sensitivity in shotgun proteomics using cost sensitive artificial neural networks and a threshold selector algorithm

BackgroundThis work presents a machine learning strategy to increase sensitivity in tandem mass spectrometry (MS/MS) data analysis for peptide/protein identification. MS/MS yields thousands of spectra in a single run which are then interpreted by software. Most of these computer programs use a protein database to match peptide sequences to the observed spectra. The peptide-spectrum matches (PSMs) must also be assessed by computational tools since manual evaluation is not practicable. The target-decoy database strategy is largely used for error estimation in PSM assessment. However, in general, that strategy does not account for sensitivity.ResultsIn a previous study, we proposed the method MUMAL that applies an artificial neural network to effectively generate a model to classify PSMs using decoy hits with increased sensitivity. Nevertheless, the present approach shows that the sensitivity can be further improved with the use of a cost matrix associated with the learning algorithm. We also demonstrate that using a threshold selector algorithm for probability adjustment leads to more coherent probability values assigned to the PSMs. Our new approach, termed MUMAL2, provides a two-fold contribution to shotgun proteomics. First, the increase in the number of correctly interpreted spectra in the peptide level augments the chance of identifying more proteins. Second, the more appropriate PSM probability values that are produced by the threshold selector algorithm impact the protein inference stage performed by programs that take probabilities into account, such as ProteinProphet. Our experiments demonstrate that MUMAL2 reached around 15% of improvement in sensitivity compared to the best current method. Furthermore, the area under the ROC curve obtained was 0.93, demonstrating that the probabilities generated by our model are in fact appropriate. Finally, Venn diagrams comparing MUMAL2 with the best current method show that the number of exclusive peptides found by our method was nearly 4-fold higher, which directly impacts the proteome coverage.ConclusionsThe inclusion of a cost matrix and a probability threshold selector algorithm to the learning task further improves the target-decoy database analysis for identifying peptides, which optimally contributes to the challenging task of protein level identification, resulting in a powerful computational tool for shotgun proteomics.

Abstract PD6-3: Recurrent ESR1 fusion transcripts are associated with endocrine resistance in estrogen receptor positive, HER2 negative breast cancer

Breast cancer proliferation measured by Ki67 immunohistochemistry after short-term antiestrogen therapy has been shown to correlate with disease-free survival. This suggests the use of biomarkers of the early effects of endocrine therapy on ER+ tumors will identify resistant cancers. Thus, we hypothesized that profiling operable ER+ tumors after short term treatment with an aromatase inhibitor would discover actionable molecular alterations causally associated with resistance to estrogen deprivation. We performed whole exome sequencing, RNA-Seq and quantitative immunofluorescence (QIF) of ER, PR, HER2, and Ki67 in biopsies from 130 patients with an operable ER+/HER2– breast cancer that had received letrozole for 10-21 days prior to surgery. Tumors were categorized by the natural log of 2-week post-letrozole Ki67 as sensitive, intermediate, or resistant. We sequenced RNA from 50 frozen tumors and performed fusion transcript analysis using 4 programmatic algorithms (dRanger, TopHat, DeFuse, Chimera Scan), resulting in 304 candidate gene fusions in 44 tumors. Primers with universal sequencing tags were designed against 3’ and 5’ sites of breakpoints mapping to RefSeq exon coding regions (n=187); fusion sequences were amplified by qRT-PCR from tumor and breast cancer cell line RNA. Single or multiple distinct product bands were visualized by gel electrophoresis in 96 tumor samples and Sanger-sequenced. Results were mapped to the human RNA reference transcriptome using BLAST. Overall, 9% of putative fusion transcripts (n=27 from 16 unique tumors) were validated by mapping to the open reading frames of predicted 3’ and 5’ genes. Fusion transcripts called by more than one program were more likely to validate (13 of 24 redundant versus 14 of 269 unique; p Using the 2-week Ki67 to stratify for response to treatment, the validated ESR1 fusions were present only in tumors that maintained high (≥7.4%) to intermediate (>2.7%) Ki67 labeling indices upon estrogen deprivation with letrozole (p=0.01). PR expression was lower (p=0.003) and ER expression higher (p=0.05) in ESR1 fusion+ tumors compared to fusion negative tumors. RNA extracted from 14 additional tumors were screened for ESR1 fusions by qRT-PCR and the ESR1:CCDC170 fusion was validated in 1 of 8 resistant/intermediate and 0 of 6 sensitive tumors. In summary, biomarkers of early response to antiestrogens are needed in order to identify ER+ cancers that are treatment resistant. In a prospective trial of operable ER+/HER2− breast tumors, we discovered recurrent intrachromosomal ESR1 fusion transcripts associated with intrinsic resistance to estrogen deprivation with letrozole. Additional work investigating the genomic basis and function of the fusion transcripts is underway. Citation Format: Jennifer M Giltnane, Justin M Balko, Thomas L Stricker, Christian Young, M Valeria Estrada, Nikhil Wagle, Eliezer van Allen, X Jasmine Mu, Violeta Sanchez, Jaime Farley, Kerry Fitzgerald, Armin Graber, Joseph A Pinto, Franco Doimi, Henry Gomez, Monica Rizzo, Thomas B Julian, Vandana Abramson, Ingrid Mayer, Mark Kelley, Ashwini Yenamandra, Ferrin C Wheeler, Melinda Sanders, Levi Garraway, Ingrid Meszoely, Carlos L Arteaga. Recurrent ESR1 fusion transcripts are associated with endocrine resistance in estrogen receptor positive, HER2 negative breast cancer [abstract]. In: Proceedings of the Thirty-Seventh Annual CTRC-AACR San Antonio Breast Cancer Symposium: 2014 Dec 9-13; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2015;75(9 Suppl):Abstract nr PD6-3.

Abstract 3388: Analytical validation and clinical verification of phosphoprotein biomarker modulation using a novel preservation system-based flow cytometry assay in multiple myeloma clinical trials

Propelled by the advent of novel oncogenic pathway-targeting drugs and increasing regulatory expectations to fulfil proof of mechanism endpoints in clinical trials, there is a critical need to reliably evaluate target inhibition by investigational drugs in biologically relevant compartments. However, the ability to monitor modulation of labile oncogenic pathway biomarkers in global hematomalignancy clinical trials is complicated by varying technical expertise at clinical sites, when specimen stability considerations do not permit analysis at a central specialty laboratory. To address this critical gap, we developed a novel preservation system, designated NovaPerm3 (NP3), that enables stabilization of phosphoprotein based biomarker specimens in a single step, requires less than 20 minutes, and can be implemented in routine clinical trial settings. Stabilized specimens can be frozen and analyzed even after 90 days in a specialized laboratory. Using the NP3 preservation system, we systematically developed and validated a flow cytometry assay that not only enables reliable identification of neoplastic plasma cells but also allows for quantitation of phosphoprotein biomarkers in the tumor cells of interest present in both the peripheral blood (PB) and bone marrow (BM). We will describe a) modulation of phosphorylation levels of the S6 ribosomal protein (pS6RP) by a novel anticancer agent in multiple myeloma patients b) qualitative concordance between NP3-flow cytometry and western blot results from multiple drug treated leukemia cell lines c) sensitive detection of neoplastic plasma cells up to 1% of the total white cells and d) specimen (pS6RP) stabilization over a 90 day period. In conclusion, our novel preservation method and clinical trial laboratory validated flow assay enables reliable quantification of signaling biomarkers in hematomalignancy trials with ease in a specialized central laboratory while reducing the procedural burden for collecting biomarker specimens in global clinical trials. Citation Format: Anil Pahuja, Shyam Sarikonda, Benjamin Lee, Armin Graber, Shabnam Tangri, Naveen Dakappagari. Analytical validation and clinical verification of phosphoprotein biomarker modulation using a novel preservation system-based flow cytometry assay in multiple myeloma clinical trials. [abstract]. In: Proceedings of the 106th Annual Meeting of the American Association for Cancer Research; 2015 Apr 18-22; Philadelphia, PA. Philadelphia (PA): AACR; Cancer Res 2015;75(15 Suppl):Abstract nr 3388. doi:10.1158/1538-7445.AM2015-3388

Abstract LB-96: A novel blood preservation system to study oncogenic signaling pathway biomarkers by flow cytometry in leukemia/lymphoma clinical trials

Propelled by the advent of new technologies and an evolving regulatory landscape, the desire to personalize cancer treatments has never been greater. There is a critical need to reliably evaluate target inhibition and pharmacodynamic activity of investigational drugs in biologically relevant compartments. However, the ability to fulfill such a task in global clinical trials is complicated by varying technical expertise available at clinical sites and considerations about specimen stability for centralized laboratory analysis. We developed a novel formalin-based preservation method that enables specimen stabilization in a single step, requiring less than 20 minutes, without the need for specialized training or instrumentation. Stabilized specimens can be frozen for shipping and batched analysis at a later time point, in a specialized laboratory. Using this preservation method, we developed a flow assay enabling identification of multiple cell-types, and quantification of intracellular biomarkers in target cellular compartments, in both the peripheral blood (PB) and bone marrow (BM). We present pharmacodynamic (PD) data for a novel anticancer investigational agent intended for acute myeloid leukemia and multiple myeloma and its effect on multiple target biomarkers of the PI3K signaling pathway, including phosphorylation of the S6 ribosomal protein (S6). Our novel fixation method allowed detection of pS6 modulation in a dose dependent manner in both tumor cells and PB or BM in response to the novel investigational agent. Assay sensitivity and concordance were evaluated by comparing flow assay with western blot analysis, with each assay performed at a different site using the same batched frozen samples; exceptional preservation of phosphorylated proteins for more than 72 hours could be observed, when frozen immediately following fixation. In conclusion, our novel preservation method enables reliable quantification of signaling biomarkers in centralized laboratories at different time points post sample (PB or BM) acquisition requiring minimal processing at collection sites. Citation Format: Anil Pahuja, Abdel Saci, Shyam Sarikonda, Armin Graber, Benjamin Lee, Jelveh Lameh, Shabnam Tangri, Naveen Dakappagari. A novel blood preservation system to study oncogenic signaling pathway biomarkers by flow cytometry in leukemia/lymphoma clinical trials. [abstract]. In: Proceedings of the 105th Annual Meeting of the American Association for Cancer Research; 2014 Apr 5-9; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2014;74(19 Suppl):Abstract nr LB-96. doi:10.1158/1538-7445.AM2014-LB-96

Abstract 5601: Identification of critical steps involved in optimization of next generation sequencing (NGS) using a retrospective meta-analysis of a large clinical testing cohort

Proceedings: AACR Annual Meeting 2014; April 5-9, 2014; San Diego, CAnnIntroduction: There is increased use of NGS technology in clinical applications. Promise of this platforms applicability in the assessment of somatic mutations is seen in improvements in sensitivity over other sequencing methods, higher throughput and rapidly decreasing cost. We analyzed a large cohort, n=1210, of sample data from routine testing since 2012 and find that results are robust. However, there are several areas that users of NGS technology can optimize to assure the highest levels of accuracy for clinical-grade sequencing results.nnMethods: NGS was performed as targeted sequencing of multiple exons (132 amplicons) from 5 genes (ASXL1, RUNX1, ETV6, EZH2, and TP53). Patients had a prior diagnosis of, or were suspected of having, myelodysplastic syndrome (MDS). Validated sample types included bone marrow aspirates and peripheral blood. DNA was extracted from samples and processed in duplicate using a clinically validated testing procedure. Libraries were created using a Fluidigm Access Array system and sequencing was performed on the Illumina MiSeq platform. The limit of detection of this assay is 5% mutant allele with a minimum depth of coverage of 500x. Alignment and variant calling was performed using NextGENe software. Custom Perl software was used to provide quality checking of results, duplicate run comparisons, and annotation from existing databases for known germline and somatic variants.nnResults: From 1210 random patient samples run in duplicate, 738 (61%) samples demonstrated 2081 variant calls in only one of the two replicates. The quality and quantity of input DNA is paramount to achieving stable results (DNA input accounts for 75% of singlicate results in this study). In addition, specific regions of sequence have been identified as ‘hot spots’ of high variability, accounting for 21% of the singlicate calls in this study. For example, RUNX1 accounts for 46% of the singlicate calls and 84% of those are concentrated in exon 3, a 254bp, GC-rich region. Finally, the resulting data must be well managed by analytical and bioinformatics tools to assure accuracy but meet the requirements of the high throughput and robust clinical setting. Although these factors can be mitigated, the use of duplicate testing of a large cohort of hematologic patient specimens allowed identification of these sources of variation, providing a cautionary note to others using NGS in a clinical setting.nnConclusions: Duplicate NGS runs from about a year of clinical testing demonstrate robust results. However, laboratories need to address factors that may confound results such as input sample quantity/quality, amplicon design, and post-analytical processing. These can introduce the potential for false-positive/negative results in routine testing without additional confirmation.nnCitation Format: Danielle A. Murphy, Christopher Scott, Kerry Fitzgerald, Maya Panjikaran, Yu Xia, Armin Graber, Julie Kines, Adam Baer, Matthew McGinniss, Jason Christiansen. Identification of critical steps involved in optimization of next generation sequencing (NGS) using a retrospective meta-analysis of a large clinical testing cohort. [abstract]. In: Proceedings of the 105th Annual Meeting of the American Association for Cancer Research; 2014 Apr 5-9; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2014;74(19 Suppl):Abstract nr 5601. doi:10.1158/1538-7445.AM2014-5601

Abstract 2843: Development of a binary diagnostic immunofluorescence assay by AQUA® technology for accurate detection of HER-2 levels in breast cancer specimens

HER2 is a prognostic and predictive marker for breast cancer patients and its expression is routinely evaluated by immunochemistry (IHC). Scoring of IHC slides is prone to operator subjectivity and equivocal results make selection of appropriate therapy difficult, ultimately affecting patient outcomes. We describe development and validation of a reproducible quantitative immunofluorescence assay to accurately assess HER-2 levels in Formalin-Fixed Paraffin-Embedded (FFPE) breast cancer specimens by AQUA Technology that avoids the ambiguity of equivocal results and enables critical treatment decisions. Methods: A tissue microarray (n=80) composed of breast cancer cases with known IHC and fluorescence in situ hybridization (FISH) status was used to characterize and optimize assay performance for three HER-2 antibody clones; A0485, CB11, and SP3. Eight dilutions of each antibody were tested with four different antigen retrieval conditions. A total of 108 assay conditions were evaluated by receiver operator characteristic (ROC) analysis for sensitivity and specificity. Equal weight was given to sensitivity and specificity to select the most robust assay. The top six assay conditions were then assessed using 45 whole tissue breast cancer specimens to identify one condition with highest sensitivity and specificity. The selected assay condition was evaluated on an additional 90 breast cancer specimens (training set) annotated for IHC and FISH scores to determine the binary cut point for the HER-2 AQUA assay®, the cut point was then confirmed on a validation set composed of 90 independent breast cancer specimens. The final assay was analytically validated in accordance with College of American Pathologists (CAP) guidelines utilizing over 120 independent breast cancer specimens. Results: Based on an evaluation of over 400 breast cancer specimens with nearly equal distribution of 0, 1+, 2+ and 3+ cases, HER-2 antibody clone, SP3, which recognizes the extracellular domain of the receptor, clearly segregated HER2 positive specimens from HER2 negative breast cancer cases. Conclusion: The availability of a highly reproducible quantitative binary test facilitates rapid treatment decisions with appropriate HER-2 targeting biologics on the market. Citation Format: Lakshmi Krupa Chandrasekaran, Jennifer Bordeaux, Sue Beruti, Naveen Dakappagari, Mike Nerenberg, Jelveh Lameh, Armin Graber, David Rimm, Bruce Robbins, Nagesh Rao. Development of a binary diagnostic immunofluorescence assay by AQUA® technology for accurate detection of HER-2 levels in breast cancer specimens. [abstract]. In: Proceedings of the 105th Annual Meeting of the American Association for Cancer Research; 2014 Apr 5-9; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2014;74(19 Suppl):Abstract nr 2843. doi:10.1158/1538-7445.AM2014-2843

Abstract 4878: Application of a novel nano-immunoassay platform to assess changes in cIAP1 in response to the SMAC-mimetic, LCL161

Introduction: cIAP1 is member of a class of inhibitor of apoptosis proteins (IAP) that function in a regulatory role to prevent unintended cell death by apoptosis. LCL161 is a small molecule antagonist of IAPs that reverses cIAP1 inhibition of caspases, causing apoptosis. The present study explores the use of the Nano Pro 1000, a novel proteomic platform to measure the changes in cellular cIAP1 in response to treatment with LCL161. Methods: SKOV3 cells, peripheral blood mononuclear cells (PBMC) and hair follicles collected from healthy donors were treated with LCL161 for different times and subjected to capillary iso-electric focusing. Immunodetection was performed using antibodies directed against cIAP1 and the signal was quantified using HRP-labeled chemiluminescence reagents. Data from the Nano Pro were compared to results from western blots run in parallel. Results: We report that results from the nano-immunoassay are reproducible and correlate well with the western blotting data. Modulation in the cellular levels of cIAP1 in response to LCL161 treatment could be captured in samples containing as little as 1 ng of total protein. Similar modulation of cIAP1 was seen upon exposure to LCL161 in the studied test matrices. Conclusions: This novel proteomic tool is more sensitive, specific and quantitative compared to the currently available option of western blotting assays which are labor intensive, and have a relatively low throughput. Measurable signals corresponding to cIAP1 levels can be detected in matrices like PBMCs and hair follicles. The modulation of cIAP1 levels in these matrices mirror that seen in SKOV3 cells. Citation Format: {Authors}. {Abstract title} [abstract]. In: Proceedings of the 102nd Annual Meeting of the American Association for Cancer Research; 2011 Apr 2-6; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2011;71(8 Suppl):Abstract nr 4878. doi:10.1158/1538-7445.AM2011-4878