Pan-Cancer Integrative Histology-Genomic Analysis via Interpretable Multimodal Deep Learning

Abstract

Cancer prognostication and therapeutic response prediction is driven by prognostic markers in both histopathology slides and molecular profiles. The rapidly emerging field of deep learning-based computational pathology has demonstrated promise in developing objective prognostic models from histology whole slide images. Several studies have focused on predicting genomic and transcriptomic alterations from histology images. However, most prognostic models are either based on histology or genomics alone and do not address how histology and genomics can be integrated to develop joint image-omic assays and prognostic models. To overcome these challanges, we present a pan-cancer integrative platform for biomarker discovery in both histology slides and molecular profile data (http://pancancer.ai). We used corresponding gigapixel whole-slide images, RNA-Seq abundance, copy number variation, and mutation data from 5,720 patients across 14 cancer types to train a weakly supervised, interpretable, multimodal deep learning algorithm that is able to not only fuse these heterogenous modalities for survival analysis, but also discover prognostic features from these modalities that corroborate with poor and favorable survival outcomes using multimodal interpretability. In a 5-fold cross validation, we compared our model with unimodal deep learning models trained on histology slides and molecular profiles alone, and demonstrate performance increase in risk stratification on 9 out of 14 cancers, as well as shifts in feature importance when conditioned on more than one modality. To validate our setup as a platform for driving biomarker discovery, we analyzed both morphological and molecular features that were identified by high attribution regions in every patient, and discovered statistically significant associations between high tumor-infilitrating lymphocyte (TIL) presence in HE 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-002.