Natalie Shih

Mitosis detection in breast cancer pathology images by combining handcrafted and convolutional neural network features

Abstract. Breast cancer (BCa) grading plays an important role in predicting disease aggressiveness and patient outcome. A key component of BCa grade is the mitotic count, which involves quantifying the number of cells in the process of dividing (i.e., undergoing mitosis) at a specific point in time. Currently, mitosis counting is done manually by a pathologist looking at multiple high power fields (HPFs) on a glass slide under a microscope, an extremely laborious and time consuming process. The development of computerized systems for automated detection of mitotic nuclei, while highly desirable, is confounded by the highly variable shape and appearance of mitoses. Existing methods use either handcrafted features that capture certain morphological, statistical, or textural attributes of mitoses or features learned with convolutional neural networks (CNN). Although handcrafted features are inspired by the domain and the particular application, the data-driven CNN models tend to be domain agnostic and attempt to learn additional feature bases that cannot be represented through any of the handcrafted features. On the other hand, CNN is computationally more complex and needs a large number of labeled training instances. Since handcrafted features attempt to model domain pertinent attributes and CNN approaches are largely supervised feature generation methods, there is an appeal in attempting to combine these two distinct classes of feature generation strategies to create an integrated set of attributes that can potentially outperform either class of feature extraction strategies individually. We present a cascaded approach for mitosis detection that intelligently combines a CNN model and handcrafted features (morphology, color, and texture features). By employing a light CNN model, the proposed approach is far less demanding computationally, and the cascaded strategy of combining handcrafted features and CNN-derived features enables the possibility of maximizing the performance by leveraging the disconnected feature sets. Evaluation on the public ICPR12 mitosis dataset that has 226 mitoses annotated on 35 HPFs (400× magnification) by several pathologists and 15 testing HPFs yielded an F-measure of 0.7345. Our approach is accurate, fast, and requires fewer computing resources compared to existent methods, making this feasible for clinical use.

Automatic detection of invasive ductal carcinoma in whole slide images with convolutional neural networks

This paper presents a deep learning approach for automatic detection and visual analysis of invasive ductal carcinoma (IDC) tissue regions in whole slide images (WSI) of breast cancer (BCa). Deep learning approaches are learn-from-data methods involving computational modeling of the learning process. This approach is similar to how human brain works using different interpretation levels or layers of most representative and useful features resulting into a hierarchical learned representation. These methods have been shown to outpace traditional approaches of most challenging problems in several areas such as speech recognition and object detection. Invasive breast cancer detection is a time consuming and challenging task primarily because it involves a pathologist scanning large swathes of benign regions to ultimately identify the areas of malignancy. Precise delineation of IDC in WSI is crucial to the subsequent estimation of grading tumor aggressiveness and predicting patient outcome. DL approaches are particularly adept at handling these types of problems, especially if a large number of samples are available for training, which would also ensure the generalizability of the learned features and classifier. The DL framework in this paper extends a number of convolutional neural networks (CNN) for visual semantic analysis of tumor regions for diagnosis support. The CNN is trained over a large amount of image patches (tissue regions) from WSI to learn a hierarchical part-based representation. The method was evaluated over a WSI dataset from 162 patients diagnosed with IDC. 113 slides were selected for training and 49 slides were held out for independent testing. Ground truth for quantitative evaluation was provided via expert delineation of the region of cancer by an expert pathologist on the digitized slides. The experimental evaluation was designed to measure classifier accuracy in detecting IDC tissue regions in WSI. Our method yielded the best quantitative results for automatic detection of IDC regions in WSI in terms of F-measure and balanced accuracy (71.80%, 84.23%), in comparison with an approach using handcrafted image features (color, texture and edges, nuclear textural and architecture), and a machine learning classifier for invasive tumor classification using a Random Forest. The best performing handcrafted features were fuzzy color histogram (67.53%, 78.74%) and RGB histogram (66.64%, 77.24%). Our results also suggest that at least some of the tissue classification mistakes (false positives and false negatives) were less due to any fundamental problems associated with the approach, than the inherent limitations in obtaining a very highly granular annotation of the diseased area of interest by an expert pathologist.

Multi-Field-of-View Framework for Distinguishing Tumor Grade in ER+ Breast Cancer From Entire Histopathology Slides

Modified Bloom-Richardson (mBR) grading is known to have prognostic value in breast cancer (BCa), yet its use in clinical practice has been limited by intra- and interobserver variability. The development of a computerized system to distinguish mBR grade from entire estrogen receptor-positive (ER+) BCa histopathology slides will help clinicians identify grading discrepancies and improve overall confidence in the diagnostic result. In this paper, we isolate salient image features characterizing tumor morphology and texture to differentiate entire hematoxylin and eosin (H and E) stained histopathology slides based on mBR grade. The features are used in conjunction with a novel multifield-of-view (multi-FOV) classifier-a whole-slide classifier that extracts features from a multitude of FOVs of varying sizes-to identify important image features at different FOV sizes. Image features utilized include those related to the spatial arrangement of cancer nuclei (i.e., nuclear architecture) and the textural patterns within nuclei (i.e., nuclear texture). Using slides from 126 ER+ patients (46 low, 60 intermediate, and 20 high mBR grade), our grading system was able to distinguish low versus high, low versus intermediate, and intermediate versus high grade patients with area under curve values of 0.93, 0.72, and 0.74, respectively. Our results suggest that the multi-FOV classifier is able to 1) successfully discriminate low, medium, and high mBR grade and 2) identify specific image features at different FOV sizes that are important for distinguishing mBR grade in Hand E stained ER+ BCa histology slides.

Self-renewal of CD133(hi) cells by IL6/Notch3 signalling regulates endocrine resistance in metastatic breast cancer.

The mechanisms of metastatic progression from hormonal therapy (HT) are largely unknown in luminal breast cancer. Here we demonstrate the enrichment of CD133hi/ERlo cancer cells in clinical specimens following neoadjuvant endocrine therapy and in HT refractory metastatic disease. We develop experimental models of metastatic luminal breast cancer and demonstrate that HT can promote the generation of HT-resistant, self-renewing CD133hi/ERlo/IL6hi cancer stem cells (CSCs). HT initially abrogates oxidative phosphorylation (OXPHOS) generating self-renewal-deficient cancer cells, CD133hi/ERlo/OXPHOSlo. These cells exit metabolic dormancy via an IL6-driven feed-forward ERlo-IL6hi-Notchhi loop, activating OXPHOS, in the absence of ER activity. The inhibition of IL6R/IL6-Notch pathways switches the self-renewal of CD133hi CSCs, from an IL6/Notch-dependent one to an ER-dependent one, through the re-expression of ER. Thus, HT induces an OXPHOS metabolic editing of luminal breast cancers, paradoxically establishing HT-driven self-renewal of dormant CD133hi/ERlo cells mediating metastatic progression, which is sensitive to dual targeted therapy.

Multi-field-of-view strategy for image-based outcome prediction of multi-parametric estrogen receptor-positive breast cancer histopathology: Comparison to Oncotype DX

In this paper, we attempt to quantify the prognostic information embedded in multi-parametric histologic biopsy images to predict disease aggressiveness in estrogen receptor-positive (ER+) breast cancers (BCa). The novel methodological contribution is in the use of a multi-field-of-view (multi-FOV) framework for integrating image-based information from differently stained histopathology slides. The multi-FOV approach involves a fixed image resolution while simultaneously integrating image descriptors from many FOVs corresponding to different sizes. For each study, the corresponding risk score (high scores reflecting aggressive disease and vice versa), predicted by a molecular assay (Oncotype DX), is available and serves as the surrogate ground truth for long-term patient outcome. Using the risk scores, a trained classifier is used to identify disease aggressiveness for each FOV size. The predictions for each FOV are then combined to yield the final prediction of disease aggressiveness (good, intermediate, or poor outcome). Independent multi-FOV classifiers are constructed for (1) 50 image features describing the spatial arrangement of cancer nuclei (via Voronoi diagram, Delaunay triangulation, and minimum spanning tree graphs) in H and E stained histopathology and (2) one image feature describing the vascular density in CD34 IHC stained histopathology. In a cohort of 29 patients, the multi-FOV classifiers obtained by combining information from the H and E and CD34 IHC stained channels were able to distinguish low- and high-risk patients with an accuracy of 0.91 ± 0.02 and a positive predictive value of 0.94 ± 0.10, suggesting that a purely image-based assay could potentially replace more expensive molecular assays for making disease prognostic predictions.

Supervised multi-view canonical correlation analysis (sMVCCA): integrating histologic and proteomic features for predicting recurrent prostate cancer.

In this work, we present a new methodology to facilitate prediction of recurrent prostate cancer (CaP) following radical prostatectomy (RP) via the integration of quantitative image features and protein expression in the excised prostate. Creating a fused predictor from high-dimensional data streams is challenging because the classifier must 1) account for the “curse of dimensionality” problem, which hinders classifier performance when the number of features exceeds the number of patient studies and 2) balance potential mismatches in the number of features across different channels to avoid classifier bias towards channels with more features. Our new data integration methodology, supervised Multi-view Canonical Correlation Analysis (sMVCCA), aims to integrate infinite views of highdimensional data to provide more amenable data representations for disease classification. Additionally, we demonstrate sMVCCA using Spearmans rank correlation which, unlike Pearsons correlation, can account for nonlinear correlations and outliers. Forty CaP patients with pathological Gleason scores 6-8 were considered for this study. 21 of these men revealed biochemical recurrence (BCR) following RP, while 19 did not. For each patient, 189 quantitative histomorphometric attributes and 650 protein expression levels were extracted from the primary tumor nodule. The fused histomorphometric/proteomic representation via sMVCCA combined with a random forest classifier predicted BCR with a mean AUC of 0.74 and a maximum AUC of 0.9286. We found sMVCCA to perform statistically significantly (p <; 0.05) better than comparative state-of-the-art data fusion strategies for predicting BCR. Furthermore, Kaplan-Meier analysis demonstrated improved BCR-free survival prediction for the sMVCCA-fused classifier as compared to histology or proteomic features alone.

Cascaded ensemble of convolutional neural networks and handcrafted features for mitosis detection

Breast cancer (BCa) grading plays an important role in predicting disease aggressiveness and patient outcome. A key component of BCa grade is mitotic count, which involves quantifying the number of cells in the process of dividing (i.e. undergoing mitosis) at a specific point in time. Currently mitosis counting is done manually by a pathologist looking at multiple high power fields on a glass slide under a microscope, an extremely laborious and time consuming process. The development of computerized systems for automated detection of mitotic nuclei, while highly desirable, is confounded by the highly variable shape and appearance of mitoses. Existing methods use either handcrafted features that capture certain morphological, statistical or textural attributes of mitoses or features learned with convolutional neural networks (CNN). While handcrafted features are inspired by the domain and the particular application, the data-driven CNN models tend to be domain agnostic and attempt to learn additional feature bases that cannot be represented through any of the handcrafted features. On the other hand, CNN is computationally more complex and needs a large number of labeled training instances. Since handcrafted features attempt to model domain pertinent attributes and CNN approaches are largely unsupervised feature generation methods, there is an appeal to attempting to combine these two distinct classes of feature generation strategies to create an integrated set of attributes that can potentially outperform either class of feature extraction strategies individually. In this paper, we present a cascaded approach for mitosis detection that intelligently combines a CNN model and handcrafted features (morphology, color and texture features). By employing a light CNN model, the proposed approach is far less demanding computationally, and the cascaded strategy of combining handcrafted features and CNN-derived features enables the possibility of maximizing performance by leveraging the disconnected feature sets. Evaluation on the public ICPR12 mitosis dataset that has 226 mitoses annotated on 35 High Power Fields (HPF, x400 magnification) by several pathologists and 15 testing HPFs yielded an F-measure of 0.7345. Apart from this being the second best performance ever recorded for this MITOS dataset, our approach is faster and requires fewer computing resources compared to extant methods, making this feasible for clinical use.

Co-Occurring Gland Angularity in Localized Subgraphs: Predicting Biochemical Recurrence in Intermediate-Risk Prostate Cancer Patients

Quantitative histomorphometry (QH) refers to the application of advanced computational image analysis to reproducibly describe disease appearance on digitized histopathology images. QH thus could serve as an important complementary tool for pathologists in interrogating and interpreting cancer morphology and malignancy. In the US, annually, over 60,000 prostate cancer patients undergo radical prostatectomy treatment. Around 10,000 of these men experience biochemical recurrence within 5 years of surgery, a marker for local or distant disease recurrence. The ability to predict the risk of biochemical recurrence soon after surgery could allow for adjuvant therapies to be prescribed as necessary to improve long term treatment outcomes. The underlying hypothesis with our approach, co-occurring gland angularity (CGA), is that in benign or less aggressive prostate cancer, gland orientations within local neighborhoods are similar to each other but are more chaotically arranged in aggressive disease. By modeling the extent of the disorder, we can differentiate surgically removed prostate tissue sections from (a) benign and malignant regions and (b) more and less aggressive prostate cancer. For a cohort of 40 intermediate-risk (mostly Gleason sum 7) surgically cured prostate cancer patients where half suffered biochemical recurrence, the CGA features were able to predict biochemical recurrence with 73% accuracy. Additionally, for 80 regions of interest chosen from the 40 studies, corresponding to both normal and cancerous cases, the CGA features yielded a 99% accuracy. CGAs were shown to be statistically signicantly () better at predicting BCR compared to state-of-the-art QH methods and postoperative prostate cancer nomograms.

Accurate and reproducible invasive breast cancer detection in whole-slide images: A Deep Learning approach for quantifying tumor extent

With the increasing ability to routinely and rapidly digitize whole slide images with slide scanners, there has been interest in developing computerized image analysis algorithms for automated detection of disease extent from digital pathology images. The manual identification of presence and extent of breast cancer by a pathologist is critical for patient management for tumor staging and assessing treatment response. However, this process is tedious and subject to inter- and intra-reader variability. For computerized methods to be useful as decision support tools, they need to be resilient to data acquired from different sources, different staining and cutting protocols and different scanners. The objective of this study was to evaluate the accuracy and robustness of a deep learning-based method to automatically identify the extent of invasive tumor on digitized images. Here, we present a new method that employs a convolutional neural network for detecting presence of invasive tumor on whole slide images. Our approach involves training the classifier on nearly 400 exemplars from multiple different sites, and scanners, and then independently validating on almost 200 cases from The Cancer Genome Atlas. Our approach yielded a Dice coefficient of 75.86%, a positive predictive value of 71.62% and a negative predictive value of 96.77% in terms of pixel-by-pixel evaluation compared to manually annotated regions of invasive ductal carcinoma.

Distinct microbiological signatures associated with triple negative breast cancer

Infectious agents are the third highest human cancer risk factor and may have a greater role in the origin and/or progression of cancers, and related pathogenesis. Thus, knowing the specific viruses and microbial agents associated with a cancer type may provide insights into cause, diagnosis and treatment. We utilized a pan-pathogen array technology to identify the microbial signatures associated with triple negative breast cancer (TNBC). This technology detects low copy number and fragmented genomes extracted from formalin-fixed paraffin embedded archival tissues. The results, validated by PCR and sequencing, define a microbial signature present in TNBC tissue which was underrepresented in normal tissue. Hierarchical clustering analysis displayed two broad microbial signatures, one prevalent in bacteria and parasites and one prevalent in viruses. These signatures demonstrate a new paradigm in our understanding of the link between microorganisms and cancer, as causative or commensal in the tumor microenvironment and provide new diagnostic potential.