Geert J. S. Litjens

A survey on deep learning in medical image analysis

Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, segmentation, registration, and other tasks. Concise overviews are provided of studies per application area: neuro, retinal, pulmonary, digital pathology, breast, cardiac, abdominal, musculoskeletal. We end with a summary of the current state-of-the-art, a critical discussion of open challenges and directions for future research.

Pulmonary Nodule Detection in CT Images: False Positive Reduction Using Multi-View Convolutional Networks

We propose a novel Computer-Aided Detection (CAD) system for pulmonary nodules using multi-view convolutional networks (ConvNets), for which discriminative features are automatically learnt from the training data. The network is fed with nodule candidates obtained by combining three candidate detectors specifically designed for solid, subsolid, and large nodules. For each candidate, a set of 2-D patches from differently oriented planes is extracted. The proposed architecture comprises multiple streams of 2-D ConvNets, for which the outputs are combined using a dedicated fusion method to get the final classification. Data augmentation and dropout are applied to avoid overfitting. On 888 scans of the publicly available LIDC-IDRI dataset, our method reaches high detection sensitivities of 85.4% and 90.1% at 1 and 4 false positives per scan, respectively. An additional evaluation on independent datasets from the ANODE09 challenge and DLCST is performed. We showed that the proposed multi-view ConvNets is highly suited to be used for false positive reduction of a CAD system.

Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis

Pathologists face a substantial increase in workload and complexity of histopathologic cancer diagnosis due to the advent of personalized medicine. Therefore, diagnostic protocols have to focus equally on efficiency and accuracy. In this paper we introduce ‘deep learning’ as a technique to improve the objectivity and efficiency of histopathologic slide analysis. Through two examples, prostate cancer identification in biopsy specimens and breast cancer metastasis detection in sentinel lymph nodes, we show the potential of this new methodology to reduce the workload for pathologists, while at the same time increasing objectivity of diagnoses. We found that all slides containing prostate cancer and micro- and macro-metastases of breast cancer could be identified automatically while 30–40% of the slides containing benign and normal tissue could be excluded without the use of any additional immunohistochemical markers or human intervention. We conclude that ‘deep learning’ holds great promise to improve the efficacy of prostate cancer diagnosis and breast cancer staging.

Large scale deep learning for computer aided detection of mammographic lesions

&NA; Recent advances in machine learning yielded new techniques to train deep neural networks, which resulted in highly successful applications in many pattern recognition tasks such as object detection and speech recognition. In this paper we provide a head‐to‐head comparison between a state‐of‐the art in mammography CAD system, relying on a manually designed feature set and a Convolutional Neural Network (CNN), aiming for a system that can ultimately read mammograms independently. Both systems are trained on a large data set of around 45,000 images and results show the CNN outperforms the traditional CAD system at low sensitivity and performs comparable at high sensitivity. We subsequently investigate to what extent features such as location and patient information and commonly used manual features can still complement the network and see improvements at high specificity over the CNN especially with location and context features, which contain information not available to the CNN. Additionally, a reader study was performed, where the network was compared to certified screening radiologists on a patch level and we found no significant difference between the network and the readers. HighlightsA system based on deep learning is shown to outperform a state‐of‐the art CAD system.Adding complementary handcrafted features to the CNN is shown to increase performance.The system based on deep learning is shown to perform at the level of a radiologist. Graphical abstract Figure. No caption available.

Computer-Aided Detection of Prostate Cancer in MRI

Prostate cancer is one of the major causes of cancer death for men in the western world. Magnetic resonance imaging (MRI) is being increasingly used as a modality to detect prostate cancer. Therefore, computer-aided detection of prostate cancer in MRI images has become an active area of research. In this paper we investigate a fully automated computer-aided detection system which consists of two stages. In the first stage, we detect initial candidates using multi-atlas-based prostate segmentation, voxel feature extraction, classification and local maxima detection. The second stage segments the candidate regions and using classification we obtain cancer likelihoods for each candidate. Features represent pharmacokinetic behavior, symmetry and appearance, among others. The system is evaluated on a large consecutive cohort of 347 patients with MR-guided biopsy as the reference standard. This set contained 165 patients with cancer and 182 patients without prostate cancer. Performance evaluation is based on lesion-based free-response receiver operating characteristic curve and patient-based receiver operating characteristic analysis. The system is also compared to the prospective clinical performance of radiologists. Results show a sensitivity of 0.42, 0.75, and 0.89 at 0.1, 1, and 10 false positives per normal case. In clinical workflow the system could potentially be used to improve the sensitivity of the radiologist. At the high specificity reading setting, which is typical in screening situations, the system does not perform significantly different from the radiologist and could be used as an independent second reader instead of a second radiologist. Furthermore, the system has potential in a first-reader setting.

Assessment of Prostate Cancer Aggressiveness Using Dynamic Contrast-enhanced Magnetic Resonance Imaging at 3 T

BACKGROUND A challenge in the diagnosis of prostate cancer (PCa) is the accurate assessment of aggressiveness. OBJECTIVE To validate the performance of dynamic contrast-enhanced (DCE) magnetic resonance imaging (MRI) of the prostate at 3 tesla (T) for the assessment of PCa aggressiveness, with prostatectomy specimens as the reference standard. DESIGN, SETTINGS, AND PARTICIPANTS A total of 45 patients with PCa scheduled for prostatectomy were included. This study was approved by the institutional review board; the need for informed consent was waived. OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS Subjects underwent a clinical MRI protocol including DCE-MRI. Blinded to DCE-images, PCa was indicated on T2-weighted images based on histopathology results from prostatectomy specimens with the use of anatomical landmarks for the precise localization of the tumor. PCa was classified as low-, intermediate-, or high-grade, according to Gleason score. DCE-images were used as an overlay on T2-weighted images; mean and quartile values from semi-quantitative and pharmacokinetic model parameters were extracted per tumor region. Statistical analysis included Spearmans ρ, the Kruskal-Wallis test, and a receiver operating characteristics (ROC) analysis. RESULTS AND LIMITATIONS Significant differences were seen for the mean and 75th percentile (p75) values of wash-in (p = 0.024 and p = 0.017, respectively), mean wash-out (p = 0.044), and p75 of transfer constant (K(trans)) (p = 0.035), all between low-grade and high-grade PCa in the peripheral zone. ROC analysis revealed the best discriminating performance between low-grade versus intermediate-grade plus high-grade PCa in the peripheral zone for p75 of wash-in, K(trans), and rate constant (Kep) (area under the curve: 0.72). Due to a limited number of tumors in the transition zone, a definitive conclusion for this region of the prostate could not be drawn. CONCLUSIONS Quantitative parameters (K(trans) and Kep) and semi-quantitative parameters (wash-in and wash-out) derived from DCE-MRI at 3 T have the potential to assess the aggressiveness of PCa in the peripheral zone. P75 of wash-in, K(trans), and Kep offer the best possibility to discriminate low-grade from intermediate-grade plus high-grade PCa.

Evaluation of prostate segmentation algorithms for MRI: the PROMISE12 challenge

Prostate MRI image segmentation has been an area of intense research due to the increased use of MRI as a modality for the clinical workup of prostate cancer. Segmentation is useful for various tasks, e.g. to accurately localize prostate boundaries for radiotherapy or to initialize multi-modal registration algorithms. In the past, it has been difficult for research groups to evaluate prostate segmentation algorithms on multi-center, multi-vendor and multi-protocol data. Especially because we are dealing with MR images, image appearance, resolution and the presence of artifacts are affected by differences in scanners and/or protocols, which in turn can have a large influence on algorithm accuracy. The Prostate MR Image Segmentation (PROMISE12) challenge was setup to allow a fair and meaningful comparison of segmentation methods on the basis of performance and robustness. In this work we will discuss the initial results of the online PROMISE12 challenge, and the results obtained in the live challenge workshop hosted by the MICCAI2012 conference. In the challenge, 100 prostate MR cases from 4 different centers were included, with differences in scanner manufacturer, field strength and protocol. A total of 11 teams from academic research groups and industry participated. Algorithms showed a wide variety in methods and implementation, including active appearance models, atlas registration and level sets. Evaluation was performed using boundary and volume based metrics which were combined into a single score relating the metrics to human expert performance. The winners of the challenge where the algorithms by teams Imorphics and ScrAutoProstate, with scores of 85.72 and 84.29 overall. Both algorithms where significantly better than all other algorithms in the challenge (p<0.05) and had an efficient implementation with a run time of 8min and 3s per case respectively. Overall, active appearance model based approaches seemed to outperform other approaches like multi-atlas registration, both on accuracy and computation time. Although average algorithm performance was good to excellent and the Imorphics algorithm outperformed the second observer on average, we showed that algorithm combination might lead to further improvement, indicating that optimal performance for prostate segmentation is not yet obtained. All results are available online at http://promise12.grand-challenge.org/.

Differentiation of Prostatitis and Prostate Cancer by Using Diffusion-weighted MR Imaging and MR-guided Biopsy at 3 T

PURPOSE To determine if prostatitis and prostate cancer (PCa) can be distinguished by using apparent diffusion coefficients (ADCs) on magnetic resonance (MR) images, with specimens obtained at MR-guided biopsy as the standard of reference. MATERIALS AND METHODS The need for institutional review board approval and informed consent was waived. MR-guided biopsies were performed in 130 consecutive patients with cancer-suspicious regions (CSRs) on multiparametric MR images obtained at 3 T. In this retrospective study, 88 patients met the inclusion criteria. During the biopsy procedure, an axial diffusion-weighted sequence was performed and ADC maps were generated (repetition time msec/echo time msec, 2000/67; section thickness, 4 mm; in-plane resolution, 1.8 × 1.8 mm; and b values of 0, 100, 500, and 800 sec/mm(2)). Subsequently, a confirmation image with the needle left in situ was acquired and projected on the ADC map. The corresponding ADCs at the biopsy location were compared with the histopathologic outcomes of the biopsy specimens. Linear mixed-model regression analyses were used to test for ADC differences between the histopathologic groups. RESULTS The study included 116 biopsy specimens. Median ADCs of normal prostate tissue, prostatitis, low-grade PCa (Gleason grade components 2 or 3), and high-grade PCa (Gleason grade components 4 or 5) were 1.22 × 10(-3) mm(2)/sec (standard deviation, ± 0.21), 1.08 × 10(-3) mm(2)/sec (± 0.18), 0.88 × 10(-3) mm(2)/sec (± 0.15), and 0.88 × 10(-3) mm(2)/sec (± 0.13), respectively. Although the median ADCs of biopsy specimens with prostatitis were significantly higher compared with low- and high-grade PCa (P < .001), there is a considerable overlap between the tissue types. CONCLUSION Diffusion-weighted imaging is a noninvasive technique that shows differences between prostatitis and PCa in both the peripheral zone and central gland, although its usability in clinical practice is limited as a result of significant overlap in ADCs.

Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer

Importance Application of deep learning algorithms to whole-slide pathology images can potentially improve diagnostic accuracy and efficiency. Objective Assess the performance of automated deep learning algorithms at detecting metastases in hematoxylin and eosin–stained tissue sections of lymph nodes of women with breast cancer and compare it with pathologists’ diagnoses in a diagnostic setting. Design, Setting, and Participants Researcher challenge competition (CAMELYON16) to develop automated solutions for detecting lymph node metastases (November 2015-November 2016). A training data set of whole-slide images from 2 centers in the Netherlands with (n = 110) and without (n = 160) nodal metastases verified by immunohistochemical staining were provided to challenge participants to build algorithms. Algorithm performance was evaluated in an independent test set of 129 whole-slide images (49 with and 80 without metastases). The same test set of corresponding glass slides was also evaluated by a panel of 11 pathologists with time constraint (WTC) from the Netherlands to ascertain likelihood of nodal metastases for each slide in a flexible 2-hour session, simulating routine pathology workflow, and by 1 pathologist without time constraint (WOTC). Exposures Deep learning algorithms submitted as part of a challenge competition or pathologist interpretation. Main Outcomes and Measures The presence of specific metastatic foci and the absence vs presence of lymph node metastasis in a slide or image using receiver operating characteristic curve analysis. The 11 pathologists participating in the simulation exercise rated their diagnostic confidence as definitely normal, probably normal, equivocal, probably tumor, or definitely tumor. Results The area under the receiver operating characteristic curve (AUC) for the algorithms ranged from 0.556 to 0.994. The top-performing algorithm achieved a lesion-level, true-positive fraction comparable with that of the pathologist WOTC (72.4% [95% CI, 64.3%-80.4%]) at a mean of 0.0125 false-positives per normal whole-slide image. For the whole-slide image classification task, the best algorithm (AUC, 0.994 [95% CI, 0.983-0.999]) performed significantly better than the pathologists WTC in a diagnostic simulation (mean AUC, 0.810 [range, 0.738-0.884]; P < .001). The top 5 algorithms had a mean AUC that was comparable with the pathologist interpreting the slides in the absence of time constraints (mean AUC, 0.960 [range, 0.923-0.994] for the top 5 algorithms vs 0.966 [95% CI, 0.927-0.998] for the pathologist WOTC). Conclusions and Relevance In the setting of a challenge competition, some deep learning algorithms achieved better diagnostic performance than a panel of 11 pathologists participating in a simulation exercise designed to mimic routine pathology workflow; algorithm performance was comparable with an expert pathologist interpreting whole-slide images without time constraints. Whether this approach has clinical utility will require evaluation in a clinical setting.

Stain Specific Standardization of Whole-Slide Histopathological Images

Variations in the color and intensity of hematoxylin and eosin (H&E) stained histological slides can potentially hamper the effectiveness of quantitative image analysis. This paper presents a fully automated algorithm for standardization of whole-slide histopathological images to reduce the effect of these variations. The proposed algorithm, called whole-slide image color standardizer (WSICS), utilizes color and spatial information to classify the image pixels into different stain components. The chromatic and density distributions for each of the stain components in the hue-saturation-density color model are aligned to match the corresponding distributions from a template whole-slide image (WSI). The performance of the WSICS algorithm was evaluated on two datasets. The first originated from 125 H&E stained WSIs of lymph nodes, sampled from 3 patients, and stained in 5 different laboratories on different days of the week. The second comprised 30 H&E stained WSIs of rat liver sections. The result of qualitative and quantitative evaluations using the first dataset demonstrate that the WSICS algorithm outperforms competing methods in terms of achieving color constancy. The WSICS algorithm consistently yields the smallest standard deviation and coefficient of variation of the normalized median intensity measure. Using the second dataset, we evaluated the impact of our algorithm on the performance of an already published necrosis quantification system. The performance of this system was significantly improved by utilizing the WSICS algorithm. The results of the empirical evaluations collectively demonstrate the potential contribution of the proposed standardization algorithm to improved diagnostic accuracy and consistency in computer-aided diagnosis for histopathology data.