Jeroen van der Laak

A survey on deep learning in medical image analysis

Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, segmentation, registration, and other tasks. Concise overviews are provided of studies per application area: neuro, retinal, pulmonary, digital pathology, breast, cardiac, abdominal, musculoskeletal. We end with a summary of the current state-of-the-art, a critical discussion of open challenges and directions for future research.

Skeletal Muscle Ultrasound: Correlation Between Fibrous Tissue and Echo Intensity

In this study, we examined the correlation between muscle ultrasound and muscle structure. Echo intensity (EI) of 14 muscles of two golden retriever muscular dystrophy dogs was correlated to the percentage interstitial fibrous tissue and fat in muscle biopsy. A significant correlation between interstitial fibrous tissue and EI was found (r = 0.87; p < 0.001). The separate influence of interstitial fat on muscle EI could not be established as only little fat was present. We conclude that fibrous tissue causes increased muscle EI. The high correlation between interstitial fibrous tissue and EI makes ultrasound a reliable method to determine severity of structural muscle changes.

Peroxisome proliferator-activated receptor gamma activation promotes infiltration of alternatively activated macrophages into adipose tissue.

Obesity is associated with infiltration of macrophages into adipose tissue. Adipose macrophages may contribute to an elevated inflammatory status by secreting a variety of proinflammatory mediators, including tumor necrosis factor α and interleukin-6 (IL-6). Recent data suggest that during diet-induced obesity the phenotype of adipose-resident macrophages changes from alternatively activated macrophages toward a more classical and pro-inflammatory phenotype. Here, we explore the effect of peroxisome proliferator-activated receptor γ activation on obesity-induced inflammation in 129SV mice fed a high fat diet for 20 weeks. High fat feeding increased bodyweight gain, adipose tissue mass, and liver triglycerides. Rosiglitazone treatment further increased adipose mass, reduced liver triglycerides, and changed adipose tissue morphology toward smaller adipocytes. Surprisingly, rosiglitazone markedly increased the number of macrophages in adipose tissue, as shown by immunohistochemical analysis and quantification of macrophage marker genes CD68 and F4/80+. In adipose tissue, markers for classically activated macrophages including IL-18 were down-regulated, whereas markers characteristic for alternatively activated macrophages (arginase 1, IL-10) were up-regulated by rosiglitazone. Importantly, conditioned media from rosiglitazone-treated alternatively activated macrophages neutralized the inhibitory effect of macrophages on 3T3-L1 adipocyte differentiation, suggesting that alternatively activated macrophages may be involved in mediating the effects of rosiglitazone on adipose tissue morphology and mass. Our results suggest that short term rosiglitazone treatment increases infiltration of alternatively activated macrophages in adipose tissue. The alternatively activated macrophages might play a role in peroxisome proliferator-activated receptorγ-dependent expansion and remodeling of adipose tissue.

Ki-67 as a prognostic marker in mantle cell lymphoma—consensus guidelines of the pathology panel of the European MCL Network

Mantle cell lymphoma (MCL) has a heterogeneous clinical course and is mainly an aggressive B cell non-Hodgkin lymphoma; however, there are some indolent cases The Ki-67 index, defined by the percentage of Ki-67-positive lymphoma cells on histopathological slides, has been shown to be a very powerful prognostic biomarker. The pathology panel of the European MCL Network evaluated methods to assess the Ki-67 index including stringent counting, digital image analysis, and estimation by eyeballing. Counting of 2 × 500 lymphoma cells is the gold standard to assess the Ki-67 index since this value has been shown to predict survival in prospective randomized trials of the European MCL Network. Estimation by eyeballing and digital image analysis showed a poor concordance with the gold standard (concordance correlation coefficients [CCC] between 0.29 and 0.61 for eyeballing and CCC of 0.24 and 0.37 for two methods of digital image analysis, respectively). Counting a reduced number of lymphoma cells (2 × 100 cells) showed high interobserver agreement (CCC = 0.74). Pitfalls of the Ki-67 index are discussed and guidelines and recommendations for assessing the Ki-67 index in MCL are given.

Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis

Pathologists face a substantial increase in workload and complexity of histopathologic cancer diagnosis due to the advent of personalized medicine. Therefore, diagnostic protocols have to focus equally on efficiency and accuracy. In this paper we introduce ‘deep learning’ as a technique to improve the objectivity and efficiency of histopathologic slide analysis. Through two examples, prostate cancer identification in biopsy specimens and breast cancer metastasis detection in sentinel lymph nodes, we show the potential of this new methodology to reduce the workload for pathologists, while at the same time increasing objectivity of diagnoses. We found that all slides containing prostate cancer and micro- and macro-metastases of breast cancer could be identified automatically while 30–40% of the slides containing benign and normal tissue could be excluded without the use of any additional immunohistochemical markers or human intervention. We conclude that ‘deep learning’ holds great promise to improve the efficacy of prostate cancer diagnosis and breast cancer staging.

Microscopic analysis of anastomotic healing in the intestine of normal and diabetic rats.

PURPOSE: The mechanisms that cause diabetes to impair the development of anastomotic strength in the intestine are poorly understood. We investigated whether short-term uncontrolled diabetes causes alterations in microscopic aspects of anastomoses from the ileum and colon. METHODS: Eighteen Wistar rats were rendered diabetic one week before operation by intravenous streptozotocin injection (50 mg/kg), resulting in nonfasting blood glucose levels of approximately 20 mmol/l. Another 18 age-matched rats were used as controls with a normal blood glucose range of 5 to 7 mmol/l. All rats underwent resection and anastomosis of both the ileum and colon. Animals were killed at one, three, or seven days after operation. Cellular and architectural parameters of anastomotic healing were scored in hematoxylin and eosin-stained sections. Anastomotic collagen content was analyzed by image analysis in picrosirius red-stained sections. RESULTS: Anastomotic necrosis, edema, and epithelial recovery were not affected by diabetes. In diabetic rats, the number of polymorphonuclear cells and macrophages was significantly (P=0.025 and 0.0002, respectively) increased in ileal anastomoses one and three days after operation. In colonic anastomoses, the number of polymorphonuclear cells was increased at one (P=0.001) and seven (P=0.014) days after operation. Repair of the submucosal-muscular layer in colonic anastomoses from diabetic rats was impaired seven days after surgery (P=0.0071), but in ileal anastomoses no difference was found. In the anastomotic area, collagen deposition at postoperative Days 1, 3, and 7 remained unaffected by diabetes. CONCLUSION: Experimental diabetes leads to alterations in cellular components involved in the early phase of repair of intestinal anastomoses but not to a reduced accumulation of wound collagen.

Hue-Saturation-Density (HSD) Model for Stain Recognition in Digital Images From Transmitted Light Microscopy

BACKGROUND Transmitted light microscopy is used in pathology to examine stained tissues. Digital image analysis is gaining importance as a means to quantify alterations in tissues. A prerequisite for accurate and reproducible quantification is the possibility to recognise stains in a standardised manner, independently of variations in the staining density. METHODS The usefulness of three colour models was studied using data from computer simulations and experimental data from an immuno-doublestained tissue section. Direct use of the three intensities obtained by a colour camera results in the red-green-blue (RGB) model. By decoupling the intensity from the RGB data, the hue-saturation-intensity (HSI) model is obtained. However, the major part of the variation in perceived intensities in transmitted light microscopy is caused by variations in staining density. Therefore, the hue-saturation-density (HSD) transform was defined as the RGB to HSI transform, applied to optical density values rather than intensities for the individual RGB channels. RESULTS In the RGB model, the mixture of chromatic and intensity information hampers standardisation of stain recognition. In the HSI model, mixtures of stains that could be distinguished from other stains in the RGB model could not be separated. The HSD model enabled all possible distinctions in a two-dimensional, standardised data space. CONCLUSIONS In the RGB model, standardised recognition is only possible by using complex and time-consuming algorithms. The HSI model is not suitable for stain recognition in transmitted light microscopy. The newly derived HSD model was found superior to the existing models for this purpose.

Intratumoral Recombinant Human Interleukin-12 Administration in Head and Neck Squamous Cell Carcinoma Patients Modifies Locoregional Lymph Node Architecture and Induces Natural Killer Cell Infiltration in the Primary Tumor

The objective of this study was to evaluate the histologic and immunohistopathologic effects of intratumorally given recombinant human interleukin-12 on the immune cells in the primary tumors and regional lymph nodes. Ten previously untreated patients with head and neck squamous cell carcinoma (HNSCC) were injected in the primary tumor twice to thrice, once weekly, at two dose levels of 100 or 300 ng/kg, before surgery. These patients were compared with 20 non-IL-12-treated control HNSCC patients. In the primary tumor, the number of CD56+ natural killer (NK) cells was increased in IL-12-treated patients compared with control patients. In some IL-12-treated patients, an impressive peritumoral invasion of CD20+ B cells was noticed. No differences were seen in the CD8+ or CD4+ T lymphocytes. Interestingly, major differences were apparent in the architecture of the enlarged lymph nodes of IL-12-treated patients; in particular, the distribution of B cells differed and fewer primary and secondary follicles with smaller germinal centers were observed. In addition, a decrease of dendritic cell lysosyme-associated membrane glycoprotein–positive cells in the paracortex was noted, resulting in a reduction of paracortical hyperplasia. In the lymph nodes, especially the CD56+ NK cells but also the CD8+ and CD4+ T lymphocytes, produced a high amount of IFN-γ. Patients, irrespectively of IL-12 treatment, with a high number of CD56+ cells in the primary tumor had a better overall survival than those with a low number. In conclusion, after i.t. IL-12 treatment in HNSCC patients, the largest effect was seen on the NK cells, with a higher number in the primary tumor and a high IFN-γ mRNA expression in the lymph nodes. Significant effects were noted on B cells, with altered lymph node architecture in every IL-12-treated patient and excessive peritumoral infiltration in some patients.

MIB1, A PROMISING MARKER FOR THE CLASSIFICATION OF CERVICAL INTRAEPITHELIAL NEOPLASIA

Formalin‐fixed and paraffin‐embedded tissue specimens of normal and dysplastic cervical epithelia (five CIN1, seven CIN2, five CIN3, and five normal) were assessed by an immunoperoxidase technique, using the monoclonal antibody MIB1, regonizing a formalin‐fixation‐resistant epitope on the cell proliferation‐associated Ki‐67 antigen. An image analysis system was used to measure four parameters associated with proliferative activity: the Ki‐67 labelling index (LI), the number of Ki‐67‐positive nuclei per unit length of basement membrane, and the maximum value and 90th percentile of the relative distances of Ki‐67‐positive nuclei from the basement membrane. All these four proliferation‐related parameters were highly correlated with the grade of dysplastic change in the epithelium (0·90

Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer

Importance Application of deep learning algorithms to whole-slide pathology images can potentially improve diagnostic accuracy and efficiency. Objective Assess the performance of automated deep learning algorithms at detecting metastases in hematoxylin and eosin–stained tissue sections of lymph nodes of women with breast cancer and compare it with pathologists’ diagnoses in a diagnostic setting. Design, Setting, and Participants Researcher challenge competition (CAMELYON16) to develop automated solutions for detecting lymph node metastases (November 2015-November 2016). A training data set of whole-slide images from 2 centers in the Netherlands with (n = 110) and without (n = 160) nodal metastases verified by immunohistochemical staining were provided to challenge participants to build algorithms. Algorithm performance was evaluated in an independent test set of 129 whole-slide images (49 with and 80 without metastases). The same test set of corresponding glass slides was also evaluated by a panel of 11 pathologists with time constraint (WTC) from the Netherlands to ascertain likelihood of nodal metastases for each slide in a flexible 2-hour session, simulating routine pathology workflow, and by 1 pathologist without time constraint (WOTC). Exposures Deep learning algorithms submitted as part of a challenge competition or pathologist interpretation. Main Outcomes and Measures The presence of specific metastatic foci and the absence vs presence of lymph node metastasis in a slide or image using receiver operating characteristic curve analysis. The 11 pathologists participating in the simulation exercise rated their diagnostic confidence as definitely normal, probably normal, equivocal, probably tumor, or definitely tumor. Results The area under the receiver operating characteristic curve (AUC) for the algorithms ranged from 0.556 to 0.994. The top-performing algorithm achieved a lesion-level, true-positive fraction comparable with that of the pathologist WOTC (72.4% [95% CI, 64.3%-80.4%]) at a mean of 0.0125 false-positives per normal whole-slide image. For the whole-slide image classification task, the best algorithm (AUC, 0.994 [95% CI, 0.983-0.999]) performed significantly better than the pathologists WTC in a diagnostic simulation (mean AUC, 0.810 [range, 0.738-0.884]; P < .001). The top 5 algorithms had a mean AUC that was comparable with the pathologist interpreting the slides in the absence of time constraints (mean AUC, 0.960 [range, 0.923-0.994] for the top 5 algorithms vs 0.966 [95% CI, 0.927-0.998] for the pathologist WOTC). Conclusions and Relevance In the setting of a challenge competition, some deep learning algorithms achieved better diagnostic performance than a panel of 11 pathologists participating in a simulation exercise designed to mimic routine pathology workflow; algorithm performance was comparable with an expert pathologist interpreting whole-slide images without time constraints. Whether this approach has clinical utility will require evaluation in a clinical setting.