Ronald M. Summers

Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning

Remarkable progress has been made in image recognition, primarily due to the availability of large-scale annotated datasets and deep convolutional neural networks (CNNs). CNNs enable learning data-driven, highly representative, hierarchical image features from sufficient training data. However, obtaining datasets as comprehensively annotated as ImageNet in the medical imaging domain remains a challenge. There are currently three major techniques that successfully employ CNNs to medical image classification: training the CNN from scratch, using off-the-shelf pre-trained CNN features, and conducting unsupervised CNN pre-training with supervised fine-tuning. Another effective method is transfer learning, i.e., fine-tuning CNN models pre-trained from natural image dataset to medical image tasks. In this paper, we exploit three important, but previously understudied factors of employing deep convolutional neural networks to computer-aided detection problems. We first explore and evaluate different CNN architectures. The studied models contain 5 thousand to 160 million parameters, and vary in numbers of layers. We then evaluate the influence of dataset scale and spatial image context on performance. Finally, we examine when and why transfer learning from pre-trained ImageNet (via fine-tuning) can be useful. We study two specific computer-aided detection (CADe) problems, namely thoraco-abdominal lymph node (LN) detection and interstitial lung disease (ILD) classification. We achieve the state-of-the-art performance on the mediastinal LN detection, and report the first five-fold cross-validation classification results on predicting axial CT slices with ILD categories. Our extensive empirical evaluation, CNN model analysis and valuable insights can be extended to the design of high performance CAD systems for other medical imaging tasks.Remarkable progress has been made in image recognition, primarily due to the availability of large-scale annotated datasets and deep convolutional neural networks (CNNs). CNNs enable learning data-driven, highly representative, hierarchical image features from sufficient training data. However, obtaining datasets as comprehensively annotated as ImageNet in the medical imaging domain remains a challenge. There are currently three major techniques that successfully employ CNNs to medical image classification: training the CNN from scratch, using off-the-shelf pre-trained CNN features, and conducting unsupervised CNN pre-training with supervised fine-tuning. Another effective method is transfer learning, i.e., fine-tuning CNN models pre-trained from natural image dataset to medical image tasks. In this paper, we exploit three important, but previously understudied factors of employing deep convolutional neural networks to computer-aided detection problems. We first explore and evaluate different CNN architectures. The studied models contain 5 thousand to 160 million parameters, and vary in numbers of layers. We then evaluate the influence of dataset scale and spatial image context on performance. Finally, we examine when and why transfer learning from pre-trained ImageNet (via fine-tuning) can be useful. We study two specific computer-aided detection (CADe) problems, namely thoraco-abdominal lymph node (LN) detection and interstitial lung disease (ILD) classification. We achieve the state-of-the-art performance on the mediastinal LN detection, and report the first five-fold cross-validation classification results on predicting axial CT slices with ILD categories. Our extensive empirical evaluation, CNN model analysis and valuable insights can be extended to the design of high performance CAD systems for other medical imaging tasks.

Guest Editorial Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique

The papers in this special section focus on the technology and applications supported by deep learning. Deep learning is a growing trend in general data analysis and has been termed one of the 10 breakthrough technologies of 2013. Deep learning is an improvement of artificial neural networks, consisting of more layers that permit higher levels of abstraction and improved predictions from data. To date, it is emerging as the leading machine-learning tool in the general imaging and computer vision domains. In particular, convolutional neural networks (CNNs) have proven to be powerful tools for a broad range of computer vision tasks. Deep CNNs automatically learn mid-level and high-level abstractions obtained from raw data (e.g., images). Recent results indicate that the generic descriptors extracted from CNNs are extremely effective in object recognition and localization in natural images. Medical image analysis groups across the world are quickly entering the field and applying CNNs and other deep learning methodologies to a wide variety of applications.

Machine learning and radiology

In this paper, we give a short introduction to machine learning and survey its applications in radiology. We focused on six categories of applications in radiology: medical image segmentation, registration, computer aided detection and diagnosis, brain function or activity analysis and neurological disease diagnosis from fMR images, content-based image retrieval systems for CT or MRI images, and text analysis of radiology reports using natural language processing (NLP) and natural language understanding (NLU). This survey shows that machine learning plays a key role in many radiology applications. Machine learning identifies complex patterns automatically and helps radiologists make intelligent decisions on radiology data such as conventional radiographs, CT, MRI, and PET images and radiology reports. In many applications, the performance of machine learning-based automatic detection and diagnosis systems has shown to be comparable to that of a well-trained and experienced radiologist. Technology development in machine learning and radiology will benefit from each other in the long run. Key contributions and common characteristics of machine learning techniques in radiology are discussed. We also discuss the problem of translating machine learning applications to the radiology clinical setting, including advantages and potential barriers.

Gray-scale skeletonization of small vessels in magnetic resonance angiography

Interpretation of magnetic resonance angiography (MRA) is problematic due to complexities of vascular shape and to artifacts such as the partial volume effect. The authors present new methods to assist in the interpretation of MRA. These include methods for detection of vessel paths and for determination of branching patterns of vascular trees. They are based on the ordered region growing (ORG) algorithm that represents the image as an acyclic graph, which can be reduced to a skeleton by specifying vessel endpoints or by a pruning process. Ambiguities in the vessel branching due to vessel overlap are effectively resolved by heuristic methods that incorporate a priori knowledge of bifurcation spacing. Vessel paths are detected at interactive speeds on a 500-MHz processor using vessel endpoints. These methods apply best to smaller vessels where the image intensity peaks at the center of the lumen which, for the abdominal MRA, includes vessels whose diameter is less than 1 cm.

A New 2.5D Representation for Lymph Node Detection Using Random Sets of Deep Convolutional Neural Network Observations

Automated Lymph Node (LN) detection is an important clinical diagnostic task but very challenging due to the low contrast of surrounding structures in Computed Tomography (CT) and to their varying sizes, poses, shapes and sparsely distributed locations. State-of-the-art studies show the performance range of 52.9% sensitivity at 3.1 false-positives per volume (FP/vol.), or 60.9% at 6.1 FP/vol. for mediastinal LN, by one-shot boosting on 3D HAAR features. In this paper, we first operate a preliminary candidate generation stage, towards -100% sensitivity at the cost of high FP levels (-40 per patient), to harvest volumes of interest (VOI). Our 2.5D approach consequently decomposes any 3D VOI by resampling 2D reformatted orthogonal views N times, via scale, random translations, and rotations with respect to the VOI centroid coordinates. These random views are then used to train a deep Convolutional Neural Network (CNN) classifier. In testing, the CNN is employed to assign LN probabilities for all N random views that can be simply averaged (as a set) to compute the final classification probability per VOI. We validate the approach on two datasets: 90 CT volumes with 388 mediastinal LNs and 86 patients with 595 abdominal LNs. We achieve sensitivities of 70%/83% at 3 FP/vol. and 84%/90% at 6 FP/vol. in mediastinum and abdomen respectively, which drastically improves over the previous state-of-the-art work.

Improving Computer-Aided Detection Using Convolutional Neural Networks and Random View Aggregation

Automated computer-aided detection (CADe) has been an important tool in clinical practice and research. State-of-the-art methods often show high sensitivities at the cost of high false-positives (FP) per patient rates. We design a two-tiered coarse-to-fine cascade framework that first operates a candidate generation system at sensitivities ~ 100% of but at high FP levels. By leveraging existing CADe systems, coordinates of regions or volumes of interest (ROI or VOI) are generated and function as input for a second tier, which is our focus in this study. In this second stage, we generate 2D (two-dimensional) or 2.5D views via sampling through scale transformations, random translations and rotations. These random views are used to train deep convolutional neural network (ConvNet) classifiers. In testing, the ConvNets assign class (e.g., lesion, pathology) probabilities for a new set of random views that are then averaged to compute a final per-candidate classification probability. This second tier behaves as a highly selective process to reject difficult false positives while preserving high sensitivities. The methods are evaluated on three data sets: 59 patients for sclerotic metastasis detection, 176 patients for lymph node detection, and 1,186 patients for colonic polyp detection. Experimental results show the ability of ConvNets to generalize well to different medical imaging CADe applications and scale elegantly to various data sets. Our proposed methods improve performance markedly in all cases. Sensitivities improved from 57% to 70%, 43% to 77%, and 58% to 75% at 3 FPs per patient for sclerotic metastases, lymph nodes and colonic polyps, respectively.

Stress-induced reversible and mild-to-moderate irreversible thallium defects : Are they equally accurate for predicting recovery of regional left ventricular function after revascularization?

BACKGROUND In patients with coronary artery disease, stress-redistribution-reinjection thallium scintigraphy provides important information regarding myocardial ischemia and viability. Although both reversible and mild-to-moderate irreversible thallium defects retain metabolically active, viable myocardium, we hypothesized that stress-induced reversible thallium defects may better differentiate reversible from irreversible regional left ventricular dysfunction after revascularization. METHODS AND RESULTS Twenty-four patients with chronic coronary artery disease underwent prerevascularization and postrevascularization exercise-redistribution-reinjection thallium single photon emission CT, gated MRI, and radionuclide angiography. After revascularization, mean left ventricular ejection fraction increased from 30+/-9% to 37+/-13% at rest (P<0.001). Before revascularization, abnormal contraction at rest was observed in 56 of 110 reversible and 20 of 37 mild-to-moderate irreversible thallium defects (51% and 54%, respectively). After revascularization, regional contraction improved in 44 of 56 reversible compared with 6 of 20 mild-to-moderate irreversible thallium defects (79% and 30%, respectively; P<0.001). The final thallium content (maximum tracer uptake on redistribution-reinjection images) was significantly higher in regions with reversible defects that improved than in those that did not improve after revascularization (86+/-16% versus 66+/-9%, P<0.001). In contrast, final thallium content was similar in regions with mild-to-moderate irreversible defects that improved and in those that did not improve after revascularization (69+/-9% versus 65+/-10%, P=NS). Furthermore, when asynergic regions were grouped according to the final thallium content, at 60% threshold value, functional recovery was observed in 83% of regions with reversible defects compared with 33% of regions with mild-to-moderate irreversible defects (P<0.001). CONCLUSIONS These findings suggest that although both reversible and mild-to-moderate irreversible thallium defects after stress retain viable myocardium, the identification of reversible thallium defect on stress in an asynergic region more accurately predicts recovery of function after revascularization. Even at a similar mass of viable myocardial tissue (as reflected by the final thallium content), the presence of inducible ischemia is associated with an increased likelihood of functional recovery.

Automated spinal column extraction and partitioning

This paper presents an approach to automatically segment and partition the spinal column from routine 5 mm chest and/or abdominal CT images. The segmented spinal column has great value in image registration, content based image retrieval, spine deformity analysis, and organ localization. In our method, first a simple thresholding is employed to obtain the initial spine segmentation. Then a hybrid method based on the watershed algorithm and directed graph search is applied to extract the spinal canal. After that, a four-part vertebra model (vertebral body, spinous process, and left/right transverse processes) is fitted to segment the vertebral region and separate it from adjacent ribs and other structures. Curved reformations in sagittal and coronal directions are generated and aggregated intensity profiles along the spinal cord are analyzed to partition the spinal column into vertebrae. The algorithm has been tested on 71 CT scans. Results showed that our algorithm successfully extracted and partitioned 69 spinal columns, with only 2 cases that had one missed partition at the T1-T2 level

Magnetic resonance imaging detection of occult skin and subcutaneous abnormalities in juvenile dermatomyositis : Implications for diagnosis and therapy

OBJECTIVE To assess the utility of magnetic resonance imaging (MRI) of skin, subcutaneous tissue, and fascia in evaluating disease activity in juvenile dermatomyositis (DM). METHODS Short tau inversion recovery (STIR) MRI of the proximal thighs and buttocks, cutaneous assessment, and other measures of disease activity were prospectively obtained in 26 children meeting criteria for probable or definite juvenile DM. Also undergoing STIR MRI assessment were 8 subjects who were being evaluated for muscle disorders and who were not diagnosed as having juvenile DM. RESULTS Skin, subcutaneous, or fascial edema of the thighs and buttocks were seen on STIR MRI in up to 85% of juvenile DM patients at baseline evaluation compared with no more than 38% of the comparison group without juvenile DM. In juvenile DM, STIR MRI skin and subcutaneous edema scores correlated (r(s) = 0.51, P = 0.008), as did fascial and muscle edema scores (r(s) = 0.58, P = 0.002). Skin global disease activity scores correlated with MRI skin edema scores (r(s) = 0.41, P = 0.04), and serum aldolase levels correlated with both MRI skin and subcutaneous edema scores (r = 0.44 and 0.40, P = 0.03 and 0.05 respectively). The extent and severity of STIR MRI changes in the skin, subcutaneous tissue, and fascia were not predicted by most other measures of juvenile DM disease activity. Five juvenile DM patients with thigh MRI subcutaneous edema developed clinically apparent calcinosis at the same location within 9 months. CONCLUSION Edema or inflammation in the skin, subcutaneous tissue, and fascia, found on STIR MRI, is common in juvenile DM patients and is often undetected by standard assessments. These MRI changes can precede the development of calcinosis. STIR MRI may be a useful adjunct for assessing disease activity and guiding the treatment of juvenile DM.

Colonic polyp segmentation in CT colonography-based on fuzzy clustering and deformable models

An automatic method to segment colonic polyps in computed tomography (CT) colonography is presented in this paper. The method is based on a combination of knowledge-guided intensity adjustment, fuzzy c-mean clustering, and deformable models. The computer segmentations were compared with manual segmentations to validate the accuracy of our method. An average 76.3% volume overlap percentage among 105 polyp detections was reported in the validation, which was very good considering the small polyp size. Several experiments were performed to investigate the intraoperator and interoperator repeatability of manual colonic polyp segmentation. The investigation demonstrated that the computer-human repeatability was as good as the interoperator repeatability. The polyp segmentation was also applied in computer-aided detection (CAD) to reduce the number of false positive (FP) detections and provide volumetric features for polyp classification. Our segmentation method was able to eliminate 30% of FP detections. The volumetric features computed from the segmentation can further reduce FP detections by 50% at 80% sensitivity.