Jianming Liang

Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?

Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch? To address this question, we considered four distinct medical imaging applications in three specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from three different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that 1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; 2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; 3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and 4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch? To address this question, we considered four distinct medical imaging applications in three specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from three different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that 1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; 2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; 3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and 4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.

Automated Polyp Detection in Colonoscopy Videos Using Shape and Context Information

This paper presents the culmination of our research in designing a system for computer-aided detection (CAD) of polyps in colonoscopy videos. Our system is based on a hybrid context-shape approach, which utilizes context information to remove non-polyp structures and shape information to reliably localize polyps. Specifically, given a colonoscopy image, we first obtain a crude edge map. Second, we remove non-polyp edges from the edge map using our unique feature extraction and edge classification scheme. Third, we localize polyp candidates with probabilistic confidence scores in the refined edge maps using our novel voting scheme. The suggested CAD system has been tested using two public polyp databases, CVC-ColonDB, containing 300 colonoscopy images with a total of 300 polyp instances from 15 unique polyps, and ASU-Mayo database, which is our collection of colonoscopy videos containing 19,400 frames and a total of 5,200 polyp instances from 10 unique polyps. We have evaluated our system using free-response receiver operating characteristic (FROC) analysis. At 0.1 false positives per frame, our system achieves a sensitivity of 88.0% for CVC-ColonDB and a sensitivity of 48% for the ASU-Mayo database. In addition, we have evaluated our system using a new detection latency analysis where latency is defined as the time from the first appearance of a polyp in the colonoscopy video to the time of its first detection by our system. At 0.05 false positives per frame, our system yields a polyp detection latency of 0.3 seconds.

Multiple Instance Learning of Pulmonary Embolism Detection with Geodesic Distance along Vascular Structure

We propose a novel classification approach for automatically detecting pulmonary embolism (PE) from computed-tomography-angiography images. Unlike most existing approaches that require vessel segmentation to restrict the search space for PEs, our toboggan-based candidate generator is capable of searching the entire lung for any suspicious regions quickly and efficiently. We then exploit the spatial information supplied in the vascular structure as a post-candidate-generation step by designing classifiers with geodesic distances between candidates along the vascular tree. Moreover, a PE represents a cluster of voxels in an image, and thus multiple candidates can be associated with a single PE and the PE is identified if any of its candidates is correctly classified. The proposed algorithm also provides an efficient solution to the problem of learning with multiple positive instances. Our clinical studies with 177 clinical cases demonstrate that the proposed approach outperforms existing detection methods, achieving 81 % sensitivity on an independent test set at 4 false positives per study.

Interactive Medical Image Segmentation with United Snakes

Snakes have become a standard image analysis technique with several variants now in common use. We have developed a software package called “United Snakes”. It unifies the most important snake variants, including finite difference, B-spline, and Hermite polynomial snakes, within the framework of a general finite element formulation with a choice of shape functions. Furthermore, we have incorporated into united snakes a recently proposed snake-like technique known as “livewire”, via a method for imposing hard constraints on snakes. Here, we demonstrate that the combination of techniques in united snakes yields generality, accuracy, ease of use, and robustness in several medical image analysis applications, including the segmentation of neuronal dendrites in EM images, dynamic chest image analysis, and the quantification of growth plates.

Automatic polyp detection in colonoscopy videos using an ensemble of convolutional neural networks

Computer-aided polyp detection in colonoscopy videos has been the subject of research for over the past decade. However, despite significant advances, automatic polyp detection is still an unsolved problem. In this paper, we propose a new polyp detection method based on a unique 3-way image presentation and convolutional neural networks. Our method learns a variety of polyp features such as color, texture, shape, and temporal information in multiple scales, enabling a more accurate polyp localization. Given a polyp candidate, a set of convolution neural networks - each specialized in one type of features - are applied in the vicinity of the candidate and then their results are aggregated to either accept or reject the candidate. Our experimental results based on our collection of videos, which to our knowledge is the largest annotated polyp database, shows a remarkable performance improvement over the state-of-the-art, significantly reducing the number of false positives in nearly all operating points. In addition, we propose a new performance curve, demonstrating that our new method significantly decreases polyp detection latency, which is defined as the time from the first appearance of a polyp in the video to the time of its first detection by our method.

Computer-aided detection of pulmonary embolism: Influence on radiologists’ detection performance with respect to vessel segments

The purpose was to assess the sensitivity of a CAD software prototype for the detection of pulmonary embolism in MDCT chest examinations with regard to vessel level and to assess the influence on radiologists’ detection performance. Forty-three patients with suspected PE were included in this retrospective study. MDCT chest examinations with a standard PE protocol were acquired at a 16-slice MDCT. All patient data were read by three radiologists (R1, R2, R3), and all thrombi were marked. A CAD prototype software was applied to all datasets, and each finding of the software was analyzed with regard to vessel level. The standard of reference was assessed in a consensus read. Sensitivity for the radiologists and CAD software was assessed. Thirty-three patients were positive for PE, with a total of 215 thrombi. The mean overall sensitivity for the CAD software alone was 83% (specificity, 80%). Radiologist sensitivity was 77% = R3, 82% = R2, and R1 = 87%. With the aid of the CAD software, sensitivities increased to 98% (R1), 93% (R2), and 92% (R3) (p<0.0001). CAD performance at the lobar level was 87%, at the segmental 90% and at the subsegmental 77%. With the use of CAD for PE, the detection performance of radiologists can be improved.

Accurate polyp segmentation for 3D CT colongraphy using multi-staged probabilistic binary learning and compositional model

Accurate and automatic colonic polyp segmentation and measurement in Computed Tomography (CT) has significant importance for 3D polyp detection, classification, and more generally computer aided diagnosis of colon cancers. In this paper, we propose a three-staged probabilistic binary classification approach for automatically segmenting polyp voxels from their surrounding tissues in CT. Our system integrates low-, and mid-level information for discriminative learning under local polar coordinates which align on the 3D colon surface around detected polyp. More importantly, our supervised learning system has flexible modeling capacity, which offers a principled means of encoding semantic, clinical expert annotations of colonic polyp tissue identification and segmentation. The learning generality to unseen data is bounded by boosting [12, 11] and stacked generality [14]. Extensive experimental results on polyp segmentation performance evaluation and robustness testing with disturbances (using both training data and unseen data) are provided to validate our presented approach. The reliability of polyp segmentation and measurement has been largely increased to 98:2% (ie. errors les 3 mm), compared with other state of art work [4, 15] of about 75% ~ 80%.

Computer-Aided Pulmonary Embolism Detection Using a Novel Vessel-Aligned Multi-planar Image Representation and Convolutional Neural Networks

Computer-aided detection (CAD) can play a major role in diagnosing pulmonary embolism (PE) at CT pulmonary angiography (CTPA). However, despite their demonstrated utility, to achieve a clinically acceptable sensitivity, existing PE CAD systems generate a high number of false positives, imposing extra burdens on radiologists to adjudicate these superfluous CAD findings. In this study, we investigate the feasibility of convolutional neural networks (CNNs) as an effective mechanism for eliminating false positives. A critical issue in successfully utilizing CNNs for detecting an object in 3D images is to develop a “right” image representation for the object. Toward this end, we have developed a vessel-aligned multi-planar image representation of emboli. Our image representation offers three advantages: (1) efficiency and compactness—concisely summarizing the 3D contextual information around an embolus in only 2 image channels, (2) consistency—automatically aligning the embolus in the 2-channel images according to the orientation of the affected vessel, and (3) expandability—naturally supporting data augmentation for training CNNs. We have evaluated our CAD approach using 121 CTPA datasets with a total of 326 emboli, achieving a sensitivity of 83% at 2 false positives per volume. This performance is superior to the best performing CAD system in the literature, which achieves a sensitivity of 71% at the same level of false positives. We have further evaluated our system using the entire 20 CTPA test datasets from the PE challenge. Our system outperforms the winning system from the challenge at 0mm localization error but is outperformed by it at 2mm and 5mm localization errors. In our view, the performance at 0mm localization error is more important than those at 2mm and 5mm localization errors.

A Two-Level Approach Towards Semantic Colon Segmentation: Removing Extra-Colonic Findings

Computer aided detection (CAD) of colonic polyps in computed tomographic colonography has tremendously impacted colorectal cancer diagnosis using 3D medical imaging. It is a prerequisite for all CAD systems to extract the air-distended colon segments from 3D abdomen computed tomography scans. In this paper, we present a two-level statistical approach of first separating colon segments from small intestine, stomach and other extra-colonic parts by classification on a new geometric feature set; then evaluating the overall performance confidence using distance and geometry statistics over patients. The proposed method is fully automatic and validated using both the classification results in the first level and its numerical impacts on false positive reduction of extra-colonic findings in a CAD system. It shows superior performance than the state-of-art knowledge or anatomy based colon segmentation algorithms.

Simultaneous Detection and Registration for Ileo-Cecal Valve Detection in 3D CT Colonography

Object detection and recognition has achieved a significant progress in recent years. However robust 3D object detection and segmentation in noisy 3D data volumes remains a challenging problem. Localizing an object generally requires its spatial configuration (i.e., pose, size) being aligned with the trained object model, while estimation of an objects spatial configuration is only valid at locations where the object appears. Detecting object while exhaustively searching its spatial parameters, is computationally prohibitive due to the high dimensionality of 3D search space. In this paper, we circumvent this computational complexity by proposing a novel framework capable of incrementally learning the object parameters (IPL) of location, pose and scale. This method is based on a sequence of binary encodings of the projected true positives from the original 3D object annotations (i.e., the projections of the global optima from the global space into the sections of subspaces). The training samples in each projected subspace are labeled as positive or negative, according their spatial registration distances towards annotations as ground-truth. Each encoding process can be considered as a general binary classification problem and is implemented using probabilistic boosting tree algorithm. We validate our approach with extensive experiments and performance evaluations for Ileo-Cecal Valve (ICV) detection in both clean and tagged 3D CT colonography scans. Our final ICV detection system also includes an optional prior learning procedure for IPL which further speeds up the detection.