Michael B. Gotway

Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?

Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch? To address this question, we considered four distinct medical imaging applications in three specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from three different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that 1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; 2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; 3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and 4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch? To address this question, we considered four distinct medical imaging applications in three specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from three different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that 1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; 2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; 3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and 4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.

Computer-Aided Pulmonary Embolism Detection Using a Novel Vessel-Aligned Multi-planar Image Representation and Convolutional Neural Networks

Computer-aided detection (CAD) can play a major role in diagnosing pulmonary embolism (PE) at CT pulmonary angiography (CTPA). However, despite their demonstrated utility, to achieve a clinically acceptable sensitivity, existing PE CAD systems generate a high number of false positives, imposing extra burdens on radiologists to adjudicate these superfluous CAD findings. In this study, we investigate the feasibility of convolutional neural networks (CNNs) as an effective mechanism for eliminating false positives. A critical issue in successfully utilizing CNNs for detecting an object in 3D images is to develop a “right” image representation for the object. Toward this end, we have developed a vessel-aligned multi-planar image representation of emboli. Our image representation offers three advantages: (1) efficiency and compactness—concisely summarizing the 3D contextual information around an embolus in only 2 image channels, (2) consistency—automatically aligning the embolus in the 2-channel images according to the orientation of the affected vessel, and (3) expandability—naturally supporting data augmentation for training CNNs. We have evaluated our CAD approach using 121 CTPA datasets with a total of 326 emboli, achieving a sensitivity of 83% at 2 false positives per volume. This performance is superior to the best performing CAD system in the literature, which achieves a sensitivity of 71% at the same level of false positives. We have further evaluated our system using the entire 20 CTPA test datasets from the PE challenge. Our system outperforms the winning system from the challenge at 0mm localization error but is outperformed by it at 2mm and 5mm localization errors. In our view, the performance at 0mm localization error is more important than those at 2mm and 5mm localization errors.

Fine-Tuning Convolutional Neural Networks for Biomedical Image Analysis: Actively and Incrementally

Intense interest in applying convolutional neural networks (CNNs) in biomedical image analysis is wide spread, but its success is impeded by the lack of large annotated datasets in biomedical imaging. Annotating biomedical images is not only tedious and time consuming, but also demanding of costly, specialty - oriented knowledge and skills, which are not easily accessible. To dramatically reduce annotation cost, this paper presents a novel method called AIFT (active, incremental fine-tuning) to naturally integrate active learning and transfer learning into a single framework. AIFT starts directly with a pre-trained CNN to seek worthy samples from the unannotated for annotation, and the (fine-tuned) CNN is further fine-tuned continuously by incorporating newly annotated samples in each iteration to enhance the CNNs performance incrementally. We have evaluated our method in three different biomedical imaging applications, demonstrating that the cost of annotation can be cut by at least half. This performance is attributed to the several advantages derived from the advanced active and incremental capability of our AIFT method.

Erdheim Chester Disease treated successfully with cladribine.

A 61-year-old previously healthy male with a history of progressive fatigue, lower extremity edema, and dyspnea for 4 months was hospitalized with pericardial and pleural effusions (Figure 1A, B). Lung, pleural, and pericardial biopsies were consistent with Erdheim-Chester disease. He was treated with systemic steroids, and ultimately tried on PEG-interferon. He deteriorated clinically and the disease progressed to include CNS manifestations. Ultimately he was treated with Cladribine, at a dose 0.014 mg/kg on day 1, followed by 0.09 mg/kg/day = 6.4 mg IV for 6 additional days. He received 2 further cycles of 0.14 mg kg/day for 7 days (1 month apart). After 3 cycles he improved significantly both clinically and radiographically. Six months post-treatment objective testing showed improvement in cardiac, neurologic, and pulmonary disease. Erdheim Chester Disease (ECD) is a rare non Langerhans cell histiocytosis. Only several hundred cases have been reported in the literature. Treatment for ECD is reserved for those with symptomatic disease, asymptomatic CNS involvement, or evidence of organ dysfunction. There is no standard treatment regimen: Current options include corticosteroids, Interferon alpha (IFN), systemic chemotherapy, and radiation therapy. The occurrence of the V600EBRAF mutation in about 50% of patients can make these patients amenable to targeted therapy with BRAF kinase inhibitors (e.g. Vemurafenib). More recently the presence of N/KRAS, and PIK3CA mutations have provided further rational for targeted therapies. The cytokine profile in patients with ECD suggests monocyte activation cladribine, a purine analogue toxic to monocytes, has also been studied as a treatment for ECD, especially in patients who test negative for the BRAF mutation.

Chest Pain and Diarrhea: A Case of Campylobacter Jejuni-Associated Myocarditis

BACKGROUND Diarrhea and chest pain are common symptoms in patients presenting to the emergency department (ED). However, rarely is a relationship between these two symptoms established in a single patient. OBJECTIVE Describe a case of Campylobacter-associated myocarditis. CASE REPORT A 43-year-old man with a history of hypertension presented to the ED with angina-like chest pain and a 3-day history of diarrhea. Electrocardiogram revealed ST-segment elevation in the lateral leads. Coronary angiogram revealed no obstructive coronary artery disease. Troponin T rose to 1.75 ng/mL. Cardiac magnetic resonance imaging showed subepicardial and mid-myocardial enhancement, particularly in the anterolateral wall and interventricular septum, consistent with a diagnosis of myocarditis. Stool studies were positive for Campylobacter jejuni. CONCLUSIONS Campylobacter-associated myocarditis is rare, but performing the appropriate initial diagnostic testing, including stool cultures, is critical to making the diagnosis. Identifying the etiology of myocarditis as bacterial will ensure that appropriate treatment with antibiotics occurs in addition to any cardiology medications needed for supportive care.

On the necessity of fine-tuned convolutional neural networks for medical imaging

This study aims to address two central questions. First, are fine-tuned convolutional neural networks (CNNs) necessary for medical imaging applications? In response, we considered four medical vision tasks from three different medical imaging modalities, and studied the necessity of fine-tuned CNNs under varying amounts of training data. Second, to what extent the knowledge is to be transferred? In response, we proposed a layer-wise fine-tuning scheme to examine how the extent or depth of fine-tuning contributes to the success of knowledge transfer. Our experiments consistently showed that the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch. The performance gap widened when reduced training sets were used for training and fine-tuning. Our results further revealed that the required level of fine-tuning differed from one application to another, suggesting that neither shallow tuning nor deep tuning may be the optimal choice for a particular application. Layer-wise fine-tuning may offer a practical way to reach the best performance for the application at hand based on the amount of available data. We conclude that knowledge transfer from natural images is necessary and that the level of tuning should be chosen experimentally.

Pulmonary pyoderma gangrenosum without cutaneous manifestations.

Pyoderma gangrenosum is a chronic sterile skin disorder that is frequently seen in association with systemic disorders such as inflammatory bowel disease. Extracutaneous pyoderma gangrenosum is rare and most commonly occurs in the lungs. It is particularly unusual for extracutaneous pyoderma gangrenosum to manifest prior to skin findings and without an associated systemic disorder. A 19‐year‐old white man presented with shortness of breath and a productive cough. His skin exam was normal. Unenhanced chest computed tomography showed peripheral consolidations, areas of cavitation, nodules and bilateral pleural effusions. A bronchoalveolar lavage and an autoimmune panel were unremarkable. Right lung wedge biopsies via thoracostomy was performed and showed pulmonary pyoderma gangrenosum. He was treated with corticosteroids and has returned back to his baseline. This is the first case of pulmonary pyoderma gangrenosum without any associated underlying systemic disorder and without any cutaneous manifestations to date. Serial follow‐ups are necessary to assess for the development of an associated systemic disorder or skin lesions.

A novel online boosting algorithm for automatic anatomy detection

This paper presents a novel online learning method for automatically detecting anatomic structures in medical images. Conventional off-line learning methods require collecting a complete set of representative samples prior to training a detector. Once the detector is trained, its performance is fixed. To improve the performance, the detector must be completely retrained, demanding the maintenance of historical training samples. Our proposed online approach eliminates the need for storing historical training samples and is capable of continually improving performance with new samples. We evaluate our approach with three distinct thoracic structures, demonstrating that our approach yields performance competitive with the off-line approach. Furthermore, we investigate the properties of our proposed method in comparison with an online learning method suggested by Grabner and Bischof (IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2006, vol. 1, pp. 260–267, 2006), which is the state of the art, indicating that our proposed method runs faster, offers more stability, improves handling of “catastrophic forgetting”, and simultaneously achieves a satisfactory level of adaptability. The enhanced performance is attributed to our novel online learning structure coupled with more accurate weaker learners based on histograms.

Pulmonary artery sheath haematoma with pulmonary arterial compression: a rare complication of type A aortic dissection mistaken for aortitis

An 89-year-old woman with a history of hypertension and temporal arteritis treated with corticosteroid therapy presented complaining of moderate right-sided chest pain with shortness of breath beginning 1 day earlier. She had no fever, vomiting, syncope, or light headedness. CT aortography showed hyperattenuating aortic wall thickening with unenhanced imaging (arrow, Panel A , and crescentic low attenuation thickening …

Revision of failed, recurrent or complicated pectus excavatum after Nuss, Ravitch or cardiac surgery

Pectus excavatum (PE) can recur after both open and minimally invasive repair of pectus excavatum (MIRPE) techniques. The cause of recurrence may differ based on the initial repair procedure performed. Recurrence risks for the open repair are due to factors which include incomplete previous repair, repair at too young of age, excessive dissection, early removal or lack of support structures, and incomplete healing of the chest wall. For patients presenting after failed or recurrent primary MIRPE repair, issues with support bars including placement, number, migration, and premature removal can all be associated with failure. Connective tissue disorders can complicate and increase recurrence risk in both types of PE repairs. Identifying the factors that contributed to the previous procedures failure is critical for prevention of another recurrence. A combination of surgical techniques may be necessary to successfully repair some patients.