Jeremy Kawahara

Deep features to classify skin lesions

Diagnosing an unknown skin lesion is the first step to determine appropriate treatment. We demonstrate that a linear classifier, trained on features extracted from a convolutional neural network pretrained on natural images, distinguishes among up to ten skin lesions with a higher accuracy than previously published state-of-the-art results on the same dataset. Further, in contrast to competing works, our approach requires no lesion segmentations nor complex preprocessing. We gain consistent additional improvements to accuracy using a per image normalization, a fully convolutional network to extract multi-scale features, and by pooling over an augmented feature space. Compared to state-of-the-art, our proposed approach achieves a favourable accuracy of 85.8% over 5-classes (compared to 75.1%) with noticeable improvements in accuracy for underrepresented classes (e.g., 60% compared to 15.6%). Over the entire 10-class dataset of 1300 images captured from a standard (non-dermoscopic) camera, our method achieves an accuracy of 81.8% outperforming the 67% accuracy previously reported.

BrainNetCNN: Convolutional neural networks for brain networks; towards predicting neurodevelopment

Abstract We propose BrainNetCNN, a convolutional neural network (CNN) framework to predict clinical neurodevelopmental outcomes from brain networks. In contrast to the spatially local convolutions done in traditional image‐based CNNs, our BrainNetCNN is composed of novel edge‐to‐edge, edge‐to‐node and node‐to‐graph convolutional filters that leverage the topological locality of structural brain networks. We apply the BrainNetCNN framework to predict cognitive and motor developmental outcome scores from structural brain networks of infants born preterm. Diffusion tensor images (DTI) of preterm infants, acquired between 27 and 46 weeks gestational age, were used to construct a dataset of structural brain connectivity networks. We first demonstrate the predictive capabilities of BrainNetCNN on synthetic phantom networks with simulated injury patterns and added noise. BrainNetCNN outperforms a fully connected neural‐network with the same number of model parameters on both phantoms with focal and diffuse injury patterns. We then apply our method to the task of joint prediction of Bayley‐III cognitive and motor scores, assessed at 18 months of age, adjusted for prematurity. We show that our BrainNetCNN framework outperforms a variety of other methods on the same data. Furthermore, BrainNetCNN is able to identify an infants postmenstrual age to within about 2 weeks. Finally, we explore the high‐level features learned by BrainNetCNN by visualizing the importance of each connection in the brain with respect to predicting the outcome scores. These findings are then discussed in the context of the anatomy and function of the developing preterm infant brain. HighlightsFirst deep convolutional neural network architecture designed for connectomes.Novel convolutional layers for leveraging topological locality in brain networks.Prediction of neurodevelopmental outcomes in preterm infants.Visualization of brain connections learned to be important for prediction.

Globally optimal spinal cord segmentation using a minimal path in high dimensions

Spinal cord segmentation is an important step to empirically quantify spinal cord atrophy that can occur in neurological diseases such as multiple sclerosis (MS). In this work, we propose a novel method to find the globally optimal segmentation of the spinal cord using a high dimensional minimal path search. The spinal cord cross-sectional shapes are represented using principal component analysis (in the probability simplex) which captures most of spinal cords axial cross-sectional variation and partial volume effects. We propose modifications to the A* minimal path search algorithm that drastically reduce the required memory and run-time to make our high dimensional minimal path optimization computationally feasible. Finally, we validate our results over five vertebrae levels of both healthy and MS clinical MR volumes (20 volumes total) and show improvements on volume agreement with expert segmentations and less user interaction when compared to current state-of-the-art methods.

Multi-resolution-Tract CNN with Hybrid Pretrained and Skin-Lesion Trained Layers

Correctly classifying a skin lesion is one of the first steps towards treatment. We propose a novel convolutional neural network (CNN) architecture for skin lesion classification designed to learn based on information from multiple image resolutions while leveraging pretrained CNNs. While traditional CNNs are generally trained on a single resolution image, our CNN is composed of multiple tracts, where each tract analyzes the image at a different resolution simultaneously and learns interactions across multiple image resolutions using the same field-of-view. We convert a CNN, pretrained on a single resolution, to work for multi-resolution input. The entire network is fine-tuned in a fully learned end-to-end optimization with auxiliary loss functions. We show how our proposed novel multi-tract network yields higher classification accuracy, outperforming state-of-the-art multi-scale approaches when compared over a public skin lesion dataset.

Augmenting Auto-context with Global Geometric Features for Spinal Cord Segmentation

Anatomical shape variations are typically difficult to model and parametric or hand-crafted models can lead to ill-fitting segmentations. This difficulty can be addressed with a framework like auto-context, that learns to jointly detect and regularize a segmentation. However, mis-segmentation can still occur when a desired structure, such as the spinal cord, has few locally distinct features. High-level knowledge at a global scale (e.g. an MRI contains a single connected spinal cord) is needed to regularize these candidate segmentations. To encode high-level knowledge, we propose to augment the auto-context framework with global geometric features extracted from the detected candidate shapes. Our classifier then learns these high-level rules and rejects falsely detected shapes. To validate our method we segment the spinal cords from 20 MRI volumes composed of patients with and without multiple sclerosis and demonstrate improvements in accuracy, speed, and manual effort required when compared to state-of-the-art methods.

Towards multi-modal image-guided tumour identification in robot-assisted partial nephrectomy

Tumour identification is a critical step in robot-assisted partial nephrectomy (RAPN) during which the surgeon determines the tumour localization and resection margins. To help the surgeon in achieving this step, our research work aims at leveraging both pre- and intra-operative imaging modalities (CT, MRI, laparoscopic US, stereo endoscopic video) to provide an augmented reality view of kidney-tumour boundaries with uncertainty-encoded information. We present herein the progress of this research work including segmentation of preoperative scans, biomechanical simulation of deformations, stereo surface reconstruction from stereo endoscopic camera, pre-operative to intra-operative data registration, and augmented reality visualization.

Pareto-optimal multi-objective dimensionality reduction deep auto-encoder for mammography classification

BACKGROUND AND OBJECTIVE Feature reduction is an essential stage in computer aided breast cancer diagnosis systems. Multilayer neural networks can be trained to extract relevant features by encoding high-dimensional data into low-dimensional codes. Optimizing traditional auto-encoders works well only if the initial weights are close to a proper solution. They are also trained to only reduce the mean squared reconstruction error (MRE) between the encoder inputs and the decoder outputs, but do not address the classification error. The goal of the current work is to test the hypothesis that extending traditional auto-encoders (which only minimize reconstruction error) to multi-objective optimization for finding Pareto-optimal solutions provides more discriminative features that will improve classification performance when compared to single-objective and other multi-objective approaches (i.e. scalarized and sequential). METHODS In this paper, we introduce a novel multi-objective optimization of deep auto-encoder networks, in which the auto-encoder optimizes two objectives: MRE and mean classification error (MCE) for Pareto-optimal solutions, rather than just MRE. These two objectives are optimized simultaneously by a non-dominated sorting genetic algorithm. RESULTS We tested our method on 949 X-ray mammograms categorized into 12 classes. The results show that the features identified by the proposed algorithm allow a classification accuracy of up to 98.45%, demonstrating favourable accuracy over the results of state-of-the-art methods reported in the literature. CONCLUSIONS We conclude that adding the classification objective to the traditional auto-encoder objective and optimizing for finding Pareto-optimal solutions, using evolutionary multi-objective optimization, results in producing more discriminative features.

Automatic Labelling of Tumourous Frames in Free-Hand Laparoscopic Ultrasound Video

Laparoscopic ultrasound (US) is often used during partial nephrectomy surgeries to identify tumour boundaries within the kidney. However, visual identification is challenging as tumour appearance varies across patients and US images exhibit significant noise levels. To address these challenges, we present the first fully automatic method for detecting the presence of kidney tumour in free-hand laparoscopic ultrasound sequences in near real-time. Our novel approach predicts the probability that a frame contains tumourous tissue using random forests and encodes this probability combined with a regularization term within a graph. Using Dijkstras algorithm we find a globally optimal labelling (tumour vs. non-tumour) of each frame. We validate our method on a challenging clinical dataset composed of five patients, with a total of 2025 2D ultrasound frames, and demonstrate the ability to detect the presence of kidney tumour with a sensitivity and specificity of 0.774 and 0.916, respectively.

Novel Morphological and Appearance Features for Predicting Physical Disability from MR Images in Multiple Sclerosis Patients

Physical disability in patients with multiple sclerosis is determined by functional ability and quantified with numerical scores. In vivo studies using magnetic resonance imaging (MRI) have found that these scores correlate with spinal cord atrophy (loss of tissue), where atrophy is commonly measured by spinal cord volume or cross-sectional area. However, this correlation is generally weak to moderate, and improved measures would strengthen the utility of imaging biomarkers. We propose novel spinal cord morphological and MRI-based appearance features. Select features are used to train regression models to predict patients’ physical disability scores. We validate our models using 30 MRI scans of different patients with varying levels of disability. Our results suggest that regression models trained with multiple spinal cord features predict clinical disability better than a model based on the volume of the spinal cord alone.

Graph Geodesics to Find Progressively Similar Skin Lesion Images

Skin conditions represent an enormous health care burden worldwide, and as datasets of skin images grow, there is continued interest in computerized approaches to analyze skin images. In order to explore and gain insights into datasets of skin images, we propose a graph based approach to visualize a progression of similar skin images between pairs of images. In our graph, a node represents both a clinical and dermoscopic image of the same lesion, and an edge between nodes captures the visual dissimilarity between lesions, where dissimilarity is computed by comparing the image responses of a pretrained convolutional neural network. We compute the geodesic/shortest path between nodes to determine a path of progressively visually similar skin lesions. To quantitatively evaluate the quality of the returned path, we propose metrics to measure the number of transitions with respect to the lesion diagnosis, and the progression with respect to the clinical 7-point checklist. Compared to baseline experiments, our approach shows improvements to the quality of the returned paths.