Ashnil Kumar

Content-Based Medical Image Retrieval: A Survey of Applications to Multidimensional and Multimodality Data

Medical imaging is fundamental to modern healthcare, and its widespread use has resulted in the creation of image databases, as well as picture archiving and communication systems. These repositories now contain images from a diverse range of modalities, multidimensional (three-dimensional or time-varying) images, as well as co-aligned multimodality images. These image collections offer the opportunity for evidence-based diagnosis, teaching, and research; for these applications, there is a requirement for appropriate methods to search the collections for images that have characteristics similar to the case(s) of interest. Content-based image retrieval (CBIR) is an image search technique that complements the conventional text-based retrieval of images by using visual features, such as color, texture, and shape, as search criteria. Medical CBIR is an established field of study that is beginning to realize promise when applied to multidimensional and multimodality medical data. In this paper, we present a review of state-of-the-art medical CBIR approaches in five main categories: two-dimensional image retrieval, retrieval of images with three or more dimensions, the use of nonimage data to enhance the retrieval, multimodality image retrieval, and retrieval from diverse datasets. We use these categories as a framework for discussing the state of the art, focusing on the characteristics and modalities of the information used during medical image retrieval.

An Ensemble of Fine-Tuned Convolutional Neural Networks for Medical Image Classification

The availability of medical imaging data from clinical archives, research literature, and clinical manuals, coupled with recent advances in computer vision offer the opportunity for image-based diagnosis, teaching, and biomedical research. However, the content and semantics of an image can vary depending on its modality and as such the identification of image modality is an important preliminary step. The key challenge for automatically classifying the modality of a medical image is due to the visual characteristics of different modalities: some are visually distinct while others may have only subtle differences. This challenge is compounded by variations in the appearance of images based on the diseases depicted and a lack of sufficient training data for some modalities. In this paper, we introduce a new method for classifying medical images that uses an ensemble of different convolutional neural network (CNN) architectures. CNNs are a state-of-the-art image classification technique that learns the optimal image features for a given classification task. We hypothesise that different CNN architectures learn different levels of semantic image representation and thus an ensemble of CNNs will enable higher quality features to be extracted. Our method develops a new feature extractor by fine-tuning CNNs that have been initialized on a large dataset of natural images. The fine-tuning process leverages the generic image features from natural images that are fundamental for all images and optimizes them for the variety of medical imaging modalities. These features are used to train numerous multiclass classifiers whose posterior probabilities are fused to predict the modalities of unseen images. Our experiments on the ImageCLEF 2016 medical image public dataset (30 modalities; 6776 training images, and 4166 test images) show that our ensemble of fine-tuned CNNs achieves a higher accuracy than established CNNs. Our ensemble also achieves a higher accuracy than methods in the literature evaluated on the same benchmark dataset and is only overtaken by those methods that source additional training data.

A graph-based approach for the retrieval of multi-modality medical images

In this paper, we address the retrieval of multi-modality medical volumes, which consist of two different imaging modalities, acquired sequentially, from the same scanner. One such example, positron emission tomography and computed tomography (PET-CT), provides physicians with complementary functional and anatomical features as well as spatial relationships and has led to improved cancer diagnosis, localisation, and staging. The challenge of multi-modality volume retrieval for cancer patients lies in representing the complementary geometric and topologic attributes between tumours and organs. These attributes and relationships, which are used for tumour staging and classification, can be formulated as a graph. It has been demonstrated that graph-based methods have high accuracy for retrieval by spatial similarity. However, naïvely representing all relationships on a complete graph obscures the structure of the tumour-anatomy relationships. We propose a new graph structure derived from complete graphs that structurally constrains the edges connected to tumour vertices based upon the spatial proximity of tumours and organs. This enables retrieval on the basis of tumour localisation. We also present a similarity matching algorithm that accounts for different feature sets for graph elements from different imaging modalities. Our method emphasises the relationships between a tumour and related organs, while still modelling patient-specific anatomical variations. Constraining tumours to related anatomical structures improves the discrimination potential of graphs, making it easier to retrieve similar images based on tumour location. We evaluated our retrieval methodology on a dataset of clinical PET-CT volumes. Our results showed that our method enabled the retrieval of multi-modality images using spatial features. Our graph-based retrieval algorithm achieved a higher precision than several other retrieval techniques: gray-level histograms as well as state-of-the-art methods such as visual words using the scale- invariant feature transform (SIFT) and relational matrices representing the spatial arrangements of objects.

Cellular automata and anisotropic diffusion filter based interactive tumor segmentation for positron emission tomography

Tumor segmentation in positron emission tomography (PET) aids clinical diagnosis and in assessing treatment response. However, the low resolution and signal-to-noise inherent in PET images, makes accurate tumor segmentation challenging. Manual delineation is time-consuming and subjective, whereas fully automated algorithms are often limited to particular tumor types, and have difficulties in segmenting small and low-contrast tumors. Interactive segmentation may reduce the inter-observer variability and minimize the user input. In this study, we present a new interactive PET tumor segmentation method based on cellular automata (CA) and a nonlinear anisotropic diffusion filter (ADF). CA is tolerant of noise and image pattern complexity while ADF reduces noise while preserving edges. By coupling CA with ADF, our proposed approach was robust and accurate in detecting and segmenting noisy tumors. We evaluated our method with computer simulation and clinical data and it outperformed other common interactive PET segmentation algorithms.

Saliency-based Lesion Segmentation via Background Detection in Dermoscopic Images

The segmentation of skin lesions in dermoscopic images is a fundamental step in automated computer-aided diagnosis of melanoma. Conventional segmentation methods, however, have difficulties when the lesion borders are indistinct and when contrast between the lesion and the surrounding skin is low. They also perform poorly when there is a heterogeneous background or a lesion that touches the image boundaries; this then results in under- and oversegmentation of the skin lesion. We suggest that saliency detection using the reconstruction errors derived from a sparse representation model coupled with a novel background detection can more accurately discriminate the lesion from surrounding regions. We further propose a Bayesian framework that better delineates the shape and boundaries of the lesion. We also evaluated our approach on two public datasets comprising 1100 dermoscopic images and compared it to other conventional and state-of-the-art unsupervised (i.e., no training required) lesion segmentation methods, as well as the state-of-the-art unsupervised saliency detection methods. Our results show that our approach is more accurate and robust in segmenting lesions compared to other methods. We also discuss the general extension of our framework as a saliency optimization algorithm for lesion segmentation.

Dermoscopic Image Segmentation via Multistage Fully Convolutional Networks

Objective: Segmentation of skin lesions is an important step in the automated computer aided diagnosis of melanoma. However, existing segmentation methods have a tendency to over- or under-segment the lesions and perform poorly when the lesions have fuzzy boundaries, low contrast with the background, inhomogeneous textures, or contain artifacts. Furthermore, the performance of these methods are heavily reliant on the appropriate tuning of a large number of parameters as well as the use of effective preprocessing techniques, such as illumination correction and hair removal. Methods: We propose to leverage fully convolutional networks (FCNs) to automatically segment the skin lesions. FCNs are a neural network architecture that achieves object detection by hierarchically combining low-level appearance information with high-level semantic information. We address the issue of FCN producing coarse segmentation boundaries for challenging skin lesions (e.g., those with fuzzy boundaries and/or low difference in the textures between the foreground and the background) through a multistage segmentation approach in which multiple FCNs learn complementary visual characteristics of different skin lesions; early stage FCNs learn coarse appearance and localization information while late-stage FCNs learn the subtle characteristics of the lesion boundaries. We also introduce a new parallel integration method to combine the complementary information derived from individual segmentation stages to achieve a final segmentation result that has accurate localization and well-defined lesion boundaries, even for the most challenging skin lesions. Results: We achieved an average Dice coefficient of 91.18% on the ISBI 2016 Skin Lesion Challenge dataset and 90.66% on the PH2 dataset. Conclusion and Significance: Our extensive experimental results on two well-established public benchmark datasets demonstrate that our method is more effective than other state-of-the-art methods for skin lesion segmentation.

A Graph-based approach to the retrieval of volumetric PET-CT lung images

Combined positron emission tomography and computed tomography (PET-CT) scans have become a critical tool for the diagnosis, localisation, and staging of most cancers. This has led to a rapid expansion in the volume of PET-CT data that is archived in clinical environments. The ability to search these vast imaging collections has potential clinical applications in evidence-based diagnosis, physician training, and biomedical research that may lead to the discovery of new knowledge. Content-based image retrieval (CBIR) is an image search technique that complements conventional text-based retrieval by the use of image features as search criteria. Graph-based CBIR approaches have been found to be exemplary methods for medical CBIR as they provide the ability to consider disease localisation during the similarity measurement. However, the majority of graph-based CBIR studies have been based on 2D key slice approaches and did not exploit the rich volumetric data that is inherent to modern medical images, such as multi-modal PET-CT. In this paper, we present a graph-based CBIR method that exploits 3D spatial features extracted from volumetric regions of interest (ROIs). We index these features as attributes of a graph representation and use a graph-edit distance to measure the similarity of PET-CT images based on the spatial arrangement of tumours and organs in a 3D space. Our study aims to explore the capability of these graphs in 3D PET-CT CBIR. We show that our method achieves promising precision when retrieving clinical PET-CT images of patients with lung tumours.

Adapting content-based image retrieval techniques for the semantic annotation of medical images

The automatic annotation of medical images is a prerequisite for building comprehensive semantic archives that can be used to enhance evidence-based diagnosis, physician education, and biomedical research. Annotation also has important applications in the automatic generation of structured radiology reports. Much of the prior research work has focused on annotating images with properties such as the modality of the image, or the biological system or body region being imaged. However, many challenges remain for the annotation of high-level semantic content in medical images (e.g., presence of calcification, vessel obstruction, etc.) due to the difficulty in discovering relationships and associations between low-level image features and high-level semantic concepts. This difficulty is further compounded by the lack of labelled training data. In this paper, we present a method for the automatic semantic annotation of medical images that leverages techniques from content-based image retrieval (CBIR). CBIR is a well-established image search technology that uses quantifiable low-level image features to represent the high-level semantic content depicted in those images. Our method extends CBIR techniques to identify or retrieve a collection of labelled images that have similar low-level features and then uses this collection to determine the best high-level semantic annotations. We demonstrate our annotation method using retrieval via weighted nearest-neighbour retrieval and multi-class classification to show that our approach is viable regardless of the underlying retrieval strategy. We experimentally compared our method with several well-established baseline techniques (classification and regression) and showed that our method achieved the highest accuracy in the annotation of liver computed tomography (CT) images.

Automatic detection and classification of regions of FDG uptake in whole-body PET-CT lymphoma studies

[18F]-Fluorodeoxyglucose (FDG) positron emission tomography-computed tomography (PET-CT) scans of lymphoma patients usually show disease involvement as foci of increased radiotracer uptake. Existing methods for detecting abnormalities, model the characteristics of these foci; this is challenging due to the inconsistent shape and localization information about the lesions. Thresholding the degree of FDG uptake is the standard method to separate different sites of involvement. But may fragment sites into smaller regions, and may also incorrectly identify sites of normal physiological FDG uptake and normal FDG excretion (sFEPU) such as the kidneys, bladder, brain and heart. These sFEPU can obscure sites of abnormal uptake, which can make image interpretation problematic. Identifying sFEPU is therefore important for improving the sensitivity of lesion detection and image interpretation. Existing methods to identify sFEPU are inaccurate because they fail to account for the low inter-class differences between sFEPU fragments and their inconsistent localization information. In this study, we address this issue by using a multi-scale superpixel-based encoding (MSE) to group the individual sFEPU fragments into larger regions, thereby, enabling the extraction of highly discriminative image features via domain transferred convolutional neural networks. We then classify there regions into one of the sFEPU classes using a class-driven feature selection and classification model (CFSC) method that avoids overfitting to the most frequently occurring classes. Our experiments on 40 whole-body lymphoma PET-CT studies show that our method achieved better accuracy (an average F-score of 91.73%) compared to existing methods in the classification of sFEPU.

Synthesis of Positron Emission Tomography (PET) Images via Multi-channel Generative Adversarial Networks (GANs)

Positron emission tomography (PET) image synthesis plays an important role, which can be used to boost the training data for computer aided diagnosis systems. However, existing image synthesis methods have problems in synthesizing the low resolution PET images. To address these limitations, we propose multi-channel generative adversarial networks (M-GAN) based PET image synthesis method. Different to the existing methods which rely on using low-level features, the proposed M-GAN is capable to represent the features in a high-level of semantic based on the adversarial learning concept. In addition, M-GAN enables to take the input from the annotation (label) to synthesize the high uptake regions e.g., tumors and from the computed tomography (CT) images to constrain the appearance consistency and output the synthetic PET images directly. Our results on 50 lung cancer PET-CT studies indicate that our method was much closer to the real PET images when compared with the existing methods.