Thomas Martin Deserno

Ontology of Gaps in Content-Based Image Retrieval

Content-based image retrieval (CBIR) is a promising technology to enrich the core functionality of picture archiving and communication systems (PACS). CBIR has a potential for making a strong impact in diagnostics, research, and education. Research as reported in the scientific literature, however, has not made significant inroads as medical CBIR applications incorporated into routine clinical medicine or medical research. The cause is often attributed (without supporting analysis) to the inability of these applications in overcoming the “semantic gap.” The semantic gap divides the high-level scene understanding and interpretation available with human cognitive capabilities from the low-level pixel analysis of computers, based on mathematical processing and artificial intelligence methods. In this paper, we suggest a more systematic and comprehensive view of the concept of “gaps” in medical CBIR research. In particular, we define an ontology of 14 gaps that addresses the image content and features, as well as system performance and usability. In addition to these gaps, we identify seven system characteristics that impact CBIR applicability and performance. The framework we have created can be used a posteriori to compare medical CBIR systems and approaches for specific biomedical image domains and goals and a priori during the design phase of a medical CBIR application, as the systematic analysis of gaps provides detailed insight in system comparison and helps to direct future research.

MammoSys: A content-based image retrieval system using breast density patterns

In this paper, we present a content-based image retrieval system designed to retrieve mammographies from large medical image database. The system is developed based on breast density, according to the four categories defined by the American College of Radiology, and is integrated to the database of the Image Retrieval in Medical Applications (IRMA) project, that provides images with classification ground truth. Two-dimensional principal component analysis is used in breast density texture characterization, in order to effectively represent texture and allow for dimensionality reduction. A support vector machine is used to perform the retrieval process. Average precision rates are in the range from 83% to 97% considering a data set of 5024 images. The results indicate the potential of the system as the first stage of a computer-aided diagnosis framework.

Overview of the CLEF 2009 medical image annotation track

This paper describes the last round of the medical image annotation task in ImageCLEF 2009. After four years, we defined the task as a survey of all the past experience. Seven groups participated to the challenge submitting nineteen runs. They were asked to train their algorithms on 12677 images, labelled according to four different settings, and to classify 1733 images in the four annotation frameworks. The aim is to understand how each strategy answers to the increasing number of classes and to the unbalancing. A plain classification scheme using support vector machines and local descriptors outperformed the other methods.

Automatic medical image annotation in ImageCLEF 2007: Overview, results, and discussion

In this paper, the automatic medical annotation task of the 2007 CLEF cross language image retrieval campaign (ImageCLEF) is described. The paper focusses on the images used, the task setup, and the results obtained in the evaluation campaign. Since 2005, the medical automatic image annotation task exists in ImageCLEF with increasing complexity to evaluate the performance of state-of-the-art methods for completely automatic annotation of medical images based on visual properties. The paper also describes the evolution of the task from its origin in 2005-2007. The 2007 task, comprising 11,000 fully annotated training images and 1000 test images to be annotated, is a realistic task with a large number of possible classes at different levels of detail. Detailed analysis of the methods across participating groups is presented with respect to the (i) image representation, (ii) classification method, and (iii) use of the class hierarchy. The results show that methods which build on local image descriptors and discriminative models are able to provide good predictions of the image classes, mostly by using techniques that were originally developed in the machine learning and computer vision domain for object recognition in non-medical images.

Virtual reality-based simulator for training in regional anaesthesia

BACKGROUND The safe performance of regional anaesthesia (RA) requires theoretical knowledge and good manual skills. Virtual reality (VR)-based simulators may offer trainees a safe environment to learn and practice different techniques. However, currently available VR simulators do not consider individual anatomy, which limits their use for realistic training. We have developed a VR-based simulator that can be used for individual anatomy and for different anatomical regions. METHODS Individual data were obtained from magnetic resonance imaging (MRI) and magnetic resonance angiography (MRA) without contrast agent to represent morphology and the vascular system, respectively. For data handling, registration, and segmentation, an application based on the Medical Imaging Interaction Toolkit was developed. Suitable segmentation algorithms such as the fuzzy c-means clustering approach were integrated, and a hierarchical tree data structure was created to model the flexible anatomical structures of peripheral nerve cords. The simulator was implemented in the VR toolkit ViSTA using modules for collision detection, virtual humanoids, interaction, and visualization. A novel algorithm for electric impulse transmission is the core of the simulation. RESULTS In a feasibility study, MRI morphology and MRA were acquired from five subjects for the inguinal region. From these sources, three-dimensional anatomical data sets were created and nerves modelled. The resolution obtained from both MRI and MRA was sufficient for realistic simulations. Our high-fidelity simulator application allows trainees to perform virtual peripheral nerve blocks based on these data sets and models. CONCLUSIONS Subject-specific training of RA is supported in a virtual environment. We have adapted segmentation algorithms and developed a VR-based simulator for the inguinal region for use in training for different peripheral nerve blocks. In contrast to available VR-based simulators, our simulation offers anatomical variety.

Extended Query Refinement for Medical Image Retrieval

The impact of image pattern recognition on accessing large databases of medical images has recently been explored, and content-based image retrieval (CBIR) in medical applications (IRMA) is researched. At the present, however, the impact of image retrieval on diagnosis is limited, and practical applications are scarce. One reason is the lack of suitable mechanisms for query refinement, in particular, the ability to (1) restore previous session states, (2) combine individual queries by Boolean operators, and (3) provide continuous-valued query refinement. This paper presents a powerful user interface for CBIR that provides all three mechanisms for extended query refinement. The various mechanisms of man–machine interaction during a retrieval session are grouped into four classes: (1) output modules, (2) parameter modules, (3) transaction modules, and (4) process modules, all of which are controlled by a detailed query logging. The query logging is linked to a relational database. Nested loops for interaction provide a maximum of flexibility within a minimum of complexity, as the entire data flow is still controlled within a single Web page. Our approach is implemented to support various modalities, orientations, and body regions using global features that model gray scale, texture, structure, and global shape characteristics. The resulting extended query refinement has a significant impact for medical CBIR applications.

Toward a standard reference database for computer-aided mammography

Because of the lack of mammography databases with a large amount of codified images and identified characteristics like pathology, type of breast tissue, and abnormality, there is a problem for the development of robust systems for computer-aided diagnosis. Integrated to the Image Retrieval in Medical Applications (IRMA) project, we present an available mammography database developed from the union of: The Mammographic Image Analysis Society Digital Mammogram Database (MIAS), The Digital Database for Screening Mammography (DDSM), the Lawrence Livermore National Laboratory (LLNL), and routine images from the Rheinisch-Westfälische Technische Hochschule (RWTH) Aachen. Using the IRMA code, standardized coding of tissue type, tumor staging, and lesion description was developed according to the American College of Radiology (ACR) tissue codes and the ACR breast imaging reporting and data system (BI-RADS). The import was done automatically using scripts for image download, file format conversion, file name, web page and information file browsing. Disregarding the resolution, this resulted in a total of 10,509 reference images, and 6,767 images are associated with an IRMA contour information feature file. In accordance to the respective license agreements, the database will be made freely available for research purposes, and may be used for image based evaluation campaigns such as the Cross Language Evaluation Forum (CLEF). We have also shown that it can be extended easily with further cases imported from a picture archiving and communication system (PACS).

Challenges of medical image processing

In todays health care, imaging plays an important role throughout the entire clinical process from diagnostics and treatment planning to surgical procedures and follow up studies. Since most imaging modalities have gone directly digital, with continually increasing resolution, medical image processing has to face the challenges arising from large data volumes. In this paper, we discuss Kilo- to Terabyte challenges regarding (i) medical image management and image data mining, (ii) bioimaging, (iii) virtual reality in medical visualizations and (iv) neuroimaging. Due to the increasing amount of data, image processing and visualization algorithms have to be adjusted. Scalable algorithms and advanced parallelization techniques using graphical processing units have been developed. They are summarized in this paper. While such techniques are coping with the Kilo- to Terabyte challenge, the Petabyte level is already looming on the horizon. For this reason, medical image processing remains a vital field of research.

Computer-aided diagnostics of screening mammography using content-based image retrieval

Breast cancer is one of the main causes of death among women in occidental countries. In the last years, screening mammography has been established worldwide for early detection of breast cancer, and computer-aided diagnostics (CAD) is being developed to assist physicians reading mammograms. A promising method for CAD is content-based image retrieval (CBIR). Recently, we have developed a classification scheme of suspicious tissue pattern based on the support vector machine (SVM). In this paper, we continue moving towards automatic CAD of screening mammography. The experiments are based on in total 10,509 radiographs that have been collected from different sources. From this, 3,375 images are provided with one and 430 radiographs with more than one chain code annotation of cancerous regions. In different experiments, this data is divided into 12 and 20 classes, distinguishing between four categories of tissue density, three categories of pathology and in the 20 class problem two categories of different types of lesions. Balancing the number of images in each class yields 233 and 45 images remaining in each of the 12 and 20 classes, respectively. Using a two-dimensional principal component analysis, features are extracted from small patches of 128 x 128 pixels and classified by means of a SVM. Overall, the accuracy of the raw classification was 61.6 % and 52.1 % for the 12 and the 20 class problem, respectively. The confusion matrices are assessed for detailed analysis. Furthermore, an implementation of a SVM-based CBIR system for CADx in screening mammography is presented. In conclusion, with a smarter patch extraction, the CBIR approach might reach precision rates that are helpful for the physicians. This, however, needs more comprehensive evaluation on clinical data.

Content-based image retrieval applied to BI-RADS tissue classification in screening mammography

AIM To present a content-based image retrieval (CBIR) system that supports the classification of breast tissue density and can be used in the processing chain to adapt parameters for lesion segmentation and classification. METHODS Breast density is characterized by image texture using singular value decomposition (SVD) and histograms. Pattern similarity is computed by a support vector machine (SVM) to separate the four BI-RADS tissue categories. The crucial number of remaining singular values is varied (SVD), and linear, radial, and polynomial kernels are investigated (SVM). The system is supported by a large reference database for training and evaluation. Experiments are based on 5-fold cross validation. RESULTS Adopted from DDSM, MIAS, LLNL, and RWTH datasets, the reference database is composed of over 10 000 various mammograms with unified and reliable ground truth. An average precision of 82.14% is obtained using 25 singular values (SVD), polynomial kernel and the one-against-one (SVM). CONCLUSION Breast density characterization using SVD allied with SVM for image retrieval enable the development of a CBIR system that can effectively aid radiologists in their diagnosis.