Daekeun You

Image retrieval from scientific publications: Text and image content processing to separate multipanel figures

Images contained in scientific publications are widely considered useful for educational and research purposes, and their accurate indexing is critical for efficient and effective retrieval. Such image retrieval is complicated by the fact that figures in the scientific literature often combine multiple individual subfigures (panels). Multipanel figures are in fact the predominant pattern in certain types of scientific publications. The goal of this work is to automatically segment multipanel figures—a necessary step for automatic semantic indexing and in the development of image retrieval systems targeting the scientific literature. We have developed a method that uses the image content as well as the associated figure caption to: (1) automatically detect panel boundaries; (2) detect panel labels in the images and convert them to text; and (3) detect the labels and textual descriptions of each panel within the captions. Our approach combines the output of image‐content and text‐based processing steps to split the multipanel figures into individual subfigures and assign to each subfigure its corresponding section of the caption. The developed system achieved precision of 81% and recall of 73% on the task of automatic segmentation of multipanel figures.

Text- and content-based biomedical image modality classification

Image modality classification is an important task toward achieving high performance in biomedical image and article retrieval. Imaging modality captures information about its appearance and use. Examples include X-ray, MRI, Histopathology, Ultrasound, etc. Modality classification reduces the search space in image retrieval. We have developed and evaluated several modality classification methods using visual and textual features extracted from images and text data, such as figure captions, article citations, and MeSH®. Our hierarchical classification method using multimodal (mixed textual and visual) and several class-specific features achieved the highest classification accuracy of 63.2%. The performance was among the best in ImageCLEF2012 evaluation.

Literature-based biomedical image classification and retrieval

Literature-based image informatics techniques are essential for managing the rapidly increasing volume of information in the biomedical domain. Compound figure separation, modality classification, and image retrieval are three related tasks useful for enabling efficient access to the most relevant images contained in the literature. In this article, we describe approaches to these tasks and the evaluation of our methods as part of the 2013 medical track of ImageCLEF. In performing each of these tasks, the textual and visual features used to represent images are an important consideration often left unaddressed. Therefore, we also describe a gradient-based optimization strategy for determining meaningful combinations of features and apply the method to the image retrieval task. An evaluation of our optimization strategy indicates the method is capable of producing statistically significant improvements in retrieval performance. Furthermore, the results of the 2013 ImageCLEF evaluation demonstrate the effectiveness of our techniques. In particular, our text-based and mixed image retrieval methods ranked first among all the participating groups.

Interactive cross and multimodal biomedical image retrieval based on automatic region-of-interest (ROI) identification and classification

In biomedical articles, authors often use annotation markers such as arrows, letters, or symbols overlaid on figures and illustrations to highlight ROIs. These annotations are then referenced and correlated with concepts in the caption text or figure citations in the article text. This association creates a bridge between the visual characteristics of important regions within an image and their semantic interpretation. Identifying these assists in extracting ROIs that are likely to be highly relevant to the discussion in the article text. The aim of this work is to perform semantic search without knowing the concept keyword or the specific name of the visual pattern or appearance. We consider the problem of cross and multimodal retrieval of images from articles which contains components of text and images. Our proposed method localizes and recognizes the annotations by utilizing a combination of rule-based and statistical image processing techniques. The image regions are then annotated for classification using biomedical concepts obtained from a glossary of imaging terms. Similar automatic ROI extraction can be applied to query images, or to interactively mark an ROI. As a result, visual characteristics of the ROIs can be mapped to text concepts and then used to search image captions. In addition, the system can toggle the search process from purely perceptual to a conceptual one (crossmodal) based on utilizing user feedback or integrate both perceptual and conceptual search in a multimodal search process. The hypothesis, that such approaches would improve biomedical image retrieval, was validated through experiments on a biomedical article dataset of thoracic CT scans.

A robust pointer segmentation in biomedical images toward building a visual ontology for biomedical article retrieval

Pointers (arrows and symbols) are frequently used in biomedical images to highlight specific image regions of interest (ROIs) that are mentioned in figure captions and/or text discussion. Detection of pointers is the first step toward extracting relevant visual features from ROIs and combining them with textual descriptions for a multimodal (text and image) biomedical article retrieval system. Recently we developed a pointer recognition algorithm based on an edge-based pointer segmentation method, and subsequently reported improvements made on our initial approach involving the use of Active Shape Models (ASM) for pointer recognition and region growing-based method for pointer segmentation. These methods contributed to improving the recall of pointer recognition but not much to the precision. The method discussed in this article is our recent effort to improve the precision rate. Evaluation performed on two datasets and compared with other pointer segmentation methods show significantly improved precision and the highest F1 score.

A contour-based shape descriptor for biomedical image classification and retrieval

Contours, object blobs, and specific feature points are utilized to represent object shapes and extract shape descriptors that can then be used for object detection or image classification. In this research we develop a shape descriptor for biomedical image type (or, modality) classification. We adapt a feature extraction method used in optical character recognition (OCR) for character shape representation, and apply various image preprocessing methods to successfully adapt the method to our application. The proposed shape descriptor is applied to radiology images (e.g., MRI, CT, ultrasound, X-ray, etc.) to assess its usefulness for modality classification. In our experiment we compare our method with other visual descriptors such as CEDD, CLD, Tamura, and PHOG that extract color, texture, or shape information from images. The proposed method achieved the highest classification accuracy of 74.1% among all other individual descriptors in the test, and when combined with CSD (color structure descriptor) showed better performance (78.9%) than using the shape descriptor alone.

Extraction of endoscopic images for biomedical figure classification

Modality filtering is an important feature in biomedical image searching systems and may significantly improve the retrieval performance of the system. This paper presents a new method for extracting endoscopic image figures from photograph images in biomedical literature, which are found to have highly diverse content and large variability in appearance. Our proposed method consists of three main stages: tissue image extraction, endoscopic image candidate extraction, and ophthalmic image filtering. For tissue image extraction we use image patch level clustering and MRF relabeling to detect images containing skin/tissue regions. Next, we find candidate endoscopic images by exploiting the round shape characteristics that commonly appear in these images. However, this step needs to compensate for images where endoscopic regions are not entirely round. In the third step we filter out the ophthalmic images which have shape characteristics very similar to the endoscopic images. We do this by using text information, specifically, anatomy terms, extracted from the figure caption. We tested and evaluated our method on a dataset of 115,370 photograph figures, and achieved promising precision and recall rates of 87% and 84%, respectively.

Biomedical image segmentation for semantic visual feature extraction

Biomedical photographs comprise diverse optically acquired images. Accurate classification into meaningful subclasses is valuable in biomedical image retrieval systems. Conventional visual descriptors are limited in their ability to assign semantic labels to images for meaningful retrieval. In this paper we propose a Markov random field (MRF)-based biomedical image segmentation method to segment images into meaningful regions that can be associated with semantic labels. We focus on several tissue image types and develop two MRF models: (i) for tissue image detection from large photograph collection; and, (ii) for region segmentation and semantic labeling. Experimental results demonstrate that our method can detect tissue images in about 82% precision, and our proposed visual descriptors computed from the segmentation results outperform existing visual descriptors. This latter result can be effectively used in biomedical image retrieval systems for retrieving tissue images.

Annotating image ROIs with text descriptions for multimodal biomedical document retrieval

Regions of interest (ROIs) that are pointed to by overlaid markers (arrows, asterisks, etc.) in biomedical images are expected to contain more important and relevant information than other regions for biomedical article indexing and retrieval. We have developed several algorithms that localize and extract the ROIs by recognizing markers on images. Cropped ROIs then need to be annotated with contents describing them best. In most cases accurate textual descriptions of the ROIs can be found from figure captions, and these need to be combined with image ROIs for annotation. The annotated ROIs can then be used to, for example, train classifiers that separate ROIs into known categories (medical concepts), or to build visual ontologies, for indexing and retrieval of biomedical articles. We propose an algorithm that pairs visual and textual ROIs that are extracted from images and figure captions, respectively. This algorithm based on dynamic time warping (DTW) clusters recognized pointers into groups, each of which contains pointers with identical visual properties (shape, size, color, etc.). Then a rule-based matching algorithm finds the best matching group for each textual ROI mention. Our method yields a precision and recall of 96% and 79%, respectively, when ground truth textual ROI data is used.

Classification of Visual Signs in Abdominal CT Image Figures in Biomedical Literature

“Imaging signs” are a critical part of radiology’s language. They not only are important for conveying diagnosis, but may also aid in indexing radiology literature and retrieving relevant cases and images. Here we report our work towards representing and categorizing imaging signs of abdominal abnormalities in figures in the radiology literature. Given a region-of-interest (ROI) from a figure, our goal was to assign a correct imaging sign label to that ROI from the following seven: accordion, comb, ring, sandwich, small bowel feces, target, or whirl. As training and test data, we created our own “gold standard” dataset of regions containing imaging signs. We computed 2997 feature attributes to represent imaging sign characteristics for each ROI in training and test sets. Following feature selection they were reduced to 70 attributes and were input to a Support Vector Machine classifier. We applied image-enhancement methods to compensate for variable quality of the images in radiology articles. In particular we developed a method for automatic detection and removal of pointers/markers (arrows, arrowheads, and asterisk symbols) on the images. These pointers/markers are valuable for approximately locating ROIs; however, they degrade the classification because they are often (partially) included in the training ROIs. On a test set of 283 ROIs, our method achieved an overall accuracy of 70% in labeling the seven signs, which we believe is a promising result for using imaging signs to search/retrieve radiology literature. This work is also potentially valuable for the creation of a visual ontology of biomedical imaging entities.