Amit Singhal

Probabilistic spatial context models for scene content understanding

Scene content understanding facilitates a large number of applications, ranging from content-based image retrieval to other multimedia applications. Material detection refers to the problem of identifying key semantic material types (such as sky, grass, foliage, water, and snow in images). In this paper, we present a holistic approach to determining scene content, based on a set of individual material detection algorithms, as well as probabilistic spatial context models. A major limitation of individual material detectors is the significant number of misclassifications that occur because of the similarities in color and texture characteristics of various material types. We have developed a spatial context-aware material detection system that reduces misclassification by constraining the beliefs to conform to the probabilistic spatial context models. Experimental results show that the accuracy of materials detection is improved by 13% using the spatial context models over the individual material detectors themselves.

On measuring low-level saliency in photographic images

Measuring perceptual saliency of regions in a scene is important for determining regions of interest. Color, texture and shape cues are good low-level features for detecting saliency. While self saliency refers to intrinsic attributes of a region, relative saliency as used to measure how salient a region is relative to its surrounding and, thus, needs to be defined within a spatial context. A few spatial context models are investigated in this study. In particular, we propose an auto-scaled, amorphous neighborhood as the context model to obtain reliable measurements of relative saliency features. Comparison of three context models has shown that the proposed model is capable of generating predicates more consistent with perceived saliency.

A computational approach to determination of main subject regions in photographic images

Abstract We present a computational approach to main subject detection, which provides a measure of saliency or importance for different regions that are associated with different subjects in an image with unconstrained scene content. It is built primarily upon selected image semantics, with low-level vision features also contributing to the decision. The algorithm consists of region segmentation, perceptual grouping, feature extraction, and probabilistic reasoning. To accommodate the inherent ambiguity in the problem as reflected by the ground truth (probabilistic in nature), we have developed a novel training mechanism for Bayes nets based on fractional frequency counting. Using a set of images spanning the ‘photo space,’ experimental results have shown the promise of our approach in that most of the regions that independent observers ranked as the main subject are also labeled as such by our system. In addition, without reorganization and retraining, the Bayes net-based framework lends itself to performance scalable configurations to suit different applications that have different requirements of accuracy and speed. This paper focuses on a high level description of the complete system used to solve the overall problem, while providing necessary descriptions of the component algorithms.

On the application of Bayes networks to semantic understanding of consumer photographs

Belief networks, such as Bayes nets, have emerged as an effective knowledge representation and inference engine in artificial intelligence and expert systems research. Their effectiveness is due to the ability to explicitly integrate domain knowledge in the network structure and to reduce a joint probability distribution to conditionally independence relationships. Current research in content-based image processing and analysis is largely limited to low-level feature extraction and classification. The ability to extract both low-level and semantic features and perform knowledge integration of different types of features would be very useful. We present a general knowledge integration framework that incorporates Bayes networks and has been used in two applications involving semantic understanding of consumer photographs. The first application aims at detecting main photographic subjects in an image and the second aims at selecting the most appealing image in an event. With these diverse examples, we demonstrate that effective inference engines can be built according to specific domain knowledge and available training data to solve inherently uncertain vision problems.

Visual salience metrics for image inpainting

Quantitative metrics for successful image inpainting currently do not exist, with researchers instead relying upon qualitative human comparisons to evaluate their methodologies and techniques. In an attempt to rectify this situation, we propose two new metrics to capture the notions of noticeability and visual intent in order to evaluate inpainting results. The proposed metrics use a quantitative measure of visual salience based upon a computational model of human visual attention. We demonstrate how these two metrics repeatably correlate with qualitative opinion in a human observer study, correctly identify the optimum uses for exemplar-based inpainting (as specified in the original publication), and match qualitative opinion in published examples.

Selective hidden random fields: Exploiting domain-specific saliency for event classification

Classifying an event captured in an image is useful for understanding the contents of the image. The captured event provides context to refine models for the presence and appearance of various entities, such as people and objects, in the captured scene. Such contextual processing facilitates the generation of better abstractions and annotations for the image. Consider a typical set of consumer images with sports-related content. These images are taken mostly by amateur photographers, and often at a distance. In the absence of manual annotation or other sources of information such as time and location, typical recognition tasks are formidable on these images. Identifying the sporting event in these images provides a context for further recognition and annotation tasks. We propose to use the domain-specific saliency of the appearances of the playing surfaces, and ignore the noninformative parts of the image such as crowd regions, to discriminate among different sports. To this end, we present a variation of the hidden-state conditional random field that selects a subset of the observed features suitable for classification. The inferred hidden variables in this model represent a selection criteria desirable for the problem domain. For sports-related images, this selection criteria corresponds to the segmentation of the playing surface in the image. We demonstrate the utility of this model on consumer images collected from the Internet.

Inpainting quality assessment

We propose a means of objectively comparing the results of digital image inpainting algorithms by analyzing changes in predicted human attention prior to and following application. Artifacting is generalized in two catagories, in-region and out-region, depending on whether or not attention changes are primarily within the edited region or in nearby (contrasting) regions. Human qualitative scores are shown to correlate strongly with numerical scores of in-region and out-region artifacting, including the effectiveness of training supervised classifiers of increasing complexity. Results are shown on two novel human-scored datasets.

Natural object detection in outdoor scenes based on probabilistic spatial context models

Natural object detection in outdoor scenes, i.e., identifying key object types such as sky, grass, foliage, water, and snow, can facilitate content-based applications, ranging from image enhancement to other multimedia applications. A major limitation of individual object detectors is the significant number of misclassifications that occur because of the similarities in color and texture characteristics of various object types and lack of context information. We have developed a spatial context-aware object-detection system that first combines the output of individual object detectors to produce a composite belief vector for the objects potentially present in an image. Spatial context constraints, in the form of probability density functions obtained by learning, are subsequently used to reduce misclassification by constraining the beliefs to conform to the spatial context models. Experimental results show that the spatial context models improve the accuracy of natural object detection by 13% over the individual object detectors themselves.

Normalized Kemeny and Snell distance: a novel metric for quantitative evaluation of rank-order similarity of images

There are needs for evaluating rank order-based similarity between images. Region importance maps from image understanding algorithms or human observer studies are ordered rankings of the pixel locations. We address three problems with Kemeny and Snells distance (d/sub KS/), an existing measure from ordinal ranking theory, when applied to images: its high-computational cost, its bias in favor of images with sparse histograms, and its image-size dependent range of values. We present a novel computationally efficient algorithm for computing d/sub KS/ between two images and we derive a normalized form d/sub KS/ with no bias whose range is independent of image size. For evaluating similarity between images that can be considered as ordered rankings of pixels, d/sub KS/ is subjectively superior to cross correlation.

Shape regularized active contour based on dynamic programming for anatomical structure segmentation

We present a method to incorporate nonlinear shape prior constraints into segmenting different anatomical structures in medical images. Kernel space density estimation (KSDE) is used to derive the nonlinear shape statistics and enable building a single model for a class of objects with nonlinearly varying shapes. The object contour is coerced by image-based energy into the correct shape sub-distribution (e.g., left or right lung), without the need for model selection. In contrast to an earlier algorithm that uses a local gradient-descent search (susceptible to local minima), we propose an algorithm that iterates between dynamic programming (DP) and shape regularization. DP is capable of finding an optimal contour in the search space that maximizes a cost function related to the difference between the interior and exterior of the object. To enforce the nonlinear shape prior, we propose two shape regularization methods, global and local regularization. Global regularization is applied after each DP search to move the entire shape vector in the shape space in a gradient descent fashion to the position of probable shapes learned from training. The regularized shape is used as the starting shape for the next iteration. Local regularization is accomplished through modifying the search space of the DP. The modified search space only allows a certain amount of deformation of the local shape from the starting shape. Both regularization methods ensure the consistency between the resulted shape with the training shapes, while still preserving DP’s ability to search over a large range and avoid local minima. Our algorithm was applied to two different segmentation tasks for radiographic images: lung field and clavicle segmentation. Both applications have shown that our method is effective and versatile in segmenting various anatomical structures under prior shape constraints; and it is robust to noise and local minima caused by clutter (e.g., blood vessels) and other similar structures (e.g., ribs). We believe that the proposed algorithm represents a major step in the paradigm shift to object segmentation under nonlinear shape constraints.