Liam M. Mayron

Using visual attention to extract regions of interest in the context of image retrieval

Recent research on computational modeling of visual attention has demonstrated that a bottom-up approach to identifying salient regions within an image can be applied to diverse and practical problems for which conventional machine vision techniques have not succeeded in producing robust solutions. This paper proposes a new method for extracting regions of interest (ROIs) from images using models of visual attention. It is presented in the context of improving content-based image retrieval (CBIR) solutions by implementing a biologically-motivated, unsupervised technique of grouping together images whose salient ROIs are perceptually similar. In this paper we focus on the process of extracting the salient regions of an image. The excellent results obtained with the proposed method have demonstrated that the ROIs of the images can be independently indexed for comparison against other regions on the basis of similarity for use in a CBIR solution.

An attention-driven model for grouping similar images with image retrieval applications

Recent work in the computational modeling of visual attention has demonstrated that a purely bottom-up approach to identifying salient regions within an image can be successfully applied to diverse and practical problems from target recognition to the placement of advertisement. This paper proposes an application of a combination of computational models of visual attention to the image retrieval problem. We demonstrate that certain shortcomings of existing content-based image retrieval solutions can be addressed by implementing a biologically motivated, unsupervised way of grouping together images whose salient regions of interest (ROIs) are perceptually similar regardless of the visual contents of other (less relevant) parts of the image. We propose a model in which only the salient regions of an image are encoded as ROIs whose features are then compared against previously seen ROIs and assigned cluster membership accordingly. Experimental results show that the proposed approach works well for several combinations of feature extraction techniques and clustering algorithms, suggesting a promising avenue for future improvements, such as the addition of a top-down component and the inclusion of a relevance feedback mechanism.

Challenges and Opportunities in Video Coding for 3D TV

This paper explores the challenges opportunities in developing and deploying 3D TV services. The 3D TV services can be seen as a general case of the multi-view video that has been receiving significant attention lately. The keys to a successful 3D TV experience are the availability of content, the ease of use, the quality of experience, and the cost of deployment. Recent technological advances have made possible experimental systems that can be used to evaluate the 3D TV services. We have developed a 3D TV prototype and have currently conducting our first user study to evaluate the quality and experience. These experiences have allowed us to identify challenges and opportunities in developing 3D TV services

Design and evaluation of a 3D video system based on H.264 view coding

Recent advances in video compression and 3D displays have necessitated a further understanding and development of 3D video coding algorithms. The emergence of low cost autostereoscopic displays is expected to drive the growth of 3DTV services. This paper discusses key issues that affect the quality of 3D video experience on autostereoscopic displays. The characteristics of the human visual system can be exploited to compress individual stereo views at different qualities without affecting the perceptual quality of the 3D video. The H.264/AVC video coding algorithm was used to compress each view. We examine the bounds of asymmetric stereo view compression and its relationship to eye-dominance based on a user study. This paper also presents the design and development of a modular video player with stereoscopic and multi-view capabilities including a discussion of useful tools for accelerating the development and enhancing flexibility. The experimental results indicate that eye-dominance influences 3D perception and as a result will impact the coding efficiency of 3D video.

An unsupervised method for clustering images based on their salient regions of interest

We have developed a biologically-motivated, unsupervised way of grouping together images whose salient regions of interest (ROIs) are perceptually similar regardless of the visual contents of other (less relevant) parts of the image. In the implemented model cluster membership is assigned based on feature vectors extracted from salient ROIs. This paper focuses on the experimental evaluation of the proposed approach for several combinations of feature extraction techniques and unsupervised clustering algorithms. The results reported here show that this is a valid approach and encourage further research.

Extraction of Salient Regions of Interest Using Visual Attention Models

This paper presents a novel algorithm for extracting regions of interest (ROIs) from images in an unsupervised way. It relies on the information provided by two computational models of bottom-up visual attention, encoded in the form of the images salient points-of-attention (POAs) and areas-of-attention (AOAs). The proposed method combines these POAs and AOAs to generate binary masks that correspond to the ROIs within the image. First, each AOA is binarized through an adapted relaxation algorithm where the histogram entropy of the AOA measurement is the stop criterion of the iterative process. The AOAs are also smoothed with a Gaussian pyramid followed by interpolation. Next, the binary representation of the AOAs, the smoothed version of the AOAs, and the POAs are converted in a mask that covers the salient ROIs of the image. The proposed ROI extraction algorithm does not impose any constraints on the number or distribution of salient regions in the input image. Qualitative and quantitative results show that the proposed method performs very well in a wide range of images, whether natural or man-made, from simple images of objects against a homogeneous background to complex cluttered scenes.

On the Potential of Incorporating Knowledge of Human Visual Attention into Cbir Systems

Content-based image retrieval (CBIR) systems have been actively investigated over the past decade. Several existing CBIR prototypes claim to be designed based on perceptual characteristics of the human visual system, but even those who do are far from recognizing that they could benefit further by incorporating ongoing research in vision science. This paper explores the inclusion of human visual perception knowledge into the design and implementation of CBIR systems. Particularly, it addresses the latest developments in computational modeling of human visual attention. This fresh way of revisiting concepts in CBIR based on the latest findings and open questions in vision science research has the potential to overcome some of the challenges faced by CBIR systems

A Forward-Looking User Interface for CBIR and CFIR Systems

This demonstration highlights the benefits that image retrieval systems can enjoy by use of a thoughtful interface. We present a live demonstration of PRISM, a new Web-based application that, through a simple set of natural actions, allows the user the ability to elicit sophisticated responses from easily-formed queries. Additionally, the system was designed with not only content-based, but also content-free image retrieval and semantic annotation in mind. The demonstration system uses an attention-based method for extracting and determining the relevance of salient regions of interest in an unsupervised way, but can be adapted to a variety of implementations

3D TV using MPEG-2 and H.264 view coding and autostereoscopic displays

There is a renewed interest in the 3DTV research primarily due to the advances in low cost 3D display technologies. The two views required for 3DTV can be compressed using standard video compression techniques. MPEG-2 is widely used in digital TV applications today and H.264/MPEG-4 AVC is expected to be the leading video technology standard for digital video in the near future. The compression gains and quality of 3DTV will vary depending on the video coding standard used. While inter-view prediction will likely improve the compression efficiency, new approaches such as asymmetric view coding are necessary to greatly reduce bandwidth requirements for 3DTV. This demonstration will show the quality of 3D video experience on autostereoscopic displays using H.264 and MPEG-2. We will also show the benefits of using asymmetric view coding as well as the role of eye dominance on 3DTV experience.

Using a game to evaluate image retrieval, organization, and annotation

Methods of retrieving images that incorporate human- generated metadata, such as keyword annotation and collaborative filtering, are less vulnerable to the semantic gap than content-based image retrieval. However, generating such metadata is time-consuming, expensive, and difficult to evaluate. This paper discusses an interface for image retrieval that is able to simultaneously compose queries based on content- based features, keywords, and collaborative relations. We propose using a game metaphor to evaluate the system.