De Xu

Attention-driven salient edge(s) and region(s) extraction with application to CBIR

Selective visual attention plays an important role for humans to understand an image by intuitively emphasizing some salient parts. Such mechanism can be well applied in localized content-based image retrieval, due to the fact that in the context of CBIR, the user is only interested in a portion of the image and the rest of the image is irrelevant. Being aware of this, in this paper, the selective visual attention model (SVAM) is incorporated in the CBIR task to estimate the users retrieval concept. In contrast with existing learning based retrieval algorithms which need relevance feedback strategy to get users high-level semantic information, the proposed method does not need any users interaction to provide the training data. From this point of view, our method can be regarded as the purely bottom-up manner while learning based algorithms belong to the top-down manner. Specifically, an improved saliency map computing algorithm is employed first. Then, based on the saliency map, an efficient salient edges and regions detection method is introduced. Moreover, the concepts of salient edge histogram descriptors (SEHDs) and salient region adjacency graphs (SRAGs) are proposed, respectively, for images similarity comparison. Finally, an integrated strategy is adopted for content-based image retrieval. Experiments show that the proposed algorithm can characterize the human perception well and achieve satisfying retrieval performance.

Fusing color and texture features for background model

Background subtraction is a method typically used to segment moving regions in image sequences taken from a static camera by comparing each new frame to a model of the scene background. We present a novel approach that uses fuzzy integral to fuse the texture and color features for background subtraction. The method could handle various small motions of background objects such as swaying tree branches and bushes. Our method requires less computational cost. The model adapts quickly to changes in the scene that enables very sensitive detection of moving targets. The results show that the proposed method is effective and efficient in real-time and accurate background maintenance in complex environment.

Information theoretic metrics in shot boundary detection

A favorable difference metric is crucial to the shot boundary detection (SBD) performance. In this paper, we propose a new set of metrics, information theoretic metrics, to quantitatively measure the changes between frames. It includes image entropy difference, joint entropy, conditional entropy, mutual information and divergence. They all can be used to cut detection. Specially, the image entropy and joint entropy are good clues to fade detection, while mutual information, joint entropy and conditional entropy are less sensitive to illumination variations. The theoretic analysis and experimental results show that they are useful in SBD.

Shot Type Classification in Sports Video Based on Visual Attention

In this paper, we present a new method for classifying shot type in sports video based on visual attention. The problem is important for applications such as video structure analysis and content understanding. In particular, two-stage off-line learning processes perform knowledge extraction of semantic concepts and automatic shot classification, respectively. In the first stage, the extracted prominent regions are used as a good pattern in semantic concept level. Then a number of global features are defined as efficient input of the shot type classifier in the second stage. The identification of semantic concepts and classification of shot are based on human visual system. Hence, this framework can adequately capture the uncertainty or ambiguity of scales of a shot. Experimental results show the excellent performance of the approach.

Hierarchical video summarization based on video structure and highlight

Video summarization is a significant scheme to organize massive video data, and implement a meaningful rapid navigation of video. In this paper, we propose a hierarchical video summarization approach based on video structure and highlight. We extract video structure unit, and measure the unit (frame, shot and scene) importance rank based on visual and audio attention models. According to the unit importance rank, the skim ratio and key frame ratio are assigned to the different video units. Thus we achieve a hierarchical video summary. Experimental results show the excellent performance of the approach.

A novel region-based image retrieval algorithm using selective visual attention model

Selective Visual Attention Model (SVAM) plays an important role in region-based image retrieval. In this paper, a robust and accurate method for salient region detection is proposed which integrates SVAM and image segmentation. After that, the concept of salient region adjacency graphs (SRAGs) is introduced for image retrieval. The whole process consists of three levels. First in the pixel-level, the salient value of each pixel is calculated using an improved spatial-based attention model. Then in the region-level, the salient region detection method is presented. Furthermore, in the scene-level, salient region adjacency graphs (SRAGs) are introduced to represent the salient groups in the image, which take the salient regions as root nodes. Finally, the constructed SRAGs are used for image retrieval. Experiments show that the proposed method works well.

Combining Attention Model with Hierarchical Graph Representation for Region-Based Image Retrieval

The manifold-ranking algorithm has been successfully adopted in content-based image retrieval (CBIR) in recent years. However, while the global low-level features are widely utilized in current systems, region-based features have received little attention. In this paper, a novel attention-driven transductive framework based on a hierarchical graph representation is proposed for region-based image retrieval (RBIR). This approach can be characterized by two key properties: (1) Since the issue about region significance is the key problem in region-based retrieval, a visual attention model is chosen here to measure the regions significance. (2) A hierarchical graph representation which combines region-level with image-level similarities is utilized for the manifold-ranking method. A novel propagation energy function is defined which takes both low-level visual features and regional significance into consideration. Experimental results demonstrate that the proposed approach shows the satisfactory retrieval performance compared to the global-based and the block-based manifold-ranking methods.

Generic solution for image object recognition based on vision cognition theory

Human vision system can understand images quickly and accurately, but it is impossible to design a generic computer vision system to challenge this task at present. The most important reason is that computer vision community is lack of effective collaborations with visual psychologists, because current object recognition systems use only a small subset of visual cognition theory. We argue that it is possible to put forward a generic solution for image object recognition if the whole vision cognition theory of different schools and different levels can be systematically integrated into an inherent computing framework from the perspective of computer science. In this paper, we construct a generic object recognition solution, which absorbs the pith of main schools of vision cognition theory. Some examples illustrate the feasibility and validity of this solution.

A Novel Graph Kernel Based SVM Algorithm for Image Semantic Retrieval

It has been shown that support vector machines (SVM) can be used in content-based image retrieval. Existing SVM based methods only extract low-level global or region-based features to form feature vectors and use traditional non-structured kernel function. However, these methods rarely consider the image structure or some new structured kernel types. In order to bridge the semantic gap between low-level features and high-level concepts, in this paper, a novel graph kernel based SVM method is proposed, which takes into account both low-level features and structural information of the image. Firstly, according to human selective visual attention model, for a given image, salient regions are extracted and the concept of Salient Region Adjacency Graph (SRAG) is proposed to represent the image semantics. Secondly, based on the SRAG, a novel graph kernel based SVM is constructed for image semantic retrieval. Experiments show that the proposed method shows better performance in image semantic retrieval than traditional method.

Two important action scenes detection based on probability neural networks

In this paper, an effective classification approach for action scenes is proposed, which exploits the film grammar used by filmmakers as guideline to extract features, detect and classify action scenes. First, action scenes are detected by analyzing film rhythm of video sequence. Then four important features are extracted to characterize chase and fight scenes. After then the Probability Neural Networks is employed to classify the detected action scenes into fight, chase and uncertain scenes. Experimental results show that the proposed method works well over the real movie videos.