Hrishikesh Bhaumik

Hybrid soft computing approaches to content based video retrieval

Graphical abstractDisplay Omitted There has been an unrestrained growth of videos on the Internet due to proliferation of multimedia devices. These videos are mostly stored in unstructured repositories which pose enormous challenges for the task of both image and video retrieval. Users aim to retrieve videos of interest having content which is relevant to their need. Traditionally, low-level visual features have been used for content based video retrieval (CBVR). Consequently, a gap existed between these low-level features and the high level semantic content. The semantic differential was partially bridged by proliferation of research on interest point detectors and descriptors, which represented mid-level features of the content. The computational time and human interaction involved in the classical approaches for CBVR are quite cumbersome. In order to increase the accuracy, efficiency and effectiveness of the retrieval process, researchers resorted to soft computing paradigms. The entire retrieval task was automated to a great extent using individual soft computing components. Due to voluminous growth in the size of multimedia databases, augmented by an exponential rise in the number of users, integration of two or more soft computing techniques was desirable for enhanced efficiency and accuracy of the retrieval process. The hybrid approaches serve to enhance the overall performance and robustness of the system with reduced human interference. This article is targeted to focus on the relevant hybrid soft computing techniques which are in practice for content-based image and video retrieval.

Video Shot Segmentation Using Spatio-temporal Fuzzy Hostility Index and Automatic Threshold

Shot segmentation is an important preprocessing step towards content based video analysis. In this paper, we propose a Spatio-Temporal Fuzzy Hostility Index (STFHI) for determining the edges of objects present in the frames, composing the video. The edges present in the frames are treated as features of the frame. The correlation between the features is computed for successive frames of the video. An automatic threshold is set using the three-sigma rule, on the gradient of correlation values thus computed to detect hard cuts (abrupt transitions) in the video. The proposed method is able to accurately detect the hard cuts in a video. In an experimental evaluation on a heterogeneous test set, consisting of videos from sports, movie songs, music albums and documentaries, the proposed method achieves substantial improvement over the state of the art methods.

Towards redundancy reduction in storyboard representation for static video summarization

Static video summarization techniques aim to represent the salient content in a video by extracting a set of key-frames for presentation to the user. An efficient key-frame extraction process is thus vital for effective video summarization, browsing and indexing in content-based video retrieval systems. In this paper, a three phased approach for key-frame extraction is proposed which aims to represent a static summary of a video. The first phase deals with detecting the best representative frame(s) for each shot in the video. The second and third phases comprise of techniques for intra-shot and inter-shot redundancy reduction using SURF and GIST on the extracted key-frames in the first phase. At the end of each phase, comparison between the system generated summary and user summary is performed. A comparative analysis between SURF and GIST for redundancy reduction is also presented. Experimental evaluation of the results on a test set of videos from sports, movie songs, music albums and documentaries show that the proposed method achieves high precision and recall values in all the cases.

Enhancement of perceptual quality in static video summarization using minimal spanning tree approach

A video summarization technique is proposed in this work using minimal spanning tree (MST) of data points. The data points correspond to image frames of a shot in the video which is to be summarized. Correlation is chosen as a similarity metric for computing the edge weights of the MST. The representative frames for each shot are chosen by computing the density of each data point. A novel method for redundancy reduction is devised using SURF and GIST. The redundant frames are eliminated for concise representation of the video. The degree of reduction achieved by using the two approaches is also presented. The proposed method is assessed for perceptual quality against manual summarization. High values of precision and recall endorse the efficacy of the method. The system works well for different kinds of video without a priori knowledge about the type or content of it. Two datasets are considered for experimentation, one comprises short videos while the other consists of long videos. The method is found to provide satisfactory results for both the datasets.

Redundancy Elimination in Video Summarization

Video summarization is a task which aims at presenting the contents of a video to the user in a succinct manner so as to reduce the retrieval and browsing time. At the same time sufficient coverage of the contents is to be ensured. A trade-off between conciseness and coverage has to be reached as these properties are conflicting to each other. Various feature descriptors have been developed which can be used for redundancy removal in the spatial and temporal domains. This chapter takes an insight into the various strategies for redundancy removal. A method for intra-shot and inter-shot redundancy removal for static video summarization is also presented. High values of precision and recall illustrate the efficacy of the proposed method on a dataset consisting of videos with varied characteristics.

Real-Time Storyboard Generation in Videos Using a Probability Distribution Based Threshold

Storyboard generation in real-time is a task which aims at producing a meaningful summary of a video in a duration which is comparable to its length. The main challenge in real-time videos is to extract a set of key-frames on the fly, without having access to the full content. A fast and accurate generation of a dynamic threshold is important to ensure that the summary produced is a succinct representation of the video and appropriate in terms of content coverage. In this work, an efficient method has been developed for producing a static summary using a probability distribution based threshold. An open framework has been used in the redundancy reduction phase which may incorporate any of the state-of-the-art similarity matching techniques so as to eliminate the superfluous key-frames. The efficacy of the proposed method is depicted through the high recall and precision values obtained from the experimental results on several publicly available videos of different genre and length. The method has been tested on computers having different system configurations, thereby proving the efficacy of the proposed technique in real-time.

Towards Reliable Clustering of English Text Documents Using Correlation Coefficient

This paper proposes a new approach for clustering English text documents, based on finding the pair wise correlation of documents in a given set of text documents. The correlation coefficient for each pair of documents is calculated on the basis of ranks given to the words in the documents. The ranking of the words occurring in a document is computed on the basis of weights of the words calculated according to the conventional TF-IDF factor. The proposed method is found to be able to cluster a given set of text documents into a number of classes depending on their contents where the number of classes is not known a priori. It is revealed from experimental results that the proposed method of text categorization using correlation coefficient performs better than some of the other text categorization methods, including methods that use artificial neural network.

Intelligent Analysis of Multimedia Information

Multimedia represents information in novel and varied formats. One of the most prevalent examples of continuous media is video. Extracting underlying data from these videos can be an arduous task. From video indexing, surveillance, and mining, complex computational applications are required to process this data. Intelligent Analysis of Multimedia Information is a pivotal reference source for the latest scholarly research on the implementation of innovative techniques to a broad spectrum of multimedia applications by presenting emerging methods in continuous media processing and manipulation. This book offers a fresh perspective for students and researchers of information technology, media professionals, and programmers.

Dissolve detection in videos using an ensemble approach

Detection of shot transitions is an important preprocessing step for content based video retrieval systems. Out of the various types of transitions present in videos, dissolve detection is the most challenging one. This is due to the inherent complexity induced by the component frames making up the dissolve transition. In this work, a two-phased approach is presented for detecting the dissolve sequences. The first phase is concerned with identifying candidate dissolve sequences based on the parabolic nature of the mean fuzzy entropy data computed on the composing frames of the video. In the second phase, the candidates are filtered through multiple stages where each stage is based on a low-level feature of the video stream. The threshold in each stage is based on the data obtained for that feature from the constituent frames of the video. The final set of dissolve sequences are obtained at the end of the filtration stage. The proposed method is also able to detect the span of the dissolve sequence with an error of maximum one frame. Comparisons reveal that the proposed method outperforms the state-of-the-art methods in terms of both recall and precision.