Yong Rui

Adaptive key frame extraction using unsupervised clustering

Key frame extraction has been recognized as one of the important research issues in video information retrieval. Although progress has been made in key frame extraction, the existing approaches are either computationally expensive or ineffective in capturing salient visual content. We first discuss the importance of key frame selection; and then review and evaluate the existing approaches. To overcome the shortcomings of the existing approaches, we introduce a new algorithm for key frame extraction based on unsupervised clustering. The proposed algorithm is both computationally simple and able to adapt to the visual content. The efficiency and effectiveness are validated by large amount of real-world videos.

Highlights Extraction from Unscripted Video

This chapter presents a sports highlights extraction framework, which is built on a hierarchical representation that includes play/break segmentation, audio-visual marker detection, audio-visual marker association, and finer-resolution highlight classification. It decomposes the semantic and subjective concepts of “sports highlights” to events at different layers. The key component in this framework is the detection of audio and visual objects that serve as the bridge between the observed video signal and the semantics. It is a deviation from the “feature extraction + classification” paradigm for multimedia modeling, especially when the features are global features such as color histograms. Visual object detection also uses image features, but these features represent localized features, and spatial configuration of these local features. The experimental results have confirmed the advantage of this approach. This chapter has reported the results of sports highlights extraction based on audio classification and the correlation between the applause/cheering sound with exciting moments.

Video Table-of-Content Generation

This chapter presents a novel framework for scene-level video table of contents (ToC) construction. It presents an effective scene-level ToC construction technique based on intelligent, unsupervised clustering. It has the characteristics of better modeling the time locality and scene structure. Scene-level ToC has the advantage over other techniques. This chapter reviews and evaluates the video parsing techniques at various levels. There are four parameters in the proposed video ToC construction algorithm: W c , W A , groupThreshold , and sceneThreshold. In the proposed algorithm, Gaussian normalization is used for determining these four parameters. Experiments over real-world movie videos validate the effectiveness of the proposed approach. Examples are presented to demonstrate the use of the scene-based ToC to facilitate the users access to the video. The proposed approach provides an open framework for structure analysis of video features. It has four major modules: shot boundary detection and key frame extraction, spatiotemporal feature extraction, time-adaptive grouping, and scene structure construction.

Chapter 4 – Video Structure Discovery Using Unsupervised Learning

This chapter deals with video structure discovery using unsupervised learning. It presents a content-adaptive analysis and representation framework for audio event discovery from unscripted multimedia. The proposed framework is based on the observation that interesting events happen sparsely in a background of usual events. Three time series for audio event discovery were used: low-level audio features, frame-level audio classification labels, and 1-second-level audio classification. An inlier/outlier-based temporal segmentation of these three time series was performed. The segmentation was based on eigenvector analysis of the affinity matrix obtained from statistical models of the subsequences of the input time series. The detected outliers were also ranked based on deviation from the background process. Experimental results on a total of 12 hours of sports audio from three different genres—soccer, baseball, and golf from Japanese, American, and Spanish broadcasts—show that unusual events can be effectively extracted from such an inlier/outlier-based segmentation resulting from the proposed framework.

Chapter 7 – Applications

Publisher Summary This chapter reviews video summarization, browsing, and retrieval in different applications. These applications are grouped into three categories: consumer video, image/video databases management, and surveillance. For each category, some of the exemplar applications are also listed. Consumer electronic devices with content analysis technologies enable the end user to browse the recorded content in efficient ways. Content-based image/video retrieval solves the difficulties faced by text-based image retrieval. Instead of being manually annotated by text-based key words, images are indexed by their own visual content, such as color or texture. This chapter also reviews the limitations of the video summarization, browsing, and retrieval technologies. It points out the challenges of the current video summarization, browsing, and retrieval technology. The technology has evolved from laboratory research to many small, medium, or large-scale commercial deployments. At present, it is successful in small or medium-scale applications such as highlight generation for a particular sport, but still faces great technical challenges for large-scale deployments such as airport security and general surveillance.

Chapter 5 – Video Indexing

Publisher Summary This chapter focuses on video indexing. The basic video indexing task is to find objects in the search space that match the query object. The indexing task requires each object to have a consistent description and a similarity metric that helps establish how closely any two descriptions match. This chapter focuses on the descriptions based on low-level video and audio features. Video indexing lends itself to low and high-level feature-based approaches. The video indexing community has diligently examined the entire range of available low-level features and obtained interesting results. Evidence indicates that video indexing needs to bridge the gap between low-level features and content semantics to make a significant impact, both scientifically and technologically. This chapter concludes that audio features are a rich source of content semantics. Audio data consume much less bandwidth compared to video data, and audio classification enables much easier access to content semantics than visual analysis.