Gal Ashour

Automatic generation of conference video proceedings

How many times did you miss a conference talk in a parallel track and wish you had a second chance to see it? Or you just wanted to see a few talks from a recent conference you did not attend? Video proceedings, which contain videos of all the conference talks, would be of great value in these cases. With recent progress in digital video, streaming technology, large storage, Internet and especially video indexing and retrieval technology, video proceedings finally become a reality. The key challenges are efficient production with minimal labor and easy, intuitive, and rapid user access to talks and thought-for snippet of information. This paper describes an application that allows a nearly automatic, real time creation of video proceedings. All the talks are captured in video, and are automatically indexed by speech recognition and video analysis tools. Free text search in speech and efficient multi-view video browsing are combined with the conference table of contents and speakers biography to make fully searchable and browsable video proceedings. The paper covers the work flow, processing steps, and technical details of the video segmentation, visualization, and user study results. The system was used to produce video proceedings for three local conferences.

Towards automatic real time preparation of on-line video proceedings for conference talks and presentations

How many times did you miss a conference talk on a parallel track and wish you had a second chance to see it? Or you just wanted to see a few talks from a conference you did not attend? Video proceedings, which contain videos of all the conference talks, would be of a great value in these cases and in many others. With recent progress in digital video, streaming technology, large storage, Internet and especially video indexing and retrieval technology, video proceedings finally become a reality. The key challenge is to create it in an efficient way and to provide the user with easy, intuitive and rapid access to the talks and the snippets of video that he is looking for. This paper describes an application that allows a nearly automatic real time creation of video proceedings. All the talks are captured in video, and are automatically indexed by speech recognition and video analysis tools. The abstracts and speakers biography are extracted from the text proceedings and are converted to web pages. Free text search in speech and efficient multi-view video browsing are combined with a table of contents to compose fully searchable and browsable video proceedings. The paper covers different aspects of the problem, an overview of the CueVideo system and examples from two conferences that we have processed using this system.

Moving camera analytics: Emerging scenarios, challenges, and applications

In recent years, more than one billion cameras have been actively used on moving platforms, and various video analytics applications for moving cameras are emerging in diverse areas. Each of these applications can generate very large datasets to analyze. In this paper, we present practical approaches to handle and summarize the video content from moving cameras and use the application of unmanned aerial vehicles (UAVs) as an example. Our system is enabled by a geographic information system (GIS), and video summarization is used to generate efficient representations of videos on the basis of moving object detection and tracking. The approach automatically creates a panorama for videos and includes a novel registration method we developed for normalizing 3D distortion. Our experimental results on the UAV dataset show that we can accomplish a 10,000-fold data reduction without losing significant activities of interest. We also present a summary of the emerging landscape of mobile video analytics and how it may evolve in conjunction with other distributed mobile and static sensor networks.

Architecture for varying multimedia formats

In recent years, a few transitions in multimedia applications may be observed. We can identify at least two trends. Firstly, multimedia is being introduced in mainstream applications, leaving behind its traditional focus on highly professional markets, and on the gaming enhancement arena. Secondly, standardization bodies continue to work on media standards in order to provide a common approach to enable interoperability, better quality and efficiency under specified constraints. These new media standards are then added to existing archives of media, spanning a broad spectrum of legacy media standards. The result of these two trends is that a typical multimedia application, in order to be effective, needs to support many input types and provide the user with a seamless and transparent behavior. This paper discusses a pragmatic approach to this problem based on the object-oriented paradigm for real-world multimedia applications.

Optimal Sequential Grouping for Robust Video Scene Detection Using Multiple Modalities

Video scene detection is the task of dividing a video into semantic sections. To perform this fundamental task, we propose a novel and effective method for temporal grouping of scenes using an arbitrary set of features computed from the video. We formulate the task of video scene detection as a generic optimization problem to optimally group shots into scenes, and propose an efficient procedure for solving the optimization problem based on a novel dynamic programming scheme. This unique formulation directly results in a temporally consistent segmentation, and has the advantage of being parameter-free, making it applicable across various domains. We provide detailed experimental results, showing that our algorithm outperforms current state-of-the-art methods. To assess the comprehensiveness of this method even further, we present experimental results testing different types of modalities and their applicability in this formulation.

Robust and Efficient Video Scene Detection Using Optimal Sequential Grouping

Video scene detection is the task of dividing a video into semantic sections. We propose a novel and effective method for temporal grouping of scenes using an arbitrary set of features computed from the video. We formulate the task of video scene detection as a general optimization problem and provide an efficient solution using dynamic programming. Our unique formulation allows us to directly obtain a temporally consistent segmentation, unlike many existing methods, and has the advantage of being parameter-free. We also present a novel technique to estimate the number of scenes in the video using Singular Value Decomposition (SVD) as a low-rank approximation of a distance matrix. We provide detailed experimental results, showing that our algorithm outperforms current state of the art methods. In addition, we created a new Open Video Scene Detection (OVSD) dataset which we make publicly available on the web. The ground truth scene annotation was objectively created based on the movie scripts, and the open nature of the dataset makes it available for both academic and commercial use, unlike existing datasets for video scene detection.

Robust video scene detection using multimodal fusion of optimally grouped features

Video scene detection, the task of temporally dividing a video into its semantic sections, is an important process for effective analysis of heterogeneous video content. With the increased amount of video available for consumption, video scene detection becomes more and more important by providing means for effective video summarization, search and retrieval, browsing, and video understanding. We formulate the problem of video scene detection as a generic optimization problem aimed at partitioning a video given a set of features derived from multiple modalities. By optimally grouping consecutive shots into scenes, our method presents an effective and efficient solution for dividing a video into sections using a unique dynamic programming scheme. Unlike existing methods, it allows us to directly obtain temporally consistent video scene detection and has the advantage of being parameter-free, making it robust and applicable to various types of video content. Our experimental results show that our proposed multimodal approach can provide a significant gain compared to using only a single modality (e.g., either video or audio alone). Additionally, our method outperforms the state of the art in video scene detection, clearly demonstrating the effectiveness of the proposed method. As part of this work, we also provide a significant extension to our Open Video Scene Detection dataset (OVSD), which comprises open licensed videos freely available for academic and industrial use. This extension, which increases the OVSDs cumulative duration from the original 2.5 hours to over 17 hours, makes this dataset the most extensive evaluation tool for the problem of video scene detection.

Optimally Grouped Deep Features Using Normalized Cost for Video Scene Detection

Video scene detection is the task of temporally dividing a video into its semantic sections. This is an important preliminary step for effective analysis of heterogeneous video content. We present a unique formulation of this task as a generic optimization problem with a novel normalized cost function, aimed at optimal grouping of consecutive shots into scenes. The mathematical properties of the proposed normalized cost function enable robust scene detection, also in challenging real-world scenarios. We present a novel dynamic programming formulation for efficiently optimizing the proposed cost function despite an inherent dependency between subproblems. We use deep neural network models for visual and audio analysis to encode the semantic elements in the video scene, enabling effective and more accurate video scene detection. The proposed method has two key advantages compared to other approaches: it inherently provides a temporally consistent division of the video into scenes, and is also parameter-free, eliminating the need for fine-tuning for different types of content. While our method can adaptively estimate the number of scenes from the video content, we also present a new non-greedy procedure for creating a hierarchical consensus-based division tree spanning multiple levels of granularity. We provide comprehensive experimental results showing the benefits of the normalized cost function, and demonstrating that the proposed method outperforms the current state of the art in video scene detection.