Michael G. Christel

Evolving video skims into useful multimedia abstractions

This paper reports two studies that measured the effects of different “video skim” techniques on comprehension, navigation, and user satisfaction. Video skims are compact, content-rich abstractions of longer videos, condensations that preserve frame rate while greatly reducing viewing time. Their characteristics depend on the image- and audio-processing techniques used to create them. Results from the initial study helped refine video skims, which were then reassessed in the second experiment. Significant benefits were found for skims built from audio sequences meeting certain criteria.

Lessons learned from building a terabyte digital video library

The Informedia Project at Carnegie Mellon University has created a terabyte digital video library in which automatically derived descriptors for the video are used for indexing, segmenting and accessing the library contents. Begun in 1994, the project presented numerous challenges for library creation and deployment, valuable information covered in this article. The authors, developers of the project at Carnegie Mellon University, addressed these challenges by: automatically extracting information from digitized video; creating interfaces that allowed users to search for and retrieve videos based on extracted information; and validating the system through user testbeds. Through speech, image, and natural language processing, the Informedia Project has demonstrated that previously inaccessible data can be derived automatically and used to describe and index video segments.

Successful approaches in the TREC video retrieval evaluations

This paper reviews successful approaches in evaluations of video retrieval over the last three years. The task involves the search and retrieval of shots from MPEG digitized video recordings using a combination of automatic speech, image and video analysis and information retrieval technologies. The search evaluations are grouped into interactive (with a human in the loop) and non-interactive (where the human merely enters the query into the system) submissions. Most non-interactive search approaches have relied extensively on text retrieval, and only recently have image-based features contributed reliably to improved search performance. Interactive approaches have substantially outperformed all non-interactive approaches, with most systems relying heavily on the users ability to refine queries and reject spurious answers. We will examine both the successful automatic search approaches and the user interface techniques that have enabled high performance video retrieval.

Video Retrieval Based on Semantic Concepts

An approach using many intermediate semantic concepts is proposed with the potential to bridge the semantic gap between what a color, shape, and texture-based ldquolow-levelrdquo image analysis can extract from video and what users really want to find, most likely using text descriptions of their information needs. Semantic concepts such as cars, planes, roads, people, animals, and different types of scenes (outdoor, night time, etc.) can be automatically detected in the video with reasonable accuracy. This leads us to ask how can they be used automatically and how does a user (or a retrieval system) translate the users information need into a selection of related concepts that would help find the relevant video clips, from the large list of available concepts. We illustrate how semantic concept retrieval can be automatically exploited by mapping queries into query classes and through pseudo-relevance feedback. We also provide evidence how a semantic concept can be utilized by users in interactive retrieval, through interfaces that provide affordances of explicit concept selection and search, concept filtering, and relevance feedback. How many concepts we actually need and how accurately they need to be detected and linked through various relationships is specified in the ontology structure.

Interactive maps for a digital video library

The Informedia Digital Video Library contains over 1200 hours of video. Through automatic processing, descriptors are derived for the video to improve library access. A new extension to the video processing is the extraction of geographic references from these descriptors. The operational library interface shows the geographic entities addressed in a given story, highlighting the regions discussed at any point in the video through a map display, synchronized with the video playback. The map can also be used as a query mechanism, allowing users to search the terabyte library for stories taking place in a selected area of interest.

Information Visualization Within a Digital Video Library

The Informedia Digital Video Library contains over a thousand hours of video, consuming over a terabyte of disk space. This paper summarizes the multimedia abstractions used to represent this video in prior systems and introduces the visualization techniques employed to browse and navigate multiple video documents at once.

Collages as dynamic summaries for news video

This paper introduces the video collage, a novel effective interface for browsing and interpreting video collections. The paper discusses how collages are automatically produced, illustrates their use, and evaluates their effectiveness as summaries across news stories. Collages are presentations of text and images derived from multiple video sources, which provide an interactive visualization for a set of video documents, summarizing their contents and providing a navigation aid for further exploration. The dynamic creation of collages is based on user context, e.g., an originating query, coupled with automatic processing to refine the candidate imagery. Named entity identification and common phrase extraction provides descriptive text. The dynamic manipulation of collages allows user-directed browsing and reveals additional detail. The utility of collages as summaries is examined with respect to other published news summaries.

Adjustable filmstrips and skims as abstractions for a digital video library

Filmstrips and video skims are two presentation schemes for abstracting information in a digital video segment. Filmstrips present information all at once in a static form, while video skims are played and disclose information temporally. The paper discusses the evolution of the filmstrip and skim interfaces in the Informedia Digital Video Library. Filmstrips are commonly deployed as interfaces for video and image libraries, but we found initial Informedia filmstrips and skims received little use. We discuss the interface considerations motivating the redesign of filmstrips and skims to adjust their presentations dynamically based on user context and preference.

Multimedia abstractions for a digital video library

Multimedia abstractions form essential components of digital video libraries because they enable a user to determine a video’s distinguishing content without investing long viewing times or requiring high networktransfer speeds. This paper presents usage and evaluation data for abstractions implemented the Informedia Digital Video Library, and discusses implications for video delivery over the Web.

Techniques for the Creation and Exploration of Digital Video Libraries

The Information Age is fully upon us. A recent article noted that there are perhaps 50 million people using the Internet on a regular basis, and That “the current growth rate is about 15% per month (!) and this could well continue until almost all of those in the ‘developed world’ are connected” [FM94, p. 30]. In addition, the digital domain consists not only of text but increasingly of other media representations, from graphics images to audio to motion video. As the amount of information and number of users exponentially escalate, more attention focuses on the basic problems of information management: How do you digitize information? How can you then visualize it and find what you need? How do you use and manipulate it effectively? How is it stored and managed? The proliferation of technical articles and special issues addressing these questions underscore their importance; see for example the special issue on Content-based retrieval [Nar95] or digital libraries [F+95]. This chapter will survey some of that work, especially that which relates to the treatment of video and the use of digital video libraries For education.