Behzad Shahraray

Applications of video-content analysis and retrieval

Managing multimedia data requires more than collecting the data into storage archives and delivering it via networks to homes or offices. We survey technologies and applications for video-content analysis and retrieval. We also give specific examples.

Automated generation of news content hierarchy by integrating audio, video, and text information

This paper addresses the problem of generating semantically meaningful content by integrating information from different media. The goal is to automatically construct a compact yet meaningful abstraction of the multimedia data that can serve as an effective index table, allowing users to browse through large amounts of data in a non-linear fashion with flexibility, efficiency, and confidence. We propose an integrated solution in the context of broadcast news that simultaneously utilizes cues from video, audio, and test to achieve the goal. Some experimental results are presented and discussed.

On the applications of multimedia processing to communications

The challenge of multimedia processing is to provide services that seamlessly integrate text, sound, image, and video information and to do it in a way that preserves the ease of use and interactivity of conventional plain old telephone service (POTS) telephony. To achieve this goal, there are a number of technological problems that must be considered, including: compression and coding of multimedia signals, including algorithmic issues, standards issues, and transmission issues; synthesis and recognition of multimedia signals, including speech, images, handwriting, and text; organization, storage, and retrieval of multimedia signals, including the appropriate method and speed of delivery, resolution, and quality of service; access methods to the multimedia signal, including spoken natural language interfaces, agent interfaces, and media conversion tools; searching by text, speech, and image queries; browsing by accessing the text, by voice, or by indexed images. In each of these areas, a great deal of progress has been made in the past few years, driven in part by the relentless growth in multimedia personal computers and in part by the promise of broad-band access from the home and from wireless connections. Standards have also played a key role in driving new multimedia services, both on the POTS network and on the Internet. It is the purpose of this paper to review the status of the technology in each of the areas listed above and to illustrate current capabilities by describing several multimedia applications that have been implemented at AT&T Labs over the past several years.

Pictorial transcripts: multimedia processing applied to digital library creation

This paper describes a working system for the automated archiving and selective retrieval of textual, pictorial and auditory information contained in video programs. Video processing performs the task of representing the visual information using a small subset of the video frames. Linguistic processing refines the closed caption text, generates table of contents, and creates links to relevant multimedia documents. Audio and video information are compressed and indexed based on their temporal association with the selected video frames and processed text. The derived information is used to automatically generate a hypermedia rendition of the program contents. This provides a compact representation of the information contained in the video program. It also serves as a textual and pictorial index for selective retrieval of the full-motion video program. This fully automatic system generates HyperText Markup Language (HTML) renditions of television programs, and makes them available for access over the Internet within seconds of their broadcast. This digital library currently contains over 2200 hours of television programs.

A Fast, Comprehensive Shot Boundary Determination System

The proposed shot boundary determination (SBD) algorithm contains a set of finite state machine (FSM) based detectors for pure cut, fast dissolve, fade in, fade out, dissolve, and wipe. Support vector machines (SVM) are applied to the cut and dissolve detectors to further boost performance. Our SBD system was highly effective when evaluated in TRECVID 2006 (TREC video retrieval evaluation) and its performance was ranked highest overall.

Effective and scalable video copy detection

Video copy detection techniques are essential for a number of applications including discovering copyright infringement of multimedia content, monitoring commercial air time, and querying videos by example. Over the last decade, video copy detection has received rapidly growing attention from the multimedia research community. To encourage more innovative technology and benchmark the state of the art approaches in this field, the TRECVID conference series, sponsored by the NIST, initiated an evaluation task on content based copy detection in 2008. In this paper, we describe the content-based video copy detection framework developed at AT&T Labs - Research. We employed local visual features to match the video content, and adopted locality sensitve hashing and random sample consensus techniques to maintain the scalability and the robustness of our approach. Experimental results on TRECVID 2008 data show that our approach is effective and efficient.

Multimedia access and retrieval (panel session): the state of the art and future directions

Several years have passed since the research topic of content based multimedia retrieval emerged. We have witnessed the burgeoning research activities into a plenitude of new indexing, retrieval, and filtering tools for images, video, audio, music, graphics, and their combinations with text-based information. Exciting research opportunities arise when integrating knowledge from multiple disciplines, such as media content processing, database, information retrieval, and machine user interface. In the commercial domain, we have also witnessed several impressive efforts moving technologies into practical arenas.

The MIRACLE video search engine

This paper presents some of the searching and browsing features of the MIRACLE Video Search Engine. The rapid increase in the generation and dissemination of information and entertainment in video form has created a need for video search engines that facilitate finding and browsing relevant information. MIRACLE is an ongoing project at AT&T Labs aimed at addressing this need. This video search engine combines existing metadata with content-based information that is automatically extracted from the media components, or is obtained from other sources, to achieve this goal. A Web-based user interface provides browsing mechanisms that take advantage of the user’s perceptual abilities to refine the search results. The MIRACLE search engine currently operates on an archive of more than 32,000 hours of video that have been collected and automatically indexed over a ten year period.

Multimedia content acquisition and processing in the MIRACLE system

This paper describes the content acquisition, indexing and repurposing components of the MIRACLE system. MIRACLE is an ongoing research project at AT&T Labs aimed at creating automated content-based media processing algorithms and systems to collect, organize, index, mine, and repurpose video and multimedia information. While the retrieval engine and the user interface are the most visible part of the query process, performing the required processing to extract the relevant information and generate the indices for the retrieval engine is the most crucial and challenging component of the overall system. Moreover, the utility of the system is highly dependent on the quality and quantity of the acquired data. We discuss a scalable processing architecture for extracting information that enables efficient searching and browsing of more than 32,000 hours of video programs. We describe a new version of the acquisition system that takes advantage of Personal Video Recorder (PVR) functionality of Windows OS to acquire high-quality video and the associated metadata.

MediaAlert - a broadcast video monitoring and alerting system for mobile users

We present a system for automatic monitoring and timely dissemination of multimedia information to a range or mobile information appliances based on each users interest profile. Multimedia processing algorithms detect and isolate relevant video segments from over twenty television broadcast programs based on a collection or words and phrases specified by the user. Content repurposing techniques are then used to convert the information into a form that is suitable for delivery to the users mobile devices. Alerts are sent using a number of application messaging and network access protocols including email, short message service (SMS), multimedia messaging service (MMS), voice, session initiation protocol (SIP), fax, and pager protocols. The system is evaluated with respect to performance and user experiences. The MediaAlert system provides an effective and low-cost solution for the timely generation of alerts containing personal, business, and security information.