John M. Gauch

Image segmentation and analysis via multiscale gradient watershed hierarchies

Multiscale image analysis has been used successfully in a number of applications to classify image features according to their relative scales. As a consequence, much has been learned about the scale-space behavior of intensity extrema, edges, intensity ridges, and grey-level blobs. We investigate the multiscale behavior of gradient watershed regions. These regions are defined in terms of the gradient properties of the gradient magnitude of the original image. Boundaries of gradient watershed regions correspond to the edges of objects in an image. Multiscale analysis of intensity minima in the gradient magnitude image provides a mechanism for imposing a scale-based hierarchy on the watersheds associated with these minima. This hierarchy can be used to label watershed boundaries according to their scale. This provides valuable insight into the multiscale properties of edges in an image without following these curves through scale-space. In addition, the gradient watershed region hierarchy can be used for automatic or interactive image segmentation. By selecting subtrees of the region hierarchy, visually sensible objects in an image can be easily constructed.

Vision: a digital video library

The goal of the VISION (Video Indexing for SearchIng Over Networks)project is to establish a comprehensive, online digital videolibrary. We are developing automatic mechanisms to populate thelibrary and provide content-based search and retrieval overcomputer networks. The salient feature of our approach is theintegrated application of mature image or video processing,information retrieval, speech feature extraction and word-spottingtechnologies for efficient creation and exploration of the librarymaterials. First, full-motion video is captured in real-time withflexible qualities to meet the requirements of library patronsconnected via a wide range of network bandwidths. Then, the videosare automatically segmented into a number of logically meaningfulvideo clips by our novel two-step algorithm based on video andaudio contents. A closed caption decoder and/or word-spotter isbeing incorporated into the system to extract textual informationto index the video clips by their contents. Finally, allinformation is stored in a full-text information retrieval systemfor content-baaed exploration of the library over networks ofvarying bandwidths.

Finding and identifying unknown commercials using repeated video sequence detection

Automated commercial detection can be performed by matching features extracted from commercials or by detecting embedded codes that are hidden within the commercial. In both cases, it is necessary to create a database of known commercials that contain the information necessary for detection. In this paper, we present an automated technique for locating previously unknown commercials by continuously monitoring broadcast television signals. Our system has two components: repeated video sequence detection, and feature-based classification of video sequences as commercials or non-commercials. Our system utilizes customized temporal video segmentation techniques to automatically partition the digital video signal into semantically sensible shots and scenes. As each frame of the video source is processed, we extract auxiliary information to facilitate repeated sequence detection. When the video transition marking the end of the shot/scene is detected, we are able to rapidly locate all previous occurrences of the video clip. In order to classify video sequences as commercials or non-commercials, we extract a number of features from each video sequence that characterize the temporal and chromatic variations within the clip. We have evaluated three classification approaches using this information and have consistently achieved over 93% accuracy identifying new commercials and non-commercials as they are broadcast.

Identification of new commercials using repeated video sequence detection

Automated commercial detection can be performed by matching features extracted from commercials or by detecting embedded codes that are hidden within the commercial. In both cases, it is necessary to create a database of known commercials that contain the information necessary for detection. In this paper, we present an automated technique for locating previously unknown commercials by continuously monitoring broadcast television signals. Our system has two components: repeated video sequence detection, and feature-based classification of video sequences as commercials or non-commercials. Our system utilizes customized temporal video segmentation techniques to automatically partition the digital video signal into semantically sensible shots and scenes. As each frame of the video source is processed, we extract auxiliary information to facilitate repeated sequence detection. When the video transition marking the end of the shot/scene is detected, we are able to rapidly locate all previous occurrences of the video clip. In order to classify video sequences as commercials or non-commercials, we extract a number of features from each video sequence that characterize the temporal and chromatic variations within the clip. We have evaluated three classification approaches using this information and have consistently achieved over 90% accuracy identifying new commercials and non-commercials as they are broadcast.

Real time video scene detection and classification

The VISION (video indexing for searching over networks) digital video library system has been developed in our laboratory as a testbed for evaluating automatic and comprehensive mechanisms for video archive creation and content-based search, filtering and retrieval of video over local and wide area networks. In order to provide access to video footage within seconds of broadcast, we have developed a new pipelined digital video processing architecture which is capable of digitizing, processing, indexing and compressing video in real time on an inexpensive general purpose computer. These videos were automatically partitioned into short scenes using video, audio and closed-caption information. The resulting scenes are indexed based on their captions and stored in a multimedia database. A client-server-based graphical user interface was developed to enable users to remotely search this archive and view selected video segments over networks of different bandwidths. Additionally, VISION classifies the incoming videos with respect to a taxonomy of categories and will selectively send users videos which match their individual profiles.

Real time repeated video sequence identification

In this paper, we present a real time system for detecting repeated video clips from a live video source such as news broadcasts. Our system utilizes customized temporal video segmentation techniques to automatically partition the digital video signal into semantically sensible shots and scenes. As each frame of the video source is processed, we extract auxiliary information to facilitate repeated sequence detection. When the video transition marking the end of the shot/scene is detected, we are able to rapidly locate all previous occurrences of the video clip. Our objective is to use repeated sequence information in our multimedia content analysis application to deduce semantic relationships among shots/scenes in the input video. Our real time video processing techniques are independent of source and domain and can be applied to other applications such as commercial detection and improved video compression.

Recovery of surfaces with discontinuities by fusing shading and range data within a variational framework

This work provides a variational framework for fusing range and intensity data for recovering regularized surfaces. It is shown that this framework provides natural boundary conditions for the shape-from-shading problem, results in a new shape-from-shading formulation in the absence of range data, and provides a new fusion paradigm when range data is incorporated. The approach is demonstrated on simulated range and intensity images; error analysis with respect to the ground truth surface is presented. It is shown that the formulation performs well even in very noisy images.

The VISION digital video library

The goal of the VISION (Video Indexing for Searching Over Networks) project is to demonstrate the technology necessary for a comprehensive, online digital video library. We have developed automatic mechanisms to populate the library and provide content-based search and retrieval of video over computer networks. The salient feature of our approach is the integrated application of mature image or video processing, information retrieval, speech feature extraction and word-spotting technologies for efficient creation and exploration of the library materials. First, full-motion video is captured in real time with flexible qualities to meet the requirements of library patrons connected via a wide range of network bandwidths. Then, the videos are automatically segmented into a number of logically meaningful video clips by our novel two-step algorithm based on video and audio contents. A closed caption decoder has also been incorporated into the system to extract textual information to index the video clips by their contents. Finally, all information is stored in a full-text information retrieval system for content-based exploration of the library over networks of varying bandwidths.

Real-Time Commercial Recognition Using Color Moments and Hashing

In this paper, our focus is on real-time commercial recognition. In particular, our goal is to correctly identify all commercials that are stored in our commercial database within the first second of their broadcast. To meet this objective, we make use of 27 color moments to characterize the content of every video frame. This representation is much more compact than most color histogram representations, and it less sensitive to noise and other distortion. We use frame-level hashing with subsequent matching of moment vectors and video frames to perform commercial recognition. Hashing provides constant time access to millions of video frames, so this approach can perform in real-time for databases containing thousands of commercials. In our experiments with a database of 63 commercials, we achieved 96% recall, 100% precision, and 98% utility while recognizing commercials within the first 1/2 second of their broadcast.

Hybrid boundary-based and region-based deformable models for biomedical image segmentation

The problem of segmenting an image into visually sensible regions has received considerable attention. Recent techniques based on deformable models show particular promise for this problem because they produce smooth closed object boundaries. These techniques can be broadly classified into two categories: boundary-based deformable models, and region-based deformable models. Both of these approaches have distinct advantages and disadvantages. In this paper, we introduce a hybrid deformable modeling technique which combines the advantages of both approaches and avoids some of their disadvantages. This is accomplished by first minimizing a region-based functional to obtain initial edge strength estimates. Smooth closed object boundaries are then obtained by minimizing a boundary-based functional which is attracted to the initial edge locations. In this paper, we discuss the theoretical advantages of this hybrid approach over existing image segmentation methods and show how this technique can be effectively implemented and used for the segmentation of 2D biomedical images.