Bilge Günsel

Temporal video segmentation using unsupervised clustering and semantic object tracking

This paper proposes a content-based temporal video segmentation system that integrates syntactic (domain- independent) and semantic (domain-dependent) features for auto- matic management of video data. Temporal video segmentation in- cludes scene change detection and shot classification. The proposed scene change detection method consists of two steps: detection and tracking of semantic objects of interest specified by the user, and an unsupervised method for detection of cuts, and edit effects. Object detection and tracking is achieved using a region matching scheme, where the region of interest is defined by the boundary of the object. A new unsupervised scene change detec- tion method based on two-class clustering is introduced to eliminate the data dependency of threshold selection. The proposed shot classification approach relies on semantic image features and ex- ploits domain-dependent visual properties such as shape, color, and spatial configuration of tracked semantic objects. The system has been applied to segmentation and classification of TV programs col- lected from different channels. Although the paper focuses on news programs, the method can easily be applied to other TV programs with distinct semantic structure.

Two-dimensional mesh-based visual-object representation for interactive synthetic/natural digital video

This paper first provides an overview of two-dimensional (2-D) and three-dimensional mesh models for digital video processing. It then introduces 2-D mesh-based modeling of video objects as a compact representation of motion and shape for interactive, synthetic/natural video manipulation, compression, and indexing. The 2-D mesh representation and the mesh geometry and motion compression have been included in the visual tools of the upcoming MPEG-4 standard. Functionalities enabled by 2-D mesh-based visual-object representation include animation of still texture maps, transfiguration of video overlays, video morphing, and shape-and motion-based retrieval of video objects.

Content-based video abstraction

The paper addresses automatic scene change detection, key frame selection, and similarity ranking which constitute the main steps of a content based video abstraction system. Unlike other methods, the proposed algorithm performs scene change detection and key frame selection in one step. We treat scene change detection as a two-class classification problem and employ automatic threshold selection techniques originally developed for image binarization. A quantitative measure for retrieval of similar scenes according to their color content is also defined. The described scheme can be applied to both uncompressed and MPEG compressed video, and can be implemented in real time. Performance of the algorithm has been analyzed on real TV sequences, and comparison with some previously introduced techniques are provided.

Content-based access to video objects: temporal segmentation, visual summarization, and feature extraction

Abstract The classical approach to content-based video access has been ‘frame-based’, consisting of shot boundary detection, followed by selection of key frames that characterize the visual content of each shot, and then clustering of the camera shots to form story units. However, in an object-based multimedia environment, content-based random access to individual video objects becomes a desirable feature. To this effect, this paper introduces an ‘object-based’ approach to temporal video partitioning and content-based indexing, where the basic indexing unit is ‘lifespan of a video object’, rather than a ‘camera shot’ or a ‘story unit’. We propose to represent each video object by an adaptive 2D triangular mesh. A mesh-based object tracking scheme is then employed to compute the motion trajectories of all mesh node points until the object exits the field of view. A new similarity measure that is based on motion discontinuities and shape changes of the tracked object is defined to detect content changes, resulting in temporal lifespan segments. A set of ‘key snapshots’ which constitute a visual summary of the lifespan of the object is automatically selected. These key snapshots are then used to animate objects of interest using tracked motion trajectories for a moving visual representation. The proposed scheme provides such functionalities as object-based search/browsing for interactive video retrieval, surveillance video analysis, and object-based content manipulation/editing for studio postprocessing and desktop multimedia authoring. The approach is applicable to any video data where the initial appearance of object(s) can be specified, and the object motion can be modeled by a piecewise affine transformation. The system is demonstrated using different types of video: virtual studio productions (composited video), surveillance video, and TV broadcast video.

Shape similarity matching for query-by-example

This paper describes a unified approach for two-dimensional (2-D) shape matching and similarity ranking of objects by means of a modal representation. In particular, we propose a new shape-similarity metric in the eigenshape space for object/image retrieval from a visual database via query-by-example. This differs from prior work which performed point correspondence determination and similarity ranking of shapes in separate steps. The proposed method employs selected boundary and/or contour points of an object as a coarse-to-fine shape representation, and does not require extraction of connected boundaries or silhouettes. It is rotation-, translation- and scale-invariant, and can handle mild deformations of objects (e.g. due to partial occlusions or pose variations). Results comparing the unified method with an earlier two-step approach using B-spline-based modal matching and Hausdorff distance ranking are presented on retail and museum catalog style still-image databases.

Hierarchical temporal video segmentation and content characterization

This paper addresses the segmentation of a video sequence into shots, specification of edit effects and subsequent characterization of shots in terms of color and motion content. The proposed scheme uses DC images extracted from MPEG compressed video and performs an unsupervised clustering for the extraction of camera shots. The specification of edit effects, such as fade-in/out and dissolve is based on the analysis of distribution of mean value for the luminance components. This step is followed by the representation of visual content of temporal segments in terms of key frames selected by similarity analysis of mean color histograms. For characterization of the similar temporal segments, motion and color characteristics are classified into different categories using a set of different features derived from motion vectors of triangular meshes and mean histograms of video shots.

Robust watermarking of busy images

This work extends the watermarking method proposed by Kutter et al. to increase the watermark decoding performance for textured or busy images. The proposed algorithm modifies watermark embedding rule to utilize image characteristics, like local standard deviation and gradient magnitude, in order to increase the decoding accuracy for busy images. The method does not need original image for decoding and controls the watermark embedding process at encoder site, resulting in a more accurate decoding.

Moving visual representations of video objects for content-based search and browsing

This paper proposes object-based moving visual representations for quick browsing of video content. These representations are hierarchical, such that at the coarse level a sequence of alpha planes provides a moving representation of object shape and motion information for object contours. Alternatively, a 2D mesh representation provides a complete visual representation of object motion and shape. The finest level visual representation can be obtained by texture mapping onto the moving meshes. The paper also discusses trade-offs between each representation in terms of the amount of indexing information that needs to be stored, the robustness of the representation, and the accuracy of the representation.

User preference matching for smart browsing of media contents

The last decade witnessed the improvement of strong compression technics over audio-visual data and the development of world wide communications of information. These innovations gave birth to unforeseen requirements like storing these information, indexing and retrieving them for subsequent usages. Standardization of compression of multimedia contents is rapidly accepted as a need for encoding/decoding audio-visual data regardless of machine and environment. However standardization for indexing these materials still remains a puzzle which disables browsing audio-visual data regardless of machine and environment. MPEG-7 standardization group aims to create the standard syntax to access to multimedia content. This paper puts forward a next step, the extraction of user preferences and matching them with MPEG-7 coded media content for quick and smart browsing.