Jeroen Vendrig

Systematic evaluation of logical story unit segmentation

Although various logical story unit (LSU) segmentation methods based on visual content have been presented in literature, a common ground for comparison is missing. We present a systematic evaluation of the mutual dependencies of segmentation methods and their performances. LSUs are subjective and cannot be defined with full certainty. To limit subjectivity, we present definitions based on film theory. For evaluation, we introduce a method measuring the quality of a segmentation method and its economic impact rather than the amount of errors. Furthermore, the inherent complexity of the segmentation problem given a visual feature is measured. Also, we show to what extent LSU segmentation depends on the quality of shot boundary segmentation. To understand LSU segmentation, we present a unifying framework classifying segmentation methods into four essentially different types. We present results of an evaluation of the four types under similar circumstances using an unprecedented amount of 20 hours of 17 complete videos in different genres. Tools and ground truths are available for interactive use via the Internet.

Filter Image Browsing: Interactive Image Retrieval by Using Database Overviews

Human-computer interaction is a decisive factor in effective content-based access to large image repositories. In current image retrieval systems the user refines his query by selecting example images from a relevance ranking. Since the top ranked images are all similar, user feedback often results in rearrangement of the presented images only.For better incorporation of user interaction in the retrieval process, we have developed the Filter Image Browsing method. It also uses feedback through image selection. However, it is based on differences between images rather than similarities. Filter Image Browsing presents overviews of relevant parts of the database to users. Through interaction users then zoom in on parts of the image collection. By repeatedly limiting the information space, the user quickly ends up with a small amount of relevant images. The method can easily be extended for the retrieval of multimedia objects.For evaluation of the Filter Image Browsing retrieval concept, a user simulation is applied to a pictorial database containing 10,000 images acquired from the World Wide Web by a search robot. The simulation incorporates uncertainty in the definition of the information need by users. Results show Filter Image Browsing outperforms plain interactive similarity ranking in required effort from the user. Also, the method produces predictable results for retrieval sessions, so that the user quickly knows if a successful session is possible at all. Furthermore, the simulations show the overview techniques are suited for applications such as hand-held devices where screen space is limited.

Filter Image Browsing - Exploiting Interaction in Image Retrieval

In current image retrieval systems the user refines his query by selecting example images from a relevance ranking. Since the top ranked images are all similar, user feedback often results in rearrangement of the presented images only. The Filter Image Browsing method provides better incorporation of user interaction in the retrieval process, because it is based on differences between images rather than similarities. Filter Image Browsing presents overviews of the database to users and lets them iteratively zoom in on parts of the image collection. In contrast to many papers where a new system is just introduced, we performed an extensive evaluation of the methods presented using a user simulation. Results for a database containing 10,000 images show that Filter Image Browsing requires less effort from the user. The implementation of Filter Image Browsing in the ImageRETRO system is accessible via the Web.

Interactive Indexing and Retrieval of Multimedia Content

The indexing and retrieval of multimedia items is difficult due to the semantic gap between the users perception of the data and the descriptions we can derive automatically from the data using computer vision, speech recognition, and natural language processing. In this contribution we consider the nature of the semantic gap in more detail and show examples of methods that help in limiting the gap. These methods can be automatic, but in general the indexing and retrieval of multimedia items should be a collaborative process between the system and the user. We show how to employ the users interaction for limiting the semantic gap.

Evaluation of logical story unit segmentation in video sequences

We present results of an evaluation of several existing Logical Story Unit segmentation methods based on film theory. Three types of methods known from video processing literature were evaluated. An unprecedented amount of 8 hours for 6 different videos were used to evaluate the methods under the same circumstances so that an unbiased judgement is presented. As Logical Story Units cannot be defined with full certainty, a new evaluation method is introduced measuring the economic impact of results. Experiments show reasonable performance for all methods. Time constrained clustering is the most effective and consistent method.

Feature Driven Visualization of Video Content for Interactive Indexing

When using visual video features in an interactive video indexing environment, it is necessary to visualize the meaning and impact of features to people that are not image processing experts, such as video librarians. An important method to visualize the relationship between the feature and the video is projection of feature values on the original video data.In this paper, we describe the characteristics of video feature types with respect to visualization. In addition, requirements for the visualization of video features are distinguished. Several video visualization methods are evaluated against the requirements. Furthermore, for feature visualization we propose the backprojection method in combination with the evaluated video visualization methods.We have developed the VidDex system which uses backprojection on various video visualization modes. By combining the visualization modes, the requirements for the feature characteristics identified can be met.

Model based interactive story unit segmentation

Logical Story Unit segmentation in general domains, such as movies and television series, requires interaction between experts and automatic tools. We present an interaction model that allows users, who are non-experts in the field of video processing, to segment videos into Logical Story Units by tuning segmentation model parameters rather than manually adjust results of automatic methods. Suitable features are determined interactively by visualizing their values in the context of the original video data.

Components and systems for interactive video indexing

The process of video indexing determines the quality of video retrieval. We present a modularization of indexing systems in which dependencies of components are made explicit. We stress the impact of human interaction in the architectural scheme, as the semantic gap between automatic abstractions and semantic indices requires human intervention. We discuss the components for efficient indexing, interaction and visualization in detail.

Object Tracking over Multiple Uncalibrated Cameras Using Visual, Spatial and Temporal Similarities

Developing a practical multi-camera tracking solution for autonomous camera networks is a very challenging task, due to numerous constraints such as limited memory and processing power, heterogeneous visual characteristics of objects between camera views, and limited setup time and installation knowledge for camera calibration. In this paper, we propose a unified multi-camera tracking framework, which can run online in real-time and can handle both independent field of view and common field of view cases. No camera calibration, knowledge of the relative positions of cameras, or entry and exit locations of objects is required. The memory footprint of the framework is minimised by the introduction of reusing kernels. The heterogeneous visual characteristics of objects are addressed by a novel location-based kernel matching method. The proposed framework has been evaluated using real videos captured in multiple indoor settings. The framework achieves efficient memory usage without compromising tracking accuracy.