Georg Thallinger

Automatic image annotation using visual content and folksonomies

Automatic image annotation is an important and challenging task, and becomes increasingly necessary when managing large image collections. This paper describes techniques for automatic image annotation that take advantage of collaboratively annotated image databases, so called visual folksonomies. Our approach applies two techniques based on image analysis: First, classification annotates images with a controlled vocabulary and second tag propagation along visually similar images. The latter propagates user generated, folksonomic annotations and is therefore capable of dealing with an unlimited vocabulary. Experiments with a pool of Flickr images demonstrate the high accuracy and efficiency of the proposed methods in the task of automatic image annotation. Both techniques were applied in the prototypical tag recommender “tagr”.

Object re-detection using SIFT and MPEG-7 color descriptors

Information about the occurrence of objects in videos and their interactions conveys an important part of the semantics of audio-visual content and can be used to narrow the semantic gap in video analysis, retrieval and summarization. Object re-detection, which aims at finding occurrences of specific objects in a single video or a collection of still images and videos, is an object identification problem and can thus be more satisfactorily solved than a general object recognition problem. As structural information and color information are often complementary, we propose a combined object re-detection approach using SIFT and MPEG-7 color descriptors extracted around the same interest points. We evaluate the approach on two different data sets and show that the MPEG-7 ColorLayout descriptor performs best of the tested color descriptors and that the joint approach yields better results than the use of SIFT or color descriptors only.

Using Gait Features for Improving Walking People Detection

In this paper, we explore a new approach for enriching the HoG method for pedestrian detection in an unconstrained outdoor environment. The proposed algorithm is based on using gait motion since the rhythmic footprint pattern for walking people is considered the stable and characteristic feature for the detection of walking people. The novelty of our approach is motivated by the latest research for people identification using gait. The experimental results confirmed the robustness of our method to enhance HoG to detect walking people as well as to discriminate between single walking subject, groups of people and vehicles with a detection rate of 100%. Furthermore, the results revealed the potential of our method to be used in visual surveillance systems for identity tracking over different camera views.

Detecting and clustering multiple takes of one scene

In applications such as video post-production users are confronted with large amounts of redundant unedited raw material, called rushes. Viewing and organizing this material are crucial but time consuming tasks. Typically multiple but slightly different takes of the same scene can be found in the rushes video. We propose a method for detecting and clustering takes of one scene shot from the same or very similar camera positions. It uses a variant of the LCSS algorithm to find matching subsequences in sequences of visual features extracted from the source video. Hierarchical clustering is used to group the takes of one scene. The approach is evaluated in terms of correctly assigned takes using manually annotated ground truth.

Skimming rushes video using retake detection

In audiovisual post-production users are confronted with large amounts of redundant unedited raw material, called rushes. Viewing and organizing this material is a crucial but time consuming task. This paper describes an approach for creating skimmed versions of the rushes video based on the elimination of unusable content and clustering of takes. Typically multiple, but slightly differing takes of the same scene can be found in the rushes video. We propose a method for clustering takes of one scene shot from the same camera position. It uses a variant of the LCSS algorithm to find matching subsequences in sequences of extracted features from the source video. The approach is evaluated by two subjective measures for the quality of the skim and by measuring the overlap between items found in the source video and the skim.

A distance measure for repeated takes of one scene

In applications, such as post-production and archiving of audiovisual material, users are confronted with large amounts of redundant unedited raw material, called rushes. Viewing and organizing this material are crucial but time consuming tasks. Typically, multiple but slightly different takes of the same scene can be found in the rushes video. We propose a method for detecting and clustering takes of one scene shot from the same or very similar camera positions. An important subproblem is to determine the similarity of video segments. We propose a distance measure based on the Longest Common Subsequence (LCSS) model. Two variants of the proposed approach, one with a threshold parameter and one with automatically determined threshold, are compared against the Dynamic Time Warping (DTW) distance measure on six videos from the TRECVID 2007 BBC rushes summarization data set. We also evaluate the influence of the applied temporal segmentation method at the input on the results. Applications of the proposed method to automatic skimming and interactive browsing of rushes video are described.

A Video Browsing Tool for Content Management in Postproduction

We propose an interactive video browsing tool for supporting content management and selection in postproduction. The approach is based on a process model for multimedia content abstraction. A software framework based on this process model and desktop and Web-based client applications are presented. For evaluation, we apply two TRECVID style fact finding approaches (retrieval and question answering tasks) and a user survey to the evaluation of the video browsing tool. We analyze the correlation between the results of the different methods, whether different aspects can be evaluated independently with the survey, and if a learning effect can be measured with the different methods, and we also compare the full-featured desktop and the limited Web-based user interface. The results show that the retrieval task correlates better with the user experience according to the survey. The survey rather measures the general user experience while different aspects of the usability cannot be analyzed independently.

Automatic Quality Analysis for Film and Video Restoration

A considerable amount of work in larger film and video restoration projects is dedicated to manually exploring the audiovisual content in order to estimate the costs for restoration and to plan the restoration. Manual exploration is a significant cost factor. In this paper we propose automatic content analysis algorithms and summarization techniques which allow the reduction of manual inspection time in a software based restoration environment. The throughput requirement for analysis of dust and other defects is reached by sparse application of the detectors in the image sequence while retaining sufficient detection accuracy. Analysis result metadata are represented in a MPEG-7 standard compliant way. The proposed defect summary visualization tools facilitate efficient exploration of visually impaired content by the user.

A framework for multimedia content abstraction and its application to rushes exploration

Multimedia content abstraction methods such as summarisation, skimming and browsing are of growing importance for the exploration of multimedia collections, complementing search and retrieval approaches. The methods proposed in literature differ in a number of aspects (e.g. form, purpose, presentation), but nonetheless many similarities in the creation of multimedia content abstractions can be identified. Thus we propose a generic multimedia content abstraction process and a software framework that implements it. We demonstrate the practical usability of the framework by using it to build an application for browsing a collection of rushes.

SALERO: Semantic AudiovisuaL Entertainment Reusable Objects

The Integrated Project SALERO aims to advance the state of the art in digital media to the point where it becomes possible to create audiovisual content for cross-platform delivery using intelligent content tools, with greater quality at lower cost, to provide audiences with more engaging entertainment and information at home or on the move. SALERO will build on and extend research in media technologies, web semantics and context based image retrieval, to reverse the trend toward ever-increasing cost of creating media.