Arne Jacobs

Using self-similarity matrices for structure mining on news video

Video broadcast series like news or magazine broadcasts usually expose a strong temporal structure, along with a characteristic audio-visual appearance. This results in frequent patterns occurring in the video signal. We propose an algorithm for the automatic detection of such patterns that exploits the videos self-similarity induced by the patterns. The approach is applied to the problem of anchor shot detection, but can also be used for other related purposes. Tests on real-world video data show that it is possible with our method to detect anchor shots fully automatically with high reliability.

Towards next-generation visual archives: image, film and discourse

The digital turn in visual studies has played a major role in the terminological overlap between ‘archive’, ‘database’ and ‘corpus’, and it has brought about a number of positive developments such as improved accessibility and availability. At the same time, it has also raised important questions pertaining to the materiality, searchability, annotation and analysis of the data at hand. Through a series of theoretical constructs and empirical examples, this paper illustrates the necessity and benefits of interdisciplinary dialogue when tackling the multimodal corpus annotation challenge. The meaningful interrelations between semiotic modes, the combinations between manual and (semi)automated annotation, the seamless integration of coding and annotation schemes which share common logics and the contextual embedding of the presented analyses strongly suggest multimodal document analysis in all its forms will continuously benefit from a corpus-based approach.

Automatic, context-of-capture-based categorization, structure detection and segmentation of news telecasts

The objective of the work reported here is to provide an automatic, context-of-capture categorization, structure detection and segmentation of news broadcasts employing a multimodal semantic based approach. We assume that news broadcasts can be described with context-free grammars that specify their structural characteristics. We propose a system consisting of two main types of interoperating units: The recognizer unit consisting of several modules and a parser unit. The recognizer modules (audio, video and semantic recognizer) analyze the telecast and each one identifies hypothesized instances of features in the audiovisual input. A probabilistic parser analyzes the identifications provided by the recognizers. The grammar represents the possible structures a news telecast may have, so the parser can identify the exact structure of the analyzed telecast.

Association Rule Mining of Multimedia Content

The analysis of video sequences is of primary concern in the field of mass communication. One particular topic is the study of collective visual memories and neglections as they emerged in various cultures, with trans-cultural and global elements (Ludes P., Multimedia und Multi-Moderne: Schlusselbilder, Fernsehnachrichten und World Wide Web – Medienzivilisierung in der Europaischen Wahrungsunion. Westdeutscher Verlag, Opladen 2001). The vast amount of visual data from television and web offerings make comparative studies on visual material rather complex and very expensive. A standard task in this realm is to find images that are similar to each other. Similarity is typically aimed at a conceptual level comprising both syntactic as well as semantic similarity. The use of semi-automatic picture retrieval techniques would facilitate this task. An important aspect is to combine the syntactical analysis that is usually performed automatically with the semantic level obtained from annotations or the analysis of captions or closely related text. Association rules are in particular suited to extract implicit knowledge from the data base and to make this knowledge accessible for further quantitative analysis.

Inter-video Similarity for Video Parsing

In this paper we present a method for automatic detection of visual patterns in a given news video format by investigating similarities in a set of videos of that format. The approach aims at reducing the manual effort needed to create models of news broadcast formats for automatic video indexing and retrieval. Our algorithm has only very few parameters and can be run fully unsupervised. It shows good performance on a news format of the TRECVID’03 data which had already been modeled with hand-selected visual patterns and served as ground truth for evaluation.

Semantic Video Segmentation Using Probabilistic Relaxation

In this paper we propose a method for temporal segmentation of strongly structured videos on a semantic level. The proposed method is based on a naive Bayes classifier on low level visual features, followed by a two-stage probabilistic relaxation process. The first stage relaxation is on successive video frames that have been classified with the naive Bayes classifier into structural tokens and aims to improve the initial classification result. The second relaxation process is applied on successive video segments and uses knowledge from temporal relations of structural tokens that are characteristic for each broadcasting format and results so to the video segmentation on a semantic level. The experiments carried out, show that the proposed method can be successfully applied to magazine broadcastings.

Hybrid Model-based Estimation of Multiple Non-dominant Motions

The estimation of motion in videos yields information useful in the scope of video annotation, retrieval and compression. Current approaches use iterative minimization techniques based on intensity gradients in order to estimate the parameters of a 2D transform between successive frames. These approaches rely on good initial guesses of the motion parameters. For single or dominant motion there exist hybrid algorithms that estimate such initial parameters prior to the iterative minimization. We propose a technique for the generation of a set of motion hypotheses using blockmatching that also works in the presence of multiple non-dominant motions. These hypotheses are then refined using iterative techniques.