Pierangelo Migliorati

Semantic indexing of multimedia documents

We propose two approaches for semantic indexing of audio-visual documents, based on bottom-up and top-down strategies. We base the first approach on a finite-state machine using low-level motion indices extracted from an MPEG compressed bitstream. The second approach innovatively performs semantic indexing through Hidden Markov Models.

Audio classification in speech and music: a comparison between a statistical and a neural approach

We focus the attention on the problem of audio classification in speech and music for multimedia applications. In particular, we present a comparison between two different techniques for speech/music discrimination. The first method is based on Zero crossing rate and Bayesian classification. It is very simple from a computational point of view, and gives good results in case of pure music or speech. The simulation results show that some performance degradation arises when the music segment contains also some speech superimposed on music, or strong rhythmic components. To overcome these problems, we propose a second method, that uses more features, and is based on neural networks (specifically a multi-layer Perceptron). In this case we obtain better performance, at the expense of a limited growth in the computational complexity. In practice, the proposed neural network is simple to be implemented if a suitable polynomial is used as the activation function, and a real-time implementation is possible even if low-cost embedded systems are used.

Extraction of Significant Video Summaries by Dendrogram Analysis

In the current video analysis scenario, effective clustering of shots facilitates the access to the content and helps in understanding the associated semantics. This paper introduces a cluster analysis on shots which employs dendrogram representation to produce hierarchical summaries of the video document. Vector quantization codebooks are used to represent the visual content and to group the shots with similar chromatic consistency. The evaluation of the cluster codebook distortions, and the exploitation of the dependency relationships on the dendrograms, allow to obtain only a few significant summaries of the whole video. Finally the user can navigate through summaries and decide which one best suites his/her needs for eventual post-processing. The effectiveness of the proposed method is demonstrated by testing it on a collection of video-data from different kinds of programmes. Results are evaluated in terms of metrics that measure the content representational value of the summarization technique.

Hidden Markov Models for Video Skim Generation

In this paper we present a statistical framework based on hidden Markov models (HMMs) for video skimming. A chain of HMMs is used to model subsequent story units: HMM states represent different visual-concepts, transitions model the temporal dependencies in each story unit, and stochastic observations are given by single shots. The skim is generated as an observation sequence, where, in order to privilege more informative segments for entering the skim, dynamic shots are assigned higher probability of observation. The effectiveness of the method is demonstrated on a video set from different kinds of programmes, and results are evaluated in terms of metrics that assess the content representational value of the obtained video skims.

Event recognition in sport programs using low-level motion indices

In this paper we present a semantic video indexing algorithm based on finite-state machines and low-level motion indices extracted from the MPEG compressed bit-stream. The problem of semantic video indexing is actually of great interest due to the wide diffusion of large video databases. In literature we can find many video indexing algorithms, based on various types of low-level features, but the problem of semantic indexing is less studied and surely it is a great challenging one. The proposed algorithm is an example of solution to the problem of finding a semantic relevant event (e.g., scoring of a goal in a soccer game) in case of specific categories of audio-visual programmes. The simulation results show that the proposed algorithm can effectively detect the presence of goals and other relevant events in sport programs.

Emotional identity of movies

In the field of multimedia analysis, attempts that lead to an emotional characterization of content have been proposed. In this work we aim at defining the emotional identity of a feature movie by positioning it into an emotional space, as if it was a piece of art. The multimedia content is mapped into a trajectory whose coordinates are connected to filming and cinematographic techniques used by directors to convey emotions. The trajectory evolution over time provides a strong characterization of the movie, by locating different movies into different regions of the emotional space. The ability of this tool in characterizing content has been tested by retrieving emotionally similar movies from a large database, using IMDb genre classification for the evaluation of results.

Semantic indexing of sports program sequences by audio-visual analysis

Semantic indexing of sports videos is a subject of great interest to researchers working on multimedia content characterization. Sports programs appeal to large audiences and their efficient distribution over various networks should contribute to widespread usage of multimedia services. In this paper, we propose a semantic indexing algorithm for soccer programs, which uses both audio and visual information for content characterization. The video signal is processed first by extracting low-level visual descriptors from the MPEG compressed bit-stream. The temporal evolution of these descriptors during a semantic event is supposed to be governed by a controlled Markov chain. This allows to determine a list of those video segments where a semantic event of interest is likely to be found, based on the maximum likelihood criterion. The audio information is then used to refine the results of the video classification procedure by ranking the candidate video segments in the list so that the segments associated to the event of interest appear in the very first positions of the ordered list. The proposed method is applied to goal detection. Experimental results show the effectiveness of the proposed cross-modal approach.

The ToCAI Description Scheme for Indexing and Retrieval of Multimedia Documents

A framework, called Table of Content-Analytical Index (ToCAI), for the content description of multimedia material is presented. The idea for such a description scheme (DS) comes out from the structures used for indexing technical books (containing a Table of Content, typically placed at the beginning of the book, where the list of topics is organized hierarchically into chapters, sections, and an Analytical Index, typically placed at the end of the book, where keywords are listed alphabetically). The ToCAI description scheme provides similarly a hierarchical description of the time sequential structure of a multimedia document (ToC), suitable for browsing, and an “Analytical Index” (AI) of audio-visual key items for the document, suitable for effective retrieval. Besides two other sub-description schemes are proposed to specify the program category and the description of other metadata associated to the multimedia document in the general DS. The detailed structure of the DS is presented by means of a UML diagram. Moreover, some suitable automatic extraction methods for the identification of the values associated to the descriptors that compose the ToCAI are presented and discussed. Finally, a browsing application example is also proposed.

ADVANCED CONTENT-BASED SEMANTIC SCENE ANALYSIS AND INFORMATION RETRIEVAL: THE SCHEMA PROJECT

The aim of the SCHEMA Network of Excellence is to bring together a critical mass of universities, research centers, industrial partners and end users, in order to design a reference system for content-based semantic scene analysis, interpretation and understanding. Relevant research areas include: content-based multimedia analysis and automatic annotation of semantic multimedia content, combined textual and multimedia information retrieval, semantic -web, MPEG-7 and MPEG-21 standards, user interfaces and human factors. In this paper, recent advances in content-based analysis, indexing and retrieval of digital media within the SCHEMA Network are presented. These advances will be integrated in the SCHEMA module-based, expandable reference system.

Interactive visualization of video content and associated description for semantic annotation

In this paper, we present an intuitive graphic framework introduced for the effective visualization of video content and associated audio-visual description, with the aim to facilitate a quick understanding and annotation of the semantic content of a video sequence. The basic idea consists in the visualization of a 2D feature space in which the shots of the considered video sequence are located. Moreover, the temporal position and the specific content of each shot can be displayed and analysed in more detail. The selected features are decided by the user, and can be updated during the navigation session. In the main window, shots of the considered video sequence are displayed in a Cartesian plane, and the proposed environment offers various functionalities for automatically and semi-automatically finding and annotating the shot clusters in such feature space. With this tool the user can therefore explore graphically how the basic segments of a video sequence are distributed in the feature space, and can recognize and annotate the significant clusters and their structure. The experimental results show that browsing and annotating documents with the aid of the proposed visualization paradigms is easy and quick, since the user has a fast and intuitive access to the audio-video content, even if he or she has not seen the document yet.