Carsten Saathoff

Semantic annotation of images and videos for multimedia analysis

Annotations of multimedia documents typically have been pursued in two different directions. Either previous approaches have focused on low level descriptors, such as dominant color, or they have focused on the content dimension and corresponding annotations, such as person or vehicle. In this paper, we present a software environment to bridge between the two directions. M-OntoMat-Annotizer allows for linking low level MPEG-7 visual descriptions to conventional Semantic Web ontologies and annotations. We use M-OntoMat-Annotizer in order to construct ontologies that include prototypical instances of high-level domain concepts together with a formal specification of corresponding visual descriptors. Thus, we formalize the interrelationship of high- and low-level multimedia concept descriptions allowing for new kinds of multimedia content analysis and reasoning.

F--a model of events based on the foundational ontology dolce+DnS ultralight

The lack of a formal model of events hinders interoperability in distributed event-based systems. In this paper, we present a formal model of events, called Event-Model-F. The model is based on the foundational ontology DOLCE+DnS Ultralight (DUL) and provides comprehensive support to represent time and space, objects and persons, as well as mereological, causal, and correlative relationships between events. In addition, the Event-Model-F provides a flexible means for event composition, modeling event causality and event correlation, and representing different interpretations of the same event. The Event-Model-F is developed following the pattern-oriented approach of DUL, is modularized in different ontologies, and can be easily extended by domain specific ontologies.

Semantic Multimedia

Multimedia constitutes an interesting field of application for Semantic Web and Semantic Web reasoning, as the access and management of multimedia content and context depends strongly on the semantic descriptions of both. At the same time, multimedia resources constitute complex objects, the descriptions of which are involved and require the foundation on sound modeling practice in order to represent findings of low- and high level multimedia analysis and to make them accessible via Semantic Web querying of resources. This tutorial aims to provide a red thread through these different issues and to give an outline of where Semantic Web modeling and reasoning needs to further contribute to the area of semantic multimedia for the fruitful interaction between these two fields of computer science.

M-OntoMat-Annotizer: image annotation linking ontologies and multimedia low-level features

Annotations of multimedia documents typically have been pursued in two different directions. Either previous approaches have focused on low level descriptors, such as dominant color, or they have focused on the content dimension and corresponding annotations, such as person or vehicle. In this paper, we present a software environment to bridge between the two directions. M-OntoMat-Annotizer allows for linking low level MPEG-7 visual descriptions to conventional Semantic Web ontologies and annotations. We use M-OntoMat-Annotizer in order to construct ontologies that include prototypical instances of high-level domain concepts together with a formal specification of corresponding visual descriptors. Thus, we formalize the interrelationship of high- and low-level multimedia concept descriptions allowing for new kinds of multimedia content analysis, reasoning and retrieval.

SemaPlorer-Interactive semantic exploration of data and media based on a federated cloud infrastructure

SemaPlorer is an easy to use application that allows end users to interactively explore and visualize a very large, mixed-quality and semantically heterogeneous distributed semantic data set in real-time. Its purpose is to acquaint oneself about a city, touristic area, or other area a user is interested in. By visualizing the data using a map, media, and different context views, SemaPlorer advances beyond simple storage and retrieval of large numbers of triples, as the interaction with the large data set is driven by the user. SemaPlorer leverages different semantic data sources such as DBpedia, GeoNames, WordNet, and personal FOAF files. These make a significant portion of the data provided for the Billion Triple Challenge. SemaPlorer intriguingly connects with a large Flickr data set converted to RDF. The storage infrastructure bases on Amazons Elastic Computing Cloud (EC2) and Simple Storage Service. We apply NetworkedGraphs as a conceptual layer on top of EC2, realizing a large, federated data infrastructure for semantically heterogeneous data sources from within and outside of the cloud. Therefore, the application is scalable with respect to the amount of distributed components working together as well as the number of triples managed overall. Hence, SemaPlorer is flexible enough to leverage for exploration of almost arbitrary additional data sources that might be added in the future. We conducted a formative evaluation of the SemaPlorer application with 20 test subjects. The results of this evaluation are analyzed and their implication to future work discussed. SemaPlorer won the first prize at the Billion Triple Challenge of the International Semantic Web Conference in Karlsruhe, 2008.

Unlocking the semantics of multimedia presentations in the web with the multimedia metadata ontology

The semantics of rich multimedia presentations in the web such as SMIL, SVG, and Flash cannot or only to a very limited extend be understood by search engines today. This hampers the retrieval of such presentations and makes their archival and management a difficult task. Existing metadata models and metadata standards are either conceptually too narrow, focus on a specific media type only, cannot be used and combined together, or are not practically applicable for the semantic description of rich multimedia presentations. In this paper, we propose the Multimedia Metadata Ontology (M3O) for annotating rich, structured multimedia presentations. The M3O provides a generic modeling framework for representing sophisticated multimedia metadata. It allows for integrating the features provided by the existing metadata models and metadata standards. Our approach bases on Semantic Web technologies and can be easily integrated with multimedia formats such as the W3C standards SMIL and SVG. With the M3O, we unlock the semantics of rich multimedia presentations in the web by making the semantics machine-readable and machine-understandable. The M3O is used with our SemanticMM4U framework for the multi-channel generation of semantically-rich multimedia presentations.

Designing core ontologies

One of the key factors that hinders integration of distributed, heterogeneous information systems is the lack of a formal basis for modeling the complex, structured knowledge that is to be exchanged. To alleviate this situation, we present an approach based on core ontologies. Core ontologies are characterized by a high degree of axiomatization and formal precision. This is achieved by basing on a foundational ontology. In addition, core ontologies should follow a pattern-oriented design approach. By this, they are modular and extensible. Core ontologies allow for reusing the structured knowledge they define as well as integrating existing domain knowledge. The structured knowledge of the core ontologies is clearly separated from the domain-specific knowledge. Such core ontologies allow for both formally conceptualize their particular fields and to be flexibly combined to cover the needs of concrete, complex application domains. Over the last years, we have developed three independent core ontologies for events and objects, multimedia annotations and personal information management. In this paper, we present the simultaneous use and integration of our core ontologies at the example of a complex, distributed socio-technical system of emergency response. We describe our design approach for core ontologies and discuss the lessons learned in designing them. Finally, we elaborate on the beauty aspects of our core ontologies.

An ontology infrastructure for multimedia reasoning

In this paper, an ontology infrastucture for multimedia reasoning is presented, making it possible to combine low-level visual descriptors with domain specific knowledge and subsequently analyze multimedia content with a generic algorithm that makes use of this knowledge. More specifically, the ontology infrastructure consists of a domain-specific ontology, a visual descriptor ontology (VDO) and an upper ontology. In order to interpret a scene, a set of atom regions is generated by an initial segmentation and their descriptors are extracted. Considering all descriptors in association with the related prototype instances and relations, a genetic algorithm labels the atom regions. Finally, a constraint reasoning engine enables the final region merging and labelling into meaningful objects.

A core ontology on events for representing occurrences in the real world

Events are central aspect of many semantic ambient media applications such as surveillance, smart homes, automobiles, and others. Existing models for events typically do not follow a systematic development approach, are conceptually narrow with respect to event features, and their semantics is often ambiguous. This makes the communication between and integration of different event-based components and event-based semantic ambient media applications a challenging task. In this paper, we present the Event-Model-F, a formal model of events based on the foundational ontology DOLCE+DnS Ultralite (DUL). The Event-Model-F provides comprehensive support to represent time and space, objects and persons, mereological, causal, and correlative relationships between events, and different interpretations of the same event. It is developed following a pattern-oriented ontology design approach and can be easily extended by domain specific ontologies. We introduce the design and implementation of an application programming interface that allows for easy integration of the Event-Model-F in arbitrary applications. The use of the Event-Model-F is demonstrated at the example of a socio-technical system of emergency response and implemented in the SemaPlorer+ + application for creating and sharing event descriptions.

A comparative study of object-level spatial context techniques for semantic image analysis

In this paper, three approaches to utilizing object-level spatial contextual information for semantic image analysis are presented and comparatively evaluated. Contextual information is in the form of fuzzy directional relations between image regions. All techniques, namely a Genetic Algorithm (GA), a Binary Integer Programming (BIP) and an Energy-Based Model (EBM), are applied in order to estimate an optimal semantic image interpretation, after an initial set of region classification results is computed using solely visual features. Aim of this paper is the in-depth investigation of the advantages of each technique and the gain of a better insight on the use of spatial context. For this purpose, an appropriate evaluation framework, which includes several different combinations of low-level features and classification algorithms, has been developed. Extensive experiments on six datasets of varying problem complexity have been conducted for investigating the influence of typical factors (such as the utilized visual features, the employed classifier, and the number of supported concepts) on the performance of each spatial context technique, while a detailed analysis of the obtained results is also given.