Gerd Herzog

SmartWeb handheld: multimodal interaction with ontological knowledge bases and semantic web services

SMARTWEB aims to provide intuitive multimodal access to a rich selection of Web-based information services. We report on the current prototype with a smartphone client interface to the Semantic Web. An advanced ontology-based representation of facts and media structures serves as the central description for rich media content. Underlying content is accessed through conventional web service middleware to connect the ontological knowledge base and an intelligent web service composition module for external web services, which is able to translate between ordinary XML-based data structures and explicit semantic representations for user queries and system responses. The presentation module renders the media content and the results generated from the services and provides a detailed description of the content and its layout to the fusion module. The user is then able to employ multiple modalities, like speech and gestures, to interact with the presented multimedia material in a multimodal way.

Three RoboCup Simulation League Commentator Systems

� Three systems that generate real-time natural language commentary on the RoboCup simulation league are presented, and their similarities, differences, and directions for the future discussed. Although they emphasize different aspects of the commentary problem, all three systems take simulator data as input and generate appropriate, expressive, spoken commentary in real time.

MULTIPLATFORM testbed: an integration platform for multimodal dialog systems

Modern dialog and information systems are increasingly based on distributed component architectures to cope with all kinds of heterogeneity and to enable flexible re-use of existing software components. This contribution presents the MULTIPLATFORM testbed as a powerful framework for the development of integrated multimodal dialog systems. The paper provides a general overview of our approach and explicates its foundations. It describes advanced sample applications that have been realized using the integration platform and compares our approach to related works.

Utilizing Spatial Relations for Natural Language Access to an Autonomous Mobile Agent

Natural language, a primary communication medium for humans, facilitates better human-machine interaction and could be an efficient means to use intelligent robots in a more flexible manner. In this paper, we report on our joint efforts at providing natural language access to the autonomous mobile two-arm robot Kamro. The robot is able to perform complex assembly tasks. To achieve autonomous behaviour, several camera systems are used for the perception of the environment during task execution. Since natural language utterances must be interpreted with respect to the robots current environment the processing must be based on a referential semantics that is perceptually anchored. Considering localization expressions, we demonstrate how, on the one hand, verbal descriptions, and on the other hand, knowledge about the physical environment, i.e., visual and geometric information, can be connected to each other.

Incremental Natural Language Description of Dynamic Imagery

Although image understanding and natural language processing constitute two major areas of AI, they have mostly been studied independently of each other. Only a few attempts have been concerned with the integration of computer vision and the generation of natural language expressions for the description of image sequences.

A look under the hood: design and development of the first SmartWeb system demonstrator

Experience shows that decisions in the early phases of the development of a multimodal system prevail throughout the life-cycle of a project. The distributed architecture and the requirement for robust multimodal interaction in our project SmartWeb resulted in an approach that uses and extends W3C standards like EMMA and RDFS. These standards for the interface structure and content allowed us to integrate available tools and techniques. However, the requirements in our system called for various extensions, e.g., to introduce result feedback tags for an extended version of EMMA. The interconnection framework depends on a commercial telephone voice dialog system platform for the dialog-centric components while the information access processes are linked using web service technology. Also in the area of this underlying infrastructure, enhancements and extensions were necessary. The first demonstration system is operable now and will be presented at the Football World Cup 2006 in Germany.

Rocco: A RoboCup Soccer Commentator System

With the attempt to enable robots to play soccer games, the RoboCup challenge poses a demanding standard problem for AI and intelligent robotics research. The rich domain of robot soccer, however, provides a further option for the investigation of a second class of intelligent systems which are capable of understanding and describing complex time-varying scenes. Such automatic commentator systems offer an interesting research perspective for additional integration of natural language and intelligent multimedia technologies. In this paper, first results concerning the realization of a fully automated RoboCup commentator will be presented. The system called Rocco is currently able to generate TV-style live reports for arbitrary matches of the RoboCup simulator league. Based upon our generic approach towards multimedia reporting systems, step-by-step even more advanced capabilities are to be added with future versions of the initial Rocco prototype.

Generating Multimedia Presentations for RoboCup Soccer Games

The automated generation of multimedia reports for time-varying scenes on the basis of visual data constitutes a challenging research goal with a high potential for many interesting applications. In this paper, we report on our work towards an automatic commentator system for RoboCup, the Robot World-Cup Soccer. Rocco (RoboCup-Commentator) is a prototype system that has emerged from our previous work on high-level scene analysis and intelligent multimedia generation. Based on a general conception for multimedia reporting systems, we describe the initial Rocco version which is intended to generate TV-style live reports for matches of the simulator league.

Integrating Vision and Language: Towards Automatic Description of Human Movements

The integration of vision and natural language processing increasingly attracts attention in different areas of AI research. Up to now, however, there have only been a few attempts at connecting vision systems with natural language access systems. Within the SFB 314, special collaborative program on AI and knowledge-based systems, the automatic natural language description of real world image sequences constitutes a major research goal, which has been pursued during the last ten years. The aim of our approach is to obtain an incremental evaluation and simultaneous description of the perceived time-varying scenes. In this contribution we will report on new results of our joint efforts at combining the natural language access system Vitra with a vision system. We have investigated the problem of describing the movements of articulated bodies in image sequences within an integrated natural language and computer vision system. The paper will focus on our model-based approach for the recognition of pedestrians and on the further evaluation and language production in Vitra.

Large-scale software integration for spoken language and multimodal dialog systems

The development of large-scale dialog systems requires a flexible architecture model and adequate software support to cope with the challenge of presents a general framework for building integrated natural-language and multimodal dialog systems. Our approach relies on a distributed component model that enables flexible re-use and extension of existing software modules and is able to deal with a heterogeneous software environment. A practical result of our research is the development of a sophisticated integration platform, called MULTIPLATFORM, which is based on the proposed framework. This MULTIPLATFORM testbed has been used in various large and mid-size research projects to develop integrated system prototypes.