Lloyd Rutledge

Towards second and third generation web-based multimedia

First generation Web-content encodes information in handwritten (HTML) Web pages. Second generation Web content generates HTML pages on demand, e.g. by filling in templates with content retrieved dynamically from a database or transformation of structured documents using style sheets (e.g. XSLT). Third generation Web pages will make use of rich markup (e.g. XML) along with metadata (e.g. RDF) schemes to make the content not only machine readable but also machine processable - a necessary pre-requisite to the emphSemantic Web. While text-based content on the Web is already rapidly approaching the third generation, multimedia content is still trying to catch up with second generation techniques. Multimedia document processing has a number of fundamentally different requirements from text which make it more difficult to incorporate within the document processing chain. In particular, multimedia transformation uses different document and presentation abstractions, its formatting rules cannot be based on text-flow, it requires feedback from the formatting back-end and is hard to describe in the functional style of current style languages. We state the requirements for second generation processing of multimedia and describe how these have been incorporated in our prototype multimedia document transformation environment, emphCuypers. The system overcomes a number of the restrictions of the text-flow based tool sets by integrating a number of conceptually distinct processing steps in a single runtime execution environment. We describe the need for these different processing steps and describe them in turn (semantic structure, communicative device, qualitative constraints, quantitative constraints, final form presentation), and illustrate our approach by means of an example. We conclude by discussing the models and techniques required for the creation of third generation multimedia content.

Finding the story: broader applicability of semantics and discourse for hypermedia generation

Generating hypermedia presentations requires processing constituent material into coherent, unified presentations. One large challenge is creating a generic process for producing hypermedia presentations from the semantics of potentially unfamiliar domains. The resulting presentations must both respect the underlying semantics and appear as coherent, plausible and, if possible, pleasant to the user. Among the related unsolved problems is the inclusion of discourse knowledge in the generation process. One potential approach is generating a discourse structure derived from generic processing of the underlying domain semantics, transforming this to a structured progression and then using this to steer the choice of hypermedia communicative devices used to convey the actual information in the resulting presentation.This paper presents the results of the first phase of the Topia project, which explored this approach. These results include an architecture for this more domain-independent processing of semantics and discourse into hypermedia presentations. We demonstrate this architecture with an implementation using Web standards and freely available technologies.

Generating presentation constraints from rhetorical structure

Hypermedia structured in terms of the higher-level intent of its author can be adapted to a wider variety of final presentations. Many multimedia systems encode such high-level intent as constraints on either time, spatial layout or navigation. Once specified, these constraints are translated into specific presentations whose timelines, screen displays and navigational structure satisfy these constraints. This ensures that the desired spatial, temporal and navigation properties are maintained no matter how the presentation is adapted to varying circumstances. Rhetorical structure defines author intent at a still higher level. Authoring at this level requires that rhetorics can be translated to final presentations that properly reflect them. This paper explores how rhetorical structure can be translated into constraints, which are then translated into final presentations. This enables authoring in terms of rhetorics and provides the assurance that the rhetorics will remain properly conveyed in all presentation adaptation.

Making RDF presentable: integrated global and local semantic Web browsing

This paper discusses generating document structure from annotated media repositories in a domain-independent manner. This approaches the vision of a universal RDF browser. We start by applying the search-and-browse paradigm established for the WWW to RDF presentation. Furthermore, this paper adds to this paradigm the clustering-based derivation of document structure from search returns, providing simple but domain-independent hypermedia generation from RDF stores. While such generated presentations hardly meet the standards of those written by humans, they provide quick access to media repositories when the required document has not yet been written. The resulting system allows a user to specify a topic for which it generates a hypermedia document providing guided navigation through virtually any RDF repository. The impact for content providers is that as soon as one adds new media items and their annotations to a repository, they become immediately available for automatic integration into subsequently requested presentations.

Structural distinctions between hypermedia storage and presentation

In order to facilitate adaptability of hypermedia documents a distinction is often made between the underlying conceptual structure of a document and the structure of its presentation. This distinction enables greater variety in how a presentation can be adapted to best convey these underlying concepts in a given situation. What is often confusing for those applying this distinction is that although both levels of structure often share similar components: transformation form the storage of a document to its presentation sometimes occurs directly between these similar components and sometimes does not. These similarities typically fall in the categories of space, time and relationships between document portions. This paper identifies some primary similarities between the structure of hypermedia storage and presentation. It also explores how the transformation from storage to presentation often does not follow these similarities. This discussion is illustrated with the Fiets hypermedia application, which addresses the issues of storage, presentation and transformation using public domain formats and tools. The intention is to help authors who separate storage from presentation to better understand this distinction.

Towards a multimedia formatting vocabulary

Time-based, media-centric Web presentations can be described declaratively in the XML world through the development of languages such as SMIL. It is difficult, however, to fully integrate them in a complete document transformation processing chain. In order to achieve the desired processing of data-driven, time-based, media-centric presentations, the text-flow based formatting vocabularies used by style languages such as XSL, CSS and DSSSL need to be extended. The paper presents a selection of use cases which are used to derive a list of requirements for a multimedia style and transformation formatting vocabulary. The boundaries of applicability of existing text-based formatting models for media-centric transformations are analyzed. The paper then discusses the advantages and disadvantages of a fully-fledged time-based multimedia formatting model. Finally, the discussion is illustrated by describing the key properties of the example multimedia formatting vocabulary currently implemented in the back-end of our Cuypers multimedia transformation engine.

A framework for generating adaptable hypermedia documents

Being able to author a hypermedia document once for presentation under a wide variety of potential circumstances requires that it be stored in a manner that is adaptable to these circumstances. Since the nature of these circumstances is not always known at authoring time, specifying how a document adapts to them must be a process that can be performed separately from its original authoring. These distinctions include the porting of the document to different platforms and formats and the adapting of the document’s presentation to suit the needs of the user and of the current state of the presentation environment. In this paper we discuss extensions to our CMIF hypermedia authoring and presentation environment that provide adaptability through this distinction between authoring and presentation specification. This extension includes the use of HyTime for document representation and of DSSSL for presentation specification. We also discuss the Berlage architecture, our extension to HyTime that specifies the encoding of certain hypermedia concepts useful for presentation specification.

Anticipation SMIL 2.0: the developing cooperative infrastructure for multimedia on the Web

Abstract SMIL is the W3C recommendation for bringing synchronized multimedia to the Web. Version 1.0 of SMIL was accepted as a recommendation in June 1998. Work is expected to be soon underway for preparing the next version of SMIL, version 2.0. Issues that will need to be addressed in developing version 2.0 include not just adding new features but also establishing SMILs relationship with various related existing and developing W3C efforts. In this paper we offer some suggestions for how to address these issues. Potential new constructs with additional features for SMIL 2.0 are presented. Other W3C efforts and their potential relationship with SMIL 2.0 are discussed. To provide a context for discussing these issues, this paper explores various approaches for integrating multimedia information with the World Wide Web. It focuses on the modeling issues on the document level and the consequences of the basic differences between text-oriented Web-pages and networked multimedia presentations.

Media semantics: who needs it and why?

Introduction As pointed out in the keynote address at the 2001 ACM Multimedia Conference [3], the current major goal of multimedia research is directed towards provisioning information for pervasive access and use. To achieve this, what will become important are technologies that help sift useful nuggets of information from torrents of media data, which can be turned into valuable knowledge just in time and need, and tools that help provide access to these nuggets in anytime anywhere any device mode to everyone ranging from enterprise customers to independent consumers. Further we need to treat various media on an equal basis in environments that provide multimedia-based interactions, where they ultimately add value to users, whatever the nature of the interactions may be and whatever the preferred mode of media access may be. Thus, there is a fundamental need to investigate the means to elucidate, sublimate, or rationalize information and knowledge from media data. However, current user expectations are far from being met owing to generic low-level content metadata available from automated processing that deal only with representing perceived content, and not the semantics of it.

Practical application of existing hypermedia standards and tools

In order for multimedia presentations to be stored, accessed and played from a large library they should not be encoded as final form presentations, since these consume storage space and cannot easily be adapted to variations in presentation-time circumstances such as user characteristics and changes in end-user technology. Instead, a more presentation independent approach needs to be taken that allows the generation of multiple versions of a presentation based on a presentation-independent description. In order for such a generated presentation tv be widely viewable, it must be in a format that is widely implemented and adopted. Such a format for hypermedia presentations does not yet exist. However, the recent release of SMIL, whose creation and promotion is managed by the World Wide Web Consortium, promises to become such a format in the short term and be for hypermedia what HTML is for hypertext. The technology for enabling this presentation-independent approach is already available, but requires the use of large and unapproachable standards, such as DSSSL and HyTime. In this paper we show that these two standards can be used with SMIL, and by concentrating on a particular application, illustrate the use of publicly available tools to support the generation of multiple presentations from a single presentation-independent source.