Joern Ostermann

MPEG-4: Audio/video and synthetic graphics/audio for mixed media

Abstract MPEG-4 addresses coding of digital hybrids of natural and synthetic, aural and visual (A/V) information. The objective of this synthetic/natural hybrid coding (SNHC) is to facilitate content-based manipulation, interoperability, and wider user access in the delivery of animated mixed media. SNHC will support non-real-time and passive media delivery, as well as more interactive, real-time applications. Integrated spatial-temporal coding is sought for audio, video, and 2D/3D computer graphics as standardized A/V objects. Targets of standardization include mesh-segmented video coding, compression of geometry, synchronization between A/V objects, multiplexing of streamed A/V objects, and spatial-temporal integration of mixed media types. Composition, interactivity, and scripting of A/V objects can thus be supported in client terminals, as well as in content production for servers, also more effectively enabling terminals as servers. Such A/V objects can exhibit high efficiency in transmission and storage, plus content-based interactivity, spatial-temporal scalability, and combinations of transient dynamic data and persistent downloaded data. This approach can lower bandwidth of mixed media, offer tradeoffs in quality versus update for specific terminals, and foster varied distribution methods for content that exploit spatial and temporal coherence over buses and networks. MPEG-4 responds to trends at home and work to move beyond the paradigm of audio/video as a passive experience to more flexible A/V objects which combine audio/video with synthetic 2D/3D graphics and audio.

Special issue on analysis and understanding for video adaptation

Presents an introduction to a special issue on the analysis and understanding of video adaptation. The papers in the special issue are very briefly described.

Recent advances in audio and video text‐to‐speech synthesis

Harry Levitt characteristically combines a statistical or data‐driven engineering approach with extensive scientific knowledge when tackling fundamental research questions (such as how to measure co‐articulation) and practical problems (such as evaluating intelligibility). Some of Harry’s expansive research interests and contributions have focused on audio and video text‐to‐speech (TTS) synthesis. Some major recent improvements in these technologies that have resulted from applying similar approaches are discussed. The unit selection method of synthesis described differs from both the earlier techniques of formant synthesis and of concatenative synthesis from a diphone inventory. In unit selection synthesis, speech or video units are selected from among multiple candidates in a large database by optimizing a cost function. The cost functions used for unit selection are estimates of perceptual distances. The improvement in naturalness for both audio and visual TTS achieved through the unit selection techni...