Rene Kaiser

Open challenges in modelling, analysis and synthesis of human behaviour in human–human and human–machine interactions

Modelling, analysis and synthesis of behaviour are the subject of major efforts in computing science, especially when it comes to technologies that make sense of human–human and human–machine interactions. This article outlines some of the most important issues that still need to be addressed to ensure substantial progress in the field, namely (1) development and adoption of virtuous data collection and sharing practices, (2) shift in the focus of interest from individuals to dyads and groups, (3) endowment of artificial agents with internal representations of users and context, (4) modelling of cognitive and semantic processes underlying social behaviour and (5) identification of application domains and strategies for moving from laboratory to the real-world products.

Real-time Person Tracking in High-resolution Panoramic Video for Automated Broadcast Production

For enabling immersive user experiences for interactive TV services and automating camera view selection and framing, knowledge of the location of persons in a scene is essential. We describe an architecture for detecting and tracking persons in high-resolution panoramic video streams, obtained from the Omni Cam, a panoramic camera stitching video streams from 6 HD resolution tiles. We use a CUDA accelerated feature point tracker, a blob detector and a CUDA HOG person detector, which are used for region tracking in each of the tiles before fusing the results for the entire panorama. In this paper we focus on the application of the HOG person detector in real-time and the speedup of the feature point tracker by porting it to NVIDIAs Fermi architecture. Evaluations indicate significant speedup for our feature point tracker implementation, enabling the entire process in a real-time system.

Reasoning for video-mediated group communication

In this paper we present an approach to the reasoning required to support multi-location, multi-camera group-to-group video communication, which we call orchestration. Orchestration is akin to virtual directing: it has to ensure that each location displays the most adequate shots from all the other available sources. Its input is low-level cues extracted automatically from the AV streams. They are processed to detect higher-level events that determine the state of the communication. Directorial decisions are then inferred, reflecting social communication as well as stylistic criteria. Finally, they are transformed into camera and editing commands, directly executable by the AV infrastructure. Here, we present the architecture of the Orchestrator and sketch our rule-based approach to reasoning.

A Rule-Based Virtual Director Enhancing Group Communication

Audiovisual group communication systems deal with a large number of video streams, and, unlike less advanced videoconferencing systems, require intelligence for selecting adequate views for each of the connected rooms, in order to convey best what is happening in the other locations. Such a decision making component, in our implementation called Orchestration Engine (OE), acts as a Virtual Director. It processes low level events, emitted by content analysis sensors, into editing commands. The OE has two main components: one that semantically lifts low-level events into communication events and one that associates editing decisions to communication contexts. The former has to deal with uncertain and delayed information. The latter subsumes knowledge that reflects both conversation and narrative principles. Both components include contradicting bodies of knowledge. We investigate a rule-based event processing approach and reflect the scalability of our solution regarding competing and contradicting rules.

Towards a format-agnostic approach for production, delivery and rendering of immersive media

The media industry is currently being pulled in the often-opposing directions of increased realism (high resolution, stereoscopic, large screen) and personalization (selection and control of content, availability on many devices). We investigate the feasibility of an end-to-end format-agnostic approach to support both these trends. In this paper, different aspects of a format-agnostic capture, production, delivery and rendering system are discussed. At the capture stage, the concept of layered scene representation is introduced, including panoramic video and 3D audio capture. At the analysis stage, a virtual director component is discussed that allows for automatic execution of cinematographic principles, using feature tracking and saliency detection. At the delivery stage, resolution-independent audiovisual transport mechanisms for both managed and unmanaged networks are treated. In the rendering stage, a rendering process that includes the manipulation of audiovisual content to match the connected display and loudspeaker properties is introduced. Different parts of the complete system are revisited demonstrating the requirements and the potential of this advanced concept.

Metadata-driven interactive web video assembly

The recent expansion of broadband Internet access led to an exponential increase of potential consumers of video on the Web. The huge success of video upload websites shows that the online world, with its virtually unlimited possibilities of active user participation, is an ideal complement to traditional consumption-only media like TV and DVD. It is evident that users are willing to interact with content-providing systems in order to get the content they desire. In parallel to these developments, innovative tools for producing interactive, non-linear audio-visual content are being created. They support the authoring process alongside management of media and metadata, enabling on-demand assembly of videos based on the consumer’s wishes. The quality of such a dynamic video remixing system mainly depends on the expressiveness of associated metadata. Eliminating the need for manual input as far as possible, we aim at designing a system which is able to automatically enrich its own media and metadata repositories continuously. Currently, video content remixing is available on the Web mostly in very basic forms. Most platforms offer upload and simple modification of content. Although several implementations exist, to the best of our knowledge no solution uses metadata to its full extent to dynamically render a video stream based on consumers’ wishes. With the research presented in this paper, we propose a novel concept to interactive video assembly on the Web. In this approach, consumers may describe the desired content using a set of domain-specific parameters. Based on the metadata the video clips are annotated with, the system chooses clips fitting the user criteria. They are aligned in an aesthetically pleasing manner while the user furthermore is able to interactively influence content selection during playback at any time. We use a practical example to clarify the concept and further outline what it takes to implement a suchlike system.

The fascinate production scripting engine

In the realm of a format agnostic live event broadcast system, the FascinatE Scripting Engines are software components that automate taking decisions on what is visible and audible at each playout device and prepare the audiovisual content streams for display. Essentially, they act together as a Virtual Director with the production team possibly steering it via a backend user interface. We present an architecture for this real-time system and describe interfaces to other production components. Details of subcomponents of the distributed engine, design decisions and technology choices are discussed.

Automatic orchestration of video streams to enhance group communication

Unlike legacy video-conferencing, which connects two nodes each equipped with a camera, recent systems facilitating for video-mediated group communication deal simultaneously with a large number of video streams. This highlights the need for orchestration,, i.e. the intelligent selection of the most adequate camera views to be displayed on each screen. In this paper we present the initial results of a study that evaluates the effects of orchestration on communication within a specific context; that of two remote groups playing a collaborative board game. The results of the experiment indicate that automatic orchestration can provide improvements similar to the ones achieved when live video mixing is performed by human editors.

Socialising through orchestrated video communication

We report on the development of a video communication medium through which groups of people situated in different physical locations can naturally talk to each other, see and hear each other, and engage in social entertaining activities. Participants are free to move within their space and behave in a manner closer to collocated experiences. Essentially, this is implemented as a multi-location, multi-camera, hands-free video conferencing system between groups, with integrated support for entertaining activities. In this paper we focus on automatic orchestration, the reasoning process that applies screen grammar to best support the communication. We present a formal model for representing orchestration rules and discuss initial evaluation results.

Enabling Distributed Theatre Performances through Multi-Camera Telepresence: Capturing System Behaviour in a Script-Based Approach

Audio-visual media has been utilized in theatre in many creative ways, but most of them have been employed to enrich performances on a single stage. Telepresence systems give rise to a new concept: distributed theatre, denoting performances co-acted from two or more different stages, each potentially having its own local audience. To achieve a truly immersive experience, we posit, such systems must satisfy a key requirement: they must be able to dynamically reframe the cameras and mix the content from the various sources on each of the available screens in order to capture the most relevant expressions in each location and appropriately represent them in the other locations. We have developed a script-based approach that allows the production team to express and refine such adaptive behaviour during development and rehearsals, as well as controlling it during the performance. We ran a successful trial of a telepresence system that implements this approach. This paper gives a brief description of the overall approach and of two key software components that control the systems behaviour, with some reflections upon the specific experience of the trial, the overall approach and further research.