Steven Poulakos

Nonlinear disparity mapping for stereoscopic 3D

This paper addresses the problem of remapping the disparity range of stereoscopic images and video. Such operations are highly important for a variety of issues arising from the production, live broadcast, and consumption of 3D content. Our work is motivated by the observation that the displayed depth and the resulting 3D viewing experience are dictated by a complex combination of perceptual, technological, and artistic constraints. We first discuss the most important perceptual aspects of stereo vision and their implications for stereoscopic content creation. We then formalize these insights into a set of basic disparity mapping operators. These operators enable us to control and retarget the depth of a stereoscopic scene in a nonlinear and locally adaptive fashion. To implement our operators, we propose a new strategy based on stereoscopic warping of the input video streams. From a sparse set of stereo correspondences, our algorithm computes disparity and image-based saliency estimates, and uses them to compute a deformation of the input views so as to meet the target disparities. Our approach represents a practical solution for actual stereo production and display that does not require camera calibration, accurate dense depth maps, occlusion handling, or inpainting. We demonstrate the performance and versatility of our method using examples from live action post-production, 3D display size adaptation, and live broadcast. An additional user study and ground truth comparison further provide evidence for the quality and practical relevance of the presented work.

Disparity-Aware Stereo 3D Production Tools

Stereoscopic 3D (S3D) has reached wide levels of adoption in consumer and professional markets. However, production of high quality S3D content is still a difficult and expensive art. Various S3D production tools and systems have been released recently to assist high quality content creation. This paper presents a number of such algorithms, tools and systems developed at Disney Research Zurich, which all make use of disparity-aware processing.

Optimizing stereo-to-multiview conversion for autostereoscopic displays

We present a novel stereo‐to‐multiview video conversion method for glasses‐free multiview displays. Different from previous stereo‐to‐multiview approaches, our mapping algorithm utilizes the limited depth range of autostereoscopic displays optimally and strives to preserve the scenes artistic composition and perceived depth even under strong depth compression. We first present an investigation of how perceived image quality relates to spatial frequency and disparity. The outcome of this study is utilized in a two‐step mapping algorithm, where we (i) compress the scene depth using a non‐linear global function to the depth range of an autostereoscopic display and (ii) enhance the depth gradients of salient objects to restore the perceived depth and salient scene structure. Finally, an adapted image domain warping algorithm is proposed to generate the multiview output, which enables overall disparity range extension.

Perceptually-based compensation of light pollution in display systems

This paper addresses the problem of unintended light contributions due to physical properties of display systems. An example of such unintended contribution is crosstalk in stereoscopic 3D display systems, often referred to as ghosting. Ghosting results in a reduction of visual quality, and may lead to an uncomfortable viewing experience. The latter is due to conflicting (depth) edge cues, which can hinder the human visual system (HVS) proper fusion of stereo images (stereopsis). We propose an automatic, perceptually-based computational compensation framework, which formulates pollution elimination as a minimization problem. Our method aims to distribute the error introduced by the pollution in a perceptually optimal manner. As a consequence ghost edges are smoothed locally, resulting in a more comfortable stereo viewing experience. We show how to make the computation tractable by exploiting the structure of the resulting problem, and also propose a perceptually-based pollution prediction. We show that our general framework is applicable to other light pollution problems, such as descattering.

REVERIE: Natural human interaction in virtual immersive environments

REVERIE (REal and Virtual Engagement in Realistic Immersive Environments [1]) targets novel research to address the demanding challenges involved with developing state-of-the-art technologies for online human interaction. The REVERIE framework enables users to meet, socialise and share experiences online by integrating cutting-edge technologies for 3D data acquisition and processing, networking, autonomy and real-time rendering. In this paper, we describe the innovative research that is showcased through the REVERIE integrated framework through richly defined use-cases which demonstrate the validity and potential for natural interaction in a virtual immersive and safe environment. Previews of the REVERIE demo and its key research components can be viewed at www.youtube.com/user/REVERIEFP7.

A computational model for perception of stereoscopic window violations

Creating a computational model for stereoscopic 3D perception is a highly complex undertaking. As one step towards this goal, this paper investigates stereoscopic window violation artifacts, which often interfere with artistic freedom and constrain the comfortable depth volume. Window violations need to be compensated for in most 3D feature movies. Currently this is done in an ad-hoc manner due to a limited understanding of the problem. In this work, we present a model predicting problematic window violations that are visually disturbing. The model parameters were defined through psychophysical experiments on simple stimuli. Then the model was calibrated and validated on real, complex stereoscopic images. Finally, we present a system to provide visualization of problematic stereoscopic window violations as well as details for how to correct them.

Evaluating Accessible Graphical Interfaces for Building Story Worlds

In order to use computational intelligence to assist in narrative generation, domain knowledge of the story world must be defined, a task which is currently confined to experts. In an effort to democratize story world creation, we present an accessible graphical platform for content creators and end users to create a story world, populate it with smart characters and objects, and define narrative events that can be used to author digital stories. The system supports reuse to reduce the cost of content production and enables specification of semantics to enable computer assisted authoring. Additionally, we introduce an iterative, bi-directional workflow, which bridges the gap between story world building and story authoring. Users seamlessly transition between authoring stories and refining the story world definition to accommodate their current narrative. A user study demonstrates the efficacy of our system to support narrative generation.

Alternating attention in continuous stereoscopic depth

The decoupling of eye vergence and accommodation (V/A) has been found to negatively impact depth interpretation, visual comfort and fatigue. In this paper, we explore a hypothesis that placement of visual cues within a scene can assist a viewer in the process of maintaining the V/A decoupling. This effect is demonstrated through the use of a continuous depth plane that connects spatially distinct scene elements. Our experimental design enables us to make the following three contributions: (1) We show that a continuous depth element can improve the time it takes to transition visual attention in depth. (2) We observe that the subjective assessment of fatigue emerges before we detect a quantitative decline in performance. (3) We aim to motivate that stereoscopic 3D content creators may learn scene composition, framing and montage from visual psychophysics.

Computational narrative

Despite the maturity in solutions for animating expressive virtual characters, innovations realizing the creative intent of story writers have yet to make the same strides. This problem is further exacerbated for interactive narrative content, such as games. The key challenge is to provide an accessible, yet expressive interface for story authoring that enables the rapid prototyping, iteration, and deployment of narrative concepts, while facilitate free-form interaction. In this short course, we present the potential of computational intelligence to empower authors and content creators in creating their own interactive animated stories. There are 4 key contributions towards realizing this goal. First, we introduce a novel event-centric representation of narrative atoms which serve as the building blocks of any story. Second, we present a graphical platform for story architects to craft their own unique story worlds. Third, we present CANVAS, a computer-assisted visual authoring tool for synthesizing multi-character animations from sparsely- specified narrative events. In a process akin to storyboarding, authors lay out the key plot points in a story, and our system automatically fills in the missing details to synthesize a 3D animation that meets author constraints. Fourth, we present extensions to our logical formalisms to enable the transformation of a passive narrative into an interactive story, and how computational intelligence may be leveraged to identify and automatically resolve conflicts in the story. We analyse the authoring complexity of different story formalisms to present the benefits and tradeoffs of each. This course targets both Basic and Intermediate level attendees, with a preliminary background knowledge of Computer Animation and Artificial Intelligence recommended.

Story Version Control and Graphical Visualization for Collaborative Story Authoring

This paper presents a story version control and graphical visualization framework to enhance collaborative story authoring. We propose a media-agnostic story representation based on story beats, events, and participants that describes the flow of events in a storyline. We develop tree edit distance operations for this representation and use them to build the core features for story version control, including visual diff, conflict detection, and conflict resolution using three-way merge. Our system allows authors to work independently on the same story while providing the ability to automatically synchronize their efforts and resolve conflicts that may arise. We further enhance the collaborative authoring process using visualizations derived from the version control database that visually encode relationships between authors, characters, and story elements, during the evolution of the narrative. We demonstrate the efficacy of our system by integrating it within an existing visual storyboarding tool for authoring animated stories, and additionally use it to collaboratively author stories using video and images. We evaluate the usability of our system though two user studies. Our results reveal that untrained users are able to use and benefit from our system. Additionally, users are able to correctly interpret the graphical visualizations and perceive it to benefit collaboration during the story authoring process.