Vamsi Kiran Adhikarla

Exploring direct 3D interaction for full horizontal parallax light field displays using leap motion controller.

This paper reports on the design and evaluation of direct 3D gesture interaction with a full horizontal parallax light field display. A light field display defines a visual scene using directional light beams emitted from multiple light sources as if they are emitted from scene points. Each scene point is rendered individually resulting in more realistic and accurate 3D visualization compared to other 3D displaying technologies. We propose an interaction setup combining the visualization of objects within the Field Of View (FOV) of a light field display and their selection through freehand gesture tracked by the Leap Motion Controller. The accuracy and usefulness of the proposed interaction setup was also evaluated in a user study with test subjects. The results of the study revealed high user preference for free hand interaction with light field display as well as relatively low cognitive demand of this technique. Further, our results also revealed some limitations and adjustments of the proposed setup to be addressed in future work.

Subjective evaluation of Super Multi-View compressed contents on high-end light-field 3D displays

Super Multi-View (SMV) video content is composed of tens or hundreds of views that provide a light-field representation of a scene. This representation allows a glass-free visualization and eliminates many causes of discomfort existing in current available 3D video technologies. Efficient video compression of SMV content is a key factor for enabling future 3D video services. This paper first compares several coding configurations for SMV content and several inter-view prediction structures are also tested and compared. The experiments mainly suggest that large differences in coding efficiency can be observed from one configuration to another. Several ratios for the number of coded and synthesized views are compared, both objectively and subjectively. It is reported that view synthesis significantly affects the coding scheme. The amount of views to skip highly depends on the sequence and on the quality of the associated depth maps. Reported ranges of bitrates required to obtain a good quality for the tested SMV content are realistic and coherent with future 4K/8K needs. The reliability of the PSNR metric for SMV content is also studied. Objective and subjective results show that PSNR is able to reflect increase or decrease in subjective quality even in the presence of synthesized views. However, depending on the ratio of coded and synthesized views, the order of magnitude of the effective quality variation is biased by PSNR. Results indicate that PSNR is less tolerant to view synthesis artifacts than human viewers. Finally, preliminary observations are initiated. First, the light-field conversion step does not seem to alter the objective results for compression. Secondly, the motion parallax does not seem to be impacted by specific compression artifacts. The perception of the motion parallax is only altered by variations of the typical compression artifacts along the viewing angle, in cases where the subjective image quality is already low. To the best of our knowledge, this paper is the first to carry out subjective experiments and to report results of SMV compression for light-field 3D displays. It provides first results showing that improvement of compression efficiency is required, as well as depth estimation and view synthesis algorithms improvement, but that the use of SMV appears realistic according to next generation compression technology requirements. HighlightsStudy of the impact of compression on subjective quality for lightfield SMV content.To the best of our knowledge, this paper is the first to report results of this kind.Several SMV coding configurations are compared both objectively and subjectively.Compression efficiency, depth estimation and view synthesis require improvements.SMV appears realistic according to next generation compression technology requirements.

Freehand interaction with large-scale 3D map data

In this paper, we present our method and apparatus to visualize and interact with large-scale 3D map data on a 3D light-field display in real time. 3D map data are streamed over Internet to the display in real-time based on request sent by the application. On the user side, data is processed and visualized on a large-scale 3D light field display. We present a method to efficiently interact with the visualized 3D map using freehand gestures. We use a Leap Motion sensor that supports sufficient refresh rate for our current application and is able to accurately track and acquire information on the user hand position. This information is further processed to support natural and fast interaction. We have also developed a method to automatically adjust the maps plane to the screen plane of the display. The visualization and interaction method is scalable and allows the exploration of large-scale 3D maps down to the street level.

Overview of the applicability of H.264/MVC for real-time light-field applications

Several methods for compressing light-fields (LF) and multiview 3D video content have been proposed in the literature. The most widely accepted and standardized method is the Multi View Coding (MVC) extension of H.264, which is considered appropriate for use with stereoscopic and multiview 3D displays. In this paper we will focus on light-field 3D displays, outline typical use cases for such displays, analyze processing requirements for display-specific and display-independent light-fields, and see how these map to MVC as the underlying 3D video compression method. We also provide an overview of available MVC implementations, and the support these provide for multiview 3D video. Directions for future research and additional features supporting LF video compression are presented.

Natural 3D content on glasses-free light-field 3D cinema

This paper presents a complete framework for capturing, processing and displaying the free viewpoint video on a large scale immersive light-field display. We present a combined hardware-software solution to visualize free viewpoint 3D video on a cinema-sized screen. The new glasses-free 3D projection technology can support larger audience than the existing autostereoscopic displays. We introduce and describe our new display system including optical and mechanical design considerations, the capturing system and render cluster for producing the 3D content, and the various software modules driving the system. The indigenous display is first of its kind, equipped with front-projection light-field HoloVizio technology, controlling up to 63 MP. It has all the advantages of previous light-field displays and in addition, allows a more flexible arrangement with a larger screen size, matching cinema or meeting room geometries, yet simpler to set-up. The software system makes it possible to show 3D applications in real-time, besides the natural content captured from dense camera arrangements as well as from sparse cameras covering a wider baseline. Our software system on the GPU accelerated render cluster, can also visualize pre-recorded Multi-view Video plus Depth (MVD4) videos on this light-field glasses-free cinema system, interpolating and extrapolating missing views.

Real-time adaptive content retargeting for live multi-view capture and light field display

The discrete nature of multiprojector light field displays results in aliasing when rendering scene points at depths outside the supported depth of field causing visual discomfort. We propose an efficient on-the-fly content-aware real-time depth retargeting algorithm for live 3D light field video to increase the quality of visual perception on a cluster-driven multiprojector light field display. The proposed algorithm is embedded in an end-to-end real-time system capable of capturing and reconstructing light field from multiple calibrated cameras on a full horizontal parallax light field display. By automatically detecting salient regions of a scene, we solve an optimization to derive a non-linear operator to fit the whole scene within the comfortable viewing range of the light field display. We evaluate the effectiveness of our approach on synthetic and real world scenes.

Analysis and optimization of pixel usage of light-field conversion from multi-camera setups to 3D light-field displays

Light-field (LF) 3D displays require vast amount of views representing the original scene when using pure light-ray interpolation to convert multi-camera content to display-specific LF representation. Synthetic and real multi-camera setups are both used to feed these displays with image-based data, however the layout, number, frustum, and resolution of these cameras are mostly suboptimal. Storage and transmission of LF data is an issue, especially considering that some of the captured / rendered pixels are left unused while generating the final image. LF displays can have significantly different requirements for camera setups due to differences in Field of View (FOV), angular resolution and spatial resolution. An analysis of typical camera setups and LF display setups, and the typical patterns in pixel usage resulting from the combination of these setups are presented. Based on this analysis, an optimization method for virtual camera setups is proposed. As virtual cameras have wide range of adjustment possibilities, highly optimized setups for specific displays can be achieved.

Fast and efficient data reduction approach for multi-camera light field display telepresence systems

Cutting-edge telepresence systems equipped with multiple cameras for capturing the whole scene of a collaboration space, face the challenge of transmitting huge amount of dynamic data from multiple viewpoints. With the introduction of Light Field Displays (LFDs) in to the remote collaboration space, it became possible to produce an impression of 3D virtual presence. In addition, LFDs in current generation also rely on the images obtained from cameras arranged in various spatial configurations. To have a realistic and natural 3D collaboration using LFDs, the data in the form of multiple camera images needs to be transmitted in real time using the available bandwidth. Classical compression methods might resolve this issue to a certain level. However, in many cases the achieved compression level is by far insufficient. Moreover, the available compression schemes do not consider any of the display-related attributes. Here, we propose a method by which we reduce the data from each of the camera images by discarding unused parts of the images at the acquisition site in a predetermined way using the display model and geometry, as well as the mapping between the captured and displayed light field. The proposed method is simple to implement and can exclude the unnecessary data in an automatic way. While similar methods exist for 2D screens or display walls, this is the first such algorithm for light fields. Our experimental results show that an identical light field reconstruction can be achieved with the reduced set of data which we would have got if all the data were transmitted. Moreover, the devised method provides very good processing speed.

Design and Evaluation of Freehand Gesture Interaction for Light Field Display

The paper reports on a user study of freehand gesture interaction with a prototype of autostereoscopic 3D light field display. The interaction was based on a direct touch selection of simple objects rendered at different positions in space. The main goal of our experiment was to evaluate the overall user experience and perceived cognitive workload of such freehand interaction in 3D environment and compare it to the simplified touch-based interaction in 2D environment. The results of the experiment confirmed the hypothesis that significantly more time is required for the interaction in 3D than the interaction in 2D. Surprisingly, no significant difference was found in the results of the assessment of cognitive workload when comparing 3D and 2D. We believe the interaction scenario proposed and evaluated in this study could represent an efficient and intuitive future interaction technique for the selection and manipulation of content rendered on autostereoscopic 3D displays

Real-time Content Adaptive Depth Retargeting for Light Field Displays

Light field display systems present visual scenes using a set of directional light beams emitted from multiple light sources as if they are emitted from points in a physical scene. These displays offer better angular resolution and therefore provide more depth of field than other automultiscopic displays. However in some cases the size of a scene may still exceed the available depth range of a light field display. Thus, rendering on these displays requires suitable adaptation of 3D content for providing comfortable viewing experience. We propose a content adaptive depth retargeting method to automatically modify the scene depth to suit to the needs of a light field display. By analyzing the scene and using display specific parameters, we formulate and solve an optimization problem to non-linearly adapt the scene depth to display depth. Our method synthesizes the depth retargeted light field content in real-time for supporting interactive visualization and also preserves the 3D appearance of the displayed objects as much as possible.