Daniel Pohl

Improved pre-warping for wide angle, head mounted displays

High-quality head mounted displays are becoming available in the consumer space. These displays provide an immersive gaming experience by filling the wearers field of view. To achieve immersion with low cost, a commodity display panel is placed a short distance in front of each eye, and wide-angle optics are used to bring the image into focus. However, these optics introduce spatial and chromatic distortion into the image seen by the viewer. As a result, the images to be displayed must be pre-warped to cancel this distortion. This correction can be performed by warping the image in a post-processing step, by warping the scene geometry before rendering, or by modeling corrective optics in the virtual camera. Here, we examine the image quality and performance of several correction methods. Though image warping with a bilinear filter is common [Antonov et al. 2013], we find that bicubic filtering yields improved image quality with minimal performance impact. We also propose a new method for correcting chromatic distortion by warping the image using distortion meshes, and we propose a method for correcting spatial and chromatic distortion accurately in-camera.

See what I see: Concepts to improve the social acceptance of HMDs

Virtual Reality is reaching into the consumer space. Mobile virtual reality solutions are nowadays widely available and affordable for many smartphones, by adding a case with attached lenses around the phone to create a head-mounted display. While using these in public places or at social gatherings where the head-mounted display is given around to others, it can lead to problems regarding social acceptance, as the surrounding people are not aware of what the virtual reality user is seeing and doing. We address this problem by adding a second, front-facing screen to the head-mounted display. We build and evaluate two prototypes for this usage.

Combining eye tracking with optimizations for lens astigmatism in modern wide-angle HMDs

Virtual Reality has hit the consumer market with affordable head-mounted displays. When using these, it quickly becomes apparent that the resolution of the built-in display panels still needs to be highly increased. To overcome the resulting higher performance demands, eye tracking can be used for foveated rendering. However, as there are lens distortions in HMDs, there are more possibilities to increase the performance with smarter rendering approaches. We present a new system using optimizations for rendering considering lens astigmatism and combining this with foveated rendering through eye tracking. Depending on the current eye gaze, this delivers a rendering speed-up of up to 20%.

High quality, low latency in-home streaming of multimedia applications for mobile devices

Today, mobile devices like smartphones and tablets are becoming more powerful and exhibit enhanced 3D graphics performance. However, the overall computing power of these devices is still limited regarding usage scenarios like photo-realistic gaming, enabling an immersive virtual reality experience or real-time processing and visualization of big data. To overcome these limitations application streaming solutions are constantly gaining focus. The idea is to transfer the graphics output of an application running on a server or even a cluster to a mobile device, conveying the impression that the application is running locally. User inputs on the mobile client side are processed and sent back to the server. The main criteria for successful application streaming are low latency, since users want to interact with the scene in near real-time, as well as high image quality. Here, we present a novel application framework suitable for streaming applications from high-end machines to mobile devices. Using real-time ETC1 compression in combination with a distributed rendering architecture we fully leverage recent progress in wireless computer networking standards (IEEE 802.11ac) for mobile devices, achieving much higher image quality at half the latency compared to other inhome streaming solutions.

Using astigmatism in wide angle HMDs to improve rendering

Lenses in modern consumer HMDs introduce distortions like astigmatism: only the center area of the displayed content can be perceived sharp while with increasing distance from the center the image gets out of focus. We show with three new approaches that this undesired side effect can be used in a positive way to save calculations in blurry areas. For example, using sampling maps to lower the detail in areas where the image is blurred through astigmatism, increases performance by a factor of 2 to 3. Further, we introduce a new calibration of user-specific viewing parameters that increase the performance by about 20-75%.

Concept for using eye tracking in a head-mounted display to adapt rendering to the user's current visual field

With increasing spatial and temporal resolution in head-mounted displays (HMDs), using eye trackers to adapt rendering to the user is getting important to handle the rendering workload. Besides using methods like foveated rendering, we propose to use the current visual field for rendering, depending on the eye gaze. We use two effects for performance optimizations. First, we noticed a lens defect in HMDs, where depending on the distance of the eye gaze to the center, certain parts of the screen towards the edges are not visible anymore. Second, if the user looks up, he cannot see the lower parts of the screen anymore. For the invisible areas, we propose to skip rendering and to reuse the pixels colors from the previous frame. We provide a calibration routine to measure these two effects. We apply the current visual field to a renderer and get up to 2x speed-ups.

The next generation of in-home streaming: Light fields, 5K, 10 GbE, and foveated compression

Interacting with real-time rendered 3D content from powerful machines on smaller devices is becoming ubiquitous through commercial products that enable in-home streaming within the same local network. However, support for high resolution, low latency in-home streaming at high image quality is still a challenging problem. To enable this, we enhance an existing open source framework for in-home streaming. We add highly optimized DXT1 (DirectX Texture Compression) support for thin desktop and notebook clients. For rendered light fields, we improve the encoding algorithms for higher image quality. Within a 10 Gigabit Ethernet (10 GbE) network, we achieve streaming up to 5K resolution at 55 frames per second. Through new low-level algorithmic improvements, we increase the compression speed of ETC1 (Ericsson Texture Compression) by a factor of 5. We are the first to bring ETC2 compression to real-time speed, which increases the streamed image quality. Last, we reduce the required data rate by more than a factor of 2 through foveated compression with real-time eye tracking.

Concept for content-aware, automatic shifting for spherical panoramas

With the adaption of virtual reality in the consumer space, spherical panorama photos are gaining popularity. Through wide-angle head-mounted displays, they can be experienced in a natural way and offer the user an immersive view of the captured scene. While being used in virtual reality, the alignment of the saved image does not matter much. However, when displaying the panorama on a 2D screen, the alignment can make a difference on how pleasant the image looks. We propose an automatic method to do lossless shifting of the image to make it look better on 2D screens.