Joohwan Kim

The zone of comfort: Predicting visual discomfort with stereo displays

Recent increased usage of stereo displays has been accompanied by public concern about potential adverse effects associated with prolonged viewing of stereo imagery. There are numerous potential sources of adverse effects, but we focused on how vergence-accommodation conflicts in stereo displays affect visual discomfort and fatigue. In one experiment, we examined the effect of viewing distance on discomfort and fatigue. We found that conflicts of a given dioptric value were slightly less comfortable at far than at near distance. In a second experiment, we examined the effect of the sign of the vergence-accommodation conflict on discomfort and fatigue. We found that negative conflicts (stereo content behind the screen) are less comfortable at far distances and that positive conflicts (content in front of screen) are less comfortable at near distances. In a third experiment, we measured phoria and the zone of clear single binocular vision, which are clinical measurements commonly associated with correcting refractive error. Those measurements predicted susceptibility to discomfort in the first two experiments. We discuss the relevance of these findings for a wide variety of situations including the viewing of mobile devices, desktop displays, television, and cinema.

Towards foveated rendering for gaze-tracked virtual reality

Foveated rendering synthesizes images with progressively less detail outside the eye fixation region, potentially unlocking significant speedups for wide field-of-view displays, such as head mounted displays, where target framerate and resolution is increasing faster than the performance of traditional real-time renderers. To study and improve potential gains, we designed a foveated rendering user study to evaluate the perceptual abilities of human peripheral vision when viewing todays displays. We determined that filtering peripheral regions reduces contrast, inducing a sense of tunnel vision. When applying a postprocess contrast enhancement, subjects tolerated up to 2× larger blur radius before detecting differences from a non-foveated ground truth. After verifying these insights on both desktop and head mounted displays augmented with high-speed gaze-tracking, we designed a perceptual target image to strive for when engineering a production foveated renderer. Given our perceptual target, we designed a practical foveated rendering system that reduces number of shades by up to 70% and allows coarsened shading up to 30° closer to the fovea than Guenter et al. [2012] without introducing perceivable aliasing or blur. We filter both pre- and post-shading to address aliasing from undersampling in the periphery, introduce a novel multiresolution- and saccade-aware temporal antialising algorithm, and use contrast enhancement to help recover peripheral details that are resolvable by our eye but degraded by filtering. We validate our system by performing another user study. Frequency analysis shows our system closely matches our perceptual target. Measurements of temporal stability show we obtain quality similar to temporally filtered non-foveated renderings.

The Rate of Change of Vergence-Accommodation Conflict Affects Visual Discomfort

Stereoscopic (S3D) displays create conflicts between the distance to which the eyes must converge and the distance to which the eyes must accommodate. Such conflicts require the viewer to overcome the normal coupling between vergence and accommodation, and this effort appears to cause viewer discomfort. Vergence-accommodation coupling is driven by the phasic components of the underlying control systems, and those components respond to relatively fast changes in vergence and accommodative stimuli. Given the relationship between phasic changes and vergence-accommodation coupling, we examined how the rate of change in the vergence-accommodation conflict affects viewer discomfort. We used a stereoscopic display that allows independent manipulation of the stimuli to vergence and accommodation. We presented stimuli that simulate natural viewing (i.e., vergence and accommodative stimuli changed together) and stimuli that simulate S3D viewing (i.e., vergence stimulus changes but accommodative stimulus remains fixed). The changes occurred at 0.01, 0.05, or 0.25 Hz. The lowest rate is too slow to stimulate the phasic components while the highest rate is well within the phasic range. The results were consistent with our expectation: somewhat greater discomfort was experienced when stimulus distance changed rapidly, particularly in S3D viewing when the vergence stimulus changed but the accommodative stimulus did not. These results may help in the generation of guidelines for the creation and viewing of stereo content with acceptable viewer comfort.

Perceptually-based foveated virtual reality

Humans have two distinct vision systems: foveal and peripheral vision. Foveal vision is sharp and detailed, while peripheral vision lacks fidelity. The difference in characteristics of the two systems enable recently popular foveated rendering systems, which seek to increase rendering performance by lowering image quality in the periphery. We present a set of perceptually-based methods for improving foveated rendering running on a prototype virtual reality headset with an integrated eye tracker. Foveated rendering has previously been demonstrated in conventional displays, but has recently become an especially attractive prospect in virtual reality (VR) and augmented reality (AR) display settings with a large field-of-view (FOV) and high frame rate requirements. Investigating prior work on foveated rendering, we find that some previous quality-reduction techniques can create objectionable artifacts like temporal instability and contrast loss. Our emerging technologies installation demonstrates these techniques running live in a head-mounted display and we will compare them against our new perceptually-based foveated techniques. Our new foveation techniques enable significant reduction in rendering cost but have no discernible difference in visual quality. We show how such techniques can fulfill these requirements with potentially large reductions in rendering cost.

Insight into vergence/accommodation mismatch

Prolonged use of conventional stereo displays causes viewer discomfort and fatigue because of the vergenceaccommodation conflict. We used a novel volumetric display to examine how viewing distance, the sign of the vergenceaccommodation conflict, and the temporal properties of the conflict affect discomfort and fatigue. In the first experiment, we presented a fixed conflict at short, medium, and long viewing distances. We compared subjects’ symptoms in that condition and one in which there was no conflict. We observed more discomfort and fatigue with a given vergenceaccommodation conflict at the longer distances. The second experiment compared symptoms when the conflict had one sign compared to when it had the opposite sign at short, medium, and long distances. We observed greater symptoms with uncrossed disparities at long distances and with crossed disparities at short distances. The third experiment compared symptoms when the conflict changed rapidly as opposed to slowly. We observed more serious symptoms when the conflict changed rapidly. These findings help define comfortable viewing conditions for stereo displays.

Effect of interlacing methods of stereoscopic displays on perceived image quality

To provide two-eyed views with one device, stereoscopic 3D (S3D) displays interlace the two views either temporally or spatially: temporal interlacing (TI) alternates the two views in time with full resolution, while spatial interlacing (SI) presents the two views simultaneously but with half resolution for each eye. We investigate the effect of interlacing methods on image quality through a psychophysical experiment. We compared four experimental conditions: three S3D interlacing methods (TI, SI with raw sampling, and SI with vertical interpolation), and one nonconventional interlacing method (vertical interpolation). The stimuli were 10 natural stereo images presented at nine levels of pixel sizes (0.64, 0.78, 0.89, 1.00, 1.28, 1.55, 1.78, 2.00, and 2.56 arcmin). To test the effect of interlacing methods per se, we provided all the experimental conditions to the subjects using a single experimental setup: a mirror stereoscope. The results show that TI does not degrade the image quality for any pixel size. SI degrades the image quality when the pixel size is relatively large, but the effect of the two SI methods does not differ significantly. Comparison of SI methods against the vertical interpolation method implies that the primary cause of the degradation in image quality for SI methods is the visibility of the interlacing pattern rather than the loss of high-frequency information.

Visual discomfort and the temporal properties of the vergence-accommodation conflict

The vergence-accommodation conflict associated with viewing stereoscopic 3D (S3D) content can cause visual discomfort. Previous studies of vergence and accommodation have shown that the coupling between the two responses is driven by a fast, phasic component. We investigated how the temporal properties of vergence-accommodation conflicts affect discomfort. Using a unique volumetric display, we manipulated the stimulus to vergence and the stimulus to accommodation independently. There were two experimental conditions: 1) natural viewing in which the stimulus to vergence was perfectly correlated with the stimulus to accommodation; and 2) conflict viewing in which the stimulus to vergence varied while the stimulus to accommodation remained constant (thereby mimicking S3D viewing). The stimulus to vergence (and accommodation in natural viewing) varied at one of three temporal frequencies in those conditions. The magnitude of the conflict was the same for all three frequencies. The young adult subjects reported more visual discomfort when vergence changes were faster, particularly in the conflict condition. Thus, the temporal properties of the vergence-accommodation conflict in S3D media affect visual discomfort. The results can help content creators minimize discomfort by making conflict changes sufficiently slow.

Latency Requirements for Foveated Rendering in Virtual Reality

Foveated rendering is a performance optimization based on the well-known degradation of peripheral visual acuity. It reduces computational costs by showing a high-quality image in the user’s central (foveal) vision and a lower quality image in the periphery. Foveated rendering is a promising optimization for Virtual Reality (VR) graphics, and generally requires accurate and low-latency eye tracking to ensure correctness even when a user makes large, fast eye movements such as saccades. However, due to the phenomenon of saccadic omission, it is possible that these requirements may be relaxed. In this article, we explore the effect of latency for foveated rendering in VR applications. We evaluated the detectability of visual artifacts for three techniques capable of generating foveated images and for three different radii of the high-quality foveal region. Our results show that larger foveal regions allow for more aggressive foveation, but this effect is more pronounced for temporally stable foveation techniques. Added eye tracking latency of 80--150ms causes a significant reduction in acceptable amount of foveation, but a similar decrease in acceptable foveation was not found for shorter eye-tracking latencies of 20--40ms, suggesting that a total system latency of 50--70ms could be tolerated.

The visibility of color breakup and a means to reduce it

Color breakup is an artifact seen on displays that present colors sequentially. When the eye tracks a moving object on such a display, different colors land on different places on the retina, and this gives rise to visible color fringes at the objects leading and trailing edges. Interestingly, color breakup is also observed when the eye is stationary and an object moves by. Using a novel psychophysical procedure, we measured breakup both when viewers tracked and did not track a moving object. Breakup was somewhat more visible in the tracking than in the non-tracking condition. The video frames contained three subframes, one each for red, green, and blue. We spatially offset the green and blue stimuli in the second and third subframes, respectively, to find the values that minimized breakup. In the tracking and non-tracking conditions, spatial offsets of Δx/3 in the second subframe (where Δx is the displacement of the object in one frame) and 2Δx/3 in the third eliminated breakup. Thus, this method offers a way to minimize or even eliminate breakup whether the viewer is tracking or not. We suggest ways to implement the method with real video content. We also developed a color-breakup model based on spatiotemporal filtering in color-opponent pathways in early vision. We found close agreement between the models predictions and the experimental results. The model can be used to predict breakup for a wide variety of conditions.

Perceptually-guided foveation for light field displays

A variety of applications such as virtual reality and immersive cinema require high image quality, low rendering latency, and consistent depth cues. 4D light field displays support focus accommodation, but are more costly to render than 2D images, resulting in higher latency. The human visual system can resolve higher spatial frequencies in the fovea than in the periphery. This property has been harnessed by recent 2D foveated rendering methods to reduce computation cost while maintaining perceptual quality. Inspired by this, we present foveated 4D light fields by investigating their effects on 3D depth perception. Based on our psychophysical experiments and theoretical analysis on visual and display bandwidths, we formulate a content-adaptive importance model in the 4D ray space. We verify our method by building a prototype light field display that can render only 16% -- 30% rays without compromising perceptual quality.