Robert T. Held

Using blur to affect perceived distance and size

We present a probabilistic model of how viewers may use defocus blur in conjunction with other pictorial cues to estimate the absolute distances to objects in a scene. Our model explains how the pattern of blur in an image together with relative depth cues indicates the apparent scale of the images contents. From the model, we develop a semiautomated algorithm that applies blur to a sharply rendered image and thereby changes the apparent distance and scale of the scenes contents. To examine the correspondence between the model/algorithm and actual viewer experience, we conducted an experiment with human viewers and compared their estimates of absolute distance to the models predictions. We did this for images with geometrically correct blur due to defocus and for images with commonly used approximations to the correct blur. The agreement between the experimental data and model predictions was excellent. The model predicts that some approximations should work well and that others should not. Human viewers responded to the various types of blur in much the way the model predicts. The model and algorithm allow one to manipulate blur precisely and to achieve the desired perceived scale efficiently.

3D puppetry: a kinect-based interface for 3D animation

We present a system for producing 3D animations using physical objects (i.e., puppets) as input. Puppeteers can load 3D models of familiar rigid objects, including toys, into our system and use them as puppets for an animation. During a performance, the puppeteer physically manipulates these puppets in front of a Kinect depth sensor. Our system uses a combination of image-feature matching and 3D shape matching to identify and track the physical puppets. It then renders the corresponding 3D models into a virtual set. Our system operates in real time so that the puppeteer can immediately see the resulting animation and make adjustments on the fly. It also provides 6D virtual camera \\rev{and lighting} controls, which the puppeteer can adjust before, during, or after a performance. Finally our system supports layered animations to help puppeteers produce animations in which several characters move at the same time. We demonstrate the accessibility of our system with a variety of animations created by puppeteers with no prior animation experience.

Blur and Disparity Are Complementary Cues to Depth

Estimating depth from binocular disparity is extremely precise, and the cue does not depend on statistical regularities in the environment. Thus, disparity is commonly regarded as the best visual cue for determining 3D layout. But depth from disparity is only precise near where one is looking; it is quite imprecise elsewhere. Away from fixation, vision resorts to using other depth cues-e.g., linear perspective, familiar size, aerial perspective. But those cues depend on statistical regularities in the environment and are therefore not always reliable. Depth from defocus blur relies on fewer assumptions and has the same geometric constraints as disparity but different physiological constraints. Blur could in principle fill in the parts of visual space where disparity is imprecise. We tested this possibility with a depth-discrimination experiment. Disparity was more precise near fixation and blur was indeed more precise away from fixation. When both cues were available, observers relied on the more informative one. Blur appears to play an important, previously unrecognized role in depth perception. Our findings lead to a new hypothesis about the evolution of slit-shaped pupils and have implications for the design and implementation of stereo 3D displays.

Misperceptions in stereoscopic displays: a vision science perspective

3d shape and scene layout are often misperceived when viewing stereoscopic displays. For example, viewing from the wrong distance alters an objects perceived size and shape. It is crucial to understand the causes of such misperceptions so one can determine the best approaches for minimizing them. The standard model of misperception is geometric. The retinal images are calculated by projecting from the stereo images to the viewers eyes. Rays are back-projected from corresponding retinal-image points into space and the ray intersections are determined. The intersections yield the coordinates of the predicted percept. We develop the mathematics of this model. In many cases its predictions are close to what viewers perceive. There are three important cases, however, in which the model fails: 1) when the viewers head is rotated about a vertical axis relative to the stereo display (yaw rotation); 2) when the head is rotated about a forward axis (roll rotation); 3) when there is a mismatch between the camera convergence and the way in which the stereo images are displayed. In these cases, most rays from corresponding retinal-image points do not intersect, so the standard model cannot provide an estimate for the 3d percept. Nonetheless, viewers in these situations have coherent 3d percepts, so the visual system must use another method to estimate 3d structure. We show that the non-intersecting rays generate vertical disparities in the retinal images that do not arise otherwise. Findings in vision science show that such disparities are crucial signals in the visual systems interpretation of stereo images. We show that a model that incorporates vertical disparities predicts the percepts associated with improper viewing of stereoscopic displays. Improving the model of misperceptions will aid the design and presentation of 3d displays.

A Guide to Stereoscopic 3D Displays in Medicine

Stereoscopic displays can potentially improve many aspects of medicine. However, weighing the advantages and disadvantages of such displays remains difficult, and more insight is needed to evaluate whether stereoscopic displays are worth adopting. In this article, we begin with a review of monocular and binocular depth cues. We then apply this knowledge to examine how stereoscopic displays can potentially benefit diagnostic imaging, medical training, and surgery. It is apparent that the binocular depth information afforded by stereo displays 1) aid the detection of diagnostically relevant shapes, orientations, and positions of anatomical features, especially when monocular cues are absent or unreliable; 2) help novice surgeons orient themselves in the surgical landscape and perform complicated tasks; and 3) improve the three-dimensional anatomical understanding of students with low visual-spatial skills. The drawbacks of stereo displays are also discussed, including extra eyewear, potential three-dimensional misperceptions, and the hurdle of overcoming familiarity with existing techniques. Finally, we list suggested guidelines for the optimal use of stereo displays. We provide a concise guide for medical practitioners who want to assess the potential benefits of stereo displays before adopting them.

Perceptual Issues in Stereoscopic Signal Processing

Perceiving three-dimensional video imagery appropriately in a display requires matching parameters throughout the imaging pathway, such as inter-aperture distance at the stereoscopic camera side with parallax shifting at the display side. In addition, many tradeoffs and compromises are often made at different points in the imaging pathway, leading to common perceptual distortions. Some of these may be simple two-dimensional image distortions such as display surface noise, while others are three-dimensional distortions, such as global geometric scene distortions and localized depth errors around edges. There is an increasing use of various forms of signal processing to modify the images, either for compensation of distortions due to system limitations, display constraints, formatting and compression for efficient transmission, or making depth range adjustments dependent on the display viewing conditions. Perceptual issues are critical to the design of the entire imaging pathway and this paper will highlight some of those due to stereoscopic signal processing.

Visual Discomfort with Stereo 3D Displays when the Head is Not Upright

Properly constructed stereoscopic images are aligned vertically on the display screen, so on-screen binocular disparities are strictly horizontal. If the viewers inter-ocular axis is also horizontal, he/she makes horizontal vergence eye movements to fuse the stereoscopic image. However, if the viewers head is rolled to the side, the onscreen disparities now have horizontal and vertical components at the eyes. Thus, the viewer must make horizontal and vertical vergence movements to binocularly fuse the two images. Vertical vergence movements occur naturally, but they are usually quite small. Much larger movements are required when viewing stereoscopic images with the head rotated to the side. We asked whether the vertical vergence eye movements required to fuse stereoscopic images when the head is rolled cause visual discomfort. We also asked whether the ability to see stereoscopic depth is compromised with head roll. To answer these questions, we conducted behavioral experiments in which we simulated head roll by rotating the stereo display clockwise or counter-clockwise while the viewers head remained upright relative to gravity. While viewing the stimulus, subjects performed a psychophysical task. Visual discomfort increased significantly with the amount of stimulus roll and with the magnitude of on-screen horizontal disparity. The ability to perceive stereoscopic depth also declined with increasing roll and on-screen disparity. The magnitude of both effects was proportional to the magnitude of the induced vertical disparity. We conclude that head roll is a significant cause of viewer discomfort and that it also adversely affects the perception of depth from stereoscopic displays.