Doron Feldman

Mosaicing new views: the Crossed-Slits projection

We introduce anew kind of mosaicing, where the position of the sampling strip varies as a function of the input camera location. The new images that are generated this way correspond to a new projection model defined by two slits, termed here the Crossed-Slits (X-Slits) projection. In this projection model, every 3D point is projected by a ray defined as the line that passes through that point and intersects the two slits. The intersection of the projection rays with the imaging surface defines the image. X-Slits mosaicing provides two benefits. First, the generated mosaics are closer to perspective images than traditional pushbroom mosaics. Second, by simple manipulations of the strip sampling function, we can change the location of one of the virtual slits, providing a virtual walkthrough of a X-Slits camera; all this can be done without recovering any 3D geometry and without calibration. A number of examples where we translate the virtual camera and change its orientation are given; the examples demonstrate realistic changes in parallax, reflections, and occlusions.

Motion Segmentation and Depth Ordering Using an Occlusion Detector

We present a novel method for motion segmentation and depth ordering from a video sequence in general motion. We first compute motion segmentation based on differential properties of the spatio-temporal domain and scale-space integration. Given a motion boundary, we describe two algorithms to determine depth ordering from two- and three-frame sequences. A remarkable characteristic of our method is its ability compute depth ordering from only two frames. The segmentation and depth ordering algorithms are shown to give good results on six real sequences taken in general motion. We use synthetic data to show robustness to high levels of noise and illumination changes; we also include cases where no intensity edge exists at the location of the motion boundary or when no parametric motion model can describe the data. Finally, we describe psychophysical experiments showing that people, like our algorithm, can compute depth ordering from only two frames even when the boundary between the layers is not visible in a single frame.

New View Generation with a Bi-centric Camera

We propose a novel method for new view generation from a rectified sequence of images. Our new images correspond to a new camera model, which we call a bi-centric camera; in this model the centers of horizontal and vertical projections lie in different locations on the cameras optical axis. This model reduces to the regular pinhole camera when the two projection centers coincide, and the pushbroom camera when one projection center lies at infinity. We first analyze the properties of this camera model. We then show how to generate new bi-centric views from vertical cuts in the epipolar volume of a rectified sequence. Every vertical cut generates a new bi-centric view, where the specific parameters of the cut determine the location of the projection centers. We discuss and demonstrate applications, including the generation of images where the virtual camera lies behind occluding surfaces (e.g., behind the back wall of a room), and in unreachable positions (e.g., in front of a glass window). Our final application is the generation of movies taken by a simulated forward moving camera, using as input a movie taken by a sideways moving camera.

Generating Mosaics with Minimum Distortions

Manifold mosaicing is a fast and robust way to summarize video sequences captured by a moving camera. It is also useful for rendering compelling 3D visualizations from a video without estimating the 3D structure of the scene. However, since the result mosaics are not perspective images, their geometry is inherently distorted. These mosaics are commonly referred to as multi-perspective images, or multi-perspective mosaics. In this paper we address the following question: Given a video captured by a moving camera, what is the best multi-perspective mosaic that can be generated from it? What is the mosaic with the best combination of large field-of-view and minimal geometric distortions? We define the necessary conditions for a good mosaic and a quantitive criterion for the geometric distortions, and derive analytically the optimal mosaic under this criterion. Results on video sequences confirm that indeed the optimal mosaic has significantly better quality than those generated by other techniques.

Motion segmentation using an occlusion detector

We present a novel method for the detection of motion boundaries in a video sequence based on differential properties of the spatio-temporal domain. Regarding the video sequence as a 3D spatio-temporal function, we consider the second moment matrix of its gradients (averaged over a local window), and show that the eigenvalues of this matrix can be used to detect occlusions and motion discontinuities. Since these cannot always be determined locally (due to false corners and the aperture problem), a scale-space approach is used for extracting the location of motion boundaries. A closed contour is then constructed from the most salient boundary fragments, to provide the final segmentation. The method is shown to give good results on pairs of real images taken in general motion. We use synthetic data to show its robustness to high levels of noise and illumination changes; we also include cases where no intensity edge exists at the location of the motion boundary, or when no parametric motion model can describe the data.

Realtime IBR with omnidirectional crossed-slits projection

The crossed-slits (X-Slits) projection can be used to generate new views of a scene from a sequence of perspective images. Compared with other image-based rendering (IBR) techniques, X-Slits image generation is simple and requires a relatively small number of input images, which makes it suitable for realtime IBR. In this paper we extend this model to omnidirectional cameras and a circular slit. We show how it can be used for realtime image-based rendering of omnidirectional images, and how to optimize it for speed and quality. We analyze the inherent geometric distortions of the circular X-Slits projection, and describe a normalization mechanism to reduce distortions, creating a realistic virtual environment. Essentially the same mechanism is used to augment the X-Slits images with artificial objects, when using standard graphics tools which assume perspective projection.

Video synthesis made simple with the X-slits projection

We propose novel algorithms for image based rendering (IBR) by simulating a new family of scene-to-image projections, the crossed-slits (X-slits) projections. New X-slits images can be easily generated from a sequence of images captured by a translating pinhole camera, and those images look compelling and realistic. We show how this can be used for rendering movies of virtual walkthroughs by a very fast and simple mosaicing algorithm, providing a strong 3D sensation of parallax and occlusions, as well as specularity and transparency.