Assaf Zomet

Seamless Image Stitching in the Gradient Domain

Image stitching is used to combine several individual images having some overlap into a composite image. The quality of image stitching is measured by the similarity of the stitched image to each of the input images, and by the visibility of the seam between the stitched images.

Robust super-resolution

A robust approach for super-resolution is, presented, which is especially valuable in the presence of outliers. Such outliers may be due to motion errors, inaccurate blur models, noise, moving objects, motion blur etc. This robustness is needed since super-resolution methods are very sensitive to such errors. A robust median estimator is combined in an iterative process to achieve a super resolution algorithm. This process can increase resolution even in regions with outliers, where other super resolution methods actually degrade the image.

Mosaicing on adaptive manifolds

Image mosaicing is commonly used to increase the visual field of view by pasting together many images or video frames. Existing mosaicing methods are based on projecting all images onto a predetermined single manifold: A plane is commonly used for a camera translating sideways, a cylinder is used for a panning camera, and a sphere is used for a camera which is both panning and tilting. While different mosaicing methods should therefore be used for different types of camera motion, more general types of camera motion, such as forward motion, are practically impossible for traditional mosaicing. A new methodology to allow image mosaicing in more general cases of camera motion is presented. Mosaicing is performed by projecting thin strips from the images onto manifolds which are adapted to the camera motion. While the limitations of existing mosaicing techniques are a result of using predetermined manifolds, the use of more general manifolds overcomes these limitations.

Seamless image stitching by minimizing false edges

Various applications such as mosaicing and object insertion require stitching of image parts. The stitching quality is measured visually by the similarity of the stitched image to each of the input images, and by the visibility of the seam between the stitched images. In order to define and get the best possible stitching, we introduce several formal cost functions for the evaluation of the stitching quality. In these cost functions the similarity to the input images and the visibility of the seam are defined in the gradient domain, minimizing the disturbing edges along the seam. A good image stitching will optimize these cost functions, overcoming both photometric inconsistencies and geometric misalignments between the stitched images. We study the cost functions and compare their performance for different scenarios both theoretically and practically. Our approach is demonstrated in various applications including generation of panoramic images, object blending and removal of compression artifacts. Comparisons with existing methods show the benefits of optimizing the measures in the gradient domain.

Mosaicing new views: the Crossed-Slits projection

We introduce anew kind of mosaicing, where the position of the sampling strip varies as a function of the input camera location. The new images that are generated this way correspond to a new projection model defined by two slits, termed here the Crossed-Slits (X-Slits) projection. In this projection model, every 3D point is projected by a ray defined as the line that passes through that point and intersects the two slits. The intersection of the projection rays with the imaging surface defines the image. X-Slits mosaicing provides two benefits. First, the generated mosaics are closer to perspective images than traditional pushbroom mosaics. Second, by simple manipulations of the strip sampling function, we can change the location of one of the virtual slits, providing a virtual walkthrough of a X-Slits camera; all this can be done without recovering any 3D geometry and without calibration. A number of examples where we translate the virtual camera and change its orientation are given; the examples demonstrate realistic changes in parallax, reflections, and occlusions.

Lensless Imaging with a Controllable Aperture

In this paper we propose a novel, highly flexible camera. The camera consists of an image detector and a special aperture, but no lens. The aperture is a set of parallel light attenuating layers whose transmittances are controllable in space and time. By applying different transmittance patterns to this aperture, it is possible to modulate the incoming light in useful ways and capture images that are impossible to capture with conventional lens-based cameras. For example, the camera can pan and tilt its field of view without the use of any moving parts. It can also capture disjoint regions of interest in the scene without having to capture the regions in between them. In addition, the camera can be used as a computational sensor, where the detector measures the end result of computations performed by the attenuating layers on the scene radiance values. These and other imaging functionalities can be implemented with the same physical camera and the functionalities can be switched from one video frame to the next via software. We have built a prototype camera based on this approach using a bare image detector and a liquid crystal modulator for the aperture. We discuss in detail the merits and limitations of lensless imaging using controllable apertures.

Multi-sensor super-resolution

Image sensing is usually done with multiple sensors, like the RGB sensors in color imaging, the IR and EO sensors in surveillance and satellite imaging, etc. The resolution of each sensor can be increased by considering the images of the other sensors, and using the statistical redundancy among the sensors. Particularly, we use the fact that most discontinuities in the image of one sensor correspond to discontinuities in the other sensors. Two applications are presented: Increasing the resolution of a single color image by using the correlation among the three color channels, and enhancing noisy IR images.

Efficient super-resolution and applications to mosaics

Mosaicing and super resolution are two ways to combine information from multiple frames in video sequences. Mosaicing displays the information of multiple frames in a single panoramic image. Super-resolution uses regions which appear in multiple frames to improve resolution and reduce noise. The aim of this work is constructing a high resolution mosaic from a video sequence in an efficient way. Simple combination of the two methods is problematic, since the alignment used in mosaicing may not be accurate enough for super resolution. Another issue is the efficiency of the super resolution algorithm, which requires heavy computations, especially when applied to large images such as panoramic mosaics. This paper introduces two novelties. First, a framework for super resolution algorithms is presented, which enables the development of very efficient algorithms. Second, a method for applying super resolution to panoramic mosaics is presented. This method preserves the geometry of the original mosaic image, while improving its resolution.

Wide Baseline Matching between Unsynchronized Video Sequences

Abstract3D reconstruction of a dynamic scene from features in two cameras usually requires synchronization and correspondences between the cameras. These may be hard to achieve due to occlusions, different orientation, different scales, etc. In this work we present an algorithm for reconstructing a dynamic scene from sequences acquired by two uncalibrated non-synchronized fixed affine cameras.It is assumed that (possibly) different points are tracked in the two sequences. The only constraint relating the two cameras is that every 3D point tracked in one sequence can be described as a linear combination of some of the 3D points tracked in the other sequence. Such constraint is useful, for example, for articulated objects. We may track some points on an arm in the first sequence, and some other points on the same arm in the second sequence. On the other extreme, this model can be used for generally moving points tracked in both sequences without knowing the correct permutation. In between, this model can cover non-rigid bodies with local rigidity constraints.We present linear algorithms for synchronizing the two sequences and reconstructing the 3D points tracked in both views. Outlier points are automatically detected and discarded. The algorithm can handle both 3D objects and planar objects in a unified framework, therefore avoiding numerical problems existing in other methods.

Super-Resolution from Multiple Images Having Arbitrary Mutual Motion

Normal video sequences contain substantial overlap between successive frames, and each region in the scene appears in multiple frames. The super-resolution process creates high-resolution pictures of regions that are sampled in multiple frames, having a higher spatial resolution than the original video frames.