Angel G. Jordan

Intermediate view synthesis considering occluded and ambiguously referenced image regions

Abstract In this paper, we present an algorithm for synthesizing intermediate views from a single stereo-pair. The key contributions of this algorithm are the incorporation of scene assumptions and a disparity estimation confidence measure that lead to the accurate synthesis of occluded and ambiguously referenced regions. The synthesized views have been displayed on a multi-view binocular imaging system, with subjectively effective motion parallax and diminished eye strain.

Image understanding algorithms for remote visual inspection of aircraft surfaces

Visual inspection is, by far, the most widely used method in aircraft surface inspection. We are currently developing a prototype remote visual inspection system, designed to facilitate testing the hypothesized feasibility and advantages of remote visual inspection of aircraft surfaces. In this paper, we describe several experiments with image understanding algorithms that were developed to aid remote visual inspection, in enhancing and recognizing surface cracks and corrosion from the live imagery of an aircraft surface. Also described in this paper are the supporting mobile robot platform that delivers the live imagery, and the inspection console through which the inspector accesses the imagery for remote inspection. We discuss preliminary results of the image understanding algorithms and speculate on their future use in aircraft surface inspection.

A multiresolution framework for stereoscopic image sequence compression

Stereoscopic sequence compression typically involves the exploitation of the spatial redundancy between the left and right streams to achieve higher compressions than are possible with the independent compression of the two streams. In this paper the psychophysical property of the human visual system, that only one high resolution image in a stereo image pair is sufficient for satisfactory depth perception, has been used to further reduce the bit rates. Thus, one of the streams is independently coded along the lines of the MPEG standards, while the other stream is estimated at a lower resolution from this stream. A multiresolution framework has been adopted to facilitate such an estimation of motion and disparity vectors at different resolutions. Experimental results on typical sequences indicate that the additional stream can be compressed to about one-fifth of a highly compressed independently coded stream, without any significant loss in depth perception or perceived image quality.<<ETX>>

Algorithm for automated eye-strain reduction in real stereoscopic images and sequences

Eye strain is often experienced when viewing a stereoscopic image pair on a flat display device (e.g., a computer monitor). Violations of two relationships that contribute to this eye strain are: (1) the accommodation/convergence breakdown and (2) the conflict between interposition and disparity depth cues. We describe a simple algorithm that reduces eye strain through horizontal image translation and corresponding image cropping, based on a statistical description of the estimated disparity within a stereoscopi image pair. The desired amount of translation is based on the given stereoscopic image pair, and, therefore, requires no user intervention. In this paper, we first develop a statistical model of the estimated disparity that incorporates the possibility of erroneous estimates. An estimate of the actual disparity range is obtained by thresholding the disparity histogram to avoid the contribution of false disparity values. Based on the estimated disparity range, the image pair is translated to force all points to lie on, or behind, the screen surface. This algorithm has been applied to diverse real stereoscopic images and sequences. Stereoscopic image pairs, which were often characterized as producing eye strain and confusion, produced comfortable stereoscopy after the automated translation.

Compression and interpolation of 3D stereoscopic and multiview video

Compression and interpolation each require, given part of an image, or part of a collection or stream of images, being able to predict other parts. Compression is achieved by transmitting part of the imagery along with instructions for predicting the rest of it; of course, the instructions are usually much shorter than the unsent data. Interpolation is just a matter of predicting part of the way between two extreme images; however, whereas in compression the original image is known at the encoder, and thus the residual can be calculated, compressed, and transmitted, in interpolation the actual intermediate image is not known, so it is not possible to improve the final image quality by adding back the residual image. Practical 3D-video compression methods typically use a system with four modules: (1) coding one of the streams (the main stream) using a conventional method (e.g., MPEG), (2) calculating the disparity map(s) between corresponding points in the main stream and the auxiliary stream(s), (3) coding the disparity maps, and (4) coding the residuals. It is natural and usually advantageous to integrate motion compensation with the disparity calculation and coding. The efficient coding and transmission of the residuals is usually the only practical way to handle occlusions, and the ultimate performance of beginning-to-end systems is usually dominated by the cost of this coding. In this paper we summarize the background principles, explain the innovative features of our implementation steps, and provide quantitative measures of component and system performance.

COMPRESSION OF STEREO VIDEO STREAMS

Direct application of monocular compression schemes to stereo video streams is suboptimal. This is because these techniques do not take advantage of the high correlation between the left and right stereo image pairs. This paper introduces new still and motion stereo compression schemes that are far efficient than monocular compression schemes in compressing stereo image streams. 1. “worldline” borrowed from the field of special relativity, means the path of an object in 4-dimensional space - time. 2. Includes zero time offset 3. Matching for a block of pixels of the right image achieved similarly with appropriate changes. 4. Redundancy defined according to some similarity criterion. 5. Zero offset included.

SEGMENTATION BASED CODING OF STEREOSCOPIC IMAGE SEQUENCES

A binocular disparity based segmentation scheme to compactly represent one image of a stereoscopic image pair given the other image was proposed earlier by us. That scheme adapted the excess bitcount, needed to code the additional image, to the binocular disparity detail present in the image pair. This paper addresses the issue of extending such a segmentation in the temporal dimension to achieve efficient stereoscopic sequence compression. The easiest conceivable temporal extension would be to code one of the sequences using an MPEG-type scheme while the frames of the other stream are coded based on the segmentation. However such independent compression of one of the streams fails to take advantage of the segmentation or the additional disparity information available. To achieve better compression by exploiting this additional information, we propose the following scheme. Each frame in one of the streams is segmented based on disparity. An MPEG-type frame structure is used for motion compensated prediction of the segments in this segmented stream. The corresponding segments in the other stream are encoded by reversing the disparity-map obtained during the segmentation. Areas without correspondence in this stream, arising from binocular occlusions and disparity estimation errors, are filled in using a disparity-map based predictive error concealment method. Over a test set of several different stereoscopic image sequences, high perceived stereoscopic image qualities were achieved at an excess bandwidth that is roughly 40% above that of a highly compressed monoscopic sequence. Stereo perception can be achieved at significantly smaller excess bandwidths, albeit with a perceivable loss in the image quality.

Image enhancement and understanding for remote visual inspection of aircraft surface

We describe a library of image enhancement and understanding algorithms developed to enhance and recognize surface defects from remote live imagery of an aircraft surface. Also described are the supporting mobile robot platform that generates the remote stereoscopic imagery and the inspection console containing a graphical user interface, through which the inspector accesses the live imagery for remote inspection. We will discuss initial results of the remote imaging process and the image processing library, and speculate on their future application in aircraft inspection.

Multiresolutional region-based segmentation scheme for stereoscopic image compression

Stereoscopic image sequence transmission over existing monocular digital transmission channels, without seriously affecting the quality of one of the image streams, requires a very low bit-rate coding of the additional stream. Fixed block-size based disparity estimation schemes cannot achieve such low bit-rates without causing severe edge artifacts. Also, textureless regions lead to spurious matches which hampers the efficient coding of block disparities. In this paper, we propose a novel disparity-based segmentation approach, to achieve an efficient partition of the image into regions of more or less fixed disparity. The partitions are edge based, in order to minimize the edge artifacts after disparity compensation. The scheme leads to disparity discontinuity preserving, yet smoother and more accurate disparity fields than fixed block-size based schemes. The smoothness and the reduced number of block disparities lead to efficient coding of one image of a stereo pair given the other. The segmentation is achieved by performing a quadtree decomposition, with the disparity compensated error as the splitting criterion. The multiresolutional recursive decomposition offers a computationally efficient and non-iterative means of improving the disparity estimates while preserving the disparity discontinuities. The segmented regions can be tracked temporally to achieve very high compression ratios on a stereoscopic image stream.

Adaptive reference frame selection for generalized video signal coding

In this paper, we present a new algorithm that adaptively selects the best possible reference frame for the predictive coding of generalized, or multi-view, video signals, based on estimated prediction similarity with the desired frame. We define similarity between two frames as the absence of occlusion, and we estimate this quantity from the variance of composite displacement vector maps. The composite maps are obtained without requiring the computationally intensive process of motion estimation for each candidate reference frame. We provide prediction and compression performance results for generalized video signals using both this scheme and schemes where the reference frames were heuristically pre- selected. When the predicted frames were used in a modified MPEG encoder simulation, the signal compressed using the adaptively selected reference frames required, on average, more than 10% fewer bits to encode than the non-adaptive techniques; for individual frames, the reduction in bits was sometimes more than 80%. These gains were obtained with an acceptable computational increase and an inconsequential bit-count overhead.