Peter J. Burt

A multiresolution spline with application to image mosaics

We define a multiresolution spline technique for combining two or more images into a larger image mosaic. In this procedure, the images to be splined are first decomposed into a set of band-pass filtered component images. Next, the component images in each spatial frequency hand are assembled into a corresponding bandpass mosaic. In this step, component images are joined using a weighted average within a transition zone which is proportional in size to the wave lengths represented in the band. Finally, these band-pass mosaic images are summed to obtain the desired image mosaic. In this way, the spline is matched to the scale of features within the images themselves. When coarse features occur near borders, these are blended gradually over a relatively large distance without blurring or otherwise degrading finer image details in the neighborhood of th e border.

Enhanced image capture through fusion

The authors present an extension to the pyramid approach to image fusion. The modifications address problems that were encountered with past implementations of pyramid-based fusion. In particular, the modifications provide greater shift invariance and immunity to video noise, and provide at least a partial solution to the problem of combining components that have roughly equal salience but opposite contrasts. The fusion algorithm was found to perform well for a range of tasks without requiring adjustment of the algorithm parameters. Results were remarkably insensitive to changes in these parameters, suggesting that the procedure is both robust and generic. A composite imaging technique is outlined that may provide a powerful tool for image capture. By fusing a set of images obtained under restricted, narrowband, imaging conditions, it is often possible to construct an image that has enhanced information content when compared to a single image obtained directly with a broadband sensor.<<ETX>>

Smart sensing within a pyramid vision machine

A machine is designed, based on a pyramid architecture, that supports smart sensing and related highly efficient processing. Key elements of the design are (a) hierarchical data structures for image representation, (b) fine-to-coarse algorithms for the fast generation of image measures, (c) coarse-to-fine search strategies that rapidly locate objects or events within a scene, and (d) high-level control mechanisms that guide data gathering even as visual information is being interpreted. This system, known as the Pyramid Vision Machine, achieves high performance at modest cost. Design considerations and several applications are described. >

A three-frame algorithm for estimating two-component image motion

A fundamental assumption made in formulating optical-flow algorithms, that motion at any point in an image can be represented as a single pattern component undergoing a simple translation, fails for a number of situations that commonly occur in real-world images. An alternative formulation of the local motion assumption in which there may be two distinct patterns undergoing coherent (e.g. affine) motion within a given local analysis region is proposed. An algorithm for the analysis of two-component motion in which tracking and nulling mechanisms applied to three consecutive image frames separate and estimate the individual components is given. Precise results are obtained, even for components that differ only slightly in velocity as well as for a faint component in the presence of a dominant, masking component. The algorithm provides precise motion estimates for a set of elementary two-motion configurations and is robust in the presence of noise. >

Real-time scene stabilization and mosaic construction

We describe a real-time system designed to construct a stable view of a scene through aligning images of an incoming video stream and dynamically constructing an image mosaic. This system uses a video processing unit developed by the David Sarnoff Research Center called the Vision Front End (VFE-100) for the pyramid-based image processing tasks required to implement this process. This paper includes a description of the multiresolution coarse-to-fine image registration strategy, the techniques used for mosaic construction, the implementation of this process on the VFE-100 system, and experimental results showing image mosaics constructed with the VFE-100.<<ETX>>

Aerial video surveillance and exploitation

There is growing interest in performing aerial surveillance using video cameras. Compared to traditional framing cameras, video cameras provide the capability to observe ongoing activity within a scene and to automatically control the camera to track the activity. However, the high data rates and relatively small field of view of video cameras present new technical challenges that must be overcome before such cameras can be widely used. In this paper, we present a framework and details of the key components for real-time, automatic exploitation of aerial video for surveillance applications. The framework involves separating an aerial video into the natural components corresponding to the scene. Three major components of the scene are the static background geometry, moving objects, and appearance of the static and dynamic components of the scene. In order to delineate videos into these scene components, we have developed real time, image-processing techniques for 2-D/3-D frame-to-frame alignment, change detection, camera control, and tracking of independently moving objects in cluttered scenes. The geo-location of video and tracked objects is estimated by registration of the video to controlled reference imagery, elevation maps, and site models. Finally static, dynamic and reprojected mosaics may be constructed for compression, enhanced visualization, and mapping applications.

Computing two motions from three frames

A fundamental assumption made in formulating optical-flow algorithms is that motion at any point in any image can be represented as a single pattern undergoing a simple translation: even complex motion will appear as a uniform displacement when viewed through a sufficiently small window. This assumption fails in a number of common situations. The authors propose an alternative formulation in which there may be two distinct patterns undergoing coherent motion within a given local analysis region. They then present an algorithm for the analysis of two-component motion. They also demonstrate that the algorithm provides precise motion estimates for a set of elementary two-motion configurations, and show that it is robust in the presence of noise.<<ETX>>

Attention mechanisms for vision in a dynamic world

Three elements of attention in computer vision are described. These are related to familiar aspects of eye movement control in human vision: foveation, to examine selected regions of the visual world at high resolution; tracking, to stabilize the images of moving objects within the eye; and high-level interpretation, to anticipate where salient information will occur in a scene. Attention mechanisms allow the system to function in a data-rich world by directing system sensing and analysis resources to only that information critical to the systems current visual task. It is concluded that attention therefore includes mechanisms for directing the allocation of processing and mechanisms for rapidly interpreting the information as it is gathered.<<ETX>>

Mechanisms for isolating component patterns in the sequential analysis of multiple motion

Pyramid techniques are commonly used to provide computational efficiency in the analysis of image motion. But these techniques can play an even more important role in the analysis of multiple motion, where, for example, a transparent pattern moves in front of a differently moving background pattern. The pyramid framework then separates motion components based on their spatial and temporal frequency characteristics so that each can be estimated independently of the others. This property is key to recently proposed selective stabilization algorithms for the sequential analysis of multiple motion and for the detection of moving objects from a moving platform. The authors determine the conditions for component selection. Results can provide important guidance in practical applications of motion analysis.<<ETX>>

A VLSI pyramid chip for multiresolution image analysis

Advanced techniques in image processing and computer vision increasingly require that image data be represented at multiple resolutions and at multiple sample rates. Application areas for such pyramid image representations include image compression, image enhancement, motion analysis, and object recognition.We have developed a VLSI chip, called PYR, to perform the standard filter and resampling operations required in pyramid and inverse pyramid transforms for these applications. The PYR chip processes image samples sequentially, in raster scan format, so is suited for pipeline architectures. The user can choose from a set of standard filters, through software control, to construct Gaussian, Laplacian, Subband, and related pyramid structures.A unique feature of the design is that it includes timing signals that are passed with the image data. These signals coordinate successive processing steps in a pipeline system as image sizes and sample rates change. The chip also includes circuits for edge extension and image addition, and it can be run in “spread tap” mode to provide twice the standard sample density.The PYR chip is implemented in standard cell technology. At a clock rate of 15 MHz, a single chip can simultaneously construct a Gaussian and a Laplacian pyramid from a 512 by 480 image in 22.7 msec (44 frame/second).