Michael A. Pusateri

Kalman filter based video background estimation

Transferring responsibility for object tracking in a video scene to computer vision rather than human operators has the appeal that the computer will remain vigilant under all circumstances while operator attention can wane. However, when operating at their peak performance, human operators often outperform computer vision because of their ability to adapt to changes in the scene. While many tracking algorithms are available, background subtraction, where a background image is subtracted from the current frame to isolate the foreground objects in a scene, remains a well proven and popular technique. Under some circumstances, a background image can be obtained manually when no foreground objects are present. In the case of persistent surveillance outdoors, the background has a time evolution due to diurnal changes, weather, and seasonal changes. Such changes render a fixed background scene inadequate. We present a method for estimating the background of a scene utilizing a Kalman filter approach. Our method applies a one-dimensional Kalman filter to each pixel of the camera array to track the pixel intensity. We designed the algorithm to track the background intensity of a scene assuming that the camera view is relatively stationary and that the time evolution of the background occurs much slower than the time evolution of relevant foreground events. This allows the background subtraction algorithm to adapt automatically to changes in the scene. The algorithm is a two step process of mean intensity update and standard deviation update. These updates are derived from standard Kalman filter equations. Our algorithm also allows objects to transition between the background and foreground as appropriate by modeling the input standard deviation. For example, a car entering a parking lot surveillance camera field of view would initially be included in the foreground. However, once parked, it will eventually transition to the background. We present results validating our algorithms ability to estimate backgrounds in a variety of scenes. We demonstrate the application of our method to track objects using simple frame detection with no temporal coherency.

HTS sensors for NQR spectroscopy

Using a high temperature superconductor (HTS) sensor, we have detected a /sup 14/N nuclear quadrupole resonance (NQR) signal from a room temperature sodium nitrite (NaNO/sub 2/) sample. This demonstrates the feasibility of using such sensors for the NQR detection of contraband, e.g. explosives. The sensor is composed of two high Q-value, self-resonant HTS devices strongly coupled to each other that detect the induced magnetization of the target compound. We briefly describe our techniques for making, adjusting the resonant frequency and spoiling the Q-value of the HTS sensors.

Comparison of 2D median filter hardware implementations for real-time stereo video

The two-dimensional spatial median filter is a core algorithm for impulse noise removal in digital image processing and computer vision. While the literature presents several analyses of median filters optimized for a standard 3 times 3 pixel neighborhood configuration, a 5 times 5 neighborhood, useful for imagery exhibiting noise not conforming to the classic ldquosalt and pepperrdquo formation, has received little analysis. Research efforts on hardware implementations of median filters have been devoted primarily toward implementations with low latency and high throughput. We are developing a system that includes stereo visible near infrared sensors; both require a 5 times 5 median filter to handle intensifier noise. Since the system is a battery powered unit, optimal power usage is a critical requirement in addition to low latency and high throughput. However, optimal power usage for median filtering has received little attention in the literature. In this paper, we focus on investigating four selected hardware implementations of a 5 times 5 median filter and compare them on the basis of power efficiency. We also analyze the latency, maximum clock rates, and resource utilization for these implementations. The designs include implementations of merge sort and radix sort-based elimination algorithms, common in software implementation of median filters, and a systolic sorting array and a Batcher sorting network, common hardware sorting techniques. All designs were created in the Altera Quartus-II environment for Stratix-II field programmable gate arrays, and were designed to be fully pipelined, accepting input sets and generating median filter output values every pixel clock pulse. Of the four considered designs, the Batcher network is a clear winner in power efficiency. Also, the Batcher network exceeds the functional and performance requirements for resource usage, latency, and clock rate.

Hyper-spectral content aware resizing

Image resizing is performed for many reasons in image processing. Often, it is done to reduce or enlarge an image for display. It is also done to reduce the bandwidth needed to transmit an image. Most image resizing algorithms work based on principles of spatial or spatial frequency interpolation. One drawback to these algorithms is that they are not image content aware and can fail to preserve relevant features in an image, especially during size reduction. Recently, a content aware image resizing algorithm, called seam carving, was developed. In this paper we discuss an extension of the seam carving algorithm to hyper-spectral imagery. For a hyper-spectral image with an MxN field of view and with P spectral layers, our algorithm identifies a one pixel wide path through the image field of view containing a minimum of information and then removes it. This process is repeated until the image size is reduced to the desired dimension. Information content is assessed using normalized spatial power metrics. Several such metrics have been tested with varying results. The resulting carved hyper-spectral image has the minimum reduction in information for the resizing based upon energy metrics used to quantify information. We will present the results of seam carving applied to imagery sets of: three spectra RGB imagery from a standard still camera, two spectra imagery generated synthetically, and three spectra imagery captured with VNIR, SWIR, and LWIR cameras.

Towards real-time hardware gamma correction for dynamic contrast enhancement

Making the transition between digital video imagery acquired by a focal plane array and imagery useful to a human operator is not a simple process. The focal plane array “sees” the world in a fundamentally different way than the human eye. Gamma correction has been historically used to help bridge the gap. The gamma correction process is a non-linear mapping of intensity from input to output where the parameter gamma can be adjusted to improve the imagerys visual appeal. In analog video systems, gamma correction is performed with analog circuitry and is adjusted manually. With a digital video stream, gamma correction can be provided using mathematical operations in a digital circuit. In addition to manual control, gamma correction can also be automatically adjusted to compensate for changes in the scene.

Laplacian based image fusion

A fundamental goal in multispectral image fusion is to combine relevant information from multiple spectral ranges while displaying a constant amount of data as a single channel. Because we expect synergy between the views afforded by different parts of the spectrum, producing output imagery with increased information beyond any of the individual imagery sounds simple. While fusion algorithms achieve synergy under specific scenarios, it is often the case that they produce imagery with less information than any single band of imagery. Losses can arise from any number of problems including poor imagery in one band degrading the fusion result, loss of details from intrinsic smoothing, artifacts or discontinuities from discrete mixing, and distracting colors from unnatural color mapping. We have been developing and testing fusion algorithms with the goal of achieving synergy under a wider range of scenarios. This technique has been very successful in the worlds of image blending, mosaics, and image compositing for visible band imagery. The algorithm presented in this paper is based on direct pixel-wise fusion that merges the directional discrete laplacian content of individual imagery bands rather than the intensities directly. The laplacian captures the local difference in the four-connected neighborhood. The laplacian of each image is then mixed based on the premise that image edges contain the most pertinent information from each input image. This information is then reformed into an image by solving the two-dimensional Poisson equation. The preliminary results are promising and consistent. When fusing multiple continuous visible channels, the resulting image is similar to grayscale imaging over all of the visible channels. When fusing discontinuous and/or non-visible channels, the resulting image is subtly mixed and intuitive to understand.

Multicamera-multispectral video library - An algorithm development tool

Harsh Environment Applied Technology (HEAT) has developed a ground-based forward-looking multispectral data collection system mounted on a rugged all terrain vehicle (ATV) to allow recording imagery while moving over rough terrain. The image data collected from multiple bands of the electro-optical/infrared (EO/IR) spectrum is used to aid image fusion algorithm development for applications such as night vision goggles. The existing system consists of VNIR, SWIR, and LWIR cameras mounted on a ruggedized Pan/Tilt, a rack-mount PC with frame grabbers to capture digital images, and a 4 TB RAID for real-time image storage. The system can also record meteorological data and GPS information synchronized with the imagery. HEAT has developed a methodology for algorithm development using imagery and other important parameters about the scene of interest. The imagery collected by the data collection system during field exercises is stored in a database of imagery; the imagery can then be replayed into a model running in MATLAB on a desktop PC in the lab. The synchronized raw imagery and meteorological data would be provided as inputs to the model. The model is used to develop image fusion algorithms to display the best possible fused image to human eyes. Also, target identification algorithms are developed and optimized for best probability of detection with lowest false alarm rate. The optimized algorithms for both displaying to the human eyes and to computer aided target tracking can then be ported to a rugged field programmable gate array (FPGA)-based system to deploy in the real world environment. Sample raw imagery input from the cameras into the data collection system will be shown. Examples of fused imagery created by the fusion algorithms will also be shown.

A system and method for auto-correction of first order lens distortion

In multispectral imaging systems, correction for lens distortion is required to allow pixel by pixel fusion techniques to be applied. While correction of optical aberration can be extended to higher order terms, for many systems, a first order correction is sufficient to achieve desired results. In producing a multispectral imaging system in production quantities, the process of producing the corrections needs to be largely automated as each lens will require its own corrections. We discuss an auto-correction and bench sighting method application to a dual band imaging system. In principle, we wish to image a dual band target and completely determine the lens distortion parameters for the given optics. We begin with a scale-preserving, radial, first-order lens distortion model; this model allows the horizontal field of view to be determined independently of the distortion. It has the benefits of simple parameterization and the ability to correct mild to moderate distortion that may be expected of production optics. The correction process starts with imaging a dual band target. A feature extraction algorithm is applied to the imagery from both bands to generate a large number of correlated feature points. Using the feature points, we derive an over-determined system of equations; the solution to this system yields the distortion parameters for the lens. Using these parameters, an interpolation map can be generated unique to the lenses involved. The interpolation map is used in real-time to correct the distortion while preserving the horizontal field of view constraint on the system.

Biologically-inspired visible and infrared camera technology development

Visible band and Infrared (IR) band camera and vision system development has been inspired by the human and animal vision systems. This paper will discuss the development of the Electro-Optical/Infrared (EO/IR) spectrum cameras from the front end optics, the detector or photon to electron convertor, preprocessing such as non-uniformity correction, automatic gain control, foveal vision processing done by the human eye, the gimbal system (human or animal eye ball and head motion), and the analog and digital paths of the data (optic nerve in humans). The computer vision algorithms (human or animal brain vision processing) will not be discussed in this paper. The Integrated Design Services in the College of Engineering at Penn State University has been developing EO/IR camera and sensor based computer vision systems for several years and combined with more than twenty years of developing imaging sensor stabilized platforms will use this imaging system development expertise to describe how the human and animal vision systems inspired the design and development of the computer based vision system. This paper will illustrate a block diagram of both the human eye and a typical EO/IR camera while comparing the two imaging systems.

A comparison of bicubic and biquintic interpolators suitable for real-time hardware implementation

Digital multispectral night vision goggles incorporate both imagers and displays that often have different resolutions. While both thermal imager and micro-display technologies continue to produce larger arrays, thermal imagers still lag well behind displays and can require interpolation by a factor of 2.5 in both horizontal and vertical directions. In goggle applications, resizing the imagery streams to the size of the display must occur in real-time with minimal latency. In addition to low latency, a resizing algorithm must produce acceptable imagery, necessitating an understanding of the resized image fidelity and spatial smoothness. While both spatial and spatial frequency domain resizing techniques are available, most spatial frequency techniques require a complete frame for operation introducing unacceptable latency. Spatial domain techniques can be implemented on a neighborhood basis allowing latencies equivalent to several row clock pulses to be achieved. We have already implemented bilinear re-sampling in hardware and, while bilinear re-sampling supports moderate up-sizes with reasonable image quality, its deficiencies are apparent at interpolation ratios of two and greater. We are developing hardware implementations of both bicubic and biquintic resizing algorithms. We present the results of comparison between hardware ready versions of the bicubic and biquintic algorithms with the existing bilinear. We also discuss the hardware requirements for bicubic and biquintic compared to the existing bilinear resizing.