Paul L. McCarley

Large-format variable spatial acuity superpixel imaging: visible and infrared systems applications

This paper serves as a companion to SPIE paper 4820-36, presented in Seattle in 2002. Advances in the design and application of “Variable Spatial Acuity” focal plane arrays are reported here, with specific examples of large format imagers and applications to which they are being applied. These devices have been developed through the combined requirements of (a) covering a wide total field of view while (b) retaining the highest possible spatial resolution on the objects of interest while at the same time (c) operating at the highest possible frame rate. Many thousands of frames per second are possible with the prototype imager while maintaining high spatial resolution. The prototype device operates as a visible imager, and we are pursuing the transition of this technology into the infrared domain. This paper will concentrate on applications of the technology and will show some imagery collected with the systems developed for their use.

A comparison of foveated acquisition and tracking performance relative to uniform resolution approaches

Foveated imaging has been explored for compression and tele-presence, but gaps exist in the study of foveated imaging applied to acquisition and tracking systems. Results are presented from two sets of experiments comparing simple foveated and uniform resolution targeting (acquisition and tracking) algorithms. The first experiments measure acquisition performance when locating Gabor wavelet targets in noise, with fovea placement driven by a mutual information measure. The foveated approach is shown to have lower detection delay than a notional uniform resolution approach when using video that consumes equivalent bandwidth. The second experiments compare the accuracy of target position estimates from foveated and uniform resolution tracking algorithms. A technique is developed to select foveation parameters that minimize error in Kalman filter state estimates. Foveated tracking is shown to consistently outperform uniform resolution tracking on an abstract multiple target task when using video that consumes equivalent bandwidth. Performance is also compared to uniform resolution processing without bandwidth limitations. In both experiments, superior performance is achieved at a given bandwidth by foveated processing because limited resources are allocated intelligently to maximize operational performance. These findings indicate the potential for operational performance improvements over uniform resolution systems in both acquisition and tracking tasks.

Advanced Image Processing Package for FPGA-Based Re-Programmable Miniature Electronics

Nova Sensors produces miniature electronics for a variety of real-time digital video camera systems, including foveal sensors based on Novas Variable Acuity Superpixel Imager (VASITM) technology. An advanced image-processing package has been designed at Nova Sensors to re-configure the FPGA-based co-processor board for numerous applications including motion detection, optical, background velocimetry and target tracking. Currently, the processing package consists of 14 processing operations that cover a broad range of point- and area-applied algorithms. Flexible FPGA designs of these operations and re-programmability of the processing board allows for easy updates of the VASITM sensors, and for low-cost customization of VASITM sensors taking into account specific customer requirements. This paper describes the image processing algorithms implemented and verified in Xilinx FPGAs and provides the major technical performances with figures illustrating practical applications of the processing package.

Miniature embedded real-time image processor system for smart sensor systems

Programs at Nova Biomimetics have led to the design and development of a set of miniature electronics to be used for the application of a wide variety of point- and area-type mathematical operations to be applied in real time to the digital data produced by a variety of real-time digital video camera systems. Nova is planning to market these electronics in partial satisfaction of Small Business Innovation Research (SBIR) Program dual-use commercialization requirements.

Adaptive image processing techniques for camera-on-a-chip sensors

We report on recently developed algorithms and architectures capable of point source target detection near or on the FPA. The goals of this work are to demonstrate image processing functions near or on the FPA in a manner efficient enough to allow hardwired algorithms for Camera Systems on a Chip (SOC) implementation. These SOCs have the potential to improve the size and power requirements for existing IR sensor systems which require larger board sets and hardware enclosures. We report on the algorithm development for hardwired target detection algorithms using recorded IR Data.

Detecting low-contrast moving point targets

The Tanner Research Wave Process is a moving point target detection algorithm that uses the spatio-temporal correlation of points from a target trajectory to build a large aggregate response, thereby increasing the probability of detection for dim and low-contrast point targets moving amidst dense background and noise. The Wave Process is naturally represented as a 2-D array of linear passive analog components, with each node directly stimulated by its focal plane detector. The Wave Process can be implemented in compact, low-power hardware: analog VLSI for near-focal-plane integration, and dedicated digital for near-term applications, both with a fine-grain parallel architecture that can accommodate fast-frame-rate sensors. The Wave Process generates a real-time Region of Interest to window focal planes, reducing the data rate and sensor processing throughput requirements, thereby also reducing the overall sensor processor power, weight, and size requirements.

A bio-inspired infrared imager with on chip object computation

This paper discusses a Biologically Inspired Shortwave Infrared (SWIR) imager that performs on chip object detection using temporal and spatial processing embedded in the imager’s readout integrated circuit (ROIC). The sensor circuit is designed to detect pixel level intensity changes and correlate the change with nearby intensity changes using multiple thresholding criteria to output object exceedances. The sensor is capable of automatically outputting both normal video and also a reduced data set of binarized exceedances. Therefore this SWIR sensor with onboard temporal spatial sensing should be well suited to both manned and unmanned sensing scenarios which could benefit from automated object detection and reduced data sets.

Bandwidth efficient sensor architectures with feature extraction

We report on processing techniques to effectively control the data bandwidth in larger format focal plane array (FPA) sensors. We have developed an image processing architecture for foveating variable acuity FPAs that give a controlled reduction in the data rate via simple circuits that estimate activity on the FPA image plane. Integrated on-FPA signal processing goals are to perform pre-processing that is usually performed downstream in a dedicated processing module. Techniques for image pre-processing described in this paper allow transmitting ldquoactiverdquo pixel data while skipping unchanging pixels. These techniques for image pre-processing adjacent to the FPA allows significant reductions in the data rate, size, weight and power for small and low cost systems that cannot work with a large image processors.

FPGA-Based Processor for High Frame-Rate Target Detection on Cluttered Backgrounds Using LVASI™ Sensors

In our previous papers, the FPGA-based processing package and the co-processor board have been introduced for numerous commercial and military applications including motion detection, optical flow, background velocimetry, and target tracking. The processing package is being continually upgraded by new point- and area-applied algorithms for a variety of real-time digital video camera systems including foveal sensors based on Novas Variable Acuity Superpixel Imager (VASITM) and Large Format VASITM (LVASITM) technologies. This paper demonstrates the FPGA-based processor for high frame-rate target detection in a cluttered background using variable acuity sensors. For the 1024 x 1024 pixel LVASITM Focal Plane Array (FPA), the proposed target-detection algorithm increases the frame rate from 4 Hz for the full resolution mode up to 450 Hz for the foveal mode while maintaining full field of view and target-detection performances on cluttered backgrounds that are comparable with detection performances at the full resolution mode.

Digital emulation of analog components

Several useful but computationally expensive sensor processing tasks can be mapped to the natural behavior of networks of ideal, passive analog components. For example, spatially smoothing an image can be achieved by convolving it with a Gaussian kernel, or by applying it to a 2-D resistor-capacitor network, and then relying on the diffusive behavior of the network to provide a smoothed image. Numerical computation is replaced with physical computation. But implementing analog networks is challenging due to the limitations of real analog components. They have low precision, vary with temperature, and are non-uniform from unit to unit. Moreover, physics limits the size of analog components. For example, to achieve a particular capacitance, using material of a given dielectric constant, a VLSI capacitor must occupy a certain area on the chip. Twice the capacitance will require twice the area. We describe a set of digital circuits that emulate analog components. These circuits provide analog behavior with arbitrary precision, uniformity, noise immunity, and no temperature dependence. Their size is limited by VLSI linewidths and the circuit approach taken. Networks of these digital circuits behave as do their analog equivalents, making physical computation practical for sensor processing closely coupled to the FPA.