David C. Zhang

FPGA acceleration for feature based processing applications

Feature based vision applications rely on highly efficient extraction and analysis of features from images to reach satisfactory levels of performance and latency. In this paper, we describe the implementation of an algorithm that combines distributed feature detector (D-HCD) with a rotational invariant feature descriptor (R-HOG). Based on an algorithmic comparison with other feature detectors and descriptors, we show that our algorithms have the lowest error rate for 3D aerial scene matching. We present implementation on a low-cost Zynq FPGA that achieves 15x speedup, 5x reduction in latency over a quad core CPU. Our results show the considerable promise of our proposed implementation for fast and efficient robotic and aerial drone / UAV applications.

Unsupervised underwater fish detection fusing flow and objectiveness

Scientists today face an onerous task to manually annotate vast amount of underwater video data for fish stock assessment. In this paper, we propose a robust and unsupervised deep learning algorithm to automatically detect fish and thereby easing the burden of manual annotation. The algorithm automates fish sampling in the training stage by fusion of optical flow segments and objective proposals. We auto-generate large amounts of fish samples from the detection of flow motion and based on the flow-objectiveness overlap probability we annotate the true-false samples. We also adapt a biased training weight towards negative samples to reduce noise. In detection, in addition to fused regions, we used a Modified Non-Maximum Suppression (MNMS) algorithm to reduce false classifications on part of the fishes from the aggressive NMS approach. We exhaustively tested our algorithms using NOAA provided, luminance-only underwater fish videos. Our tests have shown that Average Precision (AP) of detection improved by about 10% compared to non-fusion approach and about another 10% by using MNMS.

Efficient Object Detection Using Embedded Binarized Neural Networks

Memory performance is a key bottleneck for deep learning systems. Binarization of both activations and weights is one promising approach that can best scale to realize the highest energy efficient system using the lowest possible precision. In this paper, we utilize and analyze the binarized neural network in doing human detection on infrared images. Our results show comparable algorithmic performance of binarized versus 32bit floating-point networks, with the added benefit of greatly simplified computation and reduced memory overhead. In addition, we present a system architecture designed specifically for computation using binary representation that achieves at least 4× speedup and the energy is improved by three orders of magnitude over GPU.

A single algorithm combining exposure and focus fusion

In scenes of significantly varying lighting conditions, under and over exposed regions can suffer from a loss of information. Similarly, the presence of spatial depth within a scene can cause some image regions to be out of focus. Several methods of addressing these issues exist, including tone mapping for true high dynamic range representation and exposure fusion for combining varied-exposure low dynamic range images, as solutions to the former, and image fusion and segmentation etc. to address the latter. This paper proposes an overhauled method of exposure fusion that solves the exposure and focus problems simultaneously, achieving a well-exposed, all-in-focus result. Smart, scene-based data acquisition techniques for reducing both required input data and computational resources are discussed. A platform for a realtime system implementation is also presented.

Motion adaptive signal integration-high dynamic range (MASI-HDR) video processing for dynamic platforms

Two of the biggest challenges in designing U×V vision systems are properly representing high dynamic range scene content using low dynamic range components and reducing camera motion blur. SRI’s MASI-HDR (Motion Adaptive Signal Integration-High Dynamic Range) is a novel technique for generating blur-reduced video using multiple captures for each displayed frame while increasing the effective camera dynamic range by four bits or more. MASI-HDR processing thus provides high performance video from rapidly moving platforms in real-world conditions in low latency real time, enabling even the most demanding applications on air, ground and water.

Low-light NV-CMOS image sensors for day/night imaging

Traditionally, daylight and night vision imaging systems have required image intensifiers plus daytime cameras. But SRI’s new NV-CMOS™ image sensor technology is designed to capture images over the full range of illumination from bright sunlight to starlight. SRI’s NV-CMOS image sensors provide the low light sensitivity approaching that of an analog image intensifier tube with the cost, power, ruggedness, flexibility and convenience of a digital CMOS imager chip. NV-CMOS provides multi-megapixels at video frame rates with low noise (<2 h+), high sensitivity across the visible and near infrared (NIR) bands (peak QE <85%), high resolution (MTF at Nyquist < 50% @ 650 nm), and extended dynamic range (<75 dB). The latest test data from the NV-CMOS imager technology will be presented. Unlike conventional image intensifiers, the NV-CMOS image sensor outputs a digital signal, ideal for recording or sharing video as well as fusion with thermal imagery. The result is a substantial reduction in size and weight, ideal for SWaP-constrained missions such as UAVs and mobile operations. SRI’s motion adaptive noise reduction processing further increases the sensitivity and reduces image smear. Enhancement of moving targets in imagery captured under extreme low light conditions imposes difficult challenges. SRI has demonstrated that image registration provides a robust solution for enhancing global scene contrast under very low SNR conditions.

Adaptive smoothing in real-time image stabilization

When using the conventional fixed smoothing factor to display the stabilized video, we have the issue of large undefined black border regions (BBR) when camera is fast panning and zooming. To minimize the size of BBR and also provide smooth visualization to the display, this paper discusses several novel methods that have demonstrated on a real-time platform. These methods include an IIR filter, a single Kalman filter and an interactive multi-model filter. The fundamentals of these methods are to adapt the smoothing factor to the motion change from time to time to ensure small BBR and least jitters. To further remove the residual BBR, the pixels inside the BBR are composited from the previous frames. To do that, we first store the previous images and their corresponding frame-to-frame (F2F) motions in a FIFO queue, and then start filling the black pixels from valid pixels in the nearest neighbor frame based on the F2F motion. If a matching is found, then the search is stopped and continues to the next pixel. If the search is exhausted, the pixel remains black. These algorithms have been implemented and tested in a TI DM6437 processor.

Extended motion adaptive signal integration technique for real-time image enhancement

Fast moving cameras often generate distorted and blurred images characterized by reduced sharpness (due to motion blur) and insufficient dynamic range. Reducing sensor integration times to minimize blur are often used but the light intensity and image Signal-to-Noise-Ratio (SNR) would be reduced as well. We propose a Motion Adaptive Signal Integration (MASI) algorithm that operates the sensor at a high frame rate, with real time alignment of individual image frames to form an enhanced quality video output. This technique enables signal integration in the digital domain, allowing both high SNR performance and low motion blur induced by the camera motion. We also show, in an Extended MASI (EMASI) algorithm, that high dynamic range can be achieved by combining high frame rate images of varying exposures. EMASI broadens the dynamic range of the sensor and extends the sensitivity to work in low light and noisy conditions. In a moving platform, it also reduces static noise in the sensor. This technology can be used in aerial surveillance, satellite imaging, border securities, wearable sensing, video conferencing and camera phone imaging applications.

Mask pyramid methodology for enhanced localization in image fusion and enhancement

Image fusion is a process that combines regions of images from different sources into a single fused image based on a salience selection rule for each region. In this paper, we proposed an algorithmic approach using a mask pyramid to better localize the selection process. A mask pyramid operates in different scales of the image to improve the fused image quality beyond a global selection rule. The proposed approach offers a generic methodology for applications in image enhancement, high dynamic range compression, depth of field extension, and image blending. The mask pyramid can also be encoded for intelligent analysis of source imagery. Several examples of this mask pyramid method are provided to demonstrate its performance in a variety of applications. A new embedded system architecture that builds upon the Acadia® II Vision Processor is proposed.

Designing Vision Systems that See Better

This chapter introduces computational sensing and imaging—a branch of computer vision that deals with embedded processing for capturing higher quality imagery, to the embedded vision developer. In this research domain, issues such as exposure, motion blur, and dynamic range are addressed by fundamentally changing how light is sensed, captured, and made available for downstream semantic processing. Some important applications enabled are in digital photography and situational awareness. We present several design examples on these camera platforms where modifications of the image-capture process result in significant improvements in image quality, allowing downstream vision analysis to perform better. Motivated by example applications, we then describe the basic architecture of an embedded vision system that dynamically tunes and adapts to the task at hand.