Ming Z. Zhang

A multisensor image fusion and enhancement system for assisting drivers in poor lighting conditions

A system of multisensor image fusion and enhancement for visibility improvement is proposed in this paper for helping drivers driving at night or under bad weather conditions. Video stream captured by a CCD camera is enhanced, then aligned and fused with another stream captured by a thermal camera to improve the visibility of roads in extremely low lighting conditions. A nonlinear image enhancement technique capable of dynamic range compression and contrast enhancement is developed to enhance the visible images prior to fusion. The thermal image and the enhanced visible image are then aligned based on prior information obtained on image registration process. Pixel-level multiresolution based image fusion method is applied to merge source images. After image fusion, a color restoration is performed on fused images with the chromatic information of visible images. The entire image processing and analysis system is being installed in an FPGA environment. Preliminary results obtained in various experiments conducted with the proposed system are encouraging

A Visibility Improvement System for Low Vision Drivers by Nonlinear Enhancement of Fused Visible and Infrared Video

Development of a visibility improvement system for helping drivers with poor vision during night and bad weather conditions is proposed in this paper. Video streams captured by a CCD camera and an infrared camera are registered and fused to obtain image details in extremely low lighting conditions. A novel image enhancement technique employing a nonlinear expansion function is developed for enhancing the fused images. The slope characteristics of the expansion function at an individual pixel location is based on the statistical properties of its neighborhood pixels. The expansion function is also capable of reducing the intensity of the overly bright image regions due to the presence of head lights from vehicles in opposite direction. The enhanced video stream is then subjected to a color restoration process to introduce natural color using the color information gathered from the CCD camera image. The entire image processing and analysis system is being installed in an FPGA environment for performing the processing in real-time. The processed video stream is displayed on a Head-Up Display located directly in front of the driver to assist him/her to drive safely in poor visibility environments. Preliminary results obtained in various experiments conducted with the proposed system are encouraging.

An efficient VLSI architecture for 2-D convolution with quadrant symmetric kernels

A high performance digital architecture for computing 2D convolution utilizing the quadrant symmetry of the kernels is proposed in this paper. Pixels in the four quadrants of the kernel region with respect to an image pixel are considered simultaneously for computing the partial products of the convolution sum. A novel data handling strategy to identify the pixels to be fed to different processing elements helps reducing the data storage requirements in the circuitry. The new design results in 75% reduction in multipliers and 50% reduction in adders when compared with the conventional systolic architecture. The proposed architecture design is capable of performing convolution operations with 14/spl times/14 kernel at a rate of 57 1024/spl times/1024 frames per second in a Xilinx s Virtex 2v2000ff896-4 FPGA.

Design of an efficient multiplier-less architecture for multi-dimensional convolution

Design of a hardware efficient multiplier-less architecture for the computation of multi-dimensional convolution is presented in this paper. The new architecture performs computations in the logarithmic domain by utilizing novel multiplier-less log2 and inverse-log2 modules. An effective data handling strategy is developed in conjunction with the logarithmic modules to eliminate the necessity of multipliers in the architecture. The proposed approach reduces hardware resources significantly compared to other approaches while it still maintains a high degree of accuracy. The architecture is developed as a combined systolic-pipelined design that produces an output in every clock cycle after the initial latency of the system. The architecture is capable of operating with a high speed clock frequency of 99 MHz based on Xilinx’s Virtex II 2v2000ff896-4 FPGA and the throughput of the system is observed as 99 MOPS (million outputs per second).

A tunable high-performance architecture for enhancement of stream video captured under non-uniform lighting conditions

A novel architecture for performing hue-saturation-value (HSV) domain enhancement of digital color images captured under non-uniform lighting conditions is proposed in this paper for video streaming applications. The approach promotes log-domain computation to eliminate all multiplications, divisions and exponentiations utilizing the compact high-speed logarithmic estimation modules. An optimized quadrant symmetric architecture is incorporated into the design of homomorphic filter for the enhancement of intensity value. Efficient modules are also presented for conversion between RGB and HSV color spaces with tunable H and S components in HSV for more flexible color rendering. The design is able to bring out details hidden in shadow regions of the image and preserve the bright parts with adjustable vividness and color shift for improvement of visual quality while maintaining its consistency. It is capable of producing 187.86 million outputs per second (MOPs) on Xilinxs Virtex II XC2V2000-4ff896 field programmable gate array (FPGA) at a clock frequency of 187.86MHz. It can process over 179.1 (1024x1024) frames per second, which is very suitable for high definition videos, and consumes approximately 70.7% and 76.8% less hardware resource with 127% and 280% performance boost when compared to the designs with machine learning algorithm in [M.Z. Zhang, M.J. Seow, V.K. Asari, A high performance architecture for color image enhancement using a machine learning approach, International Journal of Computational Intelligence Research - Special Issue on Advances in Neural Networks 2(1) (2006) 40-47], and with separated dynamic and contrast enhancements in [H.T. Ngo, M.Z. Zhang, L. Tao, V.K. Asari, Design of a high performance architecture for real-time enhancement of video stream captured in extremely low lighting environment, International Journal of Embedded Systems: Special Issue on Media and Stream Processing, in press], respectively. This approach also provide 83.4 times performance gain with more consistent fidelity in the results compared to some DSP based implementations (256x256 frame size) [G.D. Hines, Z. Rahman, D.J. Jobson, G.A. Woodell, DSP implementation of the retinex image enhancement algorithm, visual information processing XIII, in: Proceedings of the SPIE, vol. 5438, 2004, pp. 13-24; G.D. Hines, Z. Rahman, D.J. Jobson, G.A. Woodell, Single-scale retinex using digital signal processors, in: Proceedings of the Global Signal Processing Conference, September 2004, pp. 1-6] under the reflectance-illuminance category of image enhancement models.

Design of a Digital Architecture for Real-Time Video, Enhancement Based on Illuminance-Reflectance Model

A design of a high performance digital architecture for a nonlinear image enhancement technique is presented in this paper. The image enhancement is based on illuminance-reflectance model which improves the visual quality of digital images and video captured under insufficient or non-uniform lighting conditions [1]. Systolic, pipelined and parallel design techniques are utilized effectively in the proposed FPGA-based architectural design to achieve real-time performance. Estimation and folding techniques are used in the hardware algorithmic design to achieve faster, simpler and more efficient architecture. The video enhancement system is implemented using Xilinxs multimedia development board that contains a VirtexII-X2000 FPGA and it is capable of processing approximately 66 Mega-pixels (Mpixels) per second.

A Hardware Architecture for Color Image Enhancement Using a Machine Learning Approach with Adaptive Parameterization

A novel architecture for performing color image enhancement using a machine learning algorithm called Ratio Rule is proposed in this paper. The threshold width of the activation function is automatically determined from the image characteristics. The approach promotes log-domain computation to eliminate all multiplications and divisions, utilizing approximation techniques for efficient estimation of the log2 and inverse-log2. The design incorporates the dynamic thresholds of the activation functions and update rate. The improved quadrant symmetric architecture is also presented to provide very high throughput rate for homomorphic filters which is part of the pixel intensity enhancement across RGB components in the system. The pipelined design of the filter features the flexibility in reloading a wide range of kernels for different frequency responses. A new approach for the design of the uniform filters is also presented to reduce the processing element arrays (PEAs) from W PEAs to 2 PEAs for W times W window. This new concept is applied to assist in training the synaptic weights of the neural network for color balancing to restore the intensity enhanced image to its natural color existed in the original image. The concept of uniform filter is further extended to design max/min filters. It is observed that the performance of the system with parallel and pipelined architectures is able to achieve 139.3 million outputs per second (MOPS), or equivalently 54.7 billion operations per second on Xilinxs Virtex II XC2V2000-4ff896 FPGA at a clock frequency of 139.3 MHz.

A High Performance Architecture for Implementation of 2-D Convolution with Quadrant Symmetric Kernels

Abstract The design of a high performance digital architecture for computing 2-D convolution, utilizing the quadrant symmetry of the kernels, is proposed in this paper. Pixels in the four quadrants of the kernel region, with respect to an image pixel, are considered simultaneously for computing the partial products of the convolution sum. A novel data handling strategy, to identify pixels to be fed to different processing elements, helps reduce the data storage requirements significantly in the circuitry. The systolic architecture employs parallel and pipelined processing and is able to produce one output every clock cycle. The new design resulted in, approximately, a 75% reduction in number of multipliers and a 50% reduction in the number of adders, when compared to the conventional systolic architecture. The proposed architecture design is capable of performing convolution operations for 57 1,024 × 1,024 frames, or 59.77 million outputs per second, in a Xilinxs Virtex 2v2000ff896-4 FPGA at maximum clock frequency of 59.77 MHz. The error analysis performed in two image processing applications, namely noise filtering and edge detection, shows that the hardware implementation with the proposed design provides results similar to that of the software implementation.

Design of a systolic-pipelined architecture for real-time enhancement of color video stream based on an illuminance-reflectance model

A high performance digital architecture for the implementation of a nonlinear image enhancement technique is proposed in this paper. The image enhancement is based on an illuminance-reflectance model which improves the visual quality of digital images and video captured under insufficient or non-uniform lighting conditions. The algorithm shows robust performance with appropriate dynamic range compression, good contrast, accurate and consistent color rendition. The algorithm contains a large number of complex computations and thus it requires specialized hardware implementation for real-time applications. Systolic, pipelined and parallel design techniques are utilized effectively in the proposed FPGA-based architectural design to achieve real-time performance. Approximation techniques are used in the hardware algorithmic design to achieve high throughput. The video enhancement system is implemented using Xilinxs multimedia development board that contains a VirtexII-X2000 FPGA and it is capable of processing approximately 63 Mega-pixels (Mpixels) per second.

A design methodology for performance-resource optimization of a generalized 2D convolution architecture with quadrant symmetric kernels

We present a design technique to meet application driven constraints for performance-resource optimization of a generalized 2D convolution with quadrant symmetric kernels. A fully pipelined multiplierless digital architecture for computing modularized 2D convolution utilizing the quadrant symmetry of the kernels is proposed in this paper. Pixels in the four quadrants of the kernel region with respect to an image pixel are considered simultaneously with distributed queues for computing the partial results of the convolution sum in time-sliced fashion. The new architecture performs computations in log-domain by utilizing low complexity high performance log2 and inverse-log2 estimation modules. An effective data handling strategy is developed to minimize routing of data path in conjunction with the logarithmic modules to eliminate the necessity of the multipliers in the architecture. The proposed architecture is capable of performing convolution operations for 45.5 (1024×1024) frames or 47.73 million outputs per second in minimum resource configuration with 8×8 kernels in a Xilinxs Virtex XC2V2000-4ff896 field programmable gate array (FPGA) at maximum clock frequency of 190.92 MHz. Analysis shows that the performance and resource utilization between the fully parallel and fully resource constrained architectures are proportional to f and 1/f, respectively where f is the application driven reusability of the main computing components. In addition to resource reduction from optimization scheme, evaluation in Xilinxs core generator also showed that the proposed design results in 60% reduction in hardware when compared to the design using pipelined multipliers.