Qiang He

GPU-CPU implementation for super-resolution mosaicking of Unmanned Aircraft System (UAS) surveillance video

Unmanned Aircraft Systems (UAS) have been used in many military and civilian applications, particularly surveillance. One of the best ways to use the capacity of a UAS imaging system is by constructing a mosaic of the recorded video. In this paper, we present a novel algorithm to calculate a super-resolution mosaic for UAS, which is both fast and robust. In this algorithm, the features points between frames are found using SIFT (Scale-Invariant Feature Transform), and then RANSAC (Random Sample Consensus) is used to estimate the homography between two consecutive frames. Next, a low-resolution (LR) mosaic is computed. LR images are extracted from the LR mosaic, and then they are subtracted from the input frames to form LR error images. These images are used to compute an error mosaic. The regularization technique uses Huber prior information and is added to the error mosaic to form the superresolution (SR) mosaic. The proposed algorithm was implemented using both a GPU (Graphics Processing Unit) and a CPU (Central Processing Unit). The first part of the algorithm, which is the construction of the LR mosaic, is performed by the GPU, and the rest is performed by the CPU. As a result, there is a significant speed-up of the algorithm. The proposed algorithm has been tested in both the infrared (IR) and visible spectra, using real and synthetic data. The results for all these cases show a great improvement in resolution, with a PSNR of 41.10 dB for synthetic data, and greater visual detail for the real UAV surveillance data.

Detection of reflecting surfaces by a statistical model

Remote sensing is widely used assess the destruction from natural disasters and to plan relief and recovery operations. How to automatically extract useful features and segment interesting objects from digital images, including remote sensing imagery, becomes a critical task for image understanding. Unfortunately, current research on automated feature extraction is ignorant of contextual information. As a result, the fidelity of populating attributes corresponding to interesting features and objects cannot be satisfied. In this paper, we present an exploration on meaningful object extraction integrating reflecting surfaces. Detection of specular reflecting surfaces can be useful in target identification and then can be applied to environmental monitoring, disaster prediction and analysis, military, and counter-terrorism. Our method is based on a statistical model to capture the statistical properties of specular reflecting surfaces. And then the reflecting surfaces are detected through cluster analysis.

Shadow removal from textured images

Shadows and shadings are typical natural phenomena, which can often be found in images and videos acquired under strong directional lighting, such as those taken outdoors on a sunny day. Unfortunately, shadows can cause many difficulties in image processing and vision-related tasks, such like image segmentation and object recognition. Therefore, shadow removal is needed for improving the performance of these image understanding tasks. We present a new shadow removal algorithm for real textured color images. The algorithm is based on the statistical property of textures in images. The experimental results on real-world data are shown to demonstrate this algorithm.

Performance Evaluation of Optimization Methods for Super- resolution Mosaicking on UAS Surveillance Videos

Unmanned Aircraft Systems (UAS) have been widely applied into military reconnaissance and surveillance by exploiting the information collected from the digital imaging payload. However, the data analysis of UAS videos is frequently limited by motion blur; the frame-to-frame movement induced by aircraft roll, wind gusts, and less than ideal atmospheric conditions; and the noise inherent within the image sensors. Therefore, the super-resolution mosaicking on low-resolution UAS surveillance video frames, becomes an important task for UAS video processing and is a pre-step for further effective image understanding. Here we develop a novel super-resolution framework which does not require the construction of sparse matrices. This method applied image operators in spatial domain and adopted an iterated back-projection method to conduct super-resolution mosaics from UAS surveillance video frames. The Steepest Descent method, Conjugate Gradient method and Levenberg Marquardt algorithm are used to numerically solve the nonlinear optimization problem in the modeling of super-resolution mosaic. A quantity comparison in computation time and visual performance of the super-resolution using the three numerical methods is performed. The Levenberg Marquardt algorithm provides a numerical solution to the least squares curve fitting, which avoids the time-consuming computation of the inverse of the pseudo Hessian matrix in regular singular value decomposition (SVD). The Levenberg Marquardt method, interpolating between the Gauss-Newton algorithm (GNA) and the method of gradient descent, is efficient, robust, and easy to implement. The results obtained in our simulations shows a great improvement of the resolution of the low resolution mosaic of up to 47.54 dB for synthetic images, and a considerable visual improvement in sharpness and visual details for real UAS surveillance frames. The convergence is generally reached in no more than ten iterations.

Efficient super-resolution image reconstruction applied to surveillance video captured by small unmanned aircraft systems

The concept surrounding super-resolution image reconstruction is to recover a highly-resolved image from a series of low-resolution images via between-frame subpixel image registration. In this paper, we propose a novel and efficient super-resolution algorithm, and then apply it to the reconstruction of real video data captured by a small Unmanned Aircraft System (UAS). Small UAS aircraft generally have a wingspan of less than four meters, so that these vehicles and their payloads can be buffeted by even light winds, resulting in potentially unstable video. This algorithm is based on a coarse-to-fine strategy, in which a coarsely super-resolved image sequence is first built from the original video data by image registration and bi-cubic interpolation between a fixed reference frame and every additional frame. It is well known that the median filter is robust to outliers. If we calculate pixel-wise medians in the coarsely super-resolved image sequence, we can restore a refined super-resolved image. The primary advantage is that this is a noniterative algorithm, unlike traditional approaches based on highly-computational iterative algorithms. Experimental results show that our coarse-to-fine super-resolution algorithm is not only robust, but also very efficient. In comparison with five well-known super-resolution algorithms, namely the robust super-resolution algorithm, bi-cubic interpolation, projection onto convex sets (POCS), the Papoulis-Gerchberg algorithm, and the iterated back projection algorithm, our proposed algorithm gives both strong efficiency and robustness, as well as good visual performance. This is particularly useful for the application of super-resolution to UAS surveillance video, where real-time processing is highly desired.

Super-resolution mosaics from airborne video using robust gradient regularization

Unmanned Aircraft Systems (UAS) have been used in many military and civil applications, particularly surveillance. One of the best ways to use the capacity of a UAS imaging system is by constructing a mosaic or panorama of the recorded video. This paper presents a novel algorithm for the construction of super-resolution mosaicking. The algorithm is based on the Conjugate Gradient (CG) method. Geman -McClure prior is used together with four different cliques to deal with the ill-conditioned inverse problem and to preserve edges. We present the results with synthetic and real UAS surveillance data, resulting in a great improvement of the visual resolution. For the case of synthetic images, we obtained a PSNR of 47.0 dB, as well as a significant increase in the details visible for the case of real UAS frames in only ten iterations.

Performance Evaluations for Super-Resolution Mosaicing on UAS Surveillance Videos

Abstract Unmanned Aircraft Systems (UAS) have been widely applied for reconnaissance and surveillance by exploiting information collected from the digital imaging payload. The super-resolution (SR) mosaicing of low-resolution (LR) UAS surveillance video frames has become a critical requirement for UAS video processing and is important for further effective image understanding. In this paper we develop a novel super-resolution framework, which does not require the construction of sparse matrices. The proposed method implements image operations in the spatial domain and applies an iterated back-projection to construct super-resolution mosaics from the overlapping UAS surveillance video frames. The Steepest Descent method, the Conjugate Gradient method and the Levenberg-Marquardt algorithm are used to numerically solve the nonlinear optimization problem for estimating a super-resolution mosaic. A quantitative performance comparison in terms of computation time and visual quality of the super-resolution mosaics through the three numerical techniques is presented.

Super-Resolution Reconstruction by Image Fusion and Application to Surveillance Videos Captured by Small Unmanned Aircraft Systems

In practice, surveillance video captured by a small Unmanned Aircraft System (UAS) digital imaging payload is almost always blurred and degraded because of limits of the imaging equipment and less than ideal atmospheric conditions. Small UAS vehicles typically have wingspans of less than four meters and payload carrying capacities of less than 50 kilograms, which results in a high vibration environment due to winds buffeting the aircraft and thus poorly stabilized video that is not necessarily pointed at a target of interest. Superresolution image reconstruction can reconstruct a highly-resolved image of a scene from either a single image or a time series of low-resolution images based on image registration and fusion between different video frames [1, 6, 8, 18, 20, 27]. By fusing several subpixelregistered, low-resolution video frames, we can reconstruct a high-resolution panoramic image and thus improve imaging system performance. There are four primary applications for super-resolution image reconstruction: 1. Automatic Target Recognition: The interesting target is hard to identify and recognize under degraded videos and images. For a series of low-resolution images captured by a small UAS vehicle flown over an area under surveillance, we need to perform super-resolution to enhance image quality and automatically recognize targets of interest. 2. Remote Sensing: Remote sensing observes the Earth and helps monitor vegetation health, bodies of water, and climate change based on image data gathered by wireless equipments over time. We can gather additional information on a given area by increasing the spatial image resolution. 3. Environmental Monitoring: Related to remote sensing, environmental monitoring helps determine if an event is unusual or extreme, and to assist in the development of an appropriate experimental design for monitoring a region over time. With the 22

Object detection and tracking under planar constraints

Automatic object detection and tracking has been widely applied in the video surveillance systems for homeland security and data fusion in the remote sensing and airborne imagery. The typical applications include human motion analysis, vehicle detection, and architectural building detection. Here we conduct object detection and tracking under planar constraints for interesting objects. Planar surface abounds in man-made environment. It provides much useful information for image understanding and then can be adopted to improve the performance of object detection and tracking. The experiments on real data show that object detection and tracking could be successfully implemented by incorporating planar information of interesting objects.

Fundamental matrix and planar homographies in stereo vision

We describe the epipolar constraint that specifies the geometry of stereo vision. We consider the 3D structure reconstruction from multiple views through the new perspective of basing the reconstruction from directly estimated planar homographies instead of using techniques that are based on matched point pairs. Planar homography parameters can more accurately extract scene planar surfaces and directly solve for the 3D structure and camera motion parameters. The new method has the advantage that it integrates larger amount of information because the homography parameters are estimated directly from the intensities and not from an abstracted descriptor of the neighborhood. Because it does not rely on a transformation of an entire image region, the method is efficient.