Zhaoyi Wei

FPGA-based Real-time Optical Flow Algorithm Design and Implementation

Optical flow algorithms are difficult to apply to robotic vision applications in practice because of their extremely high computational and frame rate requirements. In most cases, traditional general purpose processors and sequentially executed software cannot compute optical flow in real time. In this paper, a tensor-based optical flow algorithm is developed and implemented using field programmable gate array (FPGA) technology. The resulting algorithm is significantly more accurate than previously published FPGA results and was specifically developed to be implemented using a pipelined hardware structure. The design can process 640 × 480 images at 64 fps, which is fast enough for most real-time robot navigation applications. This design has low resource requirements, making it easier to fit into small embedded systems. Error analysis on a synthetic image sequence is given to show its effectiveness. The algorithm is also tested on a real image sequence to show its robustness and limitations. The resulting limitations are analyzed and an improved scheme is then proposed. It is then shown that the performance of the design could be substantially improved with sufficient hardware resources.

Real-Time Optical Flow Calculations on FPGA and GPU Architectures: A Comparison Study

FPGA devices have often found use as higher-performance alternatives to programmable processors for implementing a variety of computations. Applications successfully implemented on FPGAs have typically contained high levels of parallelism and have often used simple statically-scheduled control and modest arithmetic. Recently introduced computing devices such as coarse grain reconfigurable arrays, multi-core processors, and graphical processing units (GPUs) promise to significantly change the computational landscape for the implementation of high-speed real-time computing tasks. One reason for this is that these architectures take advantage of many of the same application characteristics that fit well on FPGAs. One real-time computing task, optical flow, is difficult to apply in robotic vision applications in practice because of its high computational and data rate requirements, and so is a good candidate for implementation on FPGAs and other custom computing architectures. In this paper, a tensor-based optical flow algorithm is implemented on both an FPGA and a GPU and the two implementations discussed. The two implementations had similar performance, but with the FPGA implementation requiring 12× more development time. Other comparison data for these two technologies is then given for three additional applications taken from a MIMO digital communication system design, providing additional examples of the relative capabilities of these two technologies.

A Fast and Accurate Tensor-based Optical Flow Algorithm Implemented in FPGA

Many computer vision applications require real-time processing of image data. This requirement is especially critical for autonomous vehicles performing obstacle avoidance, path planning, and target tracking tasks. A quickly calculated and relatively rough motion estimate is more useful for autonomous navigation than a more accurate, but slowly calculated estimate. Recent technology advancements in small unmanned air and ground vehicles make many low-cost surveillance and military applications possible. Most of these applications demand a low power, compact, light weight, and high speed computation platform for processing image data in real time. In most cases, the traditional general purpose processor and sequentially executed software approach does not meet these requirements. In this paper, a tensor-based optical flow algorithm is modified and implemented using field programmable gate array (FPGA) for small unmanned vehicle obstacle avoidance and navigation

FPGA-Based Embedded Motion Estimation Sensor

Accurate real-time motion estimation is very critical to many computer vision tasks. However, because of its computational power and processing speed requirements, it is rarely used for real-time applications, especially for micro unmanned vehicles. In our previous work, a FPGA system was built to process optical flow vectors of 64 frames of image per second. Compared to software-based algorithms, this system achieved much higher frame rate but marginal accuracy. In this paper, a more accurate optical flow algorithm is proposed. Temporal smoothing is incorporated in the hardware structure which significantly improves the algorithm accuracy. To accommodate temporal smoothing, the hardware structure is composed of two parts: the derivative (DER) module produces intermediate results and the optical flow computation (OFC) module calculates the final optical flow vectors. Software running on a built-in processor on the FPGA chip is used in the design to direct the data flow and manage hardware components. This new design has been implemented on a compact, low power, high performance hardware platform for micro UV applications. It is able to process 15 frames of image per second and with much improved accuracy. Higher frame rate can be achieved with further optimization and additional memory space.

A Comparison Study on Implementing Optical Flow and Digital Communications on FPGAs and GPUs

FPGA devices have often found use as higher-performance alternatives to programmable processors for implementing computations. Applications successfully implemented on FPGAs typically contain high levels of parallelism and often use simple statically scheduled control and modest arithmetic. Recently introduced computing devices such as coarse-grain reconfigurable arrays, multi-core processors, and graphical processing units promise to significantly change the computational landscape and take advantage of many of the same application characteristics that fit well on FPGAs. One real-time computing task, optical flow, is difficult to apply in robotic vision applications because of its high computational and data rate requirements, and so is a good candidate for implementation on FPGAs and other custom computing architectures. This article reports on a series of experiments mapping a collection of different algorithms onto both an FPGA and a GPU. For two different optical flow algorithms the GPU had better performance, while for a set of digital comm MIMO computations, they had similar performance. In all cases the FPGA implementations required 10x the development time. Finally, a discussion of the two technology’s characteristics is given to show they achieve high performance in different ways.

Real-time accurate optical flow-based motion sensor

An accurate real-time motion sensor implemented in an FPGA is introduced in this paper. This sensor applies an optical flow algorithm based on ridge regression to solve the collinear problem existing in traditional least squares methods. It additionally applies extensive temporal smoothing of the image sequence derivatives to improve the accuracy of its optical flow estimates. Implemented on a customized embedded FPGA platform, it is capable of processing 60 320 × 240 images or 15 640 × 480 images per second. By evaluating its accuracy on synthetic sequences, it is shown here that the proposed design achieves very high accuracy compared to other known hardware-based designs.

A hardware-friendly adaptive tensor based optical flow algorithm

A tensor-based optical flow algorithm is presented in this paper. This algorithm uses a cost function that is an indication of tensor certainty to adaptively adjust weights for tensor computation. By incorporating a good initial value and an efficient search strategy, this algorithm is able to determine optimal weights in a small number of iterations. The weighting mask for the tensor computation is decomposed into rings to simplify a 2D weighting into 1D. The devised algorithm is well-suited for real-time implementation using a pipelined hardware structure and can thus be used to achieve real-time optical flow computation. This paper presents simulation results of the algorithm in software, and the results are compared with our previous work to show its effectiveness. It is shown that the proposed new algorithm automatically achieves equivalent accuracy to that previously achieved via manual tuning of the weights.

Two-frame structure from motion using optical flow probability distributions for unmanned air vehicle obstacle avoidance

See-and-avoid behaviors are an essential part of autonomous navigation for Unmanned Air Vehicles (UAVs). To be fully autonomous, a UAV must be able to navigate complex urban and near-earth environments and detect and avoid imminent collisions. While there have been significant research efforts in robotic navigation and obstacle avoidance during the past few years, this previous work has not focused on applications that use small autonomous UAVs. Specific UAV requirements such as non-invasive sensing, light payload, low image quality, high processing speed, long range detection, and low power consumption, etc., must be met in order to fully use this new technology. This paper presents single camera collision detection and avoidance algorithm. Whereas most algorithms attempt to extract the 3D information from a single optical flow value at each feature point, we propose to calculate a set of likely optical flow values and their associated probabilities—an optical flow probability distribution. Using this probability distribution, a more robust method for calculating object distance is developed. This method is developed for use on a UAV to detect obstacles, but it can be used on any vehicle where obstacle detection is needed.

Hardware-Friendly Vision Algorithms for Embedded Obstacle Detection Applications

Accurate optical flow estimation is a crucial task for many computer vision applications. However, because of its computational power and processing speed requirements, it is rarely used for real-time obstacle detection, especially for small unmanned vehicle and embedded applications. Two hardware-friendly vision algorithms are proposed in this paper to address this challenge. A ridge regression-based optical flow algorithm is developed to cope with the existing collinear problem in traditional least-squares approaches for calculating optical flow. Additionally, taking advantage of hardware parallelism, spatial and temporal smoothing operations are applied to image sequence derivatives to improve accuracy. An efficient motion field analysis algorithm using the optical flow values and based on a simplified motion model is also developed and implemented in hardware. The resulting obstacle detection algorithm is specifically designed for ground vehicles moving on planar surfaces. Results from the software simulations and hardware execution of the two proposed algorithms prove that with adequate hardware, a low power, compact obstacle detection sensor can be realized for small unmanned vehicles and embedded applications.

Motion projection for floating object detection

Floating mines are a significant threat to the safety of ships in theatres of military or terrorist conflict. Automating mine detection is difficult, due to the unpredictable environment and high requirements for robustness and accuracy. In this paper, a floating mine detection algorithm using motion analysis methods is proposed. The algorithm aims to locate suspicious regions in the scene using contrast and motion information, specifically regions that exhibit certain predefined motion patterns. Throughput of the algorithm is improved with a parallel pipelined data flow. Moreover, this data flow enables further computational performance improvements though special hardware such as field programmable gate arrays (FPGA) or Graphics Processing Units (GPUs). Experimental results show that this algorithm is able to detect mine regions in the video with reasonable false positive and minimum false negative rates.