Z Zhenyu Ye

Feasibility analysis of ultra high frame rate visual servoing on FPGA and SIMD processor

Visual servoing has been proven to obtain better performance than mechanical encoders for position acquisition. However, the often computationally intensive vision algorithms and the ever growing demands for higher frame rate make its realization very challenging. This work performs a case study on a typical industrial application, organic light emitting diode (OLED) screen printing, and demonstrates the feasibility of achieving ultra high frame rate visual servoing applications on both field programmable gate array (FPGA) and single instruction multiple data (SIMD) processors. We optimize the existing vision processing algorithm and propose a scalable FPGA implementation, which processes a frame within 102 µs. Though a dedicated FPGA implementation is extremely efficient, lack of flexibility and considerable amount of implementation time are two of its clear drawbacks. As an alternative, we propose a reconfigurable wide SIMD processor, which balances among efficiency, flexibility, and implementation effort. For input frames of 120 × 45 resolution, our SIMD can process a frame within 232 µs, sufficient to provide a throughput of 1000fps with less than 1ms latency for the whole vision servoing system. Compared to the reference realization on MicroBlaze, the proposed SIMD processor achieves a 21× performance improvement.

From Xetal-II to Xetal-Pro: On the Road Toward an Ultralow-Energy and High-Throughput SIMD Processor

Looking forward to the next generation of mobile streaming computing, the demanded energy efficiency of end-user terminals will become ever stringent. The Xetal-Pro processor, which is the latest member of the Xetal low-power single-instruction multiple data (SIMD) processor family from Philips, is presented in this paper. The predecessor of Xetal-Pro, known as Xetal-II, already ranks as one of the most computational-efficient [in terms of giga operations per second (GOPS)/Watt] processors available today, however, it cannot yet achieve the demanded energy efficiency (less than 1 pJ per operation). Unlike Xetal-II, Xetal-Pro supports ultrawide supply voltage (Vdd) scaling from the nominal supply to the subthreshold region. Although aggressive Vdd scaling causes severe throughput degradation, this can be partly compensated for by the massive parallelism in the Xetal family. Xetal-II includes a large on-chip frame memory (FM), which cannot be scaled well to an ultralow Vdd hence creating a big obstacle to increase energy efficiency. Therefore, we investigate both different FM realizations and memory organization alternatives. A hybrid memory system (HMS), which reduces the non-local memory traffic and enables further Vdd scaling, is proposed. For design space exploration of the right number of the scratchpad memory (SM) entries, the corresponding data locality analysis is provided, too. Moreover, some unique circuit implementation issues of Xetal-Pro such as the customized level-shifter are also discussed. Compared to Xetal-II operating at the nominal voltage, Xetal-Pro provides up to two times energy efficiency improvement even without Vdd scaling (essentially a consequence of data localization in the SM) when delivering the same amount of ultrahigh throughput. With Vdd scaling into the sub/near threshold region, Xetal-Pro could gain more than ten times energy reduction while still delivering a high throughput of 0.69 GOPS (counting multiply and add operations only). The new insight of Xetal-Pro sheds light on the direction of future ultralow-energy SIMD processors.

Direct motion planning for vision-based control

This paper presents direct methods for vision-based control for the application of industrial inkjet printing. In this, visual control is designed with a direct coupling between camera measurements and joint motion. Traditional visual servoing commonly has a slow visual update rate and needs an additional local joint controller to guarantee stability. By only using the product as reference and sampling with a high update rate, direct visual measurements are sufficient for controlled positioning. The proposed method is simpler and more reliable than standard motor encoders, despite the tight real-time constraints. This direct visual control method is experimentally verified with a 2D planar motion stage for micrometer positioning. To achieve accurate and fast motion, a balance is found between frame rate and image size. With a frame rate of 1600 fps and an image size of 160 × 100 pixels we show the effectiveness of the approach.

Demo: An embedded vision system for high frame rate visual servoing

The frame rate of commercial off-the-shelf industrial cameras is breaking the threshold of 1000 frames-per-second, the sample rate required in high performance motion control systems. On the one hand, it enables computer vision as a cost-effective feedback source; On the other hand, it imposes multiple challenges on the vision processing system. The authors have designed and implemented an FPGA-based embedded vision system in support of high frame rate visual servoing applications. The vision system will be demonstrated together with a mechanical system for vision based inkjet printing. This demonstration shows that, with off-the-shelf components, a robust, hard realtime, low delay embedded vision system is feasible for industrial applications. The research aspect of the experiment has been published in previous papers of the authors. This demonstration paper emphasises on the practical issues for the implementation of such system, and the lessons learned from this practice.

PhD forum: A cyber-physical system approach to embedded visual servoing

Visual servoing, which applies computer vision as a feedback source for control, is becoming a cost effective solution for high performance mechatronic systems. However, the potential of visual servoing systems is limited by the current design methodology, which explores the cyber domain and the physical domain separately. We propose to use a cyber-physical system approach to overcome such limitation.