Marijn J. H. Loomans

Low-complexity wavelet-based scalable image & video coding for home-use surveillance

We study scalable image and video coding for the surveillance of rooms and personal environments based on inexpensive cameras and portable devices. The scalability is achieved through a multi-level 2D dyadic wavelet decomposition featuring an accurate low-cost integer wavelet implementation with lifting. As our primary contribution, we present a modification to the SPECK wavelet coefficient encoding algorithm to significantly improve the efficiency of an embedded system implementation. The modification consists of storing the significance of all quadtree nodes in a buffer, where each node comprises several coefficients. This buffer is then used to efficiently construct the code with minimal and direct memory access. Our approach allows efficient parallel implementation on multi-core computer systems and gives a substantial reduction of memory access and thus power consumption. We report experimental results, showing an approximate gain factor of 1,000 in execution time compared to a straightforward SPECK implementation, when combined with code optimization on a common digital signal processor. This translates to 75 full color 4CIF 4:2:0 encoding cycles per second, clearly demonstrating the realtime capabilities of the proposed modification.

Performance vs. complexity in scalable video coding for embedded surveillance applications

In this paper, we explore the complexity-performance trade-offs for camera surveillance applications. For this purpose, we propose a Scalable Video Codec (SVC), based on wavelet transformation in which we have adopted a t+2D architecture. Complexity is adjusted by adapting the configuration of the lifting-based motion-compensated temporal filtering (MCTF). We discuss various configurations and have found an SVC that has a scalable complexity and performance, enabling embedded applications. The paper discusses the trade-off of coder complexity, e.g. motion-compensation stages, compression efficiency and end-to-end delay of the video coding chain. Our SVC has a lower complexity than H.264 SVC, but the quality performance at full resolution is close to H.264 SVC (within 1 dB for surveillance type video at 4CIF, 60Hz) and at lower resolutions sufficient for our video surveillance application.

Global Illumination Compensation for Background Subtraction Using Gaussian-Based Background Difference Modeling

This paper presents a background segmentation technique, which is able to process acceptable segmentation masks under fast global illumination changes. The histogram of the frame-based background difference is modeled with multiple kernels. The model that represents the histogram at best, is used to determine the shift in luminance due to global illumination or diaphragm changes, such that the background difference can be compensated. Experimental results have revealed that the number of incorrectly classified pixels using global illumination compensation instead of only the approximated median method reduces from 77% to 19% shortly after a fast change. The performance of the proposed technique is similar to state-of-the-art related work for global illumination changes, despite the fact that only luminance information is used. The algorithm is computationally simple and can operate at 30 frames-per-second for VGA resolution on a P-IV 3-GHz PC.

Robust automatic ship tracking in harbours using active cameras

Radar is commonly used to detect and track ships in maritime surveillance. Unfortunately the systems are costly and do not provide any visual information about the objects type. To complement the ship identity information given by a radar system, we propose a supplementary system using active visual cameras that can robustly detect and track ships in harbours. By combining a high-quality, non real-time robust object detector with a feature point tracker with low computational complexity, it is possible to track ships in real time over long intervals and large distances. In addition to controlling pan and tilt, we dynamically control camera zoom to provide a high resolution image of the tracked object over a large range of distances. The tracking system is improved by a special motion estimation model for the feature points, which also incorporates zooming of the camera. The system is robust and sustains tracking even under challenging conditions, such as multiple viewpoints, a large variety of ships and various weather conditions. During experiments, various types of ships were successfully tracked for up to 18 minutes, and over a distance of almost 1.5km in the port of Rotterdam. The proposed system is generic and can be utilized in various tracking applications, by training the detector for a different object class.

Real-time multi-level wavelet lifting scheme on a fixed-point DSP for JPEG 2000 and scalable video coding

In this paper, we discuss the design and real-time implementation of a multi-level two-dimensional Discrete Wavelet Transform (2D-DWT). The wavelet transform uses the well-known 5/3 filter coefficients and is implemented using the lifting framework. However, the transform allows complexity-scalable solutions with different latencies for scalable video coding. We have extensively utilized SIMD (Single Instruction Multiple Data) and DMA (Direct Memory Access) techniques, where the proposed process of background DMA transfers is so effective, that the ALUs are almost never starved for data input. The obtained execution performs a 4-level transform at CCIR-601 broadcast resolution in 3.65 Mcycles, including memory stalls, on a DM642 DSP. At a clock rate of 600MHz this translates to more than 160 transforms per second, satisfying the performance requirements for a real-time image/video encoding system for e.g. surveillance applications.

Two novel motion-based algorithms for surveillance video analysis on embedded platforms

This paper proposes two novel motion-vector based techniques for target detection and target tracking in surveillance videos. The algorithms are designed to operate on a resource-constrained device, such as a surveillance camera, and to reuse the motion vectors generated by the video encoder. The first novel algorithm for target detection uses motion vectors to construct a consistent motion mask, which is combined with a simple background segmentation technique to obtain a segmentation mask. The second proposed algorithm aims at multi-target tracking and uses motion vectors to assign blocks to targets employing five features. The weights of these features are adapted based on the interaction between targets. These algorithms are combined in one complete analysis application. The performance of this application for target detection has been evaluated for the i-LIDS sterile zone dataset and achieves an F1-score of 0.40-0.69. The performance of the analysis algorithm for multi-target tracking has been evaluated using the CAVIAR dataset and achieves an MOTP of around 9.7 and MOTA of 0.17-0.25. On a selection of targets in videos from other datasets, the achieved MOTP and MOTA are 8.8-10.5 and 0.32-0.49 respectively. The execution time on a PC-based platform is 36 ms. This includes the 20 ms for generating motion vectors, which are also required by the video encoder.

Real-time scalable video codec implementation for surveillance

In this paper, we discuss the design and real-time implementation of a Scalable Video Codec (SVC) for surveillance applications. We present a complexity-scalable temporal wavelet transform and the implementation of a multi-level 2D 5/3 wavelet transform, using the lifting framework. We have employed SIMD (Single Instruction Multiple Data) and DMA (Direct Memory Access) techniques, where the proposed process of background DMA transfers is so effective, that the ALUs are always supplied with input data. We have realized the execution of a 4-level transform at 4CIF (CCIR-601) broadcast resolution in 3.65 Mcycles, including memory stalls, on a TMS320DM642 DSP. At a clock rate of 600 MHz, this translates to more than 160 transforms per second. For our complete SVC, we achieve a frame rate of 12.5–15 fps depending on scene activity.

Highly-parallelized motion estimation for Scalable Video Coding

In this paper, we discuss the design of a highly-parallel motion estimator for real-time Scalable Video Coding (SVC). In an SVC, motion is commonly estimated bidirectionally and over various temporal distances. Current motion estimators are optimized for frame-by-frame estimation, and such estimators are designed without serious implementation constraints. To support efficient embedded applications, we propose a Highly Parallel Predictive Search (HPPS) motion estimator while preserving an accurate estimation performance. The motion estimation algorithm is optimized for processing on parallel cores and utilizes a novel recursive search strategy. This strategy is based on hierarchically increasing the temporal distance in the estimation algorithm while using the state of the previous hierarchical layer as an input. Due to the absence of local recursions in the algorithm, the proposed motion estimator has a constant computational load, regardless of video activity or temporal distance. We compared our proposed motion estimator to the well-known full search, ARPS3, 3DRS and EPZS motion estimators for the SVC case, and obtain a performance close to full search (0.2dB), while outperforming other algorithms in prediction.

Enhanced prediction for motion estimation in Scalable Video Coding

In this paper, we present a temporal candidate generation scheme that can be applied to motion estimators in Scalable Video Codecs (SVCs). For bidirectional motion estimation, usually a test is made for each block to determine which motion compensation direction is preferred: forward, bidirectional or backward. Instead of simply using the last computed motion vector field (backward or forward), giving an asymmetry in the estimation, we involve both vector fields to generate a single candidate field for a more stable and improved prediction. This field is generated with the aid of mode decision information of the codec. This single field of motion vector candidates serves two purposes: (1) it initializes the next recursion and (2) it is the foundation for the succeeding scale in the scalable coding. We have implemented this improved candidate system for both HPPS as EPZS motion estimators in a scalable video codec. We have found that it reduces the errors caused by occlusion of moving objects or image boundaries. For EPZS, only a small improvement is observed compared to the simple candidate scheme. However, for HPPS improvements are more significant: when looking at individual levels, motion compensation performance improves by up to 0.84 dB and when implemented in SVC, HPPS slightly outperforms EPZS.

Complexity reduction of wavelet codecs through modified quality control

Integer-to-integer wavelets are employed for low-complexity image encoders in embedded applications and used in combination with wavelet coefficient coders, such as EZW, SPIHT, SPECK and TSSP. In scalable coders bitstream creation is computationally expensive when individual coefficients are manipulated. In this paper, we study two options that move the quality control to earlier stages in the encoding process to alleviate this complexity problem: (1) to stage one of the TSSP and (2) to the integer wavelet transform. By inserting special functions based on premature bit-plane dropping, we effectively implement a quality-control step prior to the coefficient coding. For a typical usage scenario with full HD images, with option (1) we achieve an improvement of the processing speed of TSSP by 28%, while retaining the original bitstream and thus the ratedistortion performance. Furthermore, we have found that option (2) is generically applicable to other bit-plane based codecs, while offering nearly the same processing speed.