Kwok-Wai Cheung

MIRROR: an interactive content based image retrieval system

A content based image retrieval system, called MPEG-7 image retrieval refinement based on relevance feedback (MIRROR), is developed for evaluating MPEG-7 visual descriptors and developing new retrieval algorithms. The system core is based on MPEG-7 experimentation mode (XM) with Web-based user interface for query by image example retrieval. A new merged color palette approach for MPEG-7 dominant color descriptor similarity measure and relevance feedback are also developed in this system. Several MPEG-7 visual descriptors are adopted in MIRROR for performance comparison purposes.

Novel Directional Gradient Descent Searches for Fast Block Motion Estimation

Search point pattern-based fast block motion estimation algorithms provide significant speedup for motion estimation but usually suffer from being easily trapped in local minima. This may lead to low robustness in prediction accuracy particularly for video sequences with complex motions. This problem is especially serious in one-at-a-time search (OTS) and block-based gradient descent search (BBGDS), which provide very high speedup ratio. A multipath search using more than one search path has been proposed to improve the robustness of BBGDS but the computational requirement is much increased. To tackle this drawback, a novel directional gradient descent search (DGDS) algorithm using multiple OTSs and gradient descent searches on the error surface in eight directions is proposed in this letter. The search point patterns in each stage depend on the minima found in these eight directions, and thus the global minimum can be traced more efficiently. In addition, a fast version of the DGDS (FDGDS) algorithm is also described to further improve the speed of DGDS. Experimental results show that DGDS reduces computation load significantly compared with the well-known fast block motion estimation algorithms. Moreover, FDGDS can achieve faster speedup compared with the UMHexagonS algorithm in H.264/AVC implementation while maintaining very similar rate-distortion performance.

A Search Patterns Switching Algorithm for Block Motion Estimation

Center-biased fast motion estimation algorithms, e.g., block-based gradient descent search and diamond search, can perform much better than coarse-to-fine search algorithms, such as 2-D logarithmic search and three-step search. The latter type of algorithms, however, is more suitable for handling large motion content. To combine the advantages of both types of algorithms, an adaptive algorithm performing search patterns switching (SPS) is proposed in this paper. The proposed SPS algorithm classifies the motion content of a block using a simple yet efficient motion content classifier called error descent rate. Unlike other classifiers with heavy overhead, this classifier requires only the searching of a few points in the search window and then a division operation. Experimental results show that the proposed SPS algorithm is very robust.

Dominant Color Structure Descriptor for Image Retrieval

A new dominant color structure descriptor (DCSD) is proposed in this paper. It is designed to provide an efficient way to represent both color and spatial structure information with single compact descriptor. The descriptor combines the compactness of dominant color descriptor (DCD) and the retrieval accuracy of color structure descriptor (CSD) to enhance the retrieval performance in a highly efficient manner. The feature extraction and similarity measure of the descriptor are designed to address the problems of the existing descriptors while utilize the advantages of them. Experimental results show that DCSD has a significant improvement on both retrieval performance and descriptor size over DCD. An eight-color DCSD (DCSD 8) gives an averaged normalized modified retrieval rate (ANMRR) of 0.0993 using MPEG-7 common color dataset, outperforming compact configurations of scalable color descriptor and color structure descriptor with smaller descriptor size.

Integration of image quality and motion cues for face anti-spoofing

A multi-cues integration framework is proposed using a hierarchical neural network.Bottleneck representations are effective in multi-cues feature fusion.Shearlet is utilized to perform face image quality assessment.Motion-based face liveness features are automatically learned using autoencoders. Many trait-specific countermeasures to face spoofing attacks have been developed for security of face authentication. However, there is no superior face anti-spoofing technique to deal with every kind of spoofing attack in varying scenarios. In order to improve the generalization ability of face anti-spoofing approaches, an extendable multi-cues integration framework for face anti-spoofing using a hierarchical neural network is proposed, which can fuse image quality cues and motion cues for liveness detection. Shearlet is utilized to develop an image quality-based liveness feature. Dense optical flow is utilized to extract motion-based liveness features. A bottleneck feature fusion strategy can integrate different liveness features effectively. The proposed approach was evaluated on three public face anti-spoofing databases. A half total error rate (HTER) of 0% and an equal error rate (EER) of 0% were achieved on both REPLAY-ATTACK database and 3D-MAD database. An EER of 5.83% was achieved on CASIA-FASD database.

Automatic 2D-to-3D video conversion technique based on depth-from-motion and color segmentation

Most of the TV manufacturers have released 3DTVs in the summer of 2010 using shutter-glasses technology. 3D video applications are becoming popular in our daily life, especially at home entertainment. Although more and more 3D movies are being made, 3D video contents are still not rich enough to satisfy the future 3D video market. There is a rising demand on new techniques for automatically converting 2D video content to stereoscopic 3D video displays. In this paper, an automatic monoscopic video to stereoscopic 3D video conversion scheme is presented using block-based depth from motion estimation and color segmentation for depth map enhancement. The color based region segmentation provides good region boundary information, which is used to fuse with block-based depth map for eliminating the staircase effect and assigning good depth value in each segmented region. The experimental results show that this scheme can achieve relatively high quality 3D stereoscopic video output.

Depth map misalignment correction and dilation for DIBR view synthesis

The quality of the synthesized views by Depth Image Based Rendering (DIBR) highly depends on the accuracy of the depth map, especially the alignment of object boundaries of texture image. In practice, the misalignment of sharp depth map edges is the major cause of the annoying artifacts at the disoccluded regions of the synthesized views. Conventional smooth filter approach blurs the depth map to reduce the disoccluded regions. The drawbacks are the degradation of 3D perception of the reconstructed 3D videos and the destruction of the texture in background regions. Conventional edge preserving filter utilizes the color image information in order to align the depth edges with color edges. Unfortunately, the characteristics of color edges and depth edges are very different which causes annoying boundaries artifacts in the synthesized virtual views. Recent solution of reliability-based approach uses reliable warping information from other views to fill the holes. However, it is not suitable for the view synthesis in video-plus-depth based DIBR applications. In this paper, a new depth map preprocessing approach is proposed. It utilizes Watershed color segmentation method to correct the depth map misalignment and then the depth map object boundaries are extended to cover the transitional edge regions of color image. This approach can handle the sharp depth map edges lying inside or outside the object boundaries in 2D sense. The quality of the disoccluded regions of the synthesized views can be significantly improved and unknown depth values can also be estimated. Experimental results show that the proposed method achieves superior performance for view synthesis by DIBR especially for generating large baseline virtual views.

A foreground biased depth map refinement method for DIBR view synthesis

The performance of view synthesis using depth image based rendering (DIBR) highly depends on the accuracy of depth map. Inaccurate boundary alignment between texture image and depth map especially for large depth discontinuities always cause annoying artifacts in disocclusion regions of the synthesized view. Pre-filtering approach and reliability-based approach have been proposed to tackle this problem. However, pre-filtering approach blurs the depth map with drawback of degradation of the depth map and may also cause distortion in non-hole region. Reliability-based approach uses reliable warping information from other views to fill up holes and is not suitable for the view synthesis with single texture video such as video-plus-depth based DIBR applications. This paper presents a simple and efficient depth map preprocessing method with use of texture edge information to refine depth pixels around the large depth discontinuities. The refined depth map can make the whole texture edge pixels assigned with foreground depth values. It can significantly improve the quality of the synthesized view by avoiding incorrect use of foreground texture information in hole filling. The experimental results show the proposed method achieves superior performance for view synthesis by DIBR especially for large baseline.

A Compact and Efficient Color Descriptor for Image Retrieval

An important problem in color based image retrieval is the lack of efficient way to represent both the color and the spatial structure information with single descriptor. To solve this problem, a new dominant color structure descriptor (DCSD), is proposed. The descriptor combines the compactness of dominant color descriptor (DCD) and the accuracy of color structure descriptor (CSD) to enhance the retrieval performance in a highly efficient manner. The feature extraction and similarity measure of the descriptor are designed to address the problems of the existing descriptors such as color inaccuracy of DCD and redundancy of CSD. Experimental results show that DCSD has a significant improvement in retrieval performance and descriptor size over DCD. An eight-color DCSD (DCSD 8) gives an averaged normalized modified retrieval rate (ANMRR) of 0.0993 using MPEG-7 common color dataset, outperforming compact configurations of scalable color descriptor and color structure descriptor with smaller descriptor size.

No-Reference Video Quality Assessment With 3D Shearlet Transform and Convolutional Neural Networks

In this paper, we propose an efficient general-purpose no-reference (NR) video quality assessment (VQA) framework that is based on 3D shearlet transform and convolutional neural network (CNN). Taking video blocks as input, simple and efficient primary spatiotemporal features are extracted by 3D shearlet transform, which are capable of capturing natural scene statistics properties. Then, CNN and logistic regression are concatenated to exaggerate the discriminative parts of the primary features and predict a perceptual quality score. The resulting algorithm, which we name shearlet- and CNN-based NR VQA (SACONVA), is tested on well-known VQA databases of Laboratory for Image & Video Engineering, Image & Video Processing Laboratory, and CSIQ. The testing results have demonstrated that SACONVA performs well in predicting video quality and is competitive with current state-of-the-art full-reference VQA methods and general-purpose NR-VQA algorithms. Besides, SACONVA is extended to classify different video distortion types in these three databases and achieves excellent classification accuracy. In addition, we also demonstrate that SACONVA can be directly applied in real applications such as blind video denoising.