Menghan Hu

Saliency-induced reduced-reference quality index for natural scene and screen content images

Abstract Massive content composed of both natural scene and screen content has been generated with the increasing use of wireless computing and cloud computing, which call for general image quality assessment (IQA) measures working for both natural scene images (NSIs) and screen content images (SCIs). In this paper, we develop a saliency-induced reduced-reference (SIRR) IQA measure for both NSIs and SCIs. Image quality and visual saliency are two widely studied and closely related research topics. Traditionally, visual saliency is often used as a weighting map in the final pooling stage of IQA. Instead, we detect visual saliency as a quality feature since different types and levels of degradation can strongly influence saliency detection. Image quality is described by the similarity between two images’ saliency maps. In SIRR, saliency is detected through a binary image descriptor called “image signature”, which significantly reduces the reference data. We perform extensive experiments on five large-scale NSI quality assessment databases including LIVE, TID2008, CSIQ, LIVEMD, CID2013, as well as two recently constructed SCI QA databases, i.e., SIQAD and QACS. Experimental results show that SIRR is comparable to state-of-the-art full-reference and reduced-reference IQA measures in NSIs, and it can outperform most competitors in SCIs. The most important is that SIRR is a cross-content-type measure, which works efficiently for both NSIs and SCIs. The MATLAB source code of SIRR will be publicly available with this paper.

Synergetic use of thermal and visible imaging techniques for contactless and unobtrusive breathing measurement

Abstract. We present a dual-mode imaging system operating on visible and long-wave infrared wavelengths for achieving the noncontact and nonobtrusive measurements of breathing rate and pattern, no matter whether the subjects use the nose and mouth simultaneously, alternately, or individually when they breathe. The improved classifiers in tandem with the biological characteristics outperformed the custom cascade classifiers using the Viola–Jones algorithm for the cross-spectrum detection of face and nose as well as mouth. In terms of breathing rate estimation, the results obtained by this system were verified to be consistent with those measured by reference method via the Bland–Altman plot with 95% limits of agreement from −2.998 to 2.391 and linear correlation analysis with a correlation coefficient of 0.971, indicating that this method was acceptable for the quantitative analysis of breathing. In addition, the breathing waveforms extracted by the dual-mode imaging system were basically the same as the corresponding standard breathing sequences. Since the validation experiments were conducted under challenging conditions, such as the significant positional and abrupt physiological variations, we stated that this dual-mode imaging system utilizing the respective advantages of RGB and thermal cameras was a promising breathing measurement tool for residential care and clinical applications.

Combination of near-infrared and thermal imaging techniques for the remote and simultaneous measurements of breathing and heart rates under sleep situation

To achieve the simultaneous and unobtrusive breathing rate (BR) and heart rate (HR) measurements during nighttime, we leverage a far-infrared imager and an infrared camera equipped with IR-Cut lens and an infrared lighting array to develop a dual-camera imaging system. A custom-built cascade face classifier, containing the conventional Adaboost model and fully convolutional network trained by 32K images, was used to detect the face region in registered infrared images. The region of interest (ROI) inclusive of mouth and nose regions was afterwards confirmed by the discriminative regression and coordinate conversions of three selected landmarks. Subsequently, a tracking algorithm based on spatio-temporal context learning was applied for following the ROI in thermal video, and the raw signal was synchronously extracted. Finally, a custom-made time-domain signal analysis approach was developed for the determinations of BR and HR. A dual-mode sleep video database, including the videos obtained under environment where illumination intensity ranged from 0 to 3 Lux, was constructed to evaluate the effectiveness of the proposed system and algorithms. In linear regression analysis, the determination coefficient (R2) of 0.831 had been observed for the measured BR and reference BR, and this value was 0.933 for HR measurement. In addition, the Bland-Altman plots of BR and HR demonstrated that almost all the data points located within their own 95% limits of agreement. Consequently, the overall performance of the proposed technique is acceptable for BR and HR estimations during nighttime.

Movie piracy tracking using temporal psychovisual modulation

Nowadays, camcorder piracy has great impact on the motion picture industry. Although some watermarking technologies can track the movie pirate, the video content viewed in the theater may be affected and they cannot obstruct the need of pirated movie because the watermarks in pirated moves are invisible. This paper presents a new method to defeat camcorder piracy and realize content protection in the theater using a new paradigm of information display technology, called Temporal Psychovisual Modulation (TPVM), which utilizes the differences between the human-eye perception and digital camera imageforming to stack an invisible pattern on digital screen and projector. The images formed in human vision are continuous integration of the light field, while discrete sampling is used in digital video acquisition which has “blackout” period in each sampling cycle. Based on this difference, we can decompose a movie into a set of display frames with specific patterns and broadcast them out at high speed so that the audience cannot notice any disturbance, while the video frames captured by camcorder will contain highly objectionable artifacts (i.e., the patterns). The pattern embedded in the movies can also serves as tracking information to reveal the one responsibility for the camcorder piracy.

Perceptual Information Hiding Based on Multi-channel Visual Masking

Abstract Information security, the practice to protect information from unauthorized use, attracts researchers attention. In this paper, we proposed a visual masking effect based perceptual information protection scheme for display devices, such as mobile device, personal computer and ATM machine. The private information is embedded into the misleading image with the proposed algorithm, resulting in a dramatic descent in perceptual saliency of the private information for peepers viewing with naked eyes, while maintaining accessibility for authorized viewers wearing color filter. Quantitative and qualitative experiments are conducted, and results show effectiveness of our algorithm.

Arrow’s Impossibility Theorem inspired subjective image quality assessment approach

Abstract A large number of subjective image quality assessment databases have been constructed in the last decade, in which the Mean Opinion Score (MOS) (with single or double stimulus), and Paired Comparison (PC) are two dominant approaches for collecting the ground truth quality ratings and usually up to 15 or more subjects are needed for each image. In this paper, we show the fact that there is a potential “dictatorship” risk of using such averaging-over-multiple-rating type of method. Using Arrow’s Impossibility Theorem (AIT), we prove that meeting of the unanimity and independence of irrelevant alternatives (IIA) will generate a “pivotal subject”, who in fact determines the final rank of image quality. We also prove that no an ideal democratic approach to synthesize the whole opinions of subjects. Therefore, we advocate to recruit a small number of experts (a.k.a the “golden eyes”) for subjective viewing tests. In order to verify the reliability of our proposal, experiments on two different databases conducting on the general distorted images and professional images (here is Terahertz security image) are performed. In each experiment, the raw scores of images are subjectively assigned by at least 15 inexperienced viewers and 3 experts, and meanwhile the MOS or difference mean opinion score (DMOS) are obtained. Afterwards, the correlation of the scores rated by naive subjects and experts is analyzed. For general image experiment, it is revealed that DMOS of inexperience viewers are highly related to DMOS of experts based on six effective evaluation metrics. In professional image experiment, the preferences of experts also maintain favourable relevance with the opinions of inexperienced viewers in overall quality of THz image. Moreover, considering the quality assessments of illegal substance regions in THz images, the experts have higher accuracy than the inexperienced observers. In conclusion, the results of two validation experiments verify that a small number of experts are more suitable for assessing the perceptual quality of images, which can reduce cost and simplify procedure of creating databases.

Dual-mode imaging system for non-contact heart rate estimation during night

To estimate heart rate (HR) during night remotely and unobtrusively, we use a far-infrared camera and an RGB-Infrared camera to develop a dual-mode imaging system. The RGB or infrared images are first registered by the far-infrared images through the affine transformation. A custom-made cascade face classifier, which contains the conventional Adaboost model and fully convolutional network, was applied for the detection of the face in registered infrared images. The fully convolutional network was trained by 32K images from the PASCAL dataset. Subsequently, two facial tissues viz., mouth and nose were determined by the discriminative regression via the coordinate conversions of three selected landmarks. The spatio-temporal context learning was utilized to track the mouth and nose regions in the far-infrared image sequence. Then, the raw image feature was extracted from these two regions of interest. Finally, a state-of-art signal analysis method was explored to calculate HR in the time domain of the extracted raw signal. With respect to the validation experiment, we established the dual-mode sleep video database to verify the performances of the proposed system and algorithms. All videos in database were filmed under the environments where actual illumination intensity ranged from 0 to 3 Lux. The obtained results demonstrated that the determination coefficient (R2) was 0.933 for HR estimation in linear regression analysis. The Bland-Altman analysis showed that almost all the data points located within the 95% upper and lower limits of agreement, which were 4.293 and −5.293 bpm, respectively. Therefore, the proposed technique is efficient for the non-contact and unobtrusive HR estimation during night.

IPAD: Intensity potential for adaptive de-quantization

Display devices at bit-depth of 10 or higher have been mature but the mainstream media source is still at bit-depth as low as 8. To accommodate the gap, the most economic solution is to render source at low bit-depth for high bit-depth display, which is essentially the procedure of de-quantization. Traditional methods, like zero-padding or bit replication, introduce annoying false contour artifacts. To better estimate the least-significant bits, later works use filtering or interpolation approaches, which exploit only limited neighbor information, can not thoroughly remove the false contours. In this paper, we propose a novel intensity potential field to model the complicated relationships among pixels. Then, an adaptive de-quantization algorithm is proposed to convert low bit-depth images to high bit-depth ones. To the best of our knowledge, this is the first attempt to apply potential field for natural images. The proposed potential field preserves local consistency and models the complicated contexts very well. Extensive experiments on natural image datasets validate the efficiency of the proposed intensity potential field. Significant improvements have been achieved over the state-of-the-art methods on both PSNR and SSIM.

Selection of Good Display Mode for Terahertz Security Image via Image Quality Assessment

In order to provide a good display performance for THz (terahertz) security image, we designed several display modes on the custom-built THz security image database (THSID). Based on our statistical analysis of THz images, a total of 4 candidate display modes are proposed, namely averaging the highest 1%, 10%, 20%, 30% pixel values in Z-axis for a coordinate (x, y). In this paper, the subjective evaluation was first carried out, demonstrating that the second display mode, that was the averaging the highest 10% pixel values in Z-axis, got the greatest performance. Subsequently, to further support the result obtained by the subjective evaluation and the high throughout application requirement in real world, a total of 11 objective no-reference IQA (Image Quality Assessment) algorithms were implemented, including 4 opinion-aware approaches, viz. GMLF, NFERM, BLIINDS2, BRISQUE, and 7 opinion-unaware approaches viz. CPBD, FISBLIM, NIQE, QAC, SISBLIM, S3, Fish_bb. The results of objective evaluation show that the current objective IQA algorithms can hardly support the subjective evaluation. Even so, BLIINDS2 and CPBD perform relatively well for the chosen display mode above. A more suitable objective evaluation method need to be explored in the future study. This study will make some progresses on the display effect of THz image, which can promote the detection accuracy in the future applications.