Zhongpai Gao

Visual attention data for image quality assessment databases

Images usually contain areas that particularly attract peoples attention and visual attention is an important feature of human visual system (HVS). Visual attention had been shown to be effective in improving performance of existing image quality assessment (IQA) metrics. However, with the quick advancement of IQA research, the booming of open IQA databases calls for associated comprehensive and accurate visual attention dataset. Despite of the large number of existing computational attention/saliency models, the most accurate measure of human attention is still human based. In this research, we first conduct extensive eye tracking experiments for all the pristine images from the seven widely used IQA databases (LIVE, TID2008, CSIQ, Toyama, LIVE Multiply Distortion, IVC and A57 databases). Then we propose a gaze-duration adaptive weighting approach to generate saliency maps from the eye tracking data. When applied on the IQA databases, experimental results suggest that accuracy of benchmark quality metrics, e.g. PSNR and SSIM can be systematically improved, outperforming existing saliency datasets. Both the eye tracking data and the saliency maps in this research will be made publicly available at gvsp.sjtu.edu.cn.

Information security display system based on temporal psychovisual modulation

This paper introduces an information security display system using temporal psychovisual modulation (TPVM). TPVM was proposed as a new information display technology using the interplay of signal processing, optoelectronics and psychophysics. Since the human visual system cannot detect quick temporal changes above the flicker fusion frequency (about 60 Hz) and yet modern display technologies offer much higher refresh rates, there is a chance for a single display to simultaneously serve different contents to multiple observers. A TPVM display broadcasts a set of images called atom frames at a high speed, and those atom frames are then weighted by liquid crystal (LC) shutter based viewing devices that are synchronized with the display before entering the human visual system and fusing into the desired visual stimuli. And through different viewing devices, people can see different information. In this work, we develop a TPVM based information security display prototype. There are two kinds of viewers, those authorized viewers with the viewing devices who can see the secret information and those unauthorized viewers (bystanders) without the viewing devices who only see mask/disguise images. The prototype is built on a 120 Hz LCD screen with synchronized LC shutter glasses that were originally developed for stereoscopic display. The system is written in C++ language with SDKs of Nvidia 3D Vision, DirectX, CEGUI, MuPDF and etc. We also added human-computer interaction support of the system using Kinect. The information security display system developed in this work serves as a proof-of-concept of the TPVM paradigm, as well as a testbed for future research of TPVM technology.

Information security display system based on Spatial Psychovisual Modulation

Privacy protection is of increasing importance in this era of information explosion. This paper introduces an information security display system based on the idea of Spatial Psycho-visual Modulation (SPVM). With the rapid advance of modern manufacturing techniques, display devices now support very high pixel density (e.g. the retina display of Apple). Meanwhile the human visual system (HVS) cannot distinguish image signals with spatial frequency above a threshold, as predicted by the contrast sensitivity function (CSF). Therefore, it is now possible for us to devise a type of information security display using the mismatch between resolutions of modern display devices and the HVS. Given the desired visual stimuli for both the bystanders and the authorized users, we propose a method to design display signals accordingly. We select polarization as a way to effectively differentiate the bystanders and the authorized users. Operationally, applying complementary polarization to light emitted from different spatial sections of the display could in itself be a challenge. Fortunately, the development of stereoscopic display technologies has made available a lot of polarization based spatial multiplexing type of display devices. Hardware of the information security display system is based on a polarization based stereoscopic screen made by LG. Software of the information security display system is written in C++ with SDKs of DirectX and etc. Kinect is also included into our system to enhance the experience of human-computer interaction. Extended experimental results will be given in this paper to justify the effectiveness and robustness of the system. The developed system serves both as a proof-of-concept of the SPVM method, as well as a test bed for future research of SPVM based display technology.

Influence of compression artifacts on visual attention

Visual attention is an important function of the human visual system (HVS). In the long term research of visual attention, various computational models have been proposed with encouraging results. However, most of those work were conducted on images with ideal visual quality. In practice, outputs of most visual communication systems contain different levels of artifacts, e.g. noise, blurring, blockiness and etc. Therefore, it is interesting to investigate the impacts of artifacts on visual attention. In this paper, we question into the problem of how the widely encountered JPEG compression artifacts affect visual attention. We designed eye-tracking experiments on images with different levels of compression and viewing time and quantitatively compared the recorded eye movement data. We found that compression level does have impacts on visual attention, and yet this influence can be negligible for low levels of compression. For high levels of compression, the visual artifacts alter visual attention in a systematic way. Dependence of the influence on viewing duration was also analyzed and it was observed that too short or too long viewing time reduces the impact of compression artifacts on visual attention.

Factorization Algorithms for Temporal Psychovisual Modulation Display

Temporal psychovisual modulation (TPVM) is a new information display technology which aims to generate multiple visual percepts for different viewers on a single display simultaneously. In a TPVM system, the viewers wearing different active liquid crystal (LC) glasses with varying transparency levels can see different images (called personal views). The viewers without LC glasses can also see a semantically meaningful image (called shared view). The display frames and weights for the LC glasses in the TPVM system can be computed through nonnegative matrix factorization (NMF) with three additional constrains: the values of images and modulation weights should have upper bound (i.e., limited luminance of the display and transparency level of the LC); the shared view without using viewing devices should be considered (i.e., the sum of all basis images should be a meaningful image); and the sparsity of modulation weights should be considered due to the material property of LC. In this paper, we proposed to solve the constrained NMF problem by a modified version of hierarchical alternating least squares (HALS) algorithms. Through experiments, we analyze the choice of parameters in the setup of TPVM system. This work serves as a guideline for practical implementation of TPVM display system.

Dual-view medical image visualization based on spatial-temporal psychovisual modulation

Medical imaging technologies such as magnetic resonance imaging (MRI) and computerized tomographic (CT) are used to diagnose a wide range of medical diseases. Medical images are generated by detecting density differences between different tissues in the body. Multiple medical image visualization is of critical importance to diagnosis. This paper introduces a dual-view medical image visualization prototype based on spatial-temporal psychovisual modulation (STPVM). Temporal psychovisual modulation (TPVM) enables a single display to generate multiple visual content for different viewers. Spatial psychovisual modulation (SPVM) extends the idea of TPVM to spatial domain. STPVM combines TPVM and SPVM by exploiting both temporal and spatial redundancy of modern displays. Based on STPVM technology, one display can present even more images simultaneously. In this demo, two kinds of medical images e.g. T1 and T2 weighted MRI images, are presented simultaneously. Physicians can switch between either image by just moving the eye fixations. Since T1 and T2 are shown simultaneously and are aligned on the screen, it is more convenient for the physicians to get different information of the same spot from the T1 and T2 images. The developed demo is useful for physicians during surgery navigation and effectively reduces the burden of mental transfer.

Sound influences visual attention discriminately in videos

In the long term research of visual attention, various computational models have been proposed. However, most of those works do not take audio into consideration. In practice, visual signals often come along with audio. Therefore, it is natural to investigate the influence of audio on visual attention. In this paper, we focus on the problem of when will audio influence visual attention during video watching. We perform eye-tracking experiments on a set of 60 videos in audio-visual (AV) and visual (V) conditions. Based on the eye movement data, we find that the influence of audio on visual attention depends on the consistency between the visual and audio signals. If the salient objects from the visual perspective are not consistent with the salient objects from the audio perspective, audio will influence visual attention. Otherwise, audio has little influence on visual attention.

Simultaneous triple subtitles exhibition via Temporal Psychovisual Modulation

With the quick acceleration of cultural exchange and globalization, multi-language annotation for in-theatre movie exhibition is gaining more attention. The existing solution overlays a stack of subtitles in different languages over the movie, for example, the familiar combination of English + local tongue as seen in the cinema today. However, a major drawback of this straightforward piling-up type of solution is that as the required number of languages increases, the subtitle area grows and it may eventually span over the movie contents. An effective multi-language subtitling system without compromising the movie watching experience is therefore highly desired. In this paper, we propose a new Temporal Psychovisual Modulation (TPVM) based solution to the multi-language subtitling problem. TPVM is a new paradigm of information display exploiting the mismatch between high refresh rate of the modern optoelectronic displays and limited temporal resolution of the human visual system (HVS). In this work, we design a simultaneous triple subtitle exhibition system using 2 120 Hz stereoscopic DLP projectors with linear polarization filters and LCD shutter glasses. Wearing different pairs of glasses, the audience can enjoy the movie with 3 optional subtitles without interfering with the audience without the glasses and see the movie directly. Extended experimental results will be given in this paper to justify the effectiveness and robustness of the proposed multiple subtitling system.

Visible Light Communication via Temporal Psycho-Visual Modulation

In this paper we propose a new paradigm for visible light communication (VLC) using the emerging display technology of Temporal Psycho-Visual Modulation (TPVM) that exploits the interaction between human visual system and modern electro-optical display devices. Unlike traditional VLC, no specifically designed light emitter and receiver are required. In the proposed system, light projector is used as the information source and digital cameras act as information decoder. The emitted light is designed in a specific way such that it can carry meaningful information (or simply works as an illumination source) for human eyes while other message can be decoded by the digital camera due to the fundamental difference in the imaging mechanism of the human eye and digital devices. We further describe two applications of this new type of VLC in ubiquitous augmented reality and illegal camcorder-recording prevention with extensive experimental results.

Quality assessment for dual-view display system

Spatial psychovisual modulation (SPVM) is a new information display technology, which aims to generate multiple visual percepts for different viewers on a single display simultaneously. After the proposal of SPVM, lots of efforts have been made and several applications (i.e., dual-view display system) have been implemented based on this technology. The dual-view display (DVD) system is considered as an effective digital image hiding system based on SPVM theory, but little work has been dedicated to the perceptual quality assessment of DVD system. Up to now, there is no clear and standard method to evaluate the performance of the dual-view display system. It is important for the viewers to see a clear and non-aliasing image when they are front of the screen. Therefore, in this paper, we will build a DVD database and carry out a subjective experiment to evaluate the performance of the DVD system, and then we investigate and analyze the performance of prevailing no-reference (NR) image quality metrics on the particular DVD system. We have a sufficient belief that this paper can supply the guideline for the performance on the DVD system and serve as a good testing bed for future research of SPVM technology.