Toni Virtanen

Evaluating the multivariate visual quality performance of image-processing components

The estimation of image quality is a demanding task, especially when estimating different high-quality imaging products or their components. The challenge is the multivariate nature of image quality as well as the need to use naïve observers as test subjects, since they are the actual end-users of the products. Here, we use a subjective approach suitable for estimating the quality performance of different imaging device components with naïve observers—the interpretation-based quality (IBQ) approach. From two studies with 61 naïve observers, 17 natural image contents, and 13 different camera image signal processor pipelines, we determined the subjectively crucial image quality attributes and dimensions and the description of each pipelines perceived image quality performance. We found that the subjectively most important image quality dimensions were color shift/naturalness, darkness, and sharpness. The first dimension, which was related to naturalness and colors, distinguished the good-quality pipelines from the middle- and low-quality groups, and the dimensions of darkness and sharpness described why the quality failed in the low-quality pipelines. The study suggests that the high-level concept naturalness is a requirement for high-quality images, whereas quality can fail for other reasons in low-quality images, and this failure can be described by low-level concepts, such as darkness and sharpness.

Subjective experience of image quality: attributes, definitions, and decision making of subjective image quality

Subjective quality rating does not reflect the properties of the image directly, but it is the outcome of a quality decision making process, which includes quantification of subjective quality experience. Such a rich subjective content is often ignored. We conducted two experiments (with 28 and 20 observers), in order to study the effect of paper grade on image quality experience of the ink-jet prints. Image quality experience was studied using a grouping task and a quality rating task. Both tasks included an interview, but in the latter task we examined the relations of different subjective attributes in this experience. We found out that the observers use an attribute hierarchy, where the high-level attributes are more experiential, general and abstract, while low-level attributes are more detailed and concrete. This may reflect the hierarchy of the human visual system. We also noticed that while the observers show variable subjective criteria for IQ, the reliability of average subjective estimates is high: when two different observer groups estimated the same images in the two experiments, correlations between the mean ratings were between .986 and .994, depending on the image content.

Categorization of Natural Dynamic Audiovisual Scenes

This work analyzed the perceptual attributes of natural dynamic audiovisual scenes. We presented thirty participants with 19 natural scenes in a similarity categorization task, followed by a semi-structured interview. The scenes were reproduced with an immersive audiovisual display. Natural scene perception has been studied mainly with unimodal settings, which have identified motion as one of the most salient attributes related to visual scenes, and sound intensity along with pitch trajectories related to auditory scenes. However, controlled laboratory experiments with natural multimodal stimuli are still scarce. Our results show that humans pay attention to similar perceptual attributes in natural scenes, and a two-dimensional perceptual map of the stimulus scenes and perceptual attributes was obtained in this work. The exploratory results show the amount of movement, perceived noisiness, and eventfulness of the scene to be the most important perceptual attributes in naturalistically reproduced real-world urban environments. We found the scene gist properties openness and expansion to remain as important factors in scenes with no salient auditory or visual events. We propose that the study of scene perception should move forward to understand better the processes behind multimodal scene processing in real-world environments. We publish our stimulus scenes as spherical video recordings and sound field recordings in a publicly available database.

Evaluation of the visual performance of image processing pipes: information value of subjective image attributes

Subjective image quality data for 9 image processing pipes and 8 image contents (taken with mobile phone camera, 72 natural scene test images altogether) from 14 test subjects were collected. A triplet comparison setup and a hybrid qualitative/quantitative methodology were applied. MOS data and spontaneous, subjective image quality attributes to each test image were recorded. The use of positive and negative image quality attributes by the experimental subjects suggested a significant difference between the subjective spaces of low and high image quality. The robustness of the attribute data was shown by correlating DMOS data of the test images against their corresponding, average subjective attribute vector length data. The findings demonstrate the information value of spontaneous, subjective image quality attributes in evaluating image quality at variable quality levels. We discuss the implications of these findings for the development of sensitive performance measures and methods in profiling image processing systems and their components, especially at high image quality levels.

A new method for evaluating the subjective image quality of photographs: dynamic reference

The Dynamic Reference (DR) method has been developed for subjective image quality experiments in which original or undistorted images are unavailable. The DR method creates reference image series from test images. Reference images are presented to observers as a slide show prior to evaluating their quality. As the observers view the set of reference images, they determine the overall variation in quality within the set of test images. This study compared the performance of the DR method to that of the standardized absolute category rating (ACR) and paired comparison (PC) methods. We measured the performance of each method in terms of time effort and discriminability. The results showed that the DR method is faster than the PC method and more accurate than the ACR method. The DR method is especially suitable for experiments that require highly accurate results in a short time.

Image feature subsets for predicting the quality of consumer camera images and identifying quality dimensions

Abstract. Image-quality assessment measures are largely based on the assumption that an image is only distorted by one type of distortion at a time. These conventional measures perform poorly if an image includes more than one distortion. In consumer photography, captured images are subject to many sources of distortions and modifications. We searched for feature subsets that predict the quality of photographs captured by different consumer cameras. For this, we used the new CID2013 image database, which includes photographs captured by a large number of consumer cameras. Principal component analysis showed that the features classified consumer camera images in terms of sharpness and noise energy. The sharpness dimension included lightness, detail reproduction, and contrast. The support vector regression model with the found feature subset predicted human observations well compared to state-of-the-art measures.

VQone MATLAB toolbox: A graphical experiment builder for image and video quality evaluations

This article presents VQone, a graphical experiment builder, written as a MATLAB toolbox, developed for image and video quality ratings. VQone contains the main elements needed for the subjective image and video quality rating process. This includes building and conducting experiments and data analysis. All functions can be controlled through graphical user interfaces. The experiment builder includes many standardized image and video quality rating methods. Moreover, it enables the creation of new methods or modified versions from standard methods. VQone is distributed free of charge under the terms of the GNU general public license and allows code modifications to be made so that the program’s functions can be adjusted according to a user’s requirements. VQone is available for download from the project page (http://www.helsinki.fi/psychology/groups/visualcognition/).

Concurrent explanations can enhance visual decision making.

The verbalization of ones thoughts has been shown to impair judgment and decision making in some cases, particularly when targets are perceptual. This finding has been attributed to the fact that non-verbal processes are sometimes difficult to verbalize, which may cause a shift in processing that is maladaptive to the task. The study shows that concurrent written explanations can also enhance judgment and decision making in certain visual choice tasks. This finding suggests that the effect of verbalization on perceptual tasks is not dependent on whether the targets of the judgment are verbal or perceptual but rather on whether there is adequate vocabulary to execute the task and whether the task benefits from a more analytic approach.

Measuring multivariate subjective image quality for still and video cameras and image processing system components

The subjective quality of an image is a non-linear product of several, simultaneously contributing subjective factors such as the experienced naturalness, colorfulness, lightness, and clarity. We have studied subjective image quality by using a hybrid qualitative/quantitative method in order to disclose relevant attributes to experienced image quality. We describe our approach in mapping the image quality attribute space in three cases: still studio image, video clips of a talking head and moving objects, and in the use of image processing pipes for 15 still image contents. Naive observers participated in three image quality research contexts in which they were asked to freely and spontaneously describe the quality of the presented test images. Standard viewing conditions were used. The data shows which attributes are most relevant for each test context, and how they differentiate between the selected image contents and processing systems. The role of non-HVS based image quality analysis is discussed.

Performance measure of image and video quality assessment algorithms: subjective root-mean-square error

Abstract. Evaluating algorithms used to assess image and video quality requires performance measures. Traditional performance measures (e.g., Pearson’s linear correlation coefficient, Spearman’s rank-order correlation coefficient, and root mean square error) compare quality predictions of algorithms to subjective mean opinion scores (mean opinion score/differential mean opinion score). We propose a subjective root-mean-square error (SRMSE) performance measure for evaluating the accuracy of algorithms used to assess image and video quality. The SRMSE performance measure takes into account dispersion between observers. The other important property of the SRMSE performance measure is its measurement scale, which is calibrated to units of the number of average observers. The results of the SRMSE performance measure indicate the extent to which the algorithm can replace the subjective experiment (as the number of observers). Furthermore, we have presented the concept of target values, which define the performance level of the ideal algorithm. We have calculated the target values for all sample sets of the CID2013, CVD2014, and LIVE multiply distorted image quality databases.The target values and MATLAB implementation of the SRMSE performance measure are available on the project page of this study.