Lucjan Janowski

From packets to people: quality of experience as a new measurement challenge

Over the course of the last decade, the concept of Quality of Experience (QoE) has gained strong momentum, both from an academic research and an industry perspective. Being linked very closely to the subjective perception of the end user, QoE is supposed to enable a broader, more holistic understanding of the qualitative performance of networked communication systems and thus to complement the traditional, more technology-centric Quality of Service (QoS) perspective. The purpose of this chapter is twofold: firstly, it introduces the reader to QoE by discussing the origins and the evolution of the concept. Secondly, it provides an overview of the current state of the art of QoE research, with focus on work that particularly addresses QoE as a measurement challenge on the technology as well as on the end-user level. This is achieved by surveying the different streams of QoE research that have emerged in the context of Video, Voice and Web services with respect to the following aspects: fundamental relationships and perceptual principles, QoE assessment, modeling and monitoring.

The Influence of Subjects and Environment on Audiovisual Subjective Tests: An International Study

Traditionally, audio quality and video quality are evaluated separately in subjective tests. Best practices within the quality assessment community were developed before many modern mobile audiovisual devices and services came into use, such as internet video, smart phones, tablets and connected televisions. These devices and services raise unique questions that require jointly evaluating both the audio and the video within a subjective test. However, audiovisual subjective testing is a relatively under-explored field. In this paper, we address the question of determining the most suitable way to conduct audiovisual subjective testing on a wide range of audiovisual quality. Six laboratories from four countries conducted a systematic study of audiovisual subjective testing. The stimuli and scale were held constant across experiments and labs; only the environment of the subjective test was varied. Some subjective tests were conducted in controlled environments and some in public environments (a cafeteria, patio or hallway). The audiovisual stimuli spanned a wide range of quality. Results show that these audiovisual subjective tests were highly repeatable from one laboratory and environment to the next. The number of subjects was the most important factor. Based on this experiment, 24 or more subjects are recommended for Absolute Category Rating (ACR) tests. In public environments, 35 subjects were required to obtain the same Students t-test sensitivity. The second most important variable was individual differences between subjects. Other environmental factors had minimal impact, such as language, country, lighting, background noise, wall color, and monitor calibration. Analyses indicate that Mean Opinion Scores (MOS) are relative rather than absolute. Our analyses show that the results of experiments done in pristine, laboratory environments are highly representative of those devices in actual use, in a typical user environment.

Framework for the integrated video quality assessment

Through years of development Content Networks (CN) have become more sophisticated and more technically diverse. Modern CN are designed to be more adaptive to communication environment, devices and user requirements. However, one open issue is the still fluctuating quality of service provision. As a result user experience can be negatively affected. In order to maintain a satisfactory level of user experience it is crucial to develop a feasible solution to measure the extent to which video services meet users’ expectation. Assessing video quality with respect to users’ subjective opinions is a complex task. In this paper we address challenges of this task and design an integrated framework using a number of comprehensive functional modules. Our framework integrates objective quality assessment models of Artifacts Measurement (AM) and Quality of Delivery (QoD) approaches. Only the fittest models are activated by the framework considering requirements of individual evaluation tasks. We also introduce our recent work of realising key functional modules of the framework. Joint subjective experiments between two institutes have also been carried out for the purpose of model implementation and evaluation. Results from experiments verify the concept of an integrated framework and show the effectiveness of its key modules in estimating the quality level of video services.

Perceptual quality assessment for H.264/AVC compression

The paper proposes a No-Reference (NR) metric to objectively assess the H.264/AVC video quality. The proposed model takes into account the typical artefacts introduced by hybrid block-based motion compensated predictive video codecs as the one related to the H.264/AVC standard. More specifically, these artefacts are the blockiness introduced at the boundaries of each coded block and the temporal flickering due to different coding modes used for the same macroblock along the video sequence. Furthermore, a flickering metric for intra coded frames is also derived. The quality prediction accuracy of the proposed NR quality metric is validated over subjective data collected during a video subjective evaluation experiments. Moreover, the quality prediction accuracy is also compared with the one provided by the well known state-of-the-art Structural SIMilarity (SSIM) metric which works in a full-reference mode. The proposed metric achieves a higher Pearsons correlation coefficient with subjective scores than the one achieved by the SSIM metric.

QoE as a function of frame rate and resolution changes

Video bit rate reduction can be very important for all video streaming application. One of the possible ways to reduce bit rate is decreasing change in time or space domain i.e. changing frame rate or resolution. In this paper we present two no reference metrics mapping frame rate or resolution into MOS. Both models use simple to calculate parameters expressed by sequence spatial and temporal information. The models were estimated and verified upon distinctive video sequence sets. The considered frame rate change varies from 5 to 30 frames per second. The considered resolutions changes from SQCIF to SD.

Quality assessment for a visual and automatic license plate recognition

Video transmission and analysis is often utilized in applications outside of the entertainment sector, and generally speaking this class of video is used to perform specific tasks. Examples of these applications include security and public safety. The Quality of Experience (QoE) concept for video content used for entertainment differs significantly from the QoE of surveillance video used for recognition tasks. This is because, in the latter case, the subjective satisfaction of the user depends on achieving a given functionality. Recognizing the growing importance of video in delivering a range of public safety services, we focused on developing critical quality thresholds in license plate recognition tasks based on videos streamed in constrained networking conditions. Since the number of surveillance cameras is still growing it is obvious that automatic systems will be used to do the tasks. Therefore, the presented research includes also analysis of automatic recognition algorithms.

Standardized toolchain and model development for video quality assessment — The mission of the Joint Effort Group in VQEG

Since 1997, the Video Quality Experts Group (VQEG) has been active in the field of subjective and objective video quality assessment. The group has validated competitive quality metrics throughout several projects. Each of these projects requires mandatory actions such as creating a testplan and obtaining databases consisting of degraded video sequences with corresponding subjective quality ratings. Recently, VQEG started a new open initiative, the Joint Effort Group (JEG), for encouraging joint collaboration on all mandatory actions needed to validate video quality metrics. Within the JEG, effort is made to advance the field of both subjective and objective video quality measurement by providing proper software tools and subjective databases to the community. One of the subprojects of the JEG is the joint development of a hybrid H.264/AVC objective quality metric. In this paper, we introduce the JEG and provide an overview of the different ongoing activities within this newly started group.

Quality Assessment for a Licence Plate Recognition Task Based on a Video Streamed in Limited Networking Conditions

Video transmission and analysis is often utilised in applications outside of the entertainment sector, and generally speaking this class of video is used to perform a specific task. Examples of these applications are security and public safety. The Quality of Experience (QoE) concept for video content used for entertainment differs significantly from the QoE of surveillance video used for recognition tasks. This is because, in the latter case, the subjective satisfaction of the user depends on achieving a given functionality. Moreover, such sequences have to be compressed significantly because the monitored place has to be seen on-line and it can be connected by an error prone wireless connection. Recognising the growing importance of video in delivering a range of public safety services, we focused on developing critical quality thresholds in licence plate recognition tasks based on videos streamed in constrained networking conditions.

Video Quality Assessment: Subjective testing of entertainment scenes

This article describes how to perform a video quality subjective test. For companies, these tests can greatly facilitate video product development; for universities, removing perceived barriers to conducting such tests allows expanded research opportunities. This tutorial assumes no prior knowledge and focuses on proven techniques. (Certain commercial equipment, materials, and/or programs are identified in this article to adequately specify the experimental procedure. In no case does such identification imply recommendation or endorsement by the National Telecommunications and Information Administration, nor does it imply that the program or equipment identified is necessarily the best available for this application.)

An Accurate Sampling Scheme for Detecting SYN Flooding Attacks and Portscans

In this paper, we propose an accurate sampling scheme for defeating SYN flooding attacks as well as TCP portscan activity. The scheme examines TCP segments to find at least one of multiple ACK segments coming from the server. The method is simple and scalable, because it achieves good detection performance with false positive rate close to zero even for very low sampling rates. Our trace-based simulations show that the effectiveness of the proposed scheme only relies on the sampling rate regardless on the sampling method.