Raimund Schatz

Quantification of YouTube QoE via Crowdsourcing

This paper addresses the challenge of assessing and modeling Quality of Experience (QoE) for online video services that are based on TCP-streaming. We present a dedicated QoE model for You Tube that takes into account the key influence factors (such as stalling events caused by network bottlenecks) that shape quality perception of this service. As second contribution, we propose a generic subjective QoE assessment methodology for multimedia applications (like online video) that is based on crowd sourcing - a highly cost-efficient, fast and flexible way of conducting user experiments. We demonstrate how our approach successfully leverages the inherent strengths of crowd sourcing while addressing critical aspects such as the reliability of the experimental data obtained. Our results suggest that, crowd sourcing is a highly effective QoE assessment method not only for online video, but also for a wide range of other current and future Internet applications.

The Logarithmic Nature of QoE and the Role of the Weber-Fechner Law in QoE Assessment

The Weber-Fechner Law (WFL) is an important principle in psychophysics which describes the relationship be- tween the magnitude of a physical stimulus and its perceived intensity. With the sensory system of the human body, in many cases this dependency turns out to be of logarithmic nature. Re- cent quantitative QoE research shows that in several different scenarios a similar logarithmic relationship can be observed be- tween the size of a certain QoS parameter of the communication system and the resulting QoE on the user side as observed during appropriate user trials. In this paper, we discuss this surprising link in more detail. After a brief survey on the background of the WFL, we review its basic implications with respect to related work on QoE assessment for VoIP, most notably the recently published IQX hypothesis, before we present results of our own trials on QoE assessment for mobile broadband scenarios which confirm this dependency also for data services. Finally, we point out some conclusions and directions for further research.

Initial delay vs. interruptions: Between the devil and the deep blue sea

End user quality perception in the context of Internet applications is often characterized by waiting times before service consumption as well as interruptions during service consumption. In particular in case of bad network conditions, network and service providers have to trade off between these two impairment types, i.e. between the devil and the deep blue sea. In this paper we investigate this tradeoff in order to guide the design and development of Internet applications and network management approaches. The contribution of this paper is twofold. Firstly, we quantify the impact of initial delays on the user perceived Quality of Experience (QoE) for different application scenarios by means of subjective laboratory and crowdsourcing studies. We show that QoE for a given waiting time strongly depends on the concrete application at hand but that rating diversity remains fairly application-invariant. Secondly, using the example of YouTube video streaming we compare the influence of initial delays and interruptions (stallings) during watching. Our results demonstrate that users are extremely sensitive to interruptions and that services should be designed accordingly e.g. by increasing initial delay for prebuffering to overcome lack of resources.

Challenges of QoE management for cloud applications

Cloud computing is currently gaining enormous momentum due to a number of promised benefits: ease of use in terms of deployment, administration, and maintenance, along with high scalability and flexibility to create new services. However, as more personal and business applications migrate to the cloud, service quality will become an important differentiator between providers. In particular, quality of experience as perceived by users has the potential to become the guiding paradigm for managing quality in the cloud. In this article, we discuss technical challenges emerging from shifting services to the cloud, as well as how this shift impacts QoE and QoE management. Thereby, a particular focus is on multimedia cloud applications. Together with a novel QoE-based classification scheme of cloud applications, these challenges drive the research agenda on QoE management for cloud applications.

Logarithmic laws in service quality perception: where microeconomics meets psychophysics and quality of experience

Utility functions, describing the value of a good or a resource from an end user’s point of view, are widely used as an important ingredient for all sorts of microeconomic models. In the context of resource allocation in communication networks, a logarithmic version of utility usually serves as the standard example due to its simplicity and mathematical tractability. In this paper we argue that indeed there are much more (and better) reasons to consider logarithmic utilities as really paradigmatic, at least when it comes to characterizing user experience with specific telecommunication services. We justify this claim with the help of recent results from Quality of Experience (QoE) research, and demonstrate that, especially for Voice-over-IP and mobile broadband scenarios, there is increasing evidence that user experience and satisfaction follows logarithmic laws. Finally, we go even one step further and put these results into the broader context of the Weber-Fechner Law, a key principle in psychophysics describing the general relationship between the magnitude of a physical stimulus and its perceived intensity within the human sensory system.

Waiting times in quality of experience for web based services

A considerable share of applications such as web or e-mail browsing, online picture viewing and file downloads imply waiting times for their users, which is due to the turn-taking of information requests by the user and correspoding response times until each request is fulfilled. Thus, end-user quality perception in the context of interactive data services is dominated by waiting times; the longer the latter, the less satisfied the user becomes. As opposed to heavily researched multimedia experience, perception of waiting times is still not strongly explored in the context of Quality of Experience (QoE). This tutorial will contribute to closing this gap. In its first part, it addresses perception principles and discusses their applicability towards fundamental relationships between waiting times and resulting QoE. It then investigates to which extent the same relationships can also be used to describe QoE for more complex services such as web browsing. Finally, it discusses applications where waiting times determine QoE, amongst other factors. For example, the past shift from UDP media streaming to TCP media streaming (e.g. youtube.com) has extended the relevance of waiting times also to the domain of online video services. In particular, user-perceived quality suffers from initial delays when applications are launched, as well as from freezes during the delivery of the stream. These aspects, which have to be traded against each other to some extent, will be discussed mainly for HTTP video streaming in the last part of this tutorial.

SOS: The MOS is not enough!

When it comes to analysis and interpretation of the results of subjective QoE studies, one often witnesses a lack of attention to the diversity in subjective user ratings. In extreme cases, solely Mean Opinion Scores (MOS) are reported, causing the loss of important information on the user rating diversity. In this paper, we emphasize the importance of considering the Standard deviation of Opinion Scores (SOS) and analyze important characteristics of this measure. As a result, we formulate the SOS hypothesis which postulates a square relationship between the MOS and the SOS. We demonstrate the validity and applicability of the SOS hypothesis for a wide range of studies. The main benefit of the SOS hypothesis is that it allows for a compact, yet still comprehensive statistical summary of subjective user tests. Furthermore, it supports checking the reliability of test result data sets as well as their comparability across different QoE studies.

“Time is bandwidth”? Narrowing the gap between subjective time perception and Quality of Experience

Over the last couple of years, the scope of Quality of Experience (QoE) research has been constantly extended, most recently to the field of Web QoE in the context of HTTP-based applications. In this paper, we address the question whether it is sufficient to reduce typical Web QoE assessment scenarios to the temporal aspects of waiting for task completion, which would allow to attribute the resulting logarithmic laws to well-known psychological insights on human time perception. We demonstrate that while this attribution is valid for simple waiting tasks which are typical for simple data services like e.g. file downloads, the case of interactive web browsing is much more complex. We show that this is not only because technical issues prevent bandwidth and download time from being directly correlated with each other in a simple manner, but also because user perceived web page load times strongly deviate from technical page load times. Consequently, existing approaches towards assessment and modeling of web browsing QoE have to be critically reviewed and redesigned.

Internet video delivery in youtube: from traffic measurements to quality of experience

This chapter investigates HTTP video streaming over the Internet for the YouTube platform. YouTube is used as concrete example and case study for video delivery over the Internet, since it is not only the most popular online video platform, but also generates a large share of traffic on todays Internet. We will describe the YouTube infrastructure as well as the underlying mechanisms for optimizing content delivery. Such mechanisms include server selection via DNS as well as application-layer traffic management. Furthermore, the impact of delivery via the Internet on the user experienced quality (QoE) of YouTube video streaming is quantified. In this context, different QoE monitoring approaches are qualitatively compared and evaluated in terms of the accuracy of QoE estimation.

From packets to people: quality of experience as a new measurement challenge

Over the course of the last decade, the concept of Quality of Experience (QoE) has gained strong momentum, both from an academic research and an industry perspective. Being linked very closely to the subjective perception of the end user, QoE is supposed to enable a broader, more holistic understanding of the qualitative performance of networked communication systems and thus to complement the traditional, more technology-centric Quality of Service (QoS) perspective. The purpose of this chapter is twofold: firstly, it introduces the reader to QoE by discussing the origins and the evolution of the concept. Secondly, it provides an overview of the current state of the art of QoE research, with focus on work that particularly addresses QoE as a measurement challenge on the technology as well as on the end-user level. This is achieved by surveying the different streams of QoE research that have emerged in the context of Video, Voice and Web services with respect to the following aspects: fundamental relationships and perceptual principles, QoE assessment, modeling and monitoring.