Matthias Hirth

Quantification of YouTube QoE via Crowdsourcing

This paper addresses the challenge of assessing and modeling Quality of Experience (QoE) for online video services that are based on TCP-streaming. We present a dedicated QoE model for You Tube that takes into account the key influence factors (such as stalling events caused by network bottlenecks) that shape quality perception of this service. As second contribution, we propose a generic subjective QoE assessment methodology for multimedia applications (like online video) that is based on crowd sourcing - a highly cost-efficient, fast and flexible way of conducting user experiments. We demonstrate how our approach successfully leverages the inherent strengths of crowd sourcing while addressing critical aspects such as the reliability of the experimental data obtained. Our results suggest that, crowd sourcing is a highly effective QoE assessment method not only for online video, but also for a wide range of other current and future Internet applications.

Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms

Abstract Crowdsourcing is becoming more and more important for commercial purposes. With the growth of crowdsourcing platforms like Amazon Mechanical Turk or Microworkers, a huge work force and a large knowledge base can be easily accessed and utilized. But due to the anonymity of the workers, they are encouraged to cheat the employers in order to maximize their income. In this paper, we analyze two widely used crowd-based approaches to validate the submitted work. 1 Both approaches are evaluated with regard to their detection quality, their costs and their applicability to different types of typical crowdsourcing tasks.

Cost-Optimal Validation Mechanisms and Cheat-Detection for Crowdsourcing Platforms

Crowd sourcing is becoming more and more important for commercial purposes. With the growth of crowd sourcing platforms like MTurk or Micro workers, a huge work force and a large knowledge base can be easily accessed and utilized. But due to the anonymity of the workers, they are encouraged to cheat the employers in order to maximize their income. Thus, this paper presents two crowd-based approaches to validate the submitted work. Both approaches are evaluated with regard to their detection quality, their costs and their applicability to different types of typical crowd sourcing tasks.

Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment

The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.

Bridging the gap between eye tracking and crowdsourcing

Visual attention constitutes a very important feature of the human visual system (HVS). Every day when watching videos, images or browsing the Internet, people are confronted with more information than they are able to process, and analyze only part of the information in front of them. In parallel, crowdsourcing has become a particularly hot topic, enabling to scale subjective experiments to a large crowd with diversity in terms of nationalities, social background, age, etc. This paper describes a novel framework with the aim to bridge these two fields, by providing a new way of measurements of users experience in a subjective crowdsourcing experiment. This study goes beyond self-reported methods, and provide a new kind of information for the context of crowdsourcing: visual attention. The results show that it is possible to estimate visual attention, in a non-intrusive manner and without using self-reported methods or specialized equipment, with a precision as high as 14.1% in the horizontal axis and 17.9% in the vertical axis. This accuracy is sufficient for many kinds of measurements that can be efficiently executed only in non-controlled environments..

Measuring one-way delay in wireless mesh networks: an experimental investigation

Wireless Mesh networks are multi-hop networks mostly based on IEEE 802.11 technology and are considered as a viable alternative for providing broadband wireless Internet access. As a consequence, they require support for Quality of Service or advanced mechanisms for selecting Internet gateways. One important required information is the one-way delay between different nodes. In this paper, we have developed, implemented, and evaluated an one-way delay estimation technique for wireless mesh networks which is based on estimating intra node queuing and inter node forwarding delay. An IP-header option field is used to accumulate the per hop delay estimate to provide an end-to-end estimate. We also outline problems with the implementation and compare results with real one-way delays obtained from a 14 node mesh testbed. We show how estimation accuracy depends on network load and provide insights into further improvements.

AquareYoum: Application- and Quality of Experience-Aware Resource Management for YouTube in Wireless Mesh Networks

Das Internet ermoglicht den Zugang zu einer Vielzahl von Diensten wie z.B. YouTube, Skype, Cloud Storage oder IPTV. Die vom Benutzer erfahrene Quality of Experience (QoE) oder Dienstgute ist jedoch oftmals nicht optimal, da die Netze keine Kenntnis uber die Dienste haben deren Daten sie transportieren. Daher werden IPTV Pakete, die in Echtzeit ausgeliefert werden mussen, genauso behandelt wie zu einem Dokument-Upload gehorende weniger zeitkritische Pakete, auch wenn eine Priorisierung von IPTV technisch moglich ware. Werden drahtlose Mesh Netze (WMNs) als Internetzugangsnetze genutzt, ist dieses Problem besonders akut, da Ressourcen knapper sind als in drahtgebundenen Zugangsnetzen. Im Rahmen des BMBF Projektes G-Lab1 entstand daher das Konzept Aquarema (Application and QoE-aware resource management), das ein auf Informationen der Anwendungsschicht reagierendes dynamisches Ressourcenmanagement (RM) fur WMNs vorschlagt [1]. Die Software-Suite AquareYoum ist eine konkrete Realisierung dieses Konzepts. AquareYoum vermeidet eine Unterbrechung eines YouTube-Videos, indem es eine solche vorhersagt und in diesem Fall die Verteilung der WMN-Ressourcen gezielt anpasst. Das Video wird flussig abgespielt und die QoE von YouTube Nutzern in diesem WMN verbessert.

Is affective crowdsourcing reliable

Affective content annotations are typically acquired from subjective manual assessments by experts in supervised laboratory tests. While well manageable, such campaigns are expensive, time-consuming and results may not be generalizable to larger audiences. Crowdsourcing constitutes a promising approach for quickly collecting data with wide demographic scope and reasonable costs. Undeniably, affective crowdsourcing is particularly challenging in the sense that it attempts to collect subjective perceptions from humans with different cultures, languages, knowledge background, etc. In this study we analyze the validity of well-known user affective scales in a crowdsourcing context by comparing results with the ones obtained in laboratory tests. Experimental results demonstrate that pictorial scales possess promising features for affective crowdsourcing.

Increasing the Coverage of Vantage Points in Distributed Active Network Measurements by Crowdsourcing

Internet video constitutes more than half of all consumer traffic. Most of the video traffic is delivered by content delivery networks (CDNs). The huge amount of traffic from video CDNs poses problems to access providers. To understand and monitor the impact of video traffic on access networks and the topology of CDNs, distributed active measurements are needed. Recently used measurement platforms are mainly hosted in National Research and Education Networks (NRENs). However, the view of these platforms on the CDN is very limited, since the coverage of NRENs is low in developing countries. Furthermore, campus networks do not reflect the characteristics of end user access networks. We propose to use crowdsourcing to increase the coverage of vantage points in distributed active network measurements. In this study, we compare measurements of a global CDN conducted in PlanetLab with measurements assigned to workers of a crowdsourcing platform. Thus, the coverage of vantage points and the sampled part of the global video CDN are analyzed. Our results show that the capability of PlanetLab to measure global CDNs is rather low, since the vast majority of requests is directed to the US. By using a crowdsourcing platform we obtain a diverse set of vantage points that reveals more than twice as many autonomous systems deploying video servers.

Wikipedia and its network of authors from a social network perspective

Online social networks (OSNs) become more and more important in todays social and business life. Therefore, considerable effort is put in research to gain a deeper knowledge of the development of these networks and their dynamics. However, most of the existing literature is based on very limited subsets of the network data, which is often filtered by the OSN operator providing the data or biased by the crawling mechanisms used to obtain the data. This makes it difficult to analyze the temporal evolution of OSNs based on complete data. To overcome this issue, we investigate the dynamics of the publicly available collaboration network of the Wikipedia authors as an example for an OSN-like network. In particular, we study the temporal evolution of this network since its beginning and demonstrate that it exhibits prominent similarities to well known social networks such as the small-world phenomenon. This indicates that the insights gained from the analysis of Wikipedias collaboration network might be transferable to social networks in general.