Measuring HTTP/3: Adoption and Performance
Martino Trevisan, Danilo Giordano, Idilio Drago, Ali Safari Khatouni
MMeasuring HTTP/3:Adoption and Performance
Martino Trevisan † , Danilo Giordano † , Idilio Drago ‡ , Ali Safari Khatouni (cid:63) † Politecnico di Torino, ‡ University of Turin, (cid:63)
Shopify first.last@ { polito.it, unito.it, shopify.com } Abstract —The third version of the Hypertext TransferProtocol (HTTP) is currently in its final standardization phaseby the IETF. Besides better security and increased flexibility,it promises benefits in terms of performance. HTTP/3 adopts amore efficient header compression schema and replaces TCPwith QUIC, a transport protocol carried over UDP, originallyproposed by Google and currently under standardization too.Although HTTP/3 early implementations already exist andsome websites announce its support, it has been subject tofew studies.In this work, we provide a first measurement study onHTTP/3. We testify how, during 2020, it has been adoptedby some of the leading Internet companies such as Google,Facebook and Cloudflare. We run a large-scale measurementcampaign toward thousands of websites adopting HTTP/3,aiming at understanding to what extent it achieves betterperformance than HTTP/2. We find that adopting websitesoften host most web page objects on third-party servers, whichsupport only HTTP/2 or even HTTP/1.1. Our experimentsshow that HTTP/3 provides sizable benefits only in scenarioswith high latency or very poor bandwidth. Despite theadoption of QUIC, we do not find benefits in case of highpacket loss, but we observe large diversity across websiteproviders’ infrastructures.
Index Terms —HTTP/3 ; Performance ; Measurements
I. I
NTRODUCTION
The Hypertext Transfer Protocol (HTTP) is the king ofweb protocols and is used to access a plethora of servicesavailable on the Internet, from websites to social networksand collaborative platforms. HTTP was born in the early90s, and its first version 1.1 was standardized in 1997 [1].For more than a decade, this version remained unchanged.Only in 2014, HTTP/2 [2] was proposed, with substantialchanges in the framing mechanisms. HTTP/3 is the thirdversion of HTTP and is currently in the final standard-ization phase at the IETF [3]. It promises performancebenefits and security improvements compared to version2. As a major change, HTTP/3 replaces TCP as transportlayer in favor of QUIC, a UDP-based protocol originallyproposed by Google and currently being standardized bythe IETF [4]. Furthermore, it introduces a more effectiveheader compression mechanism and exploits TLS 1.3 [5]or higher to improve the level of security.HTTP/3 is expected to supplant HTTP/2 in the nextyears, and some of the leading Internet companies alreadyannounced its support during 2020 (e.g., the CloudFlareCDN and Facebook ). However, currently neither the https://blog.cloudflare.com/http3-the-past-present-and-future/ https://engineering.fb.com/2020/10/21/networking-traffic/how-facebook-is-bringing-quic-to-billions/ real state of deployment nor the performance benefits ofHTTP/3 have been measured yet.In this paper, we fill this gap by running a large-scalemeasurement study. Firstly, we collect data to understand towhat extent the web ecosystem already adopted HTTP/3.Secondly, we run an experimental campaign to measurethe benefits introduced by HTTP/3 on the users’ Qualityof Experience (QoE), and how HTTP/3 helps in differentnetwork conditions.Using the open-source HTTPArchive Dataset, we findthousands of websites supporting HTTP/3, most of themowned by a handful of Internet hyper-giants, i.e., Facebook,Google, and Cloudflare. We automatically visit the HTTP/3websites under diverse network conditions to measure theperformance benefits in terms of QoE-related metrics. Wevisit
14 707 websites while emulating artificial latency,packet loss, and limiting the bandwidth. In total, werun
735 350 visits over a period of one month. We findthat HTTP/3 benefits emerge only on particular networkconditions and strongly differ across website providers. Ourkey findings are the following: • Google, Facebook and Cloudflare are the earlyadopters of HTTP/3, representing almost the totalityof currently HTTP/3 enabled websites. • For most of them, the majority of web page objectsare still hosted on non-HTTP/3 third-party servers. • We observe sizable performance benefits only in sce-narios with high latency or poor bandwidth. • The performance speed-up largely depends on theinfrastructure hosting the website. • Websites requiring few connections to load are thosebenefiting the most.The remainder of the paper is organized as follows:Section II describes HTTP/3 and illustrates related work.Section III presents the datasets we use and how we collectthem, and Section IV illustrates our results on HTTP/3adoption and performance. Finally, Section V states thelimitations of our measurements and poses the basis forfuture work, while Section VI concludes the paper.II. B
ACKGROUND
A. HTTP/3
The HTTP/3 protocol is the third version of the well-known Hypertext Transfer Protocol, born in the 90s totransfer multimedia content and hyper-textual documentsover the nascent Internet. With its version 1.1, it hasbeen the king of web protocols for more than 20 years, a r X i v : . [ c s . N I] F e b P HTTP/2 HTTP/3TLS
TCP QUIC (+TLS 1.3)
UDPHTTP/1.1TLS (optional)
TCP
Fig. 1: Protocol stack for different HTTP versions.superseded only by its second version HTTP/2 in 2014.HTTP/2 implemented several novel features, especiallyto improve how the data is framed and transported. Itpromised to make the web faster even if its benefits havenot universally accepted [6], [7].HTTP/3 is the third version of HTTP. It is currentlyin the final standardization phase, reaching the 34th draftversion [3] and making it stable and usable for real de-ployments. The main improvements from version 2 includemore efficient header compression, advanced security fea-tures based on TLS 1.3, and, especially, the use of QUICat the transport layer. The resulting protocol stack is thusheavily modified, as we show in Figure 1. QUIC, initiallydeveloped by Google, is a transport protocol based on UDPand is currently in the standardization phase too [4]. QUICrevisits TCP, moving congestion control in user space andallowing faster handshakes. Moreover, it solves the long-standing issue of head-of-line blocking, allowing multipleindependent streams within the same connection. Indeed,QUIC allows independent retransmission for sub-streamsand decouples it from congestion control. This operationis expected to improve users’ QoE with faster websiteresponsiveness, especially in scenarios with poor networkconditions. HTTP/3 also mandates the use TLS 1.3 [5],directly incorporated at the QUIC layer. Besides the dep-recation of older cipher suites, it allows 1-RTT handshakesand 0-RTT resumption, further reducing session setup time.
B. Related Work
Given its recent conception, few works already targetedHTTP/3. Saif et al. [8] run experiments controlling bothclient and server accessing a single web page. They studythe effect of delay, packet loss and throughput but do notmeasure any major impact on performance. In contrastto them, we run a large-scale measurement campaign,controlling only the client and targeting thousands ofHTTP/3 websites residing on their original servers. Marx etal. [9] compare HTTP/3 implementations, finding alarge heterogeneity in how congestion control, prioritiza-tion and packetization work. They only run single filedownloads, but their results call for extensive in-the-wildmeasurements. Cloudflare benchmarks its own draft 27HTTP/3 implementation in [10], finding it to be − %slower than HTTP/2. However, their experiments are lim-ited to the blog.cloudflare.com website. Guillen etal. [11] proposed a control algorithm for adaptive streamingtailored for HTTP/3. TABLE I: Description of the employed datasets. Dataset Visits Goal
HTTPArchive
53 107 185
HTTP/3 Adoption
BrowserTime
735 350
HTTP/3 PerformanceGoogle firstly proposed QUIC in 2012 and, as such, hasbeen the subject of many studies. For example, Wolsing etal. [12] show that QUIC still outperforms TCP thanks to thefast connection setup. Manzoor et al. [13] show how QUICperforms worse than TCP in Wireless Mesh Networks.Wolsing et al. [14] found that QUIC reduces the overallpage retrieval time compared with standard HTTP. Kakhi etal. [15] run a large-scale measurement campaign on QUIC,finding that it outperforms TCP in most cases. However,these works target older Google’s QUIC versions, while thecurrent standard proposed at the IETF has made significantprogress [16]. Moreover, they focus uniquely on QUIC asthe transport layer, neglecting the improvement introducedin HTTP/3 and TLS 1.3, that we aim at measuring in thiswork. III. D
ATA C OLLECTION
We rely on two datasets to study (i) the adoptionof HTTP/3 and (ii) its performance on diverse networkconditions. We summarize them in Table I.
A. HTTPArchive
We study the adoption of HTTP/3 using theHTTPArchive, an open dataset available online. Thedataset contains metadata coming from visits to a listof more than M URLs provided by the Chrome UserExperience Report. . The list of URL is built using thenavigation data of real Chrome users and contains arepresentative view of the most popular website and webservices accessed worldwide. Each month, all URLs arevisited using the Google Chrome browser from a US datacenter, and the resulting navigation data is made public.For each visit, the dataset contains information about thepage performance and characteristics as well as details onall the HTTP transactions in the HAR format , includingrequest and response headers.Fundamental for our analyses, the details of HTTPresponses indicate the eventual Alt-Svc header, whichis used by servers to announce support to HTTP/3. Bysetting the
Alt-Svc header, the server has the possibilityto inform the client to make subsequent connections usingHTTP/3 and may specify the support to specific draftversions (e.g., or ).We download the HTTPArchive dataset starting fromNovember , when we observe the first websites offer-ing support to HTTP/3. We use the data to study the trend https://httparchive.org/,lastvisit:February4th2021 https://developers.google.com/web/tools/chrome-user-experience-report HTTPArchive used to adopt the Alexa top-1M website list, butswitched to the Chrome User Experience Report when Alexa discontinuedthe rank in July 2018. ABLE II: Impairment levels for controlled experiments.
Parameter Impairment levels
Latency [ms] Native, 50, 100, 200Loss [%] Native, 1, 2, 5Bandwidth [Mbit/s] Native, 1, 2, 5of HTTP/3 adoption. The data sum up to . TB, and westore them on a medium-sized Hadoop cluster for scalableprocessing. Since we are interested in studying the adoptionof HTTP/3 on websites , we discard all visits to internalpages (less than half of the totality) and keep only visitsto home pages. We refer to this dataset as
HTTPArchive . B. Controlled Experiments
We use the most recent snapshot at the time of writing(December 2020) to build the list of websites currentlysupporting HTTP/3. We find
14 707 websites announcingsupport to HTTP/3. Next, we visit them with all three ver-sions of HTTP (1.1, 2, and 3) to quantify the performanceimprovements. To this end, we rely on BrowserTime, adockerized tool to run automatic visits to web pages with alarge set of configurable parameters. We use BrowserTimeto instrument Google Chrome to run visits by using aspecific HTTP version. We achieve this using the commandline switches of Google Chrome. Important for our goal,Google Chrome allows specifying a set of domains to becontacted with HTTP/3 on the first visit, i.e., without priorindication via
Alt-Svc header. We cannot find a similarpossibility for Firefox, and, as such, our experiments arelimited to Chrome.We are interested in studying the impact of HTTP/3under different network conditions and, as such, we run ourmeasurements enforcing different network configurations .Each network configuration is defined by three parameters:(i) extra-latency, (ii) packet loss, and (iii) bandwidth limit.To enforce impairments during the visits, we rely on thewell-known Linux tc tool. For each parameter, we use different impairment levels, reported in Table II. Incase of latency, we impose it on the uplink, while packetloss and bandwidth limit are enforced on both up anddownlink. For each network configuration, we visit eachwebsite (i) enabling uniquely HTTP/1.1, (ii) with HTTP/1.1and HTTP/2, and (iii) enabling HTTP/3 too. All visits tothe same website are run consecutively and repeated times to get reliable results. Hence, we visit each website ∗ ∗ times.BrowserTime collects several statistics for each visit,including details on all HTTP transactions as well asperformance metrics. We track two of these that have beenshown to be correlated with users’ QoE [17]: • onLoad : The time at which the browser fires the onLoad event – i.e., when all elements of the page,including images, style sheets and scripts, have beendownloaded and parsed; • SpeedIndex : Proposed by Google , represents the https://web.dev/speed-index/ / / / / / / / / / / / / / / / W e b s it e s [ % ] Fig. 2: Websites announcing support to HTTP/3, separatelyby IETF draft. c l o u d fl a r e G S E N o t s p e c i fi e d n g i n x G o o g l e F r o n t e n d g w s C a d d y A p a c h e F l yg t r a n s l a t e O t h e r W e b s it e s Fig. 3:
Server in HTTP response (December 2020).time at which visible portions of the page are dis-played. It is computed by capturing the video of thebrowser screen and tracking the visual progress of thepage during rendering.We run our experiments using two high-end serversconnected to the Internet via Gbit/s Ethernet cable andlocated in our university campus premises. In total, werun
735 350 visits over a period of one month. The visitmetadata account for
GB, and we refer to this datasetwith
BrowserTime . IV. R
ESULTS
We first provide an overview of the HTTP/3 adoptionoffering a historical perspective and analyzing the currentstatus. Then, we study how HTTP/3 affects QoE-relatedperformance metrics by analyzing our experiments undercontrolled network conditions.
A. Adoption
We first study to what extent HTTP/3 has been adoptedsince its first proposal. To this end, we profit from the
HTTPArchive dataset, offering a historical perspective. Thefirst IETF draft is dated January 2017, but we observe thefirst websites adopting HTTP/3 only in late 2019. Sincethen, the number of enabled websites started to grow, andFigure 2 shows the trend for the last months of 2019 andthe entire 2020. Looking at the
Alt-Svc header, we can
20 40 60 80 100Objects/volume served on HTTP/3 [%]0 . . . . . . C D F ( W e b s it e s ) ObjectsVolume
Fig. 4: Share of objects/volume served on HTTP/3 onenabled websites.observe the HTTP/3 draft version supported by the server,shown with different colors in the figure. In case a websiteoffers more than one version, we considered the earliest forthe figure. In the first four months, the number of enabledwebsites increased slowly, reaching 0.7 % of the totality,and we notice that, at that time, only Google and Facebookused to offer HTTP/3 on their websites. In February 2020,the number of enabled websites exploded. This is due toCloudFlare, which enabled HTTP/3 on most of the websitesit hosts. The share of enabled websites reaches and passes %, with its maximum in October 2020, with . % and k websites. In November, the number of websitessuddenly dropped to less than . % ( in absoluteterms). We observe that this was caused by CloudFlaresuspending support to HTTP/3 due to performance issues,as declared online. On December, CloudFlare re-enabledHTTP/3 on a subset of websites. On that date,
14 707 websites were announcing support to HTTP/3, and, in thefollowing, we consider these for our analyses.The majority of the
14 707
HTTP/3-enabled websitesbelong to large companies running their own web serverapplications. We breakdown this in Figure 3, which in-dicates the most popular servers as indicated on theHTTP Server header. We extrapolate this figure usingagain the
HTTPArchive dataset, containing details on allHTTP headers of responses. CloudFlare, as expected, hostsmost of the HTTP/3-enabled websites (notice the log y -scale). Google is in the second position, with GSE (GoogleServlet Engine), Google Frontend and GWS (Google WebServer). Indeed, GSE is used on the Blogspot platform,represented by websites in our list. For websites,there is no server indication on HTTP responses, andwe find that of these belong to the Facebook company– e.g., facebook.com and instagram.com domains– revealing the social network is a prominent exampleof an HTTP/3 early adopter. The remaining websitesrun popular open-source servers (nginx, Apache) or morepeculiar HTTP ones (e.g., Caddy), which offer HTTP/3support in their earlier versions. Interestingly, we do notfind any websites whose server header or IP address https://community.cloudflare.com/t/community-tip-http-3-with-quic/117551, visited on 2/20/2021. Cloudflare (12455)
Facebook (445)
Google (1094)
Other (731) H TT P / S h a r e [ % ] ObjectsVolume
Fig. 5: Share of objects/volume served on HTTP/3, sepa-rately by provider.belong to popular CDNs providers such as Akamai or Ama-zon CloudFront, meaning they have not started deployingHTTP/3 yet.Next, we study to what extent objects of enabled web-sites are served using HTTP/3. Indeed, if a website sup-ports HTTP/3, it does not entail that all objects are sentthrough HTTP/3. Objects may lay on external CDNs, cloudproviders or third parties not supporting the same protocol.This is the case, for example, of ads and trackers typicallyhosted on different third-party infrastructures. We now usethe
BrowserTime dataset, which allows us to observe theprotocol used for delivering each object composing the webpage we visit. Again, we focus on the
14 707
HTTP/3-enabled websites. In Figure 4, we consider all visits runwith HTTP/3 enabled, and, for each, we compute the shareof objects served over HTTP/3. As each website is accessedmultiple times, we average the values across visits. Clearly,at least the main HTML document is always sent withHTTP/3, but the remaining objects of the page may not.The figure shows the distribution of the percentage ofobjects served over HTTP/3 (solid red line) and also depictsthe byte-wise percentage – i.e., weighting each object byits size. We first notice that in % of cases, all objectsare delivered over HTTP/3, meaning that the web pageonly contains elements hosted on HTTP/3 enabled servers.The websites having % or more of objects (volume) onHTTP/3 are ( ) % and only ( ) % have less than %. Interestingly, we notice that % of websites stillhave one or more objects retrieved using HTTP/1.1.Next, we dissect the above analysis by provider – i.e.,the company/CDN hosting the website. We obtain it bylooking at the server HTTP header, website name andserver IP address, which allow easy identification. Asdiscussed for Figure 3, we notice that HTTP/3 has beenadopted mostly by (i) Cloudflare CDN, (ii) the Facebookand (iii) Google companies. The remaining websites(i.e., Other) belong mostly to self-hosted websites runningupdated versions of the nginx web server. Figure 5 showsthe share of objects and volume served over HTTP/3,separately by provider. Cloudflare websites tend to be moreheterogeneous, with half of the objects retrieved via non-HTTP/3 servers in median. Moreover, only 24% of theebsites’ volume is served by using HTTP/3. This is likelydue to the provider’s nature; indeed, Cloudflare offers itshosting service to a very large plethora of websites. Thesewebsites may use complex web pages composed of severalthird-party objects stored on external servers not relyingon HTTP/3 yet. Conversely, Facebook and Google showa very different situation. Almost all the websites’ objectsand volumes are served with HTTP/3. Indeed Facebookand Google use their CDNs mostly to offer their ownservices. Looking at Google, the long tail of the distributionis due to Blogspot websites in which the creator mayadd content from external sources. Finally, considering theOther category, almost all the objects and volumes areserved by using HTTP/3. These websites tend to be simple,composed of a few objects stored in the same self-hostedservers together with the main HTML document.
B. Performance
We now study the impact of HTTP/3 on web page per-formance. To this end, we use the
BrowserTime dataset,in which the
14 707 have been visited multiple times underdifferent emulated network conditions. Using tc-netem ,we enforce extra latency, packet loss and limit bandwidth.We then contrast page QoE-related performance indicators(onLoad and SpeedIndex), (i) showing their absolute valueand (ii) computing the speed-up . Given a website and agiven network scenario, we obtain the speed-up as therelative deviation of the metric when using HTTP/3 ( h )instead of HTTP/2 ( h ). As we always run 5 visits for eachcase, we consider the median value. As such, the speed-upfor a website w in scenario s is computed as follows:speed-up ( w, s ) = median ( w, s, h − median ( w, s, h max ( median ( w, s, h , median ( w, s, h By definition, speed-up ( w, s ) is bound in [ − , andis negative when a website loads faster under HTTP/3,positive otherwise. We compute it for both onLoad andSpeedIndex.We illustrate how the metric values vary when imposingdifferent network impairments, focusing firstly on extra la-tency in Figure 6. Using boxplots, we show the distributionof onLoad (top) and SpeedIndex (bottom), separately byHTTP version (colored boxes). The boxes span from thefirst to the third quartile, while whiskers report the 10 th andthe 90 th percentiles; black strokes represent the median.When no extra latency is imposed ( native case), we observethat onLoad time is in median around s, while SpeedIndexaround s, without significant differences across HTTPversions. When adding extra latency, the websites loadslower as more time is needed to download the pageobjects, requiring in median 6 seconds with ms ofadditional latency. Not shown here for brevity, also packetloss and limited bandwidth cause similar degradation ofperformance indicators. In the following, we compare towhat extent HTTP/3 provides benefits in such scenarios.Figure 6 shows that HTTP/1.1 has the worst performancewith high latency, while HTTP/3 shows the greatest ben-efits. Considering additional latency of ms, websites P a g e L o a d T i m e [ s ] HTTP/1.1HTTP/2HTTP/3
Native 50 100 200Extra Latency [ms]0123456 S p ee d I nd e x [ s ] Fig. 6: onLoad (top) and SpeedIndex (bottom) with extralatency, separately for HTTP/1.1, 2 and 3.onLoad in median in . , . and . s with HTTP version1.1, 2 and 3, respectively.To better catch differences between HTTP/3 and 2, wenow study the speed-up in Figure 7, where we show thedistribution over the
14 707 websites for both onLoad (toprow) and SpeedIndex (bottom row). The three columnsrefer to scenarios with additional latency, limited band-width and packet loss, respectively. The solid red linesrepresent the native case – i.e., without imposing anynetwork impairment. Dashed lines represent scenarios withnetwork impairments, as indicated in Table II.Starting from latency, we confirm what already emergedfrom Figure 6. When latency is high, HTTP/3 gives sizablebenefits compared to HTTP/2. In the native case, weobserve no benefit. Looking at the solid red lines, we noticethat approximately in 50 % of cases websites load fasterwith HTTP/3 and in the remaining cases slower. If weimpose extra latency of ms, 69 (74) % of websites havelower onLoad time (SpeedIndex), meaning that they loadfaster. They increase to 76 (81) % with ms latency.With ms, they reach 81 (87) %, and the median speed-up is 0.08 (0.12). In a few words, HTTP/3 shows sizablebenefits in case of high latency.Focusing on experiments with bandwidth limitation (cen-tral plots in Figure 7), different considerations hold. Weobserve sizable benefits only for onLoad time with thebandwith limited to just Mbit/s, where % of websitesload faster with HTTP/3. Notice that this benefit cannotbe introduced by indirect higher latency due to queuingdelay (also called bufferbloat), as we limit the machines’queue to as little as 32 KB. In other cases, no cleartrend emerges, but we only notice a larger variability . . . . . . C D F Native50 ms100 ms200 ms-0.5 -0.3 -0.1 0 0.1 0.3 0.5Speed-up (SpeedIndex)0 . . . . . . C D F (a) Latency. -0.5 -0.3 -0.1 0 0.1 0.3 0.5Speed-up (onLoad)Native5 Mbit/s2 Mbit/s1 Mbit/s-0.5 -0.3 -0.1 0 0.1 0.3 0.5Speed-up (SpeedIndex) (b) Bandwidth. -0.5 -0.3 -0.1 0 0.1 0.3 0.5Speed-up (onLoad)Native1%2%5%-0.5 -0.3 -0.1 0 0.1 0.3 0.5Speed-up (SpeedIndex) (c) Packet loss. Fig. 7: Speed-up of HTTP/3 on different scenarios. onLoad (top) and SpeedIndex (bottom).of the speed-up measure introduced by the constrainedsetup. For example, in case of SpeedIndex, , , % websites load faster with HTTP/3 with , and Mbit/s bandwidth, respectively. Similar considerations holdfor packet loss (right-most plots in Figure 7). Despite alarger variability, we cannot identify any obvious trend,and the speed-up values are equally distributed above andbelow . In summary, we do not testify any performancebenefit of HTTP/3 in scenarios with high packet loss. Weonly observe improvements with very poor bandwidth andlimited to onLoad. C. Dissecting performance speed-up
We now focus on the performance speed-up as definedpreviously and discuss how it is impacted by (i) theprovider hosting the website and (ii) characteristics ofthe web pages and load process. As we observed sizableperformance benefit for HTTP/3 only in cases of highlatency or (very) poor bandwidth, we restrict our analysesto those cases.
1) Performance by Provider:
Figure 8 shows the distribution of performance speed-upfor onLoad, separately by provider. We focus on scenarioswith ms extra latency and Mbit/s bandwidth limit.Similarly to Figure 5, we study Cloudflare, Facebookand Google, being the companies behind the majority ofthe HTTP/3 enabled websites currently. We observe thespeed-up largely differs by provider. Focusing on latency(Figure 8a), Facebook websites show the highest perfor-mance gain ( − . in median), represented in the figurewith the blue dashed line. Moreover, % of websites are loaded faster with HTTP/3 than with HTTP/2. Cloudflare(red solid line) shows the smallest benefits, with only % of websites loading faster. Google and the remainingwebsites sit in the middle. Similar considerations hold forSpeedIndex, not shown here for brevity.With limited bandwidth (Figure 8b), we observe a verydifferent situation. Here, Facebook has in general worstperformance with HTTP/3 with % of websites loadingfaster with HTTP/2. Conversely, Google (green dashedline) shows the best improvements, with median speed-up − . and % of websites loading faster with HTTP/3.Cloudflare and the remaining websites exhibit no cleartrend with roughly half of the websites loading faster withHTTP/3. Considering the scenario with artificial packetloss, we do not observe differences across providers, asthey all approximately behave as shown in Figure 7c.
2) The impact of page characteristics:
Now we focus on how web page characteristics impactperformance. To this end, we compute various metricsdescribing the web page load process and contrast themto understand they relationship to the HTTP/3 speed-up.For each visit to the
14 707 websites in our dataset, in ad-dition to the HTTP/3 speed-up, we compute the followingmetrics: • Number of connections issued by the browser to loadthe website when using HTTP/3. • Number of domains contacted while loading the webpage, thus including third-pary domains). • Share of objects on the largest connection : tomeasure to what extent the objects composing theweb page are spit into many domains, we compute . − . − . − . − . . . . . . . . . . . . . C D F CloudflareFacebookGoogleOther (a) Latency. − . − . − . − . − . . . . . . . . . . . . . C D F CloudflareFacebookGoogleOther (b) Bandwidth.
Fig. 8: onLoad speed-up by website provider for scenarioswith extra-latency and bandwidth limit.the share of objects carried over the connection wheremost objects have been requested, divided by the totalnumber of objects. Remind that the best practicesof HTTP/3 recommend avoiding domain sharding toincrease performance. • Share of objects served on HTTP/3 : our goal isto understand how the fraction of the website objectsserved on HTTP/3 (shown in Figure 4) has an impacton the speed-up. • Page Size: to breakdown performance for small andlarge web pages.In Figure 9, we compare the distribution of the aforemen-tioned metrics, grouping websites based on classes definedby the onLoad speed-up: • H3 Faster : websites loading faster with HTTP/3, i.e.,onLoad speed-up < − . . • H3 ≈ H2 : websites having a similar loadingtime in HTTP/2 and HTTP/3, i.e., onLoad speed-up ∈ [ − . , . . • H2 Faster : websites loading faster with HTTP/2, i.e.,onLoad speed-up > . .In the figure, boxes with different colors represent thesethree classes. The y -axis represents the metrics normalizedby scaling to unit variance to ease the visualization andthe comparison. Again, we study the scenarios with extralatency (top row) and limited bandwidth (bottom row) sincethey provide the most interesting insights. In case of extralatency, H3 Faster websites are %, H2 Faster % andH3 ≈ H2 are %. With limited bandwidth, they are %, % and %, respectively. N u m b e r o f c o n n . N u m b e r o f D o m a i n s O b j e c t s o n l a r g e s t c o n n . H T T P / s h a r e P a g e s i z e N o r m a li ze d V a l u e H3 FasterH3 ≈ H2H2 Faster (a) Latency. N u m b e r o f c o n n . N u m b e r o f D o m a i n s O b j e c t s o n l a r g e s t c o n n . H T T P / s h a r e P a g e s i z e N o r m a li ze d V a l u e H3 FasterH3 ≈ H2H2 Faster (b) Bandwidth.
Fig. 9: Visit characteristics vs. HTTP/3 speed-up class(normalized values).We first focus on the left-most box group of Fig-ure 9, showing the (normalized) number of connectionsthe browser issued to load the web page. Green boxeshint that websites issuing fewer connections (smaller metricvalues) are faster with HTTP/3 than in HTTP/2. This truein both scenario, low latency and poor bandwidth. Similarconsiderations hold if we focus on the second box group,representing the number of contacted domains. Indeed, wenotice that the number of connections per website and thenumber of contacted domains are . -correlated. The thirdbox group offers a similar perspective, measuring how webpage objects are split on multiple connections/domains.The websites benefiting the most from HTTP/3 are thosewhich tend to mass objects on a single connection – seethe highest position of the green boxes, meaning moreobjects are on a single connection. This is very clearwith limited bandwidth (Figure 9b), rather than with highlatency (Figure 9a).Serving most objects with HTTP/3 (rather than withHTTP/2) has a positive impact too, as we notice from thefourth box group in Figure 9. Again, this is evident espe-cially with bandwidth limit (Figure 9b), while with extralatency (Figure 9a) it is hard to find a clear trend. Finally,interesting is the case of the web page size (last box group).In scenarios with high latency, the websites benefiting fromHTTP/3 are small, while large ones typically perform betterwith HTTP/2. When bandwidth is scarce, we observe anopposite picture: even if moderately, website loading fasterith HTTP/3 are the large ones.In summary, websites taking benefits from HTTP/3are those limiting the number of connections and third-party domains, and fully adopting HTTP/3 on all webpage objects. Page size has diverse implications dependingon the network conditions. These considerations hold inscenarios with high latency or limited bandwidth, while wedo not observe any clear trend in case of optimal networkconditions or high packet loss, where metric distributionsmostly overlap.V. M EASUREMENT LIMITATIONS AND FUTURE WORK
We dissect the performance of HTTP/3 under diversenetwork conditions, showing the impact of network latencyand bandwidth and differences across website providers.However, we run measurements uniquely using GoogleChrome, as, to the best of our knowledge, it is the onlybrowser that can be instrumented to use HTTP/3 on thefirst connection – i.e., without the need to observe the
Alt-Svc header previously. Moreover, we uniquely visitwebsites with a fresh browser profile with an empty cacheand no pre-existing connections. This limits the scopeof our study as we cannot measure how HTTP/3 affectsperformance on subsequent visits or with a warm HTTPcache.We measure a large set of websites, but we are limitedonly to the subset of those already adopting HTTP/3.Moreover, we include only a fraction of those hosted onthe CloudFlare CDN, as its HTTP/3 support is partiallydisabled at the time of writing this article. Our measure-ments need to be run continuously to thoroughly study howthe web ecosystem will react to HTTP/3 in the near future,adopting the protocol and modifying the web page structureto follow the new guidelines, in particular concerningdomain sharding.Finally, HTTP/3 and QUIC are not yet definitive IETFstandards. Although recent modifications to the IETF draftonly concerned minor protocol features, it will be funda-mental to provide a similar analysis once the final standardversions are approved. Similarly, we did not explore howdifferent endpoint configurations affect how objects aresent, eventually retransmitted and congestion control algo-rithms operate. This calls for fine-grained measurements,better suited to be run on fully-controlled lab experiments.Moreover, we only explore scenarios with uniform andrandom packet loss, while it is of certain interest to studyother loss schemes (e.g., bursty loss).VI. C
ONCLUSIONS
In this paper, we provided a first study on HTTP/3, mea-suring its adoption and dissecting performance benefits. Wetestified how some of the Internet leading companies starteddeploying it during 2020, although most of the adopterwebsite still host the majority of objects on HTTP/2 third-party servers. With a large-scale measurement campaign,we studied the performance of HTTP/3 under differentnetwork conditions, targeting thousands of websites. Wefound performance benefits emerging in scenarios withhigh latency or poor bandwidth, while in case of high packet loss HTTP/3 and 2 perform roughly the same.Moreover, we found large performance diversity dependingon the infrastructure hosting the website. In general, weobserved that websites taking benefits from HTTP/3 arethose loading objects from a limited set of third-partydomains, thus limiting the number of issued connections.A
CKNOWLEDGEMENTS
This work has been supported by the EU H2020 researchand innovation programme under grant agreement No.644399 (MONROE) and by the SmartData@PoliTO centeron Big Data and Data Science.R
EFERENCES[1] R. T. Fielding, H. Nielsen, J. Mogul, J. Gettys, and T. Berners-Lee,“Hypertext Transfer Protocol – HTTP/1.1.” RFC 2068, Jan. 1997.[2] M. Belshe, R. Peon, and M. Thomson, “Hypertext Transfer ProtocolVersion 2 (HTTP/2).” RFC 7540, May 2015.[3] M. Bishop, “Hypertext Transfer Protocol Version 3 (HTTP/3),”Internet-Draft draft-ietf-quic-http-33, Internet Engineering TaskForce, Feb. 2021. Work in Progress.[4] J. Iyengar and M. Thomson, “QUIC: A UDP-Based Multiplexedand Secure Transport,” Internet-Draft draft-ietf-quic-transport-33,Internet Engineering Task Force, Dec. 2020. Work in Progress.[5] E. Rescorla, “The Transport Layer Security (TLS) Protocol Version1.3.” RFC 8446, Aug. 2018.[6] M. Varvello, K. Schomp, D. Naylor, J. Blackburn, A. Finamore,and K. Papagiannaki, “Is the web http/2 yet?,” in
InternationalConference on Passive and Active Network Measurement , pp. 218–232, Springer, 2016.[7] M. Rajiullah, A. Lutu, A. S. Khatouni, M.-R. Fida, M. Mellia,A. Brunstrom, O. Alay, S. Alfredsson, and V. Mancuso, “Webexperience in mobile networks: Lessons from two million pagevisits,” in
The World Wide Web Conference , pp. 1532–1543, 2019.[8] D. Saif, C.-H. Lung, and A. Matrawy, “An early benchmark ofquality of experience between http/2 and http/3 using lighthouse,” arXiv preprint arXiv:2004.01978 , 2020.[9] R. Marx, J. Herbots, W. Lamotte, and P. Quax, “Same standards,different decisions: A study of quic and http/3 implementation diver-sity,” in
Proceedings of the Workshop on the Evolution, Performance,and Interoperability of QUIC , pp. 14–20, 2020.[10] S. Tellakula, “Comparing HTTP/3 vs. HTTP/2 Performance.” https://blog.cloudflare.com/http-3-vs-http-2/, 2020.[11] L. Guillen, S. Izumi, T. Abe, and T. Suganuma, “Sand/3: Sdn-assisted novel qoe control method for dynamic adaptive streamingover http/3,”
Electronics , vol. 8, no. 8, p. 864, 2019.[12] K. Wolsing, J. R¨uth, K. Wehrle, and O. Hohlfeld, “A performanceperspective on web optimized protocol stacks: Tcp+ tls+ http/2vs. quic,” in
Proceedings of the Applied Networking ResearchWorkshop , pp. 1–7, 2019.[13] J. Manzoor, L. Cerd`a-Alabern, R. Sadre, and I. Drago, “On theperformance of quic over wireless mesh networks,”
Journal ofNetwork and Systems Management , vol. 28, no. 4, pp. 1872–1901,2020.[14] G. Carlucci, L. De Cicco, and S. Mascolo, “Http over udp: An ex-perimental investigation of quic,” in
Proceedings of the 30th AnnualACM Symposium on Applied Computing , SAC ’15, (New York, NY,USA), p. 609–614, Association for Computing Machinery, 2015.[15] A. M. Kakhki, S. Jero, D. Choffnes, C. Nita-Rotaru, and A. Mislove,“Taking a long look at quic: an approach for rigorous evaluation ofrapidly evolving transport protocols,” in
Proceedings of the 2017Internet Measurement Conference , pp. 290–303, 2017.[16] M. Kosek, T. Shreedhar, and V. Bajpai, “Beyond quic v1–a first lookat recent transport layer ietf standardization efforts,” arXiv preprintarXiv:2102.07527 , 2021.[17] D. N. da Hora, A. S. Asrese, V. Christophides, R. Teixeira, andD. Rossi, “Narrowing the gap between qos metrics and web qoe us-ing above-the-fold metrics,” in