Stitching Inter-Domain Paths over IXPs
Vasileios Kotronis, Rowan Kloti, Matthias Rost, Panagiotis Georgopoulos, Bernhard Ager, Stefan Schmid, Xenofontas Dimitropoulos
SStitching Inter-Domain Paths over IXPs
Vasileios Kotronis Rowan Klöti Matthias Rost Panagiotis Georgopoulos Bernhard Ager Stefan Schmid Xenofontas Dimitropoulos , ETH Zurich, Switzerland TU Berlin, Germany Aalborg University, Denmark Foundation of Research and Technology Hellas (FORTH), Greece
ABSTRACT
Modern Internet applications, from HD video-conferencing to healthmonitoring and remote control of power-plants, pose stringent de-mands on network latency, bandwidth and availability. Centralizedinter-domain routing brokers is an approach to support such applica-tions and provide inter-domain guarantees, enabling new avenuesfor innovation. These entities centralize routing control for mission-critical traffic across domains, working in parallel to BGP. In thiswork, we propose using IXPs as natural points for stitching inter-domain paths under the control of inter-domain routing brokers.To evaluate the potential of this approach, we first map the globalsubstrate of inter-IXP pathlets that IXP members could offer, basedon measurements for 229 IXPs worldwide. We show that usingIXPs as stitching points has two useful properties. Up to 91% ofthe total IPv4 address space can be served by such inter-domainrouting brokers when working in concert with just a handful of largeIXPs and their associated ISP members. Second, path diversity onthe inter-IXP graph increases by up to times, as compared tocurrent BGP valley-free routing. To exploit the rich path diversity,we introduce algorithms that inter-domain routing brokers can useto embed paths, subject to bandwidth and latency constraints. Weshow that our algorithms scale to the sizes of the measured graphsand can serve diverse simulated path request mixes. Our work high-lights a novel direction for SDN innovation across domains, basedon logically centralized control and programmable IXP fabrics.
1. INTRODUCTION
A great success of the Internet is that it has been used in waysthat were never anticipated during its early days. Carrying voicedata and connecting stock exchange markets are just two exam-ples of such use cases. Nothing suggests that this innovation willnot persist in the future. We see though that modern applicationshave increasingly tighter requirements for bandwidth, latency and/oravailability [91]. For example, real-time HD video streaming, tele-music [32], remote control of critical infrastructure, such as powerplants [20], or even telesurgery [51] are emerging or envisioned ap-plications with strict network requirements. Presently, ISPs are ableto provide certain QoS guarantees [81] only in intra-domain settingsbased on technologies such as leased circuits and VPN tunnels, e.g.,over MPLS-TE. However, despite several research and standardiza-tion efforts, providing QoS guarantees at the inter-domain level hasseen very limited success so far [16, 19, 86, 88]. Besides, currentBGP routing can lead to inefficient paths across domains, triangleinequality violations, and long-lasting outages [9, 56, 67].During the last decade, an increasing number of proposals coming Increasingly, traditional telcos like Deutsche Telekom are planningto switch to IP telephony exclusively [2]. from diverse angles advocate inter-domain routing brokers [34, 60,64,82,83,89] as an approach to enable ISPs to cooperate and provideend-to-end (e2e) guarantees. In these schemes, ISPs provide QoS-enabled pathlets [46], which are stitched together by an inter-domainrouting mediator, e.g., a bandwidth broker [83]. Related initiativesare currently explored in the industry [3] and in standardizationbodies, in particular in the context of the PCE (Path ComputationElement) architecture [37, 55, 84].This work visits logically centralized inter-domain mediatorsin light of the evolving Internet ecosystem. Namely, the Internetis becoming denser and more flat [30, 47, 63] because public In-ternet eXchange Points (IXPs) are continuously rising in numberand size [5, 25]. In parallel, the paradigm shift towards networkvirtualization [74] and Software-Defined Networking (SDN) [71]introduces new possibilities in network management and innovation,also in the context of IXPs, e.g., as shown in the Software-DefinedeXchange (SDX) approach [49]. While SDX enables new servicesat individual IXPs, we focus on multi-IXP services.
Contribution 1: Stitching inter-domain paths via IXPs.
Wepropose using IXPs for stitching paths under the control of inter-domain routing brokers. We call such brokers
Control eXchangePoints (CXPs) . The choice of IXPs as switching points exploitstheir rich connectivity, enabling high path diversity and global clientreach with deployment in only a few well-connected IXPs. CXPsenable the utilization of additional path diversity compared to currentBGP-based inter-domain paths, which typically follow valley-freerouting policies [41, 44, 45]. CXP-stitched paths can freely crossmultiple IXPs, yielding new paths that BGP hides. Contribution 2: Mapping the IXP Multigraph.
To evaluatethe potential of CXPs, we map the global Internet substrate for path-let stitching over IXPs. In particular, we outline a novel abstractionof the Internet topology, in which vertices are IXPs and edges are vir-tual links connecting two IXPs over an ISP. We call this abstractionthe
IXP multigraph because two IXPs can be generally connectedwith multiple edges over different ISPs. This abstraction hides theinternal details of an ISP (including the technologies that can beleveraged to provide intra-domain QoS [89]), and serves a cleanseparation of concerns between intra- and inter-domain QoS routingthat is consistent with the status quo. We analyze the member ISPsof 229 IXPs using data from Euro-IX [35] and show that CXPs canservice, e.g., 40 % of the globally announced IPv4 addresses throughonly the 5 largest IXPs. This increases to 91 % if we also considerthe 1-hop customers of the IXP members. Second, we show thatby relaxing valley-free constraints, CXPs can greatly increase pathdiversity by up to 29 times compared to BGP valley-free routing.
Contribution 3: Algorithms.
We present algorithms to effi-ciently exploit the high path diversity observed in the IXP multi- CXPs can generally use any switching point between ISPs. a r X i v : . [ c s . N I] N ov raph. In particular, our algorithms aim at maximizing the numberof concurrently embedded paths, subject to bandwidth and latencyconstraints. We describe online as well as hybrid online-offline al-gorithms which sample feasible paths efficiently (i.e., in polynomialtime). These algorithms achieve different trade-offs between opti-mal acceptance ratios and fast online computation, with the hybridapproach realizing a balance between the two goals by reallocatingpaths in the background based on an optimal offline algorithm. Us-ing simulation, we show that our algorithms scale to the sizes of themeasured graphs and derive insights on which variants should beleveraged to serve diverse requests.CXPs provide a possible avenue for SDN innovation at the inter-domain level. In this context, we investigate both the algorithms thatcan serve as the controller logic of logically centralized inter-domainrouting brokers, operating on IXP multigraphs, and the interestingproperties of this particular data plane. The latter is studied both inspace (incremental deployment at IXPs) and time, as the peeringecosystem evolves over the years. Moreover, we discuss furtherchallenges for future work under the prism of a possible use case.The rest of the paper is structured as follows. Section 2 providesthe background on inter-domain service brokers and the motiva-tion behind our IXP-based approach. Section 3 maps the globalinter-IXP multigraph, based on Euro-IX and PeeringDB data, andcharacterizes its high path diversity and client reach for inter-domainQoS. Section 4 presents algorithms for embedding paths in IXPmultigraphs and Section 5 evaluates these algorithms based on acustom simulator. Section 6 discusses our work under the prism oftelesurgery as a use case, while Section 7 presents related literature.
2. SERVICE BROKERS, IXPS AND CXPS
This section first gives an overview of previous research on cen-tralized path brokers for inter-domain guaranteed services. Second,we discuss why IXPs are suitable locations for deploying the dataplane elements of path brokers. Lastly, we describe in detail theproperties of our IXP-based path brokers, which we call ControlExchange Points (CXP).
Previous research has focused on bandwidth brokers for medi-ating the concatenation of multiple guaranteed bandwidth pathlets(e.g., MINT [83]), or for scaling up the support for guaranteedbandwidth services within an ISP network (e.g., the work of Zhanget al. [89]). Similar initiatives have created bandwidth marketsand commercial brokers, such as Geant’s multi-domain Bandwidth-on-Demand service [3]. Other proposals introduce “route bazaars”between ISPs and end-users [34], where pricing mechanisms andinteractions directly affect path establishment. Routing-as-a-Servicecontrollers [64] have been proposed as potential broker implemen-tations. Others have proposed entirely outsourcing routing controlto inter-domain SDN controllers [60]. Such controllers can dealwith end-to-end path stitching using their bird’s eye view over theparticipating domains; dynamic traffic management applications canoperate on this global view. Centralized routing controller platformsbased on the Path Computation Element (PCE) architecture [37, 84]have been evaluated in the context of QoS routing schemes forhigh capacity optical networks [43]. The initial multi-domain in-tention of PCE was to help coordinate path establishment requests,and to be able to compute an end-to-end path using cooperativeper-domain PCEs. Systems like PCE are highly relevant for theimplementation of brokers and routing controllers, e.g., applied onIP/MPLS domains [82], and are backed up by IETF standardizationefforts [37, 84]. Figure 1: The CXP stitches QoS-enabled e2e paths.
Brokers and controllers for guaranteed e2e services need to exertinter-domain control through programmable data plane elements,such as OpenFlow switches. We call these elements anchors , sincethey “anchor” inter-domain traffic switching to specific locations,decoupled from the traffic management within e.g., ISP domains.The ideal anchor is adjacent to multiple geo-diverse ISPs, is provi-sioned for high bandwidth and availability, and is independent froma single ISP. We observe that IXPs have all these properties and thusprovide ideal starting points for deployment. IXPs are presentlythe hubs of multiple services surpassing their initial goal of purelayer-2 switching fabrics [25]: (i) hosting route servers for ease ofBGP-based peering [79], (ii) mobile peering with 3G providers fortraffic convergence [1], or (iii) the adoption of SDN approaches fornew inter-domain applications [49]—such as application-specificpeering—are just a few examples. They are therefore open to host-ing new services for their members, together with increasing theirpeering base.Modeling IXPs as vertices and inter-IXP pathlets as edges, theresulting topology is a dense multigraph : two IXPs can be connectedvia multiple ISPs. This is quite common because many ISPs arepresent at multiple different IXPs in parallel (cf. Section 3 fordetails). We base our study on this simple yet powerful observation,enabling us to build a novel IXP-centric abstraction of the Internettopology. Endpoints can connect to this topology via pathlets offeredby their access providers towards adjacent IXPs (see Fig. 1).
Following the observation that IXPs provide ideal locations fordata plane anchors, we introduce
Control Exchange Points (CXPs),i.e., control points which stitch pathlets across multiple administra-tive domains to construct global paths. Here we discuss in detailhow CXPs would operate and the existing or emerging control anddata plane technologies a CXP implementation could rely on. Wenote that the full implementation of a CXP is beyond the scope ofthis work.
Basics.
A CXP is a logically centralized entity, applying inter-domain control over how parts of Internet traffic are routed. Inthis context, it can, for example, provide e2e QoS or support mul-ticast services by selecting (a multitude of) appropriate paths. ACXP works in parallel to traditional routing and can control parts oftraffic independently from BGP, e.g., utilizing flow space isolationmechanisms [74]. CXPs use data plane anchors which classify andswitch traffic, such as SDN switches [71]. Software Defined InterneteXchanges (SDX) as proposed by Gupta et al. [49] could constitutean IXP-based deployment possibility. CXP control planes can bebuilt using PCEs [37]. PCEs can reduce the required inter-domainsignaling, enforce traffic access policies and hierarchically managemulti-technology domains. Moreover, a potential cooperation be-tween IXP Route Servers and PCEs could enable CXPs to respondynamically to changing requirements over a set of IXP-mediatedinter-domain connections. Besides public IXPs, anchors can bedeployed at private peering points for augmenting geographical cov-erage, if required. Between data plane anchors, traffic is shipped onvirtual links which are parts of e2e paths and act as pathlets [46].Pathlets are provided by ISPs and may be annotated with specificproperties, such as bandwidth and latency guarantees (if QoS is tobe supported), with simple connectivity as the baseline. When aclient requests an e2e path, the CXP has to find a suitable sequenceof pathlets that meet the client’s QoS requirements.
Providing Pathlets.
Pathlets can be provided by ISPs with ex-isting tunneling techniques, such as MPLS, GRE and VPNs, oremerging SDN approaches based on flow space allocation along anetwork path [71,74]. Within the ISP backbone, QoS guarantees areprovided via traffic engineering and prioritization techniques [14,89].MPLS-TE [53] is one example technology. The ISP is responsiblefor providing cross-traffic isolation internally, keeping its manage-ment policies confidential. The CXP on the other hand, providesisolation on the data plane anchors. An ISP may provide multiplepathlets between two data plane anchors with different properties forservice differentiation or fail-over. We note that CXPs do not havecontrol over how physical pathlet redundancy is achieved withinthe ISP. Availability properties (e.g., for telesurgical applications)should therefore accompany the ISP-originated pathlet advertise-ments. One way to achieve this is by annotating pathlets with SharedRisk Link Group (SRLG) IDs [29]. The incentive for ISPs to pro-vide pathlets is the revenue generated when their pathlets are usedfor e2e services; any ISP can be a provider. As shown in Fig. 1,the ISPs of the source and the destination offer access pathlets toconnect to ISP-adjacent data plane anchors, while the intermediaryISPs offer transit pathlets over their domains, between anchors.
CXP Tasks.
The CXP (i) handles new requests for QoS-enabledpaths (admission control), (ii) computes and sets up suitable paths(embeddings), (iii) monitors pathlet availability and compliancewith QoS guarantees, and (iv) performs reembedding, if required.A client negotiates her request directly with her access ISP, whichselects a suitable CXP for establishing the inter-domain route outof a set of available CXPs. The ISP forwards the client’s request tothe chosen CXP which in turn computes a suitable e2e path. TheCXP reserves capacity on the selected pathlets and then configuresthe respective data plane anchors. Accordingly, the client’s ISP hasto configure its network such that the quality sensitive traffic is sentvia a pathlet to the correct data plane anchor. A CXP monitors thebandwidth, latency and availability of a path for the duration of theclient’s reservation, using existing technologies and approaches [73,78,85]. If the client’s requirements are violated or a pathlet becomesunavailable, the CXP chooses and configures an alternative path forthe affected part(s) of the traffic; this can even be a “hot-standby”backup path carrying traffic duplicates. Besides, the CXP maychoose to better utilize the available pathlets by re-embedding pathsand defragmenting the substrate resources.
Scale-Down Factor (SDF)Property 1 2 4 8 16 32Node count 229 115 57 28 14 7Edge count 49k 29k 15k 6.5k 3.9k 1.1kDiameter 5 5 3 2 2 1Av. node degree 220 250 260 230 280 160Av. edge multiplicity 4.3 6.0 8.3 12. 25. 26.Av. shortest path len. 1.9 1.6 1.4 1.3 1.1 1.0Av. clustering coeff. 0.80 0.82 0.85 0.87 0.93 1.0
Table 1: Properties of the graphs generated from the Euro-IX datasetat various scale-down factors (SDF); larger SDFs correspond tosmaller CXP penetration and vice versa.
3. THE IXP MULTIGRAPH
In this section we measure and characterize the inter-IXP multi-graph, i.e., the substrate on which inter-domain path brokers mayoperate. This analysis is necessary to understand where inter-domaincontrol could be applied as well as the efficiency of incrementaldeployment, and is complementary to research related to scalingup CXP-like control planes [17] or investigating the trade-offs in-volved in logical centralization [65]. We thus answer the followingquestions: (i) how many IXPs need to participate so that CXPs canprovide guaranteed services to a large population of the Internet,assuming that their member ISPs would offer the necessary pathlets,and (ii) how much path choice and diversity we can gain comparedto classic BGP routing practices. We highlight this because currently,due to valley-free routing [41] and the prevalence of peer-to-peerlinks at IXPs [5], Internet paths normally cross at most one IXP.IXP-based path brokers simplify the use of paths that cross multipleIXPs.
We use four datasets to map the inter-IXP topology and theIPv4 address space: (i) the Euro-IX [35] and (ii)
PeeringDB [76]databases, from which we obtained IXP membership data, (iii) theCAIDA AS relationship data [21, 66], and (iv) the CAIDA Route-Views AS-to-prefix data [22]. Due to space constraints we reportresults only for Euro-IX, which also provides geographic coordi-nates of IXPs (used to determine distances between IXP locationsin Section 5) in contrast to PeeringDB. Analysis on PeeringDB datafurther corroborates our findings. We note that, in general, there aremultiple publicly available sources of information on IXPs, includ-ing Euro-IX, PeeringDB, PCH, IXP websites and public data fromBGP route collectors. For a comprehensive comparison of thesesources in terms of completeness and accuracy we refer the readerto the work of Klöti et al. [57], which serves as complementaryresearch to the investigation of the properties of such datasets.Using a snapshot of the Euro-IX peering database [35], we ex-tracted membership data for 6,542 ASes in 277 IXPs. After ignoringIXPs which had no members or had only members which advertisedno IP prefixes, we have 6,122 ASes in 231 IXPs. Two further IXPswhich have no connections to others are discarded. The final (con-nected) graph consists of 229 IXPs and ∼
49k edges between IXPs,crossing ISPs that peer concurrently with these IXPs. We derivesimple graphs by collapsing multi-edges to single edges, annotatedwith the initial edge multiplicity.We scale down the extracted inter-IXP topology assuming that aCXP does not have all the IXPs at its disposal, but gradually recruitsIXPs to maximize the IP address space it can serve. Each new IXPprovides access to more client address space served by its memberISPs. We determine a suitable order based on a greedy heuristic,starting with the IXP having the largest address space coverage and
10 100Edge multiplicity0.20.40.60.81.0 F r equen cy (a) F r equen cy (b) Figure 2: CCDFs of edge multiplicity and path diversityin each iteration adding the IXP which yields the greatest numberof non-overlapping addresses. We assume that whenever we add anew IXP, all its member ISPs would host pathlets that: (i) connecttheir edge clients to the new IXP (via access pathlets, cf. Fig. 1) ,and (ii) connect the new IXP to other CXP-enabled IXPs at whichthese ISPs are present (via transit pathlets, cf. Fig. 1). We make thisassumption, since our goal is to investigate the potential of an IXP-centric multigraph for CXP deployment, as the CXP approachesmore and more IXPs. Each IXP is associated with an ISP member-ship base, which we want to examine in full. The dynamics of thepathlet market will eventually determine which IXPs and ISPs willparticipate, which pathlets they will advertise and which clients willchoose to connect under diverse QoS guarantees. For such marketanalyses, investigating pathlet pricing and ISP participation, werefer the reader to works such as MINT [83] or RouteBazaar [34].
Table 1 gives an overview of the properties of the inter-IXPmultigraph at different scales. The scale-down factor 32 correspondsto a small CXP deployment on 7 IXPs, while a factor of 1 involvesall the 229 IXPs. We first observe average shortest path lengthsbetween 1 and 1.9 edges. This observation combined with the highclustering factors suggests small world properties. Furthermore,multi-edges result in very high average node degrees, e.g., of 160 inthe initial topology with 7 IXPs. Fig. 2a shows the ComplementaryCumulative Distribution Function (CCDF) of the edge multiplicity,i.e., the number of parallel ASes that connect pairs of IXPs, in thefull (unmodified) topology. We observe that a few pairs of IXPs areinterconnected by over a hundred distinct ASes, each of which is ina position to offer one or more pathlets between each pair. Betweenthe largest IXPs, which form the most likely targets for an initialdeployment, hundreds of pathlets—over member ISPs—may beavailable.Fig. 2b shows the CCDF of path diversity , which is the numberof edge-disjoint paths between each pair of IXPs, computed withthe minimum cut. These paths can cross multiple IXPs and may becomposed of multiple pathlets used in sequence. Conceptually, thecut provides the minimum number of pathlets which would haveto be removed so that no path at all is found between these IXPs.We note however, that a failure inside a single ISP (e.g., related tointernal routing) can affect many pathlets offered by this ISP. Also,different ISPs may share the same physical cables (e.g., transatlanticfiber links). As Fig. 2a and Fig. 2b show, the path diversity is muchhigher than the direct connectivity i.e., edge multiplicity betweenpairs of IXPs. Thus even when all direct ISP pathlets between anIXP pair fails, multiple indirect paths crossing other ISPs and IXPanchors may be used to replace the lost connectivity.
To be successful, reaching a large client base is important for aCXP. Therefore, we address the question of how much of the IPv4
Perc. of added p2p links0 % 25 % 50 %Scenario Description µ M µ M µ M P OINTY P EAK
Valley-free 2.9 2 3.2 2 3.3 2W
IDE P EAK + multiple peering links 10. 2 43. 3 70. 3W
ITH S TEPS + unconstrained peering 19. 3 68. 4 104. 4U
NRESTRICTED
No restrictions 42. 5 108. 7 143. 7
Table 2: AS-level policy models and their mean ( µ ) and median ( M )path diversity, with added p2p links.address space can be reached from IXPs and their members. Fig. 3adepicts the IP address coverage versus the number of participatingIXPs, assuming a greedy strategy maximizing IP address coverage.We show results both for directly adjacent IXP members as wellas those connected over a single intermediate ISP (one hop). Weobserve that we can serve over 1 billion IP addresses via only 5 CXPanchors in well-connected IXPs for directly connected customers,which is 40 % of the announced IPv4 addresses in the Internet.This increases to 2.4 billion IP addresses (91 % of announced ad-dresses) if we also consider the 1-hop customer cone of the IXPmembers. With 20 IXPs, more than 1.5 billion IP addresses (>50 %of announced addresses) can be reached directly. This allows aninitial deployment of just a few IXPs to serve large parts of the IPv4address space and enables efficient incremental adoption of inter-domain QoS-enabled services. Further use of private peering pointsmight selectively augment the required coverage, where applicable. We next evaluate the increase in path diversity gained when usinga CXP-enabled IXP multigraph with relaxed peering policies ascompared to valley-free routing of the AS-level topology. Themost constrained policy corresponds to the traditional valley-freemodel [41] (scenario P
OINTY P EAK ); this allows the sequentialcomposition of an uphill path (over customer-to-provider links),then at most one peer-to-peer (p2p) link, and a downhill path (overprovider-to-customer links), resembling a mountain with a rathernarrow peak. The upper bound on path diversity is achieved withthe unrestricted policy scenario (scenario U
NRESTRICTED ). Weinvestigate two additional scenarios by gradually relaxing the valley-free conditions. (i)
The W
IDE P EAK scenario extends valley-freerouting by allowing an arbitrary number of p2p hops between theuphill and the downhill path, instead of at most one, representing ascenario where there is exactly one CXP-mediated path traversed,passing over multiple IXPs. (ii)
The W
ITH S TEPS scenario allowsan unlimited number of p2p links anywhere in the uphill path, andalso in the downhill path. Any number of CXP-mediated paths canbe traversed either while climbing uphill or descending downhill;this results in a step-wise setup, i.e., a mountain with potentiallywide plateaus at different altitudes.To address the known deficiency in detecting p2p links using thecurrent methodology to find AS-level links [5], and to investigatethe effect of more extensive peering on the Internet topology, weaugment the AS relationship graph with p2p links derived from IXPmembership. A given percentage of the derived links (cf. Table 2) isadded to the graph, chosen uniformly at random; gradual addition isdepicted with increasing percentages . We estimate the correspond-ing policy-compliant AS-level path diversity, capitalizing on ourprior work [58]. We use a sample size of 10K pairs of AS endpoints,selected randomly, with each AS weighted by the number of IPv4addresses it announces over BGP. Larger percentages were not investigated due to the memory limi-tations of the current NetworkX [72] min-cut implementations.
50 100 150 200 250Participating IXPs0.00.51.01.52.02.5 R ea c hab l e add r e ss e s (a) Cumulative coverage of IPv4 ad-dress space originated by IXPmembers vs. participating IXPs F r equen cy UnrestrictedPointy PeakWide PeakWith Steps (b) CCDF of AS-level path diversityby scenario (with no added p2plinks from IXP membership)
Figure 3: CXP potential via IXP deploymentTable 2 shows the mean and median path diversity observed forthe various models and amounts of added p2p links, while Fig. 3bshows the distribution of path diversity for the models without addedp2p links. We observe that transitioning from P
OINTY P EAK toW
IDE P EAK greatly increases the path diversity, even without addedp2p links. W
IDE P EAK clearly has an advantage over P
OINTY P EAK even when the latter has many new links added and the former doesnot. This is true for the mean, but also the median, which is lessaffected by the highly skewed distribution; for example, for tier-1and large tier-2 ISPs we see an increase by up to a factor of 29.The W
ITH S TEPS scenario has more modest gains in median pathdiversity and lies within a factor of two of U
NRESTRICTED , whichis the upper bound. After examining the data, we observed that theadvantage of U
NRESTRICTED and W
ITH S TEPS over W
IDE P EAK stems mainly from a relatively small number of very well connectednodes. We therefore conclude that (i) relaxing constraints on peeringpolicy greatly increases path diversity, more so than simply intro-ducing new p2p links, and (ii) further relaxations of the model yieldrelatively modest benefits. Lastly, the small world properties of theInternet AS-level topology graph, also observed in the IXP multi-graph abstraction (cf. Table 1), and our analysis of shortest pathlengths show the following. Since the Internet is densely connectedon the AS level, with the number of interconnections growing withina valley-free regime, relaxing the policy constraints does not yield shorter paths but simply allows us to use more paths. We observedaverage lengths within 3-4 hops irrespective of policy, in agreementwith other related reports [62].
In this section, we use available snapshots from the PeeringDBdatabase, complementary to the Euro-IX snapshot-based analysis,in order to verify that our observations regarding the properties ofthe projected CXP multigraph are valid over time . We note that thisanalysis is not intended to be exhaustive, but rather an indicativedemonstration of the temporal evolution of the peering ecosystemand the associated IXP multigraph, on which CXPs may operate. Byknowing the past, we can extrapolate what may happen in the future,as CXPs expand within an IXP-based Internet. For our temporalanalysis we use monthly snapshots from crawling the PeeringDBwebsite over the months 3/2014 to 1/2015, effectively covering themonthly evolution of the data during the year 2014. We also processthe data extracted from SQL dumps on an almost yearly basis over2008-2012.We started with the evolution of the total number of the IXPs andASNs which participate in the peering ecosystem, over time. Weobserved that the number of IXPs has been linearly increasing ata rate of ∼
36 IXPs/year between the start of 2008 and the end of2013, while we witnessed an acceleration to a ∼
115 IXPs/year rateof increase between the start of 2014 and the end of 2014. The latteris a result of the recent influx of small IXPs mostly located in South America, Africa and Australia; we will later revisit these IXPs todetermine their impact on the CXP multigraph. On the other hand,the number of ASNs that are reported in PeeringDB seems to followa steady linear increase at a rate of ∼
460 ASNs/year. Some ofthese ASes, as we show later, may be capable of acting as inter-IXPpathlet providers, thus contributing to the density of the multigraph.In general, we observe that IXPs and their connected AS peers arerising monotonically in sheer numbers over the years; IXPs haveincreased from less than 200 in the beginning of 2008 to more than500 in the end of 2014, while the participating ASes have increasedfrom ∼
900 to ∼ ∼
32 IXPs/year. We note here that thisbehavior is a bit different than the one that we observed for all the IXPs (nodes or not). This is because the multigraph is basedon the largest connected component of the IXP-based full graph;some of the IXP nodes may be left out in case their member ASescannot connect them to the rest of the multigraph. Examples of suchIXPs are the ones in some remote parts of Africa, Australia, EastAsia and South America. Larger ISPs that may peer concurrentlyat multiple IXPs around the world are usually not members of suchsmall IXPs—at least in the beginning.Moreover, we observed that the number of inter-IXP edges in theconnected multigraph has been increasing at a rate of ∼ ∼ ∼
10k edges in 2008to over 50k edges marking the start of 2015. Further correlationwith the numbers of IXPs per ASN shows that the multigraph has a“slow” changing component increasing at ∼
5k edges per year; thelower 50% of all ASes do not contribute at all to this component,while the upper 50% is responsible for sustaining this rate over theyears. The upper 1% of the highly connected ASes is much moredynamic, contributing an extra ∼
6k edges/year.In Fig. 4a we examine the number of edges between directlyconnected
IXP pairs. We observe that 50% of the directly connectedIXP pairs in the multigraph have an edge multiplicity of 1, whichis the typical median value. These pairs are connected via a singlecarrier ISP, while each IXP of such edges can be connected to manyother IXPs via different ISPs, albeit with a low redundancy. Aswe will show later, this behavior is balanced by the indirect pathdiversity and high redundancy in terms of indirect paths betweenthe IXP nodes. In particular, as opposed to the low redundancy ofthese pairs, the remaining 49.5% of the directly connected IXP pairshave a multiplicity ranging from 2 to 50. We note that the upper0.5% reaches levels of more than 50 edges per pair, with the top0.1% striking an increasing multiplicity of over 100 in 2008, to over300 in 2014. By manual checking, we discovered that these pairscorrespond to the largest global IXPs, such as DE-CIX, AMSIX andLINX, connected over large shared ISP peering bases.In Fig. 4b, we show the distribution percentiles of the path diver-sity between all candidate IXP pairs. The diversity is calculatedas the number of edge-disjoint paths between each pair, computedwith the minimum cut. We see that our observations regarding the - -
15 2009 - -
14 2011 - - - - - -
07 2012 - - - -
26 2014 - - - - - - - - - - - - - - - - - - - - - - Time I n t e r - I X P e dg e m u l t i p li c i t y [ ] (a) Edge multiplicity distribution per-centiles between all the directly con-nected (IXP-IXP) pairs. - -
15 2009 - -
14 2011 - - - - - -
07 2012 - - - -
26 2014 - - - - - - - - - - - - - - - - - - - - - - I n t e r - I X P p a t h d i v e r s i t y [ ] Time (b) Edge-wise path diversity distribu-tion percentiles betweeen all the can-didate (IXP-IXP) pairs.
Figure 4: Edge multiplicity and edge-wise path diversity of thePeeringDB-based CXP multigraph over time.edge multiplicity of Fig. 4a are amplified by about one order ofmagnitude. That is, the connectivity-wise rich IXP pairs compose adense multigraph core, leading to a substantial 10-fold increase inthe overall path diversity as opposed to edge multiplicity.In summary, the IXP overlay graph is growing both in numberof vertices and number of edges, thus improving connectivity. Thisis mainly due to the more aggressive peering of big players likeHurricane Electric, and the introduction of many new IXPs in remoteparts of the globe during recent years. The edge multiplicity in thecorresponding multigraph leads to an order of magnitude larger pathdiversity over any IXP pair (with 1000s of paths available betweenthe upper 0.1% of the pairs). This is intensified as time progresses,especially in the recent years. A heavy tail of well-connected IXPsand aggressive AS peers is responsible for the dynamic expansionof the multigraph. In the presence of this densely connected “core",low path choice typically stems from badly connected stub ASesand not from a general graph property.
4. PATH STITCHING ALGORITHMS
As shown in Section 3 the IXP-based multigraph, on which CXPsmay operate, is very dense. In this section, we present algorithmsto exploit its rich path diversity in order to maximize the numberof concurrently embeddable routes subject to QoS guarantees, suchas maximal latency or minimal bandwidth. These algorithms serveas the application logic of a logically centralized CXP controller,operating on the global view of the IXP multigraph for inter-domainpath stitching.The problem that we need to solve is complex for several reasons. (i)
Requests from the large client base (cf. Section 3.3) dynamicallyarrive over time in a non-predictable manner, necessitating the useof online algorithms. (ii)
While a single suitable e2e path can befound in polynomial time, the IXP-based graph offers rich choice(cf. Section 3.2, Section 3.4) and requires to carefully select whichof the edges between two IXPs to use. (iii)
The online selection ofe2e paths should reflect multiple conflicting high-level objectives,namely accepting as many requests as possible, avoiding the use ofscarce low-latency, high-bandwidth edges, and preventing resourcefragmentation. We formally introduce the e2e routing problemconsidered in this work as the QoS Multigraph Routing Problem(QMRP) in Section 4.1, together with an optimal offline formula-tion. Subsequently, we present a general algorithmic frameworkto solve the QMRP in an online manner. In particular, given thecomputational complexity of the problem, we employ a sample-select approach , where in the first stage, a set of feasible paths is sampled , and subsequently one of them is selected for the actualembedding (cf. Section 4.2). Lastly, the framework is extended tosupport reconfigurations of pre-generated embeddings in order to
Integer Program 1:
Optimal Flow Formulation (OptFlow)max (cid:88) R ∈R x R (OBJ) x R = (cid:88) e ∈ δ + ( s R ) P eR − (cid:88) e ∈ δ − ( s R ) P eR ∀ R ∈ R (OF-1) (cid:88) e ∈ δ + ( v ) P eR − (cid:88) e ∈ δ − ( v ) P eR ∀ R ∈ R .v ∈ V G \ { s R , t R } (OF-2) bw e ≥ (cid:88) R ∈R bw R · P eR ∀ e ∈ E G (OF-3) lat R ≥ (cid:88) e ∈ E G lat e · P eR ∀ R ∈ R (OF-4) x R ∈{ , } ∀ R ∈ R (OF-5) P eR ∈{ , } ∀ R ∈ R , e ∈ E G (OF-6) accommodate further online requests. We model the IXPs and their pathlet interconnections as a directedmultigraph G = ( V G , E G ) , where V G is a set of IXPs (nodes/ver-tices) and E G are inter-IXP pathlets (links/edges) offered by ISPs.The ISPs annotate their pathlets e ∈ E G with their available band-width bw e ∈ R ≥ and their latency lat e ∈ R ≥ . On this substrate,we want to embed a set of e2e routing requests, henceforth denotedby R . A request R ∈ R asks for the establishment of an e2e con-nection between IP addresses s R and t R with minimal bandwidth bw R and maximal latency lat R . Note that these start and end pointsare not included in the pathlet network G . However each IP ad-dress is, by its access ISP affiliation, implicitly connected to oneor multiple IXPs (cf. Fig. 1). While we take these multiple startand end IXPs into account in the implementation of the presentedalgorithms, we assume simple IXP start and end points for the sakeof easier representation.We study how CXP operators can accept (and embed) as manyrequests as possible—a natural objective for any revenue-drivenprovider aiming at the maximization of its client base. Embedding arequest R ∈ R here refers to finding a suitable path P R , such thatthe latency of P R is less than lat R and that the path P R can carrymore than the minimal bandwidth bw R . Importantly, as inter-IXPpathlets can be used by multiple requests, the maximal availablebandwidth (i.e., capacity) of pathlets must never be exceeded.The offline version of the QoS Multigraph Routing Problem(QMRP), i.e., when R is given ahead of time, can be formulatedas an Integer Program, cf. Integer Program 1 (OptFlow): the bi-nary variable x R decides whether request R ∈ R is embeddedand the variable P eR indicates whether edge e ∈ E G is used byrequest R ∈ R . The correctness of the formulation stems fromthe following observations: (i) Constraints OF-1 and OF-2 inducea unit flow from s R towards t R if request R ∈ R is embedded(cf. [8]). δ + ( v ) and δ − ( v ) here denote the set of outgoing andincoming edges of v ∈ V G respectively. (ii) By Constraint OF-4 thepath described by variables P R must obey the maximal latency lat R . (iii) By Constraint OF-3 the available bandwidth (i.e., capacity) ofany pathlet is not exceeded. While the offline problem is interestingfor optimizing existing allocations of requests in the backgroundand further increase acceptance ratios (see Section 4.3), we are ingeneral more interested in the online variant. In this context, eachrequest R is known only at its arrival time, and the algorithm needsto compute an embedding (for the duration of the request) at thattime. lgorithm 1: Outline of Online Sample-Selection Algorithm
Input :Network G = ( V G , E G , bw e , lat e ) ,Request R = ( s R , t R , bw R , lat R ) Output :Path P R to connect s R to t R or null sample set of feasible paths P R if P R (cid:54) = ∅ then select best path P R ∈ P R and embed R accordingly else return null In order to tackle the online variant of the QMRP we proposea sample-select approach. In the first stage a set of feasible pathsis sampled from the set of all feasible paths. In the second stageone of these paths is selected for the embedding. We employ thisapproach as computing the optimal path under multiple objectivesand constraints is generally NP-hard [42], while the algorithm mightneed to handle workloads of tens or hundreds of requests per second.While investigating multiple path sampling strategies, we con-sider only a single selection strategy which aims at minimizingthe utilization of the network (in order to provide room for manyrequests) and to secondarily penalize use of high-bandwidth andlow-latency edges (since they are more scarce). We note that de-termining the best selection strategy is interesting in its own right,but lies outside the scope of this paper. The strategy used can besummarized as follows: (i)
Strictly prefer paths with a smaller hopcount. (ii)
Among paths with the same hop count, choose the onewith the minimal inverse utility, computed edge-wise ∀ e ∈ P R : InvU ( e ) = bw e min e (cid:48) ∈ E G ( u,v ) bw e (cid:48) · max e (cid:48) ∈ E G ( u,v ) lat e (cid:48) lat e / | E G ( u, v ) | , where E G ( u, v ) denotes the set of edges between nodes u, v .In the following we focus on the path sampling strategy, andpresent three different algorithmic variants. The goal of the samplingalgorithm is to efficiently compile a set of paths, giving us theflexibility of choice. In particular, we exploit the fact that computingfeasible solutions is not NP-hard:
Theorem 1.
A feasible path for a given request R can be computedin polynomial time. Proof.
The proof is constructive. We first prune all edges e ∈ E G whose bandwidth is not sufficient to support the minimal bandwidthrequirement bw R . Projecting the resulting multigraph onto a simplegraph by replacing each set of edges with the minimal latency edgeof the set, the simple graph G (cid:48) is obtained. We can now performany polynomial shortest-path algorithm to obtain the path P (cid:48) R ∈ G (cid:48) ,if such a path exists. If (cid:80) e ∈ P (cid:48) R lat e ≤ lat R , a feasible path wasconstructed; otherwise no such path can exist. Assume that thisprocess would not find a feasible path even though such a path P ∈ G exists. By replacing each edge of P with the minimallatency edge of the corresponding multi-edge set, a feasible path in G (cid:48) is constructed, proving the theorem. (cid:4) Theorem 1 is an important building block for all our path samplingalgorithms, as it allows us to: (i) abort the generation of pathsearly using a single shortest path computation, and (ii) devise pathsampling algorithms that will always return feasible paths, if theyexist (cf. Korkmaz et al. [59]). In Section 4 of our accompanyingtechnical report [61], three such algorithms are presented. The
Perturbed Dijkstra ( PD ) algorithm is essentially a k -shortest pathsvariant [33], strictly minimizing latency. The Guided Dijkstra ( GD )algorithm broadens the search space as edge selection is latency- Algorithm 2:
Offline Reconfiguration Scheme
Input :Initially rejected request R − ,Accepted requests R + with path P + R for R + ∈ R + sample set of feasible paths P R − for R − in the empty graph if P R (cid:54) = ∅ then compute conflicting requests P confl = { R + ∈ R + |∃ P R − ∈ P R − , P R + ∩ P R − (cid:54) = ∅} try to (re-)embed P confl ∪ { R − } by an offline algorithm Integer Program 2:
Heuristic Path Formulation (HeurPaths)max (cid:88) R ∈R x R (OBJ) x R = (cid:88) P R ∈P R y P R ∀ R ∈ R (HP-1) bw e ≥ (cid:88) R ∈R ,e ∈ P R bw R · y R ∀ e ∈ E G (HP-2) x R ∈{ , } ∀ R ∈ R (HP-3) y P R ∈{ , } ∀ R ∈ R , P R ∈ P R (HP-4) independent, and the Guided Random Walk ( GW ) algorithm aims atfinding arbitrary feasible paths. The run-time complexity of thesealgorithms is bounded by O ( k · ( | E G | + | V G | log | V G | )) , with k -many feasible paths to sample and | V G | nodes and | E G | edges tooperate on. The algorithms can be used for different path samplingcases, ranging from purely deterministic variants ( PD ), to semi-randomized ( GD ) and fully randomized variants ( GW ). The sample-select scheme as presented in Algorithm 1 can beused to find good embeddings of e2e path requests arriving one-by-one over time. In particular, the algorithms try to embed eacharriving request if this is possible, otherwise they reject the request.However, greedily embedding one request after the other may notbe optimal over time, and sometimes, it may be worthwhile to recon-figure existing paths in order to defragment the current allocationand make space for additional requests. Thus in the following, wepropose a hybrid online-offline scheme which performs exactly that:requests which are arriving online over time are embedded using oneof the sample-select approaches described above. However, in addi-tion, we run an offline optimization procedure in the background:this reconfigures sets of paths in order to improve acceptance ratiosfurther. Such reconfigurations may be the only possibility to accepta request.We thus extend the sample-selection scheme depicted as Algo-rithm 1 with the fallback scheme depicted as Algorithm 2. Givena just rejected request R − , feasible paths are first sampled in theempty network, i.e., without any embedded requests. If feasiblepaths exist and the request R − could in general be embedded, all re-quests that conflict with any of the found paths are selected in Line 3.In Line 4 the algorithm tries to reconfigure conflicting requests andembed R − . Note that the reconfiguration task corresponds to solv-ing the offline QMRP, where all given requests must be embedded .While generally the Integer Program 1 could be used by requiring x R = 1 for all R ∈ R + ∪ { R − } , its run-time is prohibitive. Wehave therefore developed Integer Program 2, which does not com-pute paths on its own, but is given the set of feasible paths P R of each request R ∈ R as input, computed previously via onlinepath-sampling. Again, by forcing x R = 1 for all R ∈ R + ∪ { R − } ,we can compute whether there exists a reconfiguration of embedded arameter Space (online) Space (offline/hybrid) Compared Algorithms Perturbed Dijkstra (PD) HeurPaths-PDGuided Walk (GW) HeurPaths-GWGuided Dijkstra (GD) HeurPaths-GDOptFlowScaling-down factor (SDF) 32, 16, 8, 4, 2, 1 32, 16Request Latency unif(100,150), (150,200), (200,250), (250,300) msPaths per request 5, 10, 20Number of requests per run 10,000
Table 3: Online and offline/hybrid parameter space.paths that allows accepting R − , increasing the overall acceptanceratio.We note that the proposed formulation is an adaption of theclassic multi-dimensional knapsack problem [39] which, despite itsNP-hardness, can be solved quite efficiently in practice using branch-and-bound solvers [50] when only dozens of paths are used for eachrequest. In the evaluation (cf. Section 5), we use the HeurPaths program as follows: first we produce sets of paths for all requests (5to 20 per request) using the previous path sampling algorithms onthe initial empty graph, and then we employ
HeurPaths to allocatethe requests in an offline manner using the path set input.
5. ALGORITHMIC EVALUATION
We evaluate the performance of our algorithms in terms of Accep-tance Ratio (AR), utilization i.e., the ratio of occupied bandwidth tothe total available capacity , and computation time per request. Tomaximize the revenue, the acceptance ratio should be maximizedwhile minimizing the resource utilization (so that there is room formore requests). Additionally, based on the ad-hoc online embeddingof requests, the runtime should be low in order not to block thesystem. We use our custom CXP simulator [4], and the inter-IXPmultigraphs described in Section 3. As discussed in Section 4, basedon the multigraph nature of the IXP graph existing algorithms arenot suitable and need to be adapted accordingly. In this section,we investigate the—empirical—trade-off between always choos-ing shortest paths ( Perturbed Dijkstra ) versus the more randomizedversions
Guided Dijkstra and
Guided Random Walk (cf. [61]). More-over, using the optimal Integer Program 1 as a baseline we showthat our algorithms yield near optimal solutions quickly. Hence, ourevaluation demonstrates how the orchestration on such graphs canbe performed efficiently even on realistically sized scenarios. Thisis important for the application logic of potential SDN-based CXPimplementations. We next elaborate on the setup and main insightsyielded by the evaluation process.
The search space of our simulations is composed of the cross-product of the following parameter dimensions: (i) pathlet latenciesand (ii) bandwidths, (iii) requested latencies and (iv) bandwidths, (v) graph sizes, (vi) maximal number of paths generated per request, (vii) number of requests per simulation run, and (viii) temporalcharacteristics of requests (e.g., durations). This search space hasto be explored for each evaluated algorithmic variant. Due to itslarge volume, we constrain our search space so that the simulationsmay run within reasonable time frames ( ∼ several weeks). Table 3summarizes the used parameters. We next elaborate on the inter-IXP and endpoint-to-IXP pathlet latency and bandwidth model, thechoice of the request endpoints and the temporal characteristics ofthe requests. Latency.
Pathlets connecting IXPs pass over ISP domains. Tomodel pathlet latency in a geographically diverse ISP, we utilize the This metric takes into account only inter-IXP pathlets. Hurricane Electric (HE) looking glass server [54] and perform mea-surements between pairs of routers situated at major PoPs aroundthe world. The variance of the measured latencies appears not todepend on the geographical distance d . We therefore model theRTT as a linear function (parameterized by a and b ) of d com-bined with a random variable X to reflect the uncertainty in themodel: rtt ( d ) = a · d + b + X . Through linear regression we find: a = 0 . mskm ] and b = 26 [ms]. By least squares fitting, we model X as a normal distribution N ( µ, σ ) with µ = 0 and σ = 14 [ms].We approximate the one-way latency as: lat e = 1 / · rtt ( d ) . anduse this model for both access and transit pathlets (cf. Section 2).Request latencies are selected uniformly at random from four ranges(cf. Table 3) to evaluate looser to stricter requirements. Bandwidth.
We consider unitary requests, where each embeddedrequest occupies the full bandwidth of the edge(s) it uses. Thissimplification removes the necessity to model offered and requestedbandwidth and hence reduces the search space. In contrast, werather focus on assessing our algorithms based on the topologicalcharacteristics of the inter-IXP substrate. In particular, we “fill”the multigraph with allocated bandwidth in order to discover itsinherent potential for hosting arbitrary requests. Moreover, thechosen setup of uniform (and thus blocking) paths gives insights inhow well the choice of shortest paths with respect to the hop countand the actual latencies are balanced to achieve the best resourceutilization. If one were to always prefer shorter paths with respect tothe hop count, the resource footprint would be minimized; however,this would reduce the availability of “mission-critical” pathlets ofvery low latency, hence reducing the acceptance ratio for latency-sensitive requests in the long run. Non-unitary request settings andcorresponding simulations and effects are the subject of future work.For simplicity, non-access IXP-IXP pathlets are aligned with theunitary request bandwidth setting. In reality, their bandwidth isgenerally determined by ISP competition and auctioning [83].
Request Endpoints.
We choose candidate IP addresses uni-formly at random from the IPv4 address space adjacent to the mem-bers of the IXPs under examination. After we choose a source anddestination address for a request, we retrieve their respective coordi-nates using the MaxMind GeoIP2 database [70]. These coordinates,together with the IXP locations, are used for geographical distancecalculations between endpoints and IXPs. We assume that IP-IXPpathlets are not constrained by bandwidth, since the access ISP canoffer exactly the bandwidth requested in direct collaboration withits client, even without CXP-based mediation.
Online Requests.
The requests arrive in order and are handledone-by-one in an online fashion. Each embedded request persistsduring the lifetime of the simulation (“infinite” duration), so that thepeak load in the online case corresponds to the offline case, allowingfor fair comparison at the corresponding graph scales.
Fig. 5 and Fig. 6 present key observations regarding algorithmicperformance, which we further explain and analyze below. Note thatthe ranges on the y-axes do not have a 0-baseline but are adaptedper figure. All results are based on 10 runs per simulation. Weshow average values with error-bars of 1 standard deviation. Thebaseline algorithm for the online case is the
Perturbed Dijkstra ,while
OptFlow is the offline/hybrid baseline variant. We note thatsimple-graph approaches and baselines of previous work, not tai-lored to multigraph sampling (cf. Section 7), would have to operateon orders of magnitude larger substrates, e.g., using 2 “half-edges”and one AS node to simulate a pathlet, inducing biases. In suchgraphs | V G | = O ( | E G | ) , while here | V G | (cid:28) | E G | . Which online path sampling algorithm allows for the maxi-
Min-Max latency pair [sec] A cc ep t an c e R a t i o [ % ] GWPDGD (a) Acceptance Ratio vs Required Latency:SDF=8, 20 paths/request, 10,000 requests
Paths [ A cc ep t an c e R a t i o [ % ] GDGWPD (b) Acceptance Ratio vs paths/request: SDF=8,latency in (200,250) msec, 10,000 requests
Scale down factor [ A cc ep t an c e R a t i o [ % ] GWPDGD (c) Acceptance Ratio vs SDF: 20 paths/request,latency in (200,250) msec, 10,000 requests
Paths [ U t ili z a t i on [ % ] GDGWPD (d) Utilization vs paths/request: SDF=8, latencyin (200,250) msec, 10,000 requests
Scale down factor [ U t ili z a t i on [ % ] GWPDGD (e) Utilization vs SDF: 20 paths/request, latencyin (200,250) msec, 10,000 requests
Scale down factor [ T i m e [ s e c ] GWPDGD (f) Time per request vs SDF: 20 paths/request,latency in (200,250) msec, 10,000 requests
Figure 5: Moderate scale online simulation mal acceptance ratio, at the lowest utilization?
The winner interms of acceptance ratio is the Perturbed Dijkstra approach with alead of 1-2% (cf. Fig. 5a, Fig. 5b, Fig. 5c), as opposed to GuidedDijkstra and Guided Walk. In terms of utilization, Guided Dijk-stra wins by about 2-5% followed closely by Perturbed Dijkstra,while the Guided Walk is worse within a best-case gap of about10% from its Dijkstra-based counterparts (cf. Fig. 5d), across scales(cf. Fig. 5e). The reason for the prevalence of Perturbed Dijkstraregarding acceptance ratios lies in its k -shortest path discovery; theedge-disjointness perturbation criterion, accompanied by the pathselection function (cf. Section 4.2), counteracts its tendency to con-sume precious (latency-wise) paths and leads to good embeddings.Both Dijkstra approaches embed low-latency, low-hop paths thatconsume small amounts of bandwidth on the substrate network.Especially the Guided Dijkstra performs shortest path routing onrandom samples of the network, further lowering utilization. In con-trast, the Guided Walk, due to the fully randomized path samplingprocess, embeds feasible but higher-hop paths with an importantpenalty on utilization and a small disadvantage in acceptance ratios.Its behavior in these two areas gets better as the number of calcu-lated paths increases (cf. Fig. 5b, Fig. 5d), since its progressive,random path sampling process benefits from exploring richer pathsets (cf. Section 4.2). How do hybrid variants behave w.r.t. acceptance ratios?
Heur-Paths with Guided Walk performs the best in terms of acceptanceratios and is very close to the offline optimal values. In contrast,HeurPaths with Perturbed or Guided Dijkstra leads to lower accep-tance ratios as seen in Fig. 6a, with differences up to 10% for relaxedlatency requirements. This is explained with the optimal latencyseeking stages of these algorithms that do not couple well with theheuristic hybrid allocation. Thus they fail to exploit the richnessof the substrate, being biased towards the same low-latency edges.This leads HeurPaths to saturation and limits maneuverability inpath allocation. The advantage of Guided Walk is preserved across scales (cf. Fig. 6b) and latencies (cf. Fig. 6a).
How do offline, hybrid and online algorithms compare witheach other w.r.t. acceptance ratios and utilization?
Our exper-iments on the 32-SDF and 16-SDF graphs show that the onlinealgorithms perform as good as the optimal offline and hybrid interms of acceptance ratios, but have 20-30% lower utilization. Themain reason for this is the path selection criterion for the onlinesimulation (cf. Section 4.2), which prefers low-hop paths: the on-line variants hit the optimal value through low utilization, while theoffline variants optimize based on sophisticated but computationallyexpensive allocation of requests, ignoring utilization. Note that withSDFs of 32 and 16 due to the small number of IXPs and the nature ofthe request model, many of the requests can be served directly usingtheir access ISPs and a single IXP, without occupying bandwidthon the inter-IXP graph. We did not include larger graph sizes forOptFlow due to run-time scaling issues, which we explain in thefollowing.
How do graph sizes affect run-times?
Increasing the graph size(i.e., lowering the SDF) leads to longer run-times as expected, withthe online Perturbed and Guided Dijkstras scaling worse than theGuided Walk (cf. Fig. 5f). This is because the Guided Walk simplyfinds feasible paths quickly, without taking latency optimality intoconsideration and has lower computational complexity (cf. Sec-tion 4). In contrast, the optimal offline algorithm operates roughly at1 to 3 orders of magnitude slower than the hybrid variants at scalesof 32-SDF or 16-SDF (cf. Fig. 6c), and scales very poorly for largergraphs. For the heuristic hybrid algorithm (HeurPaths) the bottle-neck is the preemptive path sampling for all requests, while the pathembedding stage has negligible time overhead. The use of Heur-Paths in collaboration with the Guided Walk yields near-optimalacceptance ratios (cf. Fig. 6a, Fig. 6b) at efficient run-times; thelatter is evident in Fig. 6c, which presents the run-time of the MixedInteger Programming computations versus the requested latencies.HeurPaths needs 10-100s to embed 10,000 paths. The path compu-
Min-Max latency pair [sec] A cc e p t a n c e R a t i o [ % ] HP-GDOPTHP-GWHP-PD (a) Acceptance Ratio vs Required Latency:SDF=16, 20 paths/request, 10,000 requests
Scale down factor [ A cc e p t a n c e R a t i o [ % ] HP-GDOPTHP-GWHP-PD (b) Acceptance Ratio vs SDF: 20 paths/request,latency in (200,250) msec, 10,000 requests (0.1, 0.15) (0.15, 0.2) (0.2, 0.25) (0.25, 0.3)
Min-Max latency pair [sec] T i m e [ s e c ] HP-GDOPTHP-GWHP-PD (c) Mixed Integer Progam Time vs Latency:SDF=16, 20 paths/request, 10,000 requests
Figure 6: Small scale offline/hybrid simulationtations can be parallelized, or be augmented by existing online paths.For example, the Guided Dijkstra and Walk can be parallelized aftertheir first Dijkstra iteration, reducing run-times on multiple cores.
What is the effect of looser latency guarantees?
The accep-tance ratio (cf. Fig. 5a, Fig. 6a) and utilization generally increasemonotonically as the latency requirements become looser, i.e., lessstrict. This behavior comes to a halt when the substrate is heav-ily utilized. The utilization ceiling is first hit by the Guided Walk,then by the Perturbed Dijkstra and last by the Guided Dijkstra. Inthe hybrid case increased latencies and therefore increased searchspaces widen the gap between HeurPaths with Guided Walk and theDijkstra-based variants as we can observe in Fig. 6a.
What have we learned from online-offline cooperation?
Wehave observed that using direct online-offline cooperation as de-scribed in Algorithm 2 increases acceptance ratios marginally ( ∼ entire current set of requests at theirdisposal, but have no incentive to optimize for utilization at the sametime. Thus they prefer to embed as many requests as possible, evenat the cost of saturating the substrate. In contrast, the pure onlinevariants cannot see all the requests concurrently; therefore, theyare doing their best to allocate each incoming request, or reject itwhen needed, without sacrificing utilization and jeopardizing futureacceptance. We note that, depending on the CXP operator’s goals,the heuristic hybrid variant can be reformed to optimize also forutilization and not only acceptance ratios, in order to efficientlydefragment the substrate when required. Summary: which algorithm should we prefer?
In our experi-ments, we observed different behaviors in terms of acceptance ratiosin the online and hybrid case. In the online case, Dijkstra-based ap-proaches prevail, while in the hybrid case fully randomized samplingperforms better. More precisely, in the online scenario PerturbedDijkstra is a better choice at small graph scales because of its highacceptance ratios and low utilization; at these scales the run-timeof all algorithms is short. We would opt for Guided Walks at largescales, when fast request allocation is desirable, especially if theincoming load of requests is high (e.g., due to higher CXP penetra-tion). In this case, rich path sets (e.g., 20 per request) are important,since they allow the Guided Walk to achieve good acceptance ratiosat reasonable utilization levels, which are close to its Dijkstra-basedcounterparts. Lastly, HeurPaths is a much better candidate for scal-ing up the hybrid version of the problem as opposed to OptFlow because it achieves similar acceptance ratios—in particular whencombined with the Guided Walk—at much shorter run-times.
6. TELESURGERY AS A USE CASE
To get a better understanding of how CXPs can be used, be-yond as plain multi-domain bandwidth brokers, we investigate thetelesurgery [51, 69] use case. Telesurgery undoubtedly has stringentrequirements on both availability and latency. Availability is essen-tial for ensuring uninterrupted surgical operations and the patient’ssafety. Latency is important for making remote surgery feasible withreal-time feedback [51]. In addition, the bandwidth requirementsare generally high, e.g., for transmission of video streams [69].Regarding availability, quick fail-over in case of emergencies ischallenging [82,88]. As a consequence, higher redundancy is needed a priori to achieve acceptable availability. One way to achieve thiswith CXPs, is to allocate multiple disjoint paths on the multigraphand send redundant packet copies on each path. One copy is thenselected by the receiver and delivered to the application. A moreefficient approach could be using Forward Error Correction (FEC)such as Reed-Solomon. For example, a CXP could allocate 12disjoint paths with 1/10 of the required capacity each; then use aFEC scheme with 12 channels including 2 times redundancy at 20 %bandwidth overhead. A CXP can check online for path failures. Ifa path is degraded, the CXP immediately allocates a replacement,leaving the rest of the operational paths intact. Obviously, lessreliable paths within ISPs mandate more redundancy to achieve highavailability.In a CXP context, the ISP’s network resources are virtualized.This example demonstrates how on-demand resource provisioningmay be used to bring prices down, by bringing up the utilization ofthe resource and amortizing its costs, analogously to how CPU andstorage are better utilized in the context of cloud computing. Clientflows can be dynamically assigned to (multiple) pathlets dependingon the resources that are available within the “CXP cloud”.CXPs may also be able to find lower latency paths than traditionalrouting. If a path is subject to a triangle inequality [67] violation(the majority of paths are [9]) and there is a well-placed CXP anchoravailable to route over, the CXP can provide a path with lowerlatency. This implies the need for a broad CXP deployment footprint.While starting with selected IXPs as CXP anchors can serve as aninitial step, it may not be sufficient for optimizing latency [6].Finally, we note the following challenges related to SDN-basedCXP implementations in the context of such use cases. (i)
Controllerdistribution and placement; the RTT between the data plane anchorsand the centralized CXP controllers is a lower bound of the reactiontimes to failures or state updates, while full distribution inducesstate consistency and concurrency challenges [65]. (ii)
Forming anccurate, real-time monitoring infrastructure for supervising path-let guarantees and measuring the performance of QoS-constrainedflows is a challenging task in its own right [73]. Nevertheless, CXPsneed to control just a handful of IXP anchors around the world,which is a promising starting point. Also, the complexity of pathletformation and state monitoring is delegated to the ISP. For example,physical link failures that affect pathlets are first handled locallywithin the ISP and then propagate on the inter-domain level only ifthe failure needs to be known to the CXP to be handled via e2e re-routing. (iii)
Path embeddings need to be protected against failuresvia CXP controller and anchor redundancy. These challenges areinteresting directions for future SDN research at the inter-domainlevel, in the context of centralized pathlet stitching as a novel multi-IXP service.
7. RELATED WORK
Internet QoS and Our Work.
Quality-of-Service is an ever-green topic that has been discussed for decades [13, 87, 90], togetherwith the challenges associated with its implementation [16, 86, 88].Such challenges have hindered its Internet-scale adoption in parallelwith classic best-effort IP routing and peering agreements [19]. Ourwork is complementary to existing work on end-to-end InternetQoS, which covers the spectrum from low-level implementation(queueing mechanisms, QoS-oriented MPLS, OpenFlow mecha-nisms) to high-level policies (SLAs, traffic isolation). We proposean IXP-centric model that can be used to support the deploymentof inter-domain QoS in the context of centralized pathlet brokersand resource controllers (cf. Section 2.1). CXPs could capitalizeon prior work for the implementation [18, 27, 28, 53] and moni-toring [73] of QoS-enabled pathlets; the scheme assumes a givenper-ISP QoS and focuses on what can be done assuming that ISPsprovide guaranteed pathlets anchored to IXPs, irrespective of how they are implemented. We note that this work, based on the conceptof logically centralized brokers offering Routing as a Service [64],is an alternative to the proposal of source-routed, composable pathsegments advocated e.g., in ARROW [77]. We believe though thatusing IXPs as the primary points where the path composition takesplace could be common ground for the deployment of such propos-als. Moreover, the CXP business model, involving the mediation ofcontracts between end-clients and pathlet providers, could benefitfrom works that facilitate the formation, establishment, and verifica-tion of end-to-end connectivity agreements based on cryptocurrenysystems [24].
IXPs.
Recently, a number of studies analyzed the important roleof IXPs [25] in terms of: (i) the flattening of the Internet topol-ogy [30, 47], (ii) the prevalence of IXP-based peering links in theInternet ecosystem [5, 12], and (iii) performance improvements,such as the reduction of average Internet delays and path lengths [7].The potential rise of SDN within IXPs, e.g., enabled by SoftwareDefined Internet eXchanges (SDX) [49], coupled with the chang-ing role of IXPs, could turn to be an avenue for inter-domain QoSservices based on the CXP paradigm. Moreover, Hu et al. [52]investigated how a version of on-demand peering policy relaxationcan take place at IXPs in order to recover from route failures. Ourmore general approach (cf. Section 3) actively uses the path diver-sity induced from different variants of routing policies, based onsequential composition of inter-IXP pathlets. Finally, we refer thereader to the work of Castro et al. [23] on remote peering at IXPs.Among other things pertaining to their study, the authors discussthe marginal utility of reaching extra ISPs in terms of the poten-tial for offloading transit traffic. In contrast, we are investigatingthe incremental deployment of IXP-based penetration in terms of: (i) end-client coverage, and (ii) path diversity potential for connect- ing these end-clients under certain quality guarantees. However, theproliferation of remote peering practices means that the IXP-basedmultigraph tends to get even richer, with remote ISPs being able tooffer pathlets (using other layer-2 resellers) to more IXPs than theones in their direct vicinity.
QoS Routing and Embeddings.
Finding suitable paths betweena pair of endpoints is a classic problem in computer science, and hasbeen studied intensively in the context of online call control [68],virtual-circuit routing [11, 15] and also specifically QoS provision-ing [42]. In the area of QoS routing, exact, approximate and heuristicalgorithms have been considered for finding paths subject to (possi-bly) multiple constraints and objectives. Based on the dense natureof the CXP multigraph and the online fashion in which requestsarrive, we have adapted two well-known heuristic algorithms fre-quently used in the context of QoS: k-shortest paths [33, 42] and thelook-ahead scheme employed by Korkmaz et al. [59]. In contrastto stochastic QoS routing algorithms as presented by Orda [75],we assume QoS guarantees over the provided ISP pathlets. Opti-mal solutions to the QoS routing problem are generally NP-hard toachieve, due to having to consider multiple objectives (minimizingcosts, avoiding scarce low-latency links etc.) or multiple constraints(latency, bandwidth, jitter etc.) [42]. The heuristic offline variantof our problem (embed as many e2e paths as possible), is a variantof unsplittable flow problems [31] and is related to the VPN [48]and virtual testbed mapping [26] problems. For a good survey, werefer the reader to Fischer et al. [38]. Schaffrath et al. [80] alsopresent a relevant virtualization architecture. The hybrid online-offline approach that enables the reconfiguration of existing e2eembeddings, was shown to increase acceptance ratios in the domainof virtual network embeddings by Fan and Ammar [36]. Frikhaand Lahoud have recently proposed to precompute QoS paths toimprove performance [40]. In contrast, the paths that have alreadybeen computed in our work are reused at a later stage (possibly indifferent contexts), thereby not introducing any additional computa-tional overhead. Lastly, Ascigil et al. [10] debunk the conventionalwisdom that logically centralized computations do not scale in termsof domain-level end-to-end Internet routes.
8. CONCLUSION
We proposed using IXPs for stitching inter-domain paths underthe control of centralized routing brokers, which provide pathswith end-to-end guarantees for mission-critical applications. Weconsidered a novel abstraction of the Internet topology: the IXPmultigraph. Based on our study using extensive peering datasets,we evaluated the potential of IXP-based pathlet stitching in thefollowing ways. (i)
In terms of IP address coverage, we showedthat even a small deployment ( ∼ (ii) In termsof AS-level path diversity, we showed the potential of generalizedrouting policies applied on the dense IXP multigraph. We observedan increase of at least one order of magnitude in path diversity,i.e., multiplicity of edge-disjoint paths, as compared to BGP inter-domain routing practices. (iii)
We exhibited the importance ofhaving suitable path sampling algorithms that take advantage of therichness of the multigraph. We further evaluated the performanceand applicability of diverse algorithmic variants—online, offline andhybrid—for different traffic requirements and graph scales; we haveshown that centralized routing variants work efficiently on the globalmultigraph view. Lastly, we placed our analysis within the scope ofa demanding application, namely telesurgery, and highlighted openchallenges.As supported by this multi-faceted evaluation of the potential ofCXPs, we believe that providing guaranteed inter-domain services isot anymore as intractable as it has been in the past. The flatteningInternet topology and the emergence of SDN provide new avenuesfor innovation on CXP-like approaches. In our on-going work weinvestigate ways to kick-start CXP markets. In particular, our goal isto still provide better than best effort paths across the Internet, evenwhen major IXPs or many ISPs do not participate yet in the market.
9. ACKNOWLEDGMENTS
This work was partly funded by European Research CouncilGrant Agreement no. 338402.
10. REFERENCES
GER , B.
ET AL . Anatomy of a Large European IXP. In
Proc.of ACM SIGCOMM (2012).[6] A
HMAD , M.,
AND G UHA , R. A Tale of Nine InternetExchange Points: Studying Path Latencies through MajorRegional IXPs. In
Proc. of IEEE LCN (2012).[7] A
HMAD , M. Z.,
AND G UHA , R. Studying the Effect ofInternet eXchange Points on Internet Link Delays. In
Proc. ofSpringSim (2010).[8] A
HUJA , R. K., M
AGNANTI , T. L.,
AND O RLIN , J. B.
Network Flows: Theory, Algorithms, and Applications .Prentice-Hall, Inc., 1993.[9] A
NDERSON
SCIGIL , O., C
ALVERT , K. L.,
AND G RIFFIOEN , J. N. Onthe Scalability of Interdomain Path Computations. In
Proc. ofIEEE IFIP Networking Conference (2014).[11] A
SPNES , J.
ET AL . On-line Load Balancing withApplications to Machine Scheduling and Virtual CircuitRouting. In
Proc. of ACM STOC (1993).[12] A
UGUSTIN , B., K
RISHNAMURTHY , B.,
AND W ILLINGER ,W. IXPs: Mapped? In
Proc. of ACM IMC (2009).[13] A
URRECOECHEA , C., C
AMPBELL , A. T.,
AND H AUW , L. ASurvey of QoS Architectures.
Multimedia Systems 6 , 3 (1998),138–151.[14] A
WDUCHE , D.
ET AL . Overview and Principles of InternetTraffic Engineering. RFC 3272, Informational, 2002.[15] A
WERBUCH , B.
ET AL . Competitive Routing of VirtualCircuits with Unknown Duration. In
Proc. of ACM-SIAMSODA (1994).[16] B
ELL , G. Failure to Thrive: QoS and the Culture ofOperational Networking. In
Proc. of ACM RIPQoS Workshop (2003).[17] B
ERDE , P.
ET AL . ONOS: Towards an Open, DistributedSDN OS. In
Proc. of ACM HotSDN Workshop (2014).[18] B
ISTARELLI , S.
ET AL . Unicast and Multicast QoS Routingwith Soft-constraint Logic Programming.
ACM TOCL 12 , 1(2010).[19] B
OUCADAIR , M.
ET AL . Considerations ofProvider-to-Provider Agreements for Internet-Scale Quality ofService (QoS). RFC 5160, Informational, 2008. [20] B
ULDYREV , S.
ET AL . Catastrophic Cascade of Failures inInterdependent Networks.
Nature 464
ASTRO , I.,
ET AL . Remote Peering: More Peering withoutInternet Flattening. In
Proc. of ACM CoNEXT (2014).[24] C
ASTRO , I.,
ET AL . Route Bazaar: Automatic InterdomainContract Negotiation. In
Proc. of HotOS XV (2015).[25] C
HATZIS , N.
ET AL . There is More to IXPs Than Meets theEye.
ACM SIGCOMM CCR 43 , 5 (2013), 19–28.[26] C
HOWDHURY , M., S
AMUEL , F.,
AND B OUTABA , R.PolyViNE: Policy-based Virtual Network Embedding AcrossMultiple Domains. In
Proc. of ACM VISA Workshop (2010).[27] C
IVANLAR , S.
ET AL . A QoS-enabled OpenFlowEnvironment for Scalable Video Streaming. In
Proc. of IEEEGC Workshops (2010).[28] C
RUVINEL , L.,
AND V AZÃO , T. Improving Performance forMultimedia Traffic with Distributed Dynamic QoSAdaptation.
Comput. Commun. 34 , 10 (2011), 1222–1234.[29] D
AHAI , X.
ET AL . Failure Protection in Layered Networkswith Shared Risk Link Groups.
IEEE Network 18 , 3 (2004),36–41.[30] D
HAMDHERE , A.,
AND D OVROLIS , C. The Internet is Flat:Modeling the Transition from a Transit Hierarchy to a PeeringMesh. In
Proc. of ACM CONEXT (2010).[31] D
INITZ , Y., G
ARG , N.,
AND G OEMANS , M. X. On theSingle-Source Unsplittable Flow Problem.
Combinatorica 19 ,1 (1999), 17–41.[32] D
RIOLI , C., A
LLOCCHIO , C.,
AND B USO , N. NetworkedPerformances and Natural Interaction via LOLA: LowLatency High Quality A/V Streaming System. In
InformationTechnologies for Performing Arts, Media Access, andEntertainment , vol. 7990. Springer, 2013.[33] E
PPSTEIN , D. Finding the k Shortest Paths.
SIAM Journal oncomputing 28 , 2 (1998), 652–673.[34] E
SQUIVEL , H.
ET AL . RouteBazaar: An EconomicFramework for Flexible Routing . Tech. rep. 1654, Univ. ofWisconsin - Madison, 2009.[35] E
URO AN , J., AND A MMAR , M. H. Dynamic TopologyConfiguration in Service Overlay Networks: A Study ofReconfiguration Policies. In
Proc. of IEEE INFOCOM (2006).[37] F
ARREL , A.
ET AL . A Path Computation Element(PCE)-Based Architecture. RFC 4655, Informational, 2006.[38] F
ISCHER , A.
ET AL . Virtual Network Embedding: A Survey.
IEEE Communications Surveys Tutorials 15 , 4 (2013),1888–1906.[39] F
RÉVILLE , A.,
AND H ANAFI , S. The Multidimensional 0-1Knapsack Problem-Bounds and Computational Aspects.
Annals of Operations Research 139 , 1 (2005), 195–227.[40] F
RIKHA , A.
AND L AHOUD , S. Performance Evaluation ofPre-computation Algorithms for Inter-domain QoS Routing.In
Proc. of ICT (2011).[41] G AO , L., AND R EXFORD , J. Stable Internet Routing WithoutGlobal Coordination. In
Proc. of ACM SIGMETRICS (2000).42] G
ARROPPO , R. G., G
IORDANO , S.,
AND T AVANTI , L. ASurvey on Multi-constrained Optimal Path Computation:Exact and Approximate Algorithms.
Comput. Netw. 54 , 17(2010), 3081–3107.[43] G
ELEJI , G.
ET AL . A Performance Analysis of Inter-DomainQoS Routing Schemes Based on Path Computation Elements.In
Proc. of HONET (2008).[44] G
ILL , P., S
CHAPIRA , M.,
AND G OLDBERG , S. A Survey ofInterdomain Routing Policies.
ACM SIGCOMM CCR 44 , 1(2013), 28–34.[45] G
IOTSAS , V.
ET AL . Inferring Complex AS Relationships. In
Proc. of ACM IMC (2014).[46] G
ODFREY , P. B.
ET AL . Pathlet Routing. In
Proc. of ACMSIGCOMM (2009).[47] G
REGORI , E.
ET AL . The Impact of IXPs on the AS-levelTopology Structure of the Internet.
Comput. Commun. 34 , 1(2011), 68–82.[48] G
UPTA , A.
ET AL . Provisioning a Virtual Private Network: ANetwork Design Problem for Multicommodity Flow. In
Proc.of ACM STOC (2001).[49] G
UPTA , A.
ET AL . SDX: A Software Defined InternetExchange. In
Proc. of ACM SIGCOMM (2014).[50] G
UROBI O PTIMIZATION
AIDEGGER , T.,
AND B ENYÓ , Z.
Extreme Telesurgery .InTech, 2010.[52] H U , C. ET AL . A Measurement Study on PotentialInter-Domain Routing Diversity.
IEEE TSNM 9 , 3 (2012),268–278.[53] H
UNT , R. Review: A Review of Quality of ServiceMechanisms in IP-based Networks - Integrated andDifferentiated Services, Multi-layer Switching, MPLS andTraffic Engineering.
Comput. Commun. 25 , 1 (2002),100–108.[54] H
URRICANE E LECTRIC I NTERNET S ERVICES
ATZ -B ASSETT , E., M
ADHYASTHA , H. V., J
OHN , J. P.,K
RISHNAMURTHY , A., W
ETHERALL , D.,
AND A NDERSON ,T. E. Studying black holes in the internet with hubble. In
NSDI (2008), pp. 247–262.[57] K
LÖTI , R., A
GER , B., K
OTRONIS , V., N
OMIKOS , G.,
AND D IMITROPOULOS , X. A Comparative Look into Public IXPDatasets.
ACM SIGCOMM CCR (2016).[58] K
LÖTI , R.
ET AL . Policy-Compliant Path Diversity andBisection Bandwidth. In
Proc. of IEEE INFOCOM (2015).[59] K
ORKMAZ , T.,
AND K RUNZ , M. Multi-constrained OptimalPath Selection. In
Proc. of IEEE INFOCOM (2001).[60] K
OTRONIS , V., D
IMITROPOULOS , X.,
AND A GER , B.Outsourcing the Routing Control Logic: Better InternetRouting Based on SDN Principles. In
Proc. of ACM HotNets (2012).[61] K
OTRONIS , V.
ET AL . Investigating the Potential of theInter-IXP Multigraph for the Provisioning of GuaranteedEnd-to-End Services. Tech. Rep. 360, ETH Zurich,Laboratory TIK, Feb 2015.[62] K
ÜHNE , M. Update on AS Path Lengths Over Time.https://labs.ripe.net/Members/mirjam/update-on-as-path-lengths-over-time, 2012. RIPE NCC Article.[63] L
ABOVITZ , C.
ET AL . Internet inter-domain traffic.
ACMSIGCOMM CCR 41 , 4 (2011), 75–86.[64] L
AKSHMINARAYANAN , K., S
TOICA , I.,
AND S HENKER , S.Routing as a Service. Tech. rep. ucb-cs-04-1327, UC Berkeley,2004.[65] L
EVIN , D.
ET AL . Logically Centralized?: State DistributionTrade-offs in Software Defined Networks. In
Proc. of ACMHotSDN Workshop (2012).[66] L
UCKIE , M.
ET AL . AS Relationships, Customer Cones, andValidation. In
Proc. of ACM IMC (2013).[67] L
UMEZANU , C.
ET AL . Triangle Inequality and RoutingPolicy Violations in the Internet. In
Proc. of PAM (2009).[68] M
ARBACH , P., M
IHATSCH , O.,
AND T SITSIKLIS , J. CallAdmission Control and Routing in Integrated ServicesNetworks Using Neuro-dynamic Programming.
IEEE JSAAC18 , 2 (2000), 197–208.[69] M
ARESCAUX , J.
ET AL . Transcontinental Robot-assistedRemote Telesurgery: Feasibility and Potential Applications.
Annals of Surgery 235 , 4 (2002), 487.[70] MAXMIND, I NC . GeoLite2 Free Downloadable Databases.http://dev.maxmind.com/geoip/geoip2/geolite2/. Datasetacquired on 2014-04-10.[71] M C K EOWN , N.
ET AL . OpenFlow: Enabling Innovation inCampus Networks.
ACM SIGCOMM CCR 38 , 2 (2008),69–74.[72] N
ETWORK
X. NetworkX. http://networkx.github.io/.[73] O
BERORTNER , E.
ET AL . Monitoring Performance-relatedQoS Properties in Service-oriented Systems: A Pattern-basedArchitectural Decision Model. In
Proc. of EuroPLoP (2012).[74] ON.LAB. OpenVirtex: Programmable Virtual Networks.http://ovx.onlab.us/.[75] O
RDA , A. Routing with End-to-end QoS Guarantees inBroadband Networks.
IEEE/ACM TON 7 , 3 (1999), 365–374.[76] P
EERING
ETER , S.,
ET AL . One Tunnel is (Often) Enough. In
Proc. ofACM SIGCOMM (2014).[78] P
RASAD , R.
ET AL . Bandwidth Estimation: Metrics,Measurement Techniques, and Tools.
IEEE Network 17 , 6(2003), 27–35.[79] R
ICHTER , P.
ET AL . Peering at Peerings: On the Role of IXPRoute Servers. In
Proc. of ACM IMC (2014).[80] S
CHAFFRATH , G.
ET AL . Network VirtualizationArchitecture: Proposal and Initial Prototype. In
Proc. of ACMVISA Workshop (2009).[81] S
PRINTSON , A.
ET AL . Reliable Routing with QoSGuarantees for Multi-Domain IP/MPLS Networks. In
Proc. ofIEEE INFOCOM (2007).[83] V
ALANCIUS , V.
ET AL . MINT: A Market for INternet Transit.In
Proc. of ACM CONEXT (2008).[84] V
ASSEUR , J.,
AND L E R OUX , J. Path Computation Element(PCE) Communication Protocol (PCEP). RFC 5440,Standards Track, 2009.[85] W
ANG , J., Z
HOU , M.,
AND L I , Y. Survey on the End-to-EndInternet Delay Measurements. In High Speed Networks andMultimedia Communications , vol. 3079. Springer, 2004,pp. 155–166.[86] X
IAO , X.
Technical, Commercial and Regulatory Challengesf QoS: An Internet Service Model Perspective . MorganKaufmann, 2008.[87] X
IAO , X.,
AND N I , L. M. Internet QoS: A Big Picture. Netwrk. Mag. of Global Internetwkg. 13 , 2 (1999), 8–18.[88] Y
ANUZZI , M.
ET AL . On the Challenges of EstablishingDisjoint QoS IP/MPLS Paths Across Multiple Domains.
IEEECommunications Magazine 44 , 12 (2006), 60–66.[89] Z
HANG , Z.-L.
ET AL . Decoupling QoS Control from CoreRouters: A Novel Bandwidth Broker Architecture for Scalable Support of Guaranteed Services.
SIGCOMM CCR30 , 4 (2000), 71–83.[90] Z
HOU , X., W EI , J., AND X U , C.-Z. Quality-of-serviceDifferentiation on the Internet: A Taxonomy. J. Netw. Comput.Appl. 30 , 1 (2007), 354–383.[91] Z
HUANG , Y.