An agile and distributed mechanism for inter-domain network slicing in next generation mobile networks
Jalal Khamse-Ashari, Gamini Senarath, Irem Bor-Yaliniz, Halim Yanikomeroglu
AAn agile and distributed mechanism forinter-domain network slicing in next generationmobile networks
Jalal Khamse-Ashari
1, 2 , Gamini Senarath , Irem Bor-Yaliniz
1, 2 , and Halim Yanikomeroglu Department of Systems and Computer Engineering, Carleton University, Ottawa, Canada Huawei Canada Research Center, Ottawa, Canada
Abstract —Network slicing is emerging as a promising methodto provide sought after versatility and flexibility to cope with ever-increasing demands. To realize such potential advantages and tomeet the challenging requirements of various network slices in anon-demand fashion, we need to develop an agile and distributedmechanism for resource provisioning to different network slicesin a heterogeneous multi-resource multi-domain mobile networkenvironment. We formulate inter-domain resource provisioning tonetwork slices in such an environment as an optimization problemwhich maximizes social welfare among network slice tenants(so that maximizing tenants’ satisfaction), while minimizingoperational expenditures for infrastructure service providers atthe same time. To solve the envisioned problem, we implement aniterative auction game among network slice tenants, on one hand,and a plurality of price-taking subnet service providers, on theother hand. We show that the proposed solution method resultsin a distributed privacy-saving mechanism which converges tothe optimal solution of the described optimization problem. Inaddition to providing analytical results to characterize the per-formance of the proposed mechanism, we also employ numericalevaluations to validate the results, demonstrate convergence ofthe presented algorithm, and show the enhanced performance ofthe proposed approach (in terms of resource utilization, fairnessand operational costs) against the existing solutions.
Index Terms —Inter-domain network slicing, virtualization,end-to-end service provisioning, distributed implementation,multi-resource allocation.
I. I
NTRODUCTION
Network slicing is expected to become an integral part ofnext generation mobile networks, since slicing helps networksto be more versatile. Built upon recently developed tech-nologies, such as network function virtualization (NFV) andsoftware defined networks (SDN), network slicing lets mul-tiple logical networks share the same physical infrastructure.This virtualization technique can enable one network providingmultiple services with extremely different requirements, whilemaintaining isolation. As opposed to many QoS assurancemethods, network slicing differentiates between different traf-fic as per their requirements, and for the same kind of trafficas per different tenants . Therefore, network slicing is a * The paper has been accepted for publication at IEEE Transactions onMobile Computing. Please refer to DOI 10.1109/TMC.2021.3061613 for thefinal version. Tenant can be thought as a group of users or a 3rd-party consumer usingthe communication services to provide other communication services to users. unique method for end-to-end granular network managementand service provisioning. Moreover, by integrating with othertechnologies (such as cloud computing, cloud-RAN [1] andmobile edge computing [2]), it introduces more flexibility indeployment resulting in significant reduction in both opera-tional and capital expenditures. Accordingly, network slicinghas attracted significant attention from both industry andacademia. While the leading standardization bodies for thenext generation wireless networks included network slicing intheir work items [3]–[6], there is also a significant numberof studies from the academia investigating characteristics anddynamics of network slicing [7]–[14].From the control plane (CP) perspective in 3GPP 5Gstandardization, a network slice can consist of several networkslice instances (NSIs). NSIs help not only to adjust the networkslice resources based on the service requirements, but alsoto differentiate between even the same type of QoS flowsbased on granular network policies. The network management(NM) perspective involves also the network slice subnet in-stance (NSSI) concept for granular management of networkdomains (e.g., radio access network (RAN), core network(CN), transport network (TN)) and provides different levels ofexposure to the tenants. Briefly, a managed NSI can consistof several NSSIs. The NSSIs can be chosen by the networkmanager based on geographical circumstances, network andaccess technology (e.g., a mmWave NSSI, 4G core NSSI),vendor differences (e.g., parts of network resources are fromvendor X and parts of network resources are from vendorY), and other factors that are significant from the networkmanagement perspective. In this study it is assumed that NSSIsare generated based on network domain and geographicalregion.To achieve the potential advantages of network slicing,network operators need to address several challenges, such asproviding QoS guarantees for different network slices/services,while efficiently utilizing the capacity of the infrastructurenetwork. Fulfilling such requirements mainly depends on theunderlying resource provisioning mechanism that is used forresource management and placement of virtualized networkfunctions (VNFs) [8], [9], [15]. There are several work inthe literature which study resource allocation to VNFs inthe context of a general topology network. Indeed, the VNFresource allocation problem can be traced-back/reduced to the a r X i v : . [ c s . G T ] F e b irtual network embedding (VNE) problem, wherein a virtualnetwork is embedded on the top of a substrate network [16].However, it usually results in a mixed integer linear program(MILP) which is shown to be NP-hard [17]. Hence, differentheuristic methods are proposed to address VNF placementand resource provisioning in the network of an infrastructureservice provider (ISP) or an enterprise network [18]–[22]. Thereader may refer to Section II for a more detailed literaturesurvey.By capturing the underlying resource model in a real mobilenetwork environment, leveraging the possibility for multi-pathconnectivity, and by exploiting the constraints that are imposedfor placement of VNFs in practice, we propose a novel for-mulation for end-to-end resource provisioning, which avoidsintractable complexities of solving an MILP. Particularly, giventhe placement constraints, there remains a limited number ofnodes in each domain, and therefore a limited number of pathswhich can provide service to NSIs in a certain region. Byconsidering pre-determined paths (each comprising a pre-determined chain of VNFs) and exploiting the possibilityfor multi-path connectivity, we avoid intractable complexitiesof solving an MILP (for placement of VNFs), while yetproviding the flexibility to optimize routing across differentpaths/chains-of-VNFs. The formulated problem indeed relaxesthe constraint of single path routing of MILPs by exploitingthe possibility for multi-path connectivity in next generationnetworks. A detailed description of the system model ispresented in Section III. The proposed solution presents a market equilibrium approach which results in an agile anddistributed mechanism for end-to-end resource provisioningto NSIs in a multi-resource multi-domain mobile networkenvironment (such as an integrated terrestrial-aerial-satellitenetwork). The proposed market-based solution best resemblesthe real-world interaction between the infrastructure serviceproviders (in a vertical heterogeneous network [23]) andvirtual mobile network operators (i.e., tenants) which acquireresources to implement different NSIs. Market-based mech-anisms have recently received considerable attention [24]–[26], since they may lead to a desirable performance in termsof resource utilization and energy-efficiency, in addition tomaximizing social welfare and user satisfaction. Of course thedistributed implementation comes at the price of a signalingoverhead to exchange certain information between the networkslice tenants and the infrastructure providers. The requiredinformation to exchange, however, is kept minimal as it islimited only to the resource prices and the allocated resources.Such an occasional signaling (which should be performedin case of an update) may not be significant compared tothe persistent measurement and monitoring signaling that isusually communicated for the sake of network managementand maintenance. Contributions:
The contributions of this paper are summa-rized as follows. • Problem formulation:
We propose a new formulation for It is assumed that the paths are a-priori determined by running a pathfinding algorithm. The detailed implementation of such an algorithm, however,is out of the scope of this paper. inter-domain resource provisioning to network slices , devel-oping a framework which unifies the allocation of differenttypes of resources (including network bandwidth as well ascomputing resources) in a heterogeneous multi-domain en-vironment. We formulate resource provisioning to networkslices in such an environment as a concave maximizationproblem which maximizes social welfare among networkslice tenants, while minimizing operational expenditures(OPEX) for infrastructure service providers at the sametime. • An auction-based solution:
To solve the proposed resourceprovisioning problem, we devise an iterative auction gameamong network slice tenants, each bidding for differentresources so as to maximize a local payoff function. Theinfrastructure service providers (owning data centers oraccess point (APs) across different domains), on the otherhand, decide on the resource prices. The described gameis shown to be at a Nash equilibrium if and only if it isat an optimal solution to the global concave optimizationproblem (which provably results in a unique optimal trafficvolume for each NSI). • Characterizing the solution:
It is shown that the proposedapproach results in a distributed privacy-saving mechanismwhich does not require sharing any private information (e.g.,resource capacities of data centers or APs, and demand pro-file or payoff function of tenants) among different parties.We further analyze the performance of the proposed mech-anism, by demonstrating certain properties (such as envy-freeness and sharing incentive ) that are deemed desirablefor efficient and fair allocation of resources. • Demonstrating the performance:
In addition to analyticalresults , we also employ numerical evaluations to showthe validity of the results, demonstrate convergence of thepresented algorithm, and show the superior performance ofthe proposed mechanism (in terms of resource utilization,fairness and operational expenditures) compared to heuris-tics and other existing solutions.
Organization:
The background and related work is pre-sented in Section II. The system model is characterized inSection III. Our proposed distributed resource provisioningmechanism is described in Section IV. Some import exten-sions to the original formulation (such as considering budget-constrained tenants, and exploiting the capabilities at themobile edge) are presented in Section V. The numerical resultson evaluating the performance of the proposed scheme arereported in Section VI. The paper is concluded in Section VII.II. B
ACKGROUND AND R ELATED W ORK
A. Background on Enabling Technologies
Network slicing is a promising solution for the next gener-ation mobile networks which allows to build multiple logicalnetworks on the top of a shared infrastructure, so that (virtual)mobile network operators may provide services tailored fordifferent network slices with different QoS requirements. Toachieve potential advantages of network slicing, considerableresearch activities are dedicated to developing the underly-ing/enabling technologies (such as, NFV, VMs, containers,tc.) that are required for implementation of virtualized net-work functions and services [27]–[31]. NFV decouples thesoftware implementation of network functions from the un-derlying hardware. Hence, different network appliances and/ormiddle-box processings (such as firewalls, traffic shapers, etc.)can be implemented on an VM running on commercial off-the-shelf hardware (such as general purpose server, storageand switches), as opposed to dedicating specialized hardwaredevices for implementation of network functions/protocolsin conventional communications networks [32], [33]. Oneof the main advantages of virtualized network functions isprogrammability, so that future changes can be applied easilyby just updating the software without the need to replacingthe hardware [33].Another line of research in this area includes the worksinvestigating complementary technologies such as CRAN(cloud/cetralized radio access network [1]), mobile edge com-puting (MEC), and fog computing [2], [25]. CRAN is a newarchitecture which is introduced as a cost-efficient solution toaddress the scalability issue with the growing user demandsin 5G mobile networks [1]. The main idea is to achievemultiplexing gain by pooling baseband units (BBU) fromseveral base stations into a centralized radio access unit. Itintroduces substantial savings on both operational expenditures(due to enhanced energy-efficiency) and capital expendituresfor implementation of BBUs. Moreover, it may improve thenetwork performance by providing the possibility to performjoint processing of signals from different base station [1].Mobile edge computing, on the other hand, is an emergingplatform which integrates new technologies, such as cloudcomputing and NFV, with the conventional telecommunicationnetworks to provide computational capabilities at the mobileedge, enabling a wide range of new applications/services [2].In [25] a market equilibrium approach has been proposed toefficiently allocate the resources at the mobile edge to budgetconstrained users.
B. Related Work
Different business models and architectural solutions havebeen presented for application of NFV and network slicing towireless/mobile networks [8]–[10]. In [9] the authors proposethe concept of hierarchical network slice as a service, enablingthe operators to provide customized end-to-end (E2E) cellularnetwork as a service. In [8] an architecture is presented forRAN virtualization in network-slicing-based 5G networks.Moreover, it discusses how to address different challenges thatare involved in RAN virtualization (such as power control,channel allocation and mobility management) in the proposedarchitecture. Indeed, the concept of network slicing can beapplied to different domains (including RAN, backhaul andcore network) individually [11], [12], [34]–[36], or providingan E2E network slice as a service [26].To achieve potential advantages of network slicing andto provide E2E QoS guarantees for different network sliceswith diverse E2E QoS requirements, an efficient E2E resourceprovisioning mechanism is required. There are several worksin the literature which study the problem of E2E resource provisioning to network slices in the context of a generaltopology network (e.g., an ISP or an enterprise network).The work in [15] proposes a high-level E2E orchestrationframework wherein the problem is broken to placement ofvirtual machines across the network, and then resource al-location to network functions on virtual nodes. However,it does not provide a technical solution for placement andresource allocation to VNFs. The work in [21] formulatesplacement of chain of VNFs as an MILP, and then proposes adynamic-programming-based heuristic which achieves a sub-optimal solution. The complex network theory is used in [37]for ranking the nodes of an infrastructure network, and thenmapping them to VNFs. A heuristic solution is proposed in[20] for joint VNF placement and online request scheduling tothe instantiated VNFs. In [19], a multi-objective optimizationproblem (which minimizes links traffic along with the numberof busy CPU cores) is formulated for optimal routing in thenetwork of an ISP. In [18] a heuristic method is proposedfor placement of elastic network functions, while it minimizesthe operational expenditures in order to address the trade-offbetween bandwidth and host resource consumption. Indeed, allof these studies and also the work in [22] can be viewed asan extension/variant of virtual network embedding problem,resulting in an MILP which is shown to be NP-hard [17].Hence, all of these studies come up with a heuristic solution.Moreover, the underlying model in none of these work iscomprehensive in the sense to consider an accurate resourcemodel for data centers (comprising multiple types of resourceswith heterogeneous resource capacities).The work in [26] is the most related study to the frameworkpresented herein. However, it does not provide the possibilityto exploit different paths/chains-of-VNFs. Indeed, the solutionin [26] is based on a model assuming fixed placement ofVNFs over a pre-determined path for each network slice.Moreover, it does not address cost-aware resource pricing andminimization of the OPEX. Lastly, as we show in Section VI,the solution of [26] may not satisfy some desirable fairnessrelated property (such as sharing incentive). Adopting a marketequilibrium approach, we develop a distributed mechanism forE2E resource provisioning to different network slices. Theproposed mechanism is shown to maximize social welfareamong network slice tenants, while minimizing the OPEXacross different domains. In our formulation, we considera number of possible forwarding paths (each comprising apre-determined chain of VNFs) for each network slice. Inthis way, we avoid intractable complexities in placement ofVNFs, while yet providing the flexibility to optimize routingacross different paths/chains-of-VNFs. Moreover, we consideran accurate resource model (comprising multiple types ofresources) for data centers.III. S
YSTEM M ODEL
Based on the developments in 5G standardization, for eachend-to-end NSI n , n = 1 , , · · · , N , one may consider threeNSSIs in different domains, namely, radio access, backhaul,and the core network. Fig. 1 shows a sample NSI comprisingAN, CRAN , and core NSSIs. Each NSSI comprises aspecific sequence of VNFs in a certain domain. Note that it ispossible to consider several NSSIs in each domain. In addition,there might be more than one service provider node (i.e.,data center or AP) in each domain. The resources (includingnetwork bandwidth and computing resources) provided by adata center or AP in a particular domain are used to implementnetwork functions for NSSIs in the same domain. Note thatvirtualization helps to tailor the network function chains withrespect to the services provided by different NSSIs. Hence,each data center or AP has to implement several networkfunction chains, depending on the network services providedby NSSIs.Given the deployed infrastructure, it is assumed that thetraffic flow in each area, l = 1 , , ..., A , can be forwarded overcertain paths towards the core network [26], [38]. Particularly,mobile users in each geographical area could be served bya number of local radio access units. Then, the traffic flowfrom each radio access unit could be forwarded over pre-defined paths towards the core network. We assume that theset of forwarding paths are pre-defined based on VNF place-ment constraints (including vendor/technology compatibility,transmission latency constraints, etc.) for each geographicalarea. Let P l denote the set of paths originating from area l . Each path p ∈ P l originating from area l comprises asequence of nodes over different domains of the network, p = { l, i, j, k } , i ∈ I , j ∈ I , k ∈ I , where I , I ,and I denote the set of node indices over RAN, CRAN,and core network, respectively. A communication link witha certain nominal capacity is assumed between every twoconsecutive nodes of a path. It is assumed that the nodesin each domain are in charge of allocating the inboundcommunication bandwidth. In general, we use a capacityvector of size M , C i = [ C i, , ..., C i,r , ..., C i,M ] , to specify theamount of available resources, such as CPU, RAM, memorybandwidth, the outbound bandwidth towards the internet, andthe inbound communication bandwidths at each node i . Theresource model for a generic node is shown on the top-rightcorner of Fig. 1.We use the notion of demand vector [39], d pn,i = [ d pn,i,r ] ,to specify the amount of different resources, r = 1 , , · · · , M ,that are required for processing one unit of traffic for NSI n when routed to node i through path p . The reader maynote that d pn,i is node-dependent, so that it may reflect avariable performance for the same NSI over different nodes.For instance, at the RAN domain the number of physicalresource blocks that are required for transmitting one unit oftraffic for NSI n might be different from one radio access pointto another [34]. It is worth noting that the demand vector con-vention follows a physical-layer abstraction model to capturemid-term statistics of the RAN (averaged over few minutes,e.g., to absorb short-term variations), while yet providing thepossibility to capture some physical layer complexities (e.g., CRAN is actually a part of 5G core network, based on 5G standardizationby SA2 group. In this article, we use the terms “CRAN” and “core” todistinguish between the domains where these functionalities are implemented. A node at the RAN represents an access-point/base-station, whereas a nodeat backhaul or core network may represent a server or a data center. variations that may occur due to UE-mobility). Moreover, thedemand vector is slice-specific, so it can account for variableperformance of different NSIs at the same node. This canbe particularly useful to capture slice dependent complexitiesat the RAN. The demand vector, d pn,i , is also considered tobe path dependent so as to account for (possibly) differentchains of network functions over different paths. Also, it mayaccount for multiple inbound communication links at eachnode (including the RAN).As an example, consider a node (i.e., server) compris-ing 16 CPU cores, 32 GBytes of RAM, 1 Gb/s memoryBW, 10 Gb/s outbound BW, and 4 input ports each with abandwidth of 2.5 Gb/s, complying with the resource modelin Fig 1. The capacity vector here is represented as C i =[16 , , , , . , . , . , . . A demand vector, for instance,can be described as d pn,i = [0 . , , . , . , . , , , ,which specifies the amount of different resources requiredfor processing one unit of traffic (e.g., 1 Gb/s) for NSI n when routed through the first input port to node i . Identifyingdemand vectors per routing paths is the key to account forlimited bandwidth of communication links. This is in contrastto existing works in the literature which only consider theoverall network bandwidth over each data center or server [26],[40]–[42]. Such studies may only address the limited capacityof the server switch in accepting the incoming traffic, but maynot capture the capacity constraint of communication links.Note that one mobile user can obtain multiple services,each provided by a different NSI. The key point is that thegranularity level is per slice per traffic-flow, rather than permobile user. Let x pn denote the capacity allocated/provisionedto NSI n over path p . It actually represents the (maximum)volume of traffic that can be forwarded for NSI n over path p . The (maximum) volume of traffic which can be forwardedfor the users/tenant of NSI n in area l is given by x n,l := (cid:88) p ∈P l x pn . (1)Indeed, the traffic admitted from users of NSI n should bekept below x n,l in each area. The resources allocated fromnode i to NSI n (to provision x pn units of traffic over eachpath p which node i belongs to) is given by a n,i = [ a n,i,r ] , a n,i,r = (cid:88) p : i ∈ p x pn d pn,i,r . (2)Generally, for an allocation, x := { x pn | n ∈ N , p ∈ P l , l =1 , , .., A } to be feasible, the following has to be satisfied: (cid:88) n ∈N a n,i,r = (cid:88) n ∈N (cid:88) p : i ∈ p x pn d pn,i,r ≤ C i,r , ∀ i, r. (3)The key notations are summarized in Table I. It is worthnoting that the resource capacity, C i,r , represents the nominalcapacity of resource r at node i . For example, the communica-tion bandwidth of an AP is defined as the maximum achievabledata rate when using the best modulation and coding scheme.Depending on their performance requirements and to rectify In practice, a hypervisor shares the CPU-time among different VMsrunning on the same server. So, any fraction of a CPU core can be assignedto each VM [39]. erver Server Server
Internet1. CPU
2. RAM
3. Memory BW
4. Network BW . N B W . N B W M . N B W Area 1
Area 2
Area A ... D a t a - ce n t er R e s o u rce M o d e l Server Server Server Server Server Server
RAN NSSI
RLC PDCP
MAC PHY
AMF SMFUPF
AUSF UDMPCF
Core-NSSI
CRAN NSSIA sample NSI
Fig. 1:
End-to-end network slice virtualization in a multi-domain mobile network environment. Sample paths from the radio access to the core networkdomain are shown in the figure. Each AP or data center is in charge of allocating different types of resources (including the inbound network bandwidth(NBW)) to VNFs of the NSSIs in each domain. The resource model for a generic node is shown on the top-right corner.
TABLE I: A summary of key notations
Notation Description C i,r Capacity of resource r at node i d pn,i = [ d pn,i,r ] The demand vector for NSI n at node i using path p a n,i = [ a n,i,r ] Vector of resources allocated from node i to NSI nx pn The traffic capacity provisioned to NSI n over path p P l The set of paths originating from area lx n,l The traffic capacity provisioned to NSI n in area lU n,l ( x n,l ) The utility (function) for NSI n in area lq i,r The OPEX for unit of resource r at node i . µ i,r The price to book one unit of resource r at node i Π n ( x n ; µ ) The net payoff function for NSI nw n,i,r The payment of NSI n to node i for resource r certain physical layer complexities, however, some NSIs mayneed to use lower order modulation and coding schemes. Suchrequirements can be flexibly reflected in the demand vectors.IV. D ISTRIBUTED R ESOURCE P ROVISIONING
In this section, we first formulate the problem of re-source provisioning to different network slices as a centralizedsystem-wide optimization problem, maximizing social welfareamong network slice tenants, while minimizing OPEX for theinfrastructure service providers at the same time. We showthat the described problem can be solved by implementing an auction game among network slice tenants on one hand,and a set of price-taking infrastructure service providers, onthe other hand. In the proposed solution, each network slicetenant maximizes a local payoff function, while a certainresource allocation and pricing scheme is implemented at eachinfrastructure node. We further characterize the performance ofthe proposed mechanism by demonstrating certain propertieswhich are highly desirable for efficient and fair allocation ofthe resources.
A. The System-Wide Objective
Each network slice tenant may gain a utility (i.e., profit ) of U n,l ( x n,l ) out of the allocated capacity in each area l . Theutility function for each NSI n can be reasonably representedby a concave function . Problem 1: The system-wide optimization problem max x (cid:88) n,l U n,l (cid:88) p ∈P l x pn − (cid:88) i,r q i,r (cid:88) n (cid:88) p : i ∈ p x pn d pn,i,r (4)s.t. (cid:88) n (cid:88) p : i ∈ p x pn d pn,i,r ≤ C i,r , ∀ i, r, (5) x pn ≥ , ∀ n, p. (6)he parameter q i,r is the OPEX for node i to provision oneunit of resource r . So the objective in Problem 1 is to maximizethe overall utility for different NSIs (i.e., maximizing socialwelfare ), while minimizing OPEX for service providers. Thefollowing solution method and the presented analytical resultsare established in general for every utility function , U n,l ( · ) ,that is continuously differentiable and strictly concave. As anexample, one may assume U n,l ( z ) from a common class ofutility functions for which the derivative (i.e., marginal benefit )is described as U (cid:48) n,l ( z ) = ( φ n,l /z ) α n , α n > [43], [44]. Forthis class of functions, the parameter α n (which typically takeson a value in the range of [1 , ) determines the shape of theutility function [43], [44]. The parameter φ n,l may present thetraffic load for each NSI n in a certain area. We use the class offunctions for the sake of numerical evaluations in Section VI.For NSIs with delay sensitive applications, we can accountfor the end-to-end delay in the utility function. In particular,the end-to-end delay for NSI n which forwards an admissible volume of φ n,l unit of traffic can be estimated by [45] D n ( x n,l ) := Lx n,l − φ n,l + h n,l Lx n,l , for x n,l > φ n,l , (7)where L is the average packet length (in bits) and h n,l is(proportional to) the total number of steps that each packet isprocessed in the network . The net revenue for NSI n then isgiven by R n ( x n,l ) := U n ( x n,l ) − β n,l D n ( x n,l ) , (8)where the parameter β n,l in (8) relates the delay to loss inprofit. For delay sensitive slices, we can consider the netrevenue instead of the utility function in (4). The parameter β n,l then could be properly adjusted to keep the end-to-enddelay less than a desired threshold. The proposed solutionmethod in the following can be viewed as a variant of themarket equilibrium approach which was originally presentedin [44] and here is extended to a multi-resource multi-domainnetwork. B. End-to-end Network Slice Management
For each NSI we assume an end-to-end slice managerwhich monitors the end-to-end network slice performance, anddecides on bidding for the resources at different domains of thenetwork. It is assumed that the network slice manager has (a-priori) found the demand vector for each service function chainof an NSI, by getting feedbacks and monitoring the resourceutilization of the VMs implementing the function chains.Particularly, assume that ˆ d n,i = [ ˆ d n,i,r ] is an (arbitrary) initialestimate of the demand vector for a service function at node i .Given that the resources are initially allocated (to the servicefunction chain) proportional to ˆ d n,i , the resource utilizationof the VM, which implements it, is described in terms of ˆ d n,i and the true value of the demand vector [46], u n,i,r := d n,i,r ˆ d n,i,r min r (cid:48) { ˆ d n,i,r (cid:48) d n,i,r (cid:48) } . (9) The admitted traffic volume could be a fraction of offered load in eachregion. In Section V-B, we discuss a potential solution to adjust the admittedtraffic volume. By definition, D n ( x n,l ) := −∞ for x n,l ≤ φ n,l . So, given the resource utilization and an ( arbitrary ) initialestimate of the demand vector, the true value of the demandvector can be found using (9). In the following, it is assumedthat the true value of the demand vector is known to theslice manager. Based on the demand vector, the slice managerknows the amount of resources that are required for forward-ing/processing a certain volume of traffic at each domain of thenetwork. Moreover, it is assumed that the utility function (as afunction of NSI traffic volume) is known to the network slicemanager. Each network slice manager then needs to maximizeits net payoff function (i.e., the gained utility minus the totalpayment), Π n ( x n ; µ ) = (cid:88) l U n,l ( x n,l ) − (cid:88) i,r (cid:88) p : i ∈ p x pn d pn,i,r µ i,r , (10)where µ i,r is the price to book one unit of resource r atnode i . The reader may note that x n,l is described in terms ofpath traffic volumes, x pn (see (1)). The network slice managerthen strives to find an allocation x n := { x pn | p ∈ P l , l =1 , , ..., A } which solves the following problem. Problem 2: Network slice optimization problem max x n Π n ( x n ; µ ) (11)Subject to: x pn ≥ . (12)Given a solution to this problem, the network slice managerdecides how much capacity should be provisioned for eachNSI n over each path, which subsequently specifies the amountof resources which should be allocated from each node toNSI n (see (2)). The bid/payment that is made by NSI n forresource r of node i is given by w n,i,r := µ i,r (cid:88) p : i ∈ p x pn d pn,i,r . (13) Note 1.
The payment matrix for each NSI n , W n := [ w n,i,r ] ,is determined based on path traffic volumes/capacities. Thatis, W n = W n ( x n ) .C. Resource Management for Nodes Each service provider node, which implements a subnet-work functionality (i.e., NSSI) for NSI n , receives some bids w n,i,r > for different resources. It is assumed that each nodedoes not price discriminate among different NSIs. That is, eachnode i chooses a certain price, µ i,r , for unit of each resource r ,and subsequently allocates an amount of a n,i,r = w n,i,r /µ i,r of resource r to each NSI n . It is assumed that node i isincurred an OPEX of q i,r for providing one unit of resource r . This may include electrical energy costs, and the costto transport traffic over the network of an internet serviceprovider. To ensure that OPEX are covered by NSI payments,the price for one unit of resource r is chosen to be µ i,r = max (cid:26) q i,r , (cid:80) n w n,i,r ( x ) C i,r (cid:27) , (14) We consider a competitive market in the presence of a plurality of serviceprovider nodes. So it is assumed that each subnet service provider node appliesthe true value of its OPEX to set the resource prices. o that resource r is cleared (that is (cid:80) n a n,i,r = C i,r ) onlywhen (cid:80) n w n,i,r /C i,r ≥ q i,r , In general, (cid:88) n a n,i,r = (cid:88) n w n,i,r µ i,r =: η i,r C i,r , (15)where η i,r := min (cid:26) (cid:80) n w n,i,r C i,r q i,r , (cid:27) , (16)so that a fraction of η i,r < of resource r is allocatedto different NSIs when (cid:80) n w n,i,r /C i,r < q i,r . We showthat employing such a simple pricing scheme in conjunctionwith the end-to-end NSI resource management mechanism ofSection IV-B results in minimizing the operational costs inthe whole network, while maximizing social welfare amongnetwork slice tenants. D. Distributed Online Mechanism
In this section we devise a distributed online mechanismwhich implements an auction game among the network slicemangers, on one hand, and a set of price-taking infras-tructure/subnet service providers, on the other hand. In theproposed mechanism, the network slice managers (each imple-menting one NSI) may iteratively update their bids for variousresources of different nodes (see (13)). Given the payments bynetwork slice managers, each node (i.e., AP or data center)decides on the resource prices (see (14)) and declares theresource prices to all network slice managers. Each networkslice manager then may find its desired (i.e., optimal) pathtraffic volumes, and accordingly updates its payments, so asto improve its achievable payoff. Given the payments from allNSIs, the resource prices are updated in a way that the capacityconstraints are met for all resources. This procedure continuesuntil no network slice is willing to update its payments. At thispoint (which is so-called a Nash equilibrium ) a stable priceis established for each resource. Since the resource prices aredetermined based on the payments from all NSIs, a tradeoff(which results in maximizing social welfare, as shown inTheorem 1) is established between the allocated resources todifferent NSIs.In the described auction game, the bidders are networkslice managers, while the suppliers are different nodes, eachoffering M types of (divisible) resources according to a certainpricing scheme (as in (14)). Hence, the resource prices arefunction of actions (i.e., allocations) taken by network slicemanagers (see (14) and Note 1). Definition 1.
The game is said to be in a
Nash Equilibrium(NE) if there exists an allocation x ∗ and a set of resourceprices µ ∗ = µ ( x ∗ ) , in compliance with (14) , so that no NSIgains payoff by unilateral deviation from its current allocation, Π n ( x ∗ n ; µ ∗ ) ≥ Π n ( x n ; µ ∗ ) , for all feasible x n , ∀ n. (17)It is worth noting that the payoff of each NSI depends onits own action (i.e., x n ) as well as the current resource prices .The inequality in (17) implies that x ∗ n is an optimal solution Alternatively, we may assume that nodes declare the allocated resources toeach NSI, so that slice managers indirectly infer the resource prices ( µ i,r = w n,i,r /a n,i,r ). to Problem 2 (i.e., the local payoff optimization for NSI n )in conjunction with the equilibrium resource prices. Given theconcavity of the payoff functions, an allocation x n := { x pn | p ∈ P l , l = 1 , , ..., A } is an optimal solution to Problem 2for NSI n if and only if for every path p ∈ P l : ∂ Π( x n ; µ ) ∂x pn = U (cid:48) n,l ( x n,l ) − (cid:88) i ∈ p (cid:88) r d pn,i,r µ i,r (cid:40) = 0 if x pn > , ≤ otherwise . (18)The condition in (18) implies that x pn > , p ∈ P l , only when (cid:88) i ∈ p (cid:88) r d pn,i,r µ i,r = min p (cid:48) ∈P l (cid:88) i ∈ p (cid:48) (cid:88) r d p (cid:48) n,i,r µ i,r . (19)The left hand side in (19) gives the cost for transmitting oneunit of NSI n ’s traffic over path p . It means that at the optimalsolution to Problem 2, the traffic for each NSI is forwardedover the least expensive path(s) in each area. Moreover, itfollows from (18) that x ∗ n,l := U (cid:48) n,l ( − min p ∈P l (cid:88) i ∈ p (cid:88) r d pn,i,r µ i,r , (20)where U (cid:48) n,l ( − ( · ) is the inverse of U (cid:48) n,l ( · ) . The function U (cid:48) n,l ( · ) is assumed to be invertible owing to concavity of U n,l ( · ) over its feasible region . Although x ∗ n,l is uniquelyspecified for each NSI in each area, yet there might be severalpossible allocations in case that the minimum transmissioncost is attained over multiple paths. In particular, let P ∗ n,l ⊆ P l denote the set of paths which result in the minimum transmis-sion cost for NSI n in area l (c.f. (19)). A possible allocationis to uniformly distribute x ∗ n,l across the least expensive paths: x p ∗ n = (cid:40) x ∗ n,l / |P ∗ n,l | if p ∈ P ∗ n,l , otherwise . (21)The proposed Distributed Resource Provisioning (DRP)mechanism is summarized in Table II. Beginning with someinitial resource prices (e.g., µ i,r = q i,r ), the slice manager foreach NSI may find an optimal allocation according to (20)and (21). However, to prevent oscillations, especially whentraffic is to be distributed over multiple paths, x n is (gradually)updated according to (23). The slice manager then finds theamount of resources which should be allocated from differentnodes (c.f. (2)), and accordingly bids for different resources ofeach node. The offered payments are in turn used to update theresource prices at each node (see (14)). Let { ˆ µ i,r } denote theupdated resource prices which are taken by node i in responseto bids made by different NSIs in the current iteration. The actual volume of traffic that can be processed for NSI n overpath p is given by ˆ x pn := x pn min i ∈ p,r µ i,r ˆ µ i,r . (22)The updated resource prices are subsequently used by networkslice managers to repeat the same procedure in the next round.The slice managers keep updating their decisions while (cid:107) ˆx n − x ∗ n (cid:107) > (cid:15) for some (cid:15) > .ABLE II: Distributed Resource Provisioning (DRP) Mecha-nism The resource prices are initially set to µ i,r = q i,r , ∀ i, r .I. Given a set of resource prices, µ , each network slice manager - Finds the optimal allocation x ∗ n per the current resource pricesaccording to (20) and (21). It then updates the current allocationaccording to x n ← (1 − η n ) x n + η n x ∗ n , (23)where η n ∈ (0 , .- Finds the amount of resources which should be allocated fromeach node (see (2)), and then bids for different resources,accordingly (see (13)).II. Given updated payments by slice managers, each serviceprovider node - Updates the resource prices according to (14), and allocates theresources to different NSIs, accordingly.- Declares the resource prices to the network slice managers.III.
Given updated resource prices, ˆ µ , each network slice manager - Finds the actual allocation, ˆ x n , according to (22).- Updates the resource prices, µ ← ˆ µ , as well as the allocation, x n ← ˆ x n . Subroutine I then is repeated while (cid:107) x n − x ∗ n (cid:107) > (cid:15) . E. Characterizing the Solution
In this section we show that the proposed distributedresource provisioning mechanism results in optimizing theglobal system-wide objective of Problem 1, maximizing socialwelfare among the network slice tenants while minimizingthe OPEX for service providers. Moreover, we show thatProblem 1 has a unique optimal solution (in terms of { x n,l } )and so is the NE of the DRP mechanism. We finally studythe properties of the resulting allocation by characterizing thesolution of Problem 1. Theorem 1.
Assume a continuously differentiable and strictlyconcave utility function, U n,l ( · ) , for each NSI n . An allocation x is an NE for DRP mechanism (in conjunction with someresource prices) if and only if it is a solution to Problem 1. The following is a direct conclusion of Theorem 1.
Remark 1.
There exists an NE for the DRP mechanism.Moreover, the resulting allocation at the NE is unique in termsof { x ∗ n,l } . The proof appears in the Appendix. According to Theo-rem 1, the DRP mechanism results in an allocation whichmaximizes the overall utility for different NSIs minus thesummation of OPEX for infrastructure service providers (see(4)). Moreover, it follows from the proof of Theorem 1 thatthe resource prices at the NE are associated with the dualvariables λ i,r corresponding to the capacity constraints in (5).In particular, µ i,r = q i,r + λ i,r , so that resource r with arestricting capacity at node i (i.e., with λ i,r > ) results in µ i,r > q i,r .The reader may note that the convergence behavior of theproposed DRP mechanism depends on the actual choice ofutility functions, which can be different for various NSIs. Thefact that different NSIs can take different utility functionsmakes it difficult to derive analytical results on the rate of the convergence. To derive such results, one needs to performa statistical or worst-case analysis which is not straightforwardand is out of the scope of this paper. Nevertheless, Theorem 1describes the NE, which is the convergence point of theDRP mechanism, as the optimal solution to Problem 1 forany choice of continuously differentiable and strictly concaveutility functions. At this point, we leave this rich topic forfuture work, and provide numerical experiments to have someobservations on the convergence behavior of the DRP mech-anism in Section VI.To further characterize the NE of the DRP mechanism(or equivalently the solution to Problem 1), we formulate anequivalent network-wide optimization problem, which prov-ably results in the same allocation. Towards that, let w n,l denote the total payment made by NSI n for the resourceswhich process the originating traffic from area l , w n,l := (cid:88) i,r µ i,r (cid:88) p ∈P l : i ∈ p x pn d pn,i,r . (24) Theorem 2.
Let { w ∗ n,l } denote the set of payments made bydifferent NSIs when the DRP mechanism is in an NE. Anallocation x ∗ serves as an NE for the DRP mechanism if andonly if it is a solution to following problem. Problem 3: Subnetworks optimization problem max x (cid:88) n,l w ∗ n,l log( x n,l ) − (cid:88) i,r q i,r (cid:88) n (cid:88) p : i ∈ p x pn d pn,i,r (25) s.t. (cid:88) n (cid:88) p : i ∈ p x pn d pn,i,r ≤ C i,r , ∀ i, r, (26) x pn ≥ , ∀ n, p, (27) where x n,l is written in terms of x pn according to (1) . According to Theorem 2, the DRP mechanism results inan allocation which satisfies weighted proportional fairnessamong different NSIs (wherein the weights are set to pay-ments) while minimizing the OPEX [44], [47]. To furthercharacterize the allocation resulting from the DRP mechanism,let a pn = { a pn,i,r | i ∈ p, ∀ r } denote the allocated resources toNSI n over path p . We denote by T n ( a n,l ) the volume oftraffic which can be processed for NSI n using the resourcesallocated to NSI n in area l , a n,l := (cid:80) p ∈P l a pn . Definition 2.
An allocation is said to satisfy envy-freenessif each NSI n in each area would not prefer the allocatedresources to another NSI when adjusted according to theirpayments, that is, T n ( a n,l ) ≥ T n ( w ∗ n,l w ∗ m,l a m,l ) . The envy freeness property embodies the notion of fair-ness [39]. The other property that we consider here is sharing-incentive , which ensures that the proposed mechanism out-performs a so-called uniform allocation . To find a genericuniform allocation, let w p ∗ n,i denote the payment made byNSI n to node i for the service over path p . Consider a uniform allocation wherein a fraction w p ∗ n,i / (cid:80) m,p (cid:48) : i ∈ p (cid:48) w p (cid:48) ∗ m,i of different resources at node i is dedicated to NSI n forservice over path p . efinition 3. An allocation is said to satisfy sharing-incentive if each NSI is provided with more traffic volume compared tothe uniform allocation . The sharing-incentive property is the key to ensure a worst-case performance guarantee for each NSI. Satisfying thisproperty also may incentify different carriers/operators to pooltheir resources together [39], because each of them mayforward more traffic volumes over the shared infrastructure(when orchestrated by the proposed mechanism) compared tothe case that each of them gets an equal (weighted) share ofall the resources. We show that the envy-freeness and sharing-incentive properties are established at the NE of the DRPmechanism.
Theorem 3.
The allocation at the NE of the DRP mechanismsatisfies both envy-freeness and sharing-incentive properties.
The proof appears in the Appendix.V. E
XTENSIONS
We may consider several possible extensions/applicationsfor the original resource provisioning mechanism presentedherein. For instance, one may find the proposed mecha-nism particularly useful for network slicing in multi-layernetworks such as the integrated vertical HetNets [23] (withaerial BSs [48], LEO satellites, etc.), owing to agility anddistributed nature of the solution. In the following we brieflydescribe some important extensions to the original mechanismof Section IV.
A. Exploiting the Capabilities at the Mobile Edge
Our proposed mechanism can be easily extended to addressthe case that the service function chain at each domain com-prises a number of sub-chains, with the possibility that somebackhaul and/or core network sub-chains can be implementedat a domain closer to the mobile edge.In particular, let {F n , F n , F n } denote the per domainnetwork function chain for NSI n . It is assumed that for thelast two domains F sn = { ˜ F sn , ˆ F sn } , s = 2 , , where ˆ F sn shouldbe placed at a domain s node, while ˜ F sn can be flexibly placedat a node either in domain s or s − . Accordingly, we mayconsider separate demand vectors, ˆ d pn,i and ˜ d pn,i , for eachsub-chain of NSI n , where d pn,i = ˆ d pn,i + ˜ d pn,i . To extendthe DRP mechanism we may redefine paths as sequence ofnodes, p = { l, i, j, k, g, h } , which host different sub-chainsof an NSI. It reduces to the original formulation (with a per-domain service function chain), when i = i and i = i . TheDRP mechanism then is implemented as before, except thatthe network slice manager now bids separately for individualsub-chains when they are located at different nodes. The DRPmechanism may exploit this flexibility to efficiently utilizethe capabilities at the mobile edge, so as to improve theperformance for NSIs with stringent QoS/delay requirements. B. Budget-Constrained Tenants
With the resource pricing strategy described in Sec-tion IV-C, the resource prices might be chosen well above the operational expenditures (i.e., µ i,r > q i,r ) when so many NSIscontend for the resources of the same node. If the allocatedtraffic volume to an NSI is less than its desired optimal trafficvolume, the corresponding network slice manager may intendto make a larger bid/payment, which in turn may increase theresource prices. This procedure may unboundedly increase theresource prices as well as the required payments from differentNSIs. In practice, however, each network slice tenant may havea limited budget, so that the bids may not go beyond a certainlimit. The key to account for limited budgets is to exploitan admission control policy which limits the admitted trafficvolume in each area, so that the required payment remains ina feasible region. Let B n,l denote the budget for NSI n inarea l , and U (cid:48) n,l ( x n,l ) = ( φ n,l /x n,l ) α n , where φ n,l representsthe admitted traffic volume. In case that w n,l > B n,l , onemay update φ n,l ← ζφ n,l , ζ < , so as to make sure that thepayments are less than or equal to the budget. C. Multi-resource Fair Allocation
In Section IV-E we showed that the proposed distributedresource provisioning mechanism provides weighted propor-tional fairness among different NSIs, wherein the weight foreach NSI n in area l is set to its payment, w n,l (see Problem 3).The proportional fairness objective in Problem 3 is shownto satisfy some highly desirable properties, such as envy-freeness and sharing incentive [49]. It should be noted that theformulation in Problem 3 is applicable for a system where allthe resources of all nodes are allocated by a central controller.With a centralized implementation, however, there are otherproperties, such as strategy proofness which are desirable tobe satisfied [39]. In particular, an allocation mechanism is saidto satisfy strategy proofness if each NSI may not be allocatedmore traffic volumes when lying about its resource demandsto the centralized controller.Dominant resource fairness (DRF) is the first multi-resourcefair allocation mechanism which satisfies strategy-proofnessin addition to the above-mentioned properties [39], whenallocating multiple types of resources from a single server . Ofall the resources requested by a network function from onenode (for each unit of traffic), its dominant resource is theone with the highest demand when demands are expressed asfractions of the overall resource capacities. Using DRF, eachnetwork function receives a fair share of its respective domi-nant resource [39]. The studies in [47], [49]–[51] investigatethe problem of multi-resource fair allocation in an environmentof heterogeneous and geographically distributed servers. Suchstudies, however, address single-hop processing of the trafficin a network of cloud computing servers, and may not bedirectly applicable to end-to-end resource provisioning in amulti-domain mobile network environment.The study in [26] strives to extend DRF to a multi-domainmobile network environment, wherein the traffic for each NSIis forwarded over a certain path towards the core network.Towards that, it identifies an end-to-end dominant resource foreach NSI, which is specified based on its end-to-end demandvector. Then, it strives to allocate each NSI a fair share of itsespective dominant resource. Particularly, a dominant sharefor NSI n over path p is defined as [26] z n := x n max i ∈ p max r d n,i,r C i,r . (28)Then the NSI traffic volumes, { x n } , are determined so asto maximize the minimum dominant share across differentNSIs. In case that different NSIs make different payments, w n , one can maximize the minimum weighted dominant share , z n /w n , across different NSIs. We refer to such an extensionto DRF as multi-domain DRF . However, as we show in thefollowing (and also in Section VI), the multi-domain DRFmechanism may not satisfy the sharing-incentive property, andalso may not result in a fair allocation. Intuitively, the mainproblem with this mechanism is that it may identify a resourceat a lightly loaded node (with a few NSIs passing through)as the end-to-end dominant resource for some NSIs. Such aresource, however, may not serve as a bottleneck over an end-to-end routing path. In this case, a smaller share of the actualbottleneck resource is allocated to such NSIs, which in turnmay violate the sharing incentive property.For instance, consider the example of Fig. 2, which showsa simple network connecting the users from two APs to aCRAN unit. The resource capacity vector for each node, C i , i = 1 , , , as well as the demand vector for each NSI n passing through node i , d n,i , are shown in the figure. In thisexample, the communication bandwidth of APs is identifiedas the end-to-end dominant resource for each NSI. Accordingto the multi-domain DRF mechanism [26], one may equalize z n for the three NSIs, which requires x = x = x / . Itcan be observed that NSI traffic volumes can be increased up x = x = x / , before the CPU turns to a bottleneck(i.e., fully booked) at the CRAN. Evidently, the second andthird NSIs do not obtain a fair share of the bottleneckresource (i.e., CPU of the CRAN) under the multi-domainDRF allocation in this example. Moreover, the provisionedcapacity to each of them is less than that achievable under auniform allocation (where x n = 20 / , for n = 1 , , ).To address this issue, we propose an extension to DRFwhich is inspired by the per-server dominant share fair (PS-DSF) allocation mechanism presented in [46], [49] for fairresource allocation from a set of multi-resource heterogeneousservers. Particularly, by the PS-DSF mechanism a per-serverdominant resource is identified for each network function withrespect to each server. Then, each server strives to maximizethe minimum per-server dominant share among differentnetwork functions [46]. For fair resource provisioning in amulti-domain mobile network environment, we may identifya per-domain dominant resource for each NSI with respect toeach node over an end-to-end routing path. Then starting fromthe last domain (i.e., the core network), at each node one mayallocate a fair-share of the per-domain dominant resource toall NSIs which are passing through the same node. The sameprocedure can be implemented at preceding domains, exceptthat the allocated traffic volume to each NSI is limited by the Here, we may drop the index p from the demand vectors, since each NSIis presumably forwarded over a single path. Resources: [CPU (Cores), RAM (GB), BW (Gb/s)]
NSIs 1 2 3 𝐝 𝑛,1 = 1, 2, 0.1 , 𝐂 = [20, 48, 2.5], 𝑛 =
1, 2, 3 𝐝 = [0.5, 1, 0.1], 𝐝 = [0.5, 1, 0.2] 𝐂 = [8, 16, 1] 𝐂 = [16, 32, 2] 𝐝 = [0.5, 1, 0.1], Fig. 2:
A sample network of 2 APs, and 1 CRAN unit, providing resourcesto three NSIs. one at subsequent domains. We refer to this mechanism as
Per-Domain DRF . For example, in Fig. 2, CPU (bandwidth) isidentified as the dominant resource for each NSI at the CRANunit (each of the APs). Starting from the CRAN, each of theNSIs receives a fair share of the per-domain dominant resource(i.e., CPU), which results in x n = 20 / , n = 1 , , . Here thetraffic volume for NSIs at each of the APs is limited by that atthe CRAN (i.e., the end-to-end bottleneck), so the end-to-endallocation for each NSI is given by x n = 20 / , n = 1 , , .In this particular example, the proportional fairness metricof Problem 3 results in the same allocation (as the per-domain DRF mechanism) when setting OPEX to zero andassuming the same weights (i.e., payments) for all NSIs.By implementing the DRP mechanism, however, each of theNSIs would make a (different) payment which maximizes itspayoff function. In Section VI we compare the performance ofthe proposed DRP mechanism (or equivalently the weightedproportional fairness metric of Problem 3) against the multi-domain DRF , and per-domain DRF , respectively, while settingthe weight for each NSI as its payment under the DRPmechanism. VI. P ERFORMANCE E VALUATION
A. Simulation Setup
In this section we evaluate the performance of the DRPmechanism by implementing the proposed algorithm in MAT-LAB, and comparing its performance against some existingsolutions in the literature. For the sake of numerical eval-uations, we consider a multi-domain network, as shown inFig. 1, comprising ten distributed RAN units (e.g., APs),two CRAN units, and a core network data center. In theRAN segment, we consider five areas, wherein 2 differentAPs are assumed in each area. Particularly, it is assumed It can be shown that the per-domain DRF mechanism inherits all of theproperties which are satisfied by PS-DSD mechanism [49]. The details ofsuch analysis, however, are out of the scope of this paper. hat APs are of , each providing four typesof resources that are CPU, RAM, memory bandwidth andcommunication bandwidth. The resource capacities for the twotypes of APs, and for the CRAN and CN data centers aregiven in Table III. The resource capacities for each CRANdata center (and CN data center, respectively) are equivalentto 3 instances of Amazon EC2 C5.4 (one instance of AmazonEC2 C5n.18). The operational costs are taken according to auniform distribution in the range of [0 . , ¢ for unit of RAMand memory bandwidth, in the range of [1 , ¢ for 1 core ofCPU, and in the range of [1 , ¢ for 1Gb/s of communicationbandwidth allocated over unit of time. It is assumed that APs inareas 2, 3, and 4 have connections to both CRAN units, whilethe APs in area 1 (area 5, respectively) are connected onlyto the first CRAN (second CRAN) unit. The communicationbandwidth at each AP represents the maximum achievabledata rate when using the best modulation and coding scheme(i.e, the nominal capacity ). Depending on the performancerequirements and or (mid-term) feedbacks from the users ina certain area, however, some NSIs may require lower ordermodulation and coding schemes, or particular beam-formingand or MIMO transmission schemes which possibly result ina lower (average) achievable data rate. Such complexities canbe flexibly reflected in the demand vector for different NSIs.The demand vector for each NSI is assumed to be fixed in oneinstantiation of the game. However, it may vary in differentinstantiations to capture (mid-term) fluctuations in the RANenvironment.Since network slicing is rather a new concept, there areno real-world traces publicly available to use. So, as in [26],the demand vector (for 100 Mb/s of traffic) is generatedindependently for each NSI n , with the values taken ac-cording to independent uniform distributions in the rangeof coeff × [0 . , . cores for CPU, in the range of [1 , GB for RAM, in the range of coeff × [50 , Mb/s forcommunication bandwidth, and in the range of [10 , Mb/sfor the memory bandwidth. Independency of the demandvector for various NSIs and across different elements ensuresheterogeneity in the generated data-set which is the mostimportant requirement to synthesize a multi-resource data-set [39], [49]. The parameter coeff is taken to be 2 fordemands at the RAN (representing more intensive processingat the RAN), and 1 otherwise. The parameter coeff is takenrandomly from the set { , , , } at the RAN for each NSI,and is chosen to be 2 at other network segments. The utilityfunction for each NSI n , U n,l ( z ) , is chosen from the classof concave utility functions with the derivative (i.e., marginalbenefit ) described as U (cid:48) n,l ( z ) = ( φ n,l /z ) α n [43], [44]. Theparameter α n determines the shape of the utility function(or the marginal benefit ) for each NSI. The parameter φ n,l may present the traffic load for each NSI n . Unless otherwisestated, the parameter α n for each NSI n is chosen accordingto a uniform distribution in the range of [1 , . We studythe performance of the proposed mechanism under differentloading conditions (as described in Section VI-B). The budgetfor each NSI is chosen to be $100 in each area. TABLE III: Data-center/AP resource capacities Node Type CPU RAM Memory BW Comm BW( cores) (GBytes) (Gb/s) (Gb/s)AP Type 1 16 32 10 1AP Type 2 8 16 5 1CRAN 48 384 4 ×
10 7CN 96 384 2 ×
50 14
B. Simulation Results
First we study the convergence behavior of the proposed dis-tributed resource provisioning mechanism. Next, we comparethe performance of the DRP mechanism (in terms of the pro-visioned capacity, resource utilization and other performancemetrics such as OPEX) against some heuristic and/or recentlydeveloped work in the literature, which show the enhancedperformance of the proposed solution.Fig. 3 shows the number of iterations that are requiredfor the DRP mechanism to converge to an (cid:15) -boundary ofthe optimal solution to Problem 1 under different loadingconditions. In this experiment, the parameter α n for each NSIis taken according to a uniform distribution in the range of [1 , . Unless otherwise stated, we consider a fixed number of N = 50 NSIs. The parameter φ n,l = φ (representing the trafficdemand) for different NSIs is chosen under the high-loadingcondition such that a fully booked resource (i.e., bottleneck )exists on each routing path. The traffic demand is reducedto half (and one fourth, respectively) for mid-loading (low-loading) condition. We also consider another high-loadingcondition where we assume N = 100 number of NSIs but φ n is halved. It is observed that a precision of − is achievedin the worst case (i.e., under a high loading condition) withinonly a few hundreds of iterations, which only takes a fewmilliseconds when implemented in such a cluster with tens ofservers. Our observations indicate that the convergence ratemainly depends on the overall traffic demand. Hence, theconvergence rate for a high loading regime remains the samewhen the number of NSIs is doubled but φ n is halved (seeFig. 3).It is worth noting that the proposed mechanism convergeswithin only a few (less than 10) iterations under a low-loadingcondition. Intuitively, there is less contention for differentresources under lower loading conditions, resulting in partiallybooked resources with a fixed pricing at almost all nodes.With less variations in pricing, the proposed auction gameconverges more rapidly to the optimal solution under lowerloading conditions.In another experiment reported in Fig. 4, we study theconvergence performance of the DRP mechanism under highloading conditions, while the parameter α n for each NSIis taken in two different ranges. It is observed that theconvergence facilitates when α n takes on values in a tighterrange. Intuitively, when α n takes on values in the range of [1 , . (compared to taking values in [1 , ), the shape ofutility function and also the marginal benefit for different NSIsget more similar and closer to each other. It means that thepayments from different NSIs will be in a tighter range. Also,the NSIs would be making a smaller change in their payments onvergence Precision ( ) -3 -2 -1 N u m b er o f I t er a t i o n s High loading (N, ? )High loading (2N, ? /2)Mid loading (N, ? /2)Low loading (N, ? /4) Fig. 3: The required number of iterations for convergence ofthe DRP mechanism to an (cid:15) -boundary of the optimal solutionto Problem 1 under different loading conditions. The parameter α n for each NSI is taken according to a uniform distributionin the range of [1 , .in response to a change in the resource prices. This impliesthat a stable price (or the NE) can be established within afewer number of iterations. Convergence Precision ( ) -3 -2 -1 N u m b er o f I t er a t i o n s (N, ? ), , [1,2](N, ? ), , [1, 1.5](2N, ? /2), , [1,2](2N, ? /2), , [1, 1.5] Fig. 4: The required number of iterations for convergence ofthe DRP mechanism to an (cid:15) -boundary of the optimal solutionto Problem 1 under high loading conditions. The convergenceperformance is compared when α n is taken (according to auniform distribution) in two different ranges.To evaluate the performance of the proposed DRP mecha-nism in terms of resource utilization , we compare it againstsome heuristic and greedy solutions which strive to allocatethe whole resources according to some fairness criteria. Inparticular, we compare the DRP mechanism with the multi-domain DRF mechanism which allocates resources to differentNSIs by employing dominant resource fairness (DRF [39])across different domains [26] (c.f. Section V-C). We showthat, however, the multi-domain DRF mechanism does notsatisfy the sharing incentive property. So, we also implementan extension to DRF, referred to as per-domain DRF (c.f.Section V-C), which is shown to satisfy the sharing-incentive property. We further compare the performance of these threemechanisms (i.e., DRP, multi-domain DRF, and per-domainDRF) with a generic uniform allocation (described in Sec-tion IV-E).To observe how each of the above-described mechanismsperforms compared to the uniform allocation, we find theallocated traffic volume to each NSI (under each allocationmechanism), and normalize it by the allocated traffic underthe uniform allocation. Such a parameter, denoted by r n foreach NSI n , represents the improvement ratio by which theallocated traffic to NSI n is increased compared to the uniformallocation. In Fig. 5 we plot r n for different NSIs, whenimplementing each of these mechanism for the same NSIsunder a high loading condition. It can be observed that withmulti-domain DRF, the allocated traffic volume for two NSIs(index 21 and 24) is less than their allocated traffic volumeunder the uniform allocation. It means that the multi-domainDRF does not satisfy the sharing incentive property. However,it is observed that both of the DRP and per-domain DRFmechanisms out-perform the uniform allocation. To affirmthis observation, we repeat the same experiment for 100times, generating demand profiles randomly each time. Then,we find the empirical probability that the improvement ratiofor an arbitrary NSI is greater than certain values, that is P ( R > r ) . Fig. 6 shows that both of the DRP and per-domainDRF mechanisms always outperform the uniform allocation.Moreover, the DRP mechanism is shown to provision (onaverage) more traffic volumes to different NSIs. T h e I m p r ov e m e n t R a ti o (r) NSIindex
X: 21Y: 0.9163 X: 24Y: 0.9154
Fig. 5: Comparing the allocated traffic volume to each NSIunder different resource provisioning mechanisms when nor-malized by the allocated traffic volume under the uniformallocation.Particularly, as plotted in Fig. 7, the end-to-end capacityprovisioned to each NSI (under a high-loading condition) isenhanced under the DRP mechanism by compared tothe uniform allocation, and by at least compared to thetwo other schemes. The end-to-end queuing delay for eachNSI, on the other hand, depends on the provisioned capacityas well as the average traffic load that is admitted to thenetwork. Specifically, if ρ n,l denotes the average traffic loadthat is admitted for an NSI in a particular area, the end-to-endqueuing delay is conversely proportional to x n,l − ρ n,l [45].Indeed, a detailed quantitative analysis of delay depends on the .5 1 1.5 2 2.5 3 3.5 4 4.5 5 P r ob ( R > r) Fig. 6: The empirical probability that the normalized trafficvolume for an arbitrary NSI is greater than certain values. T h e A v e r a g e I m p r ov e m e n t R a ti o DRPMulti-domain DRFPer-domain DRF
Fig. 7: The average improvement ratio of the end-to-endcapacity provisioned to each NSI compared to the uniformallocation.admission and flow control policy, as well as other packet-levelnetworking modules (including traffic shaping, packet segmen-tation, aggregation, or duplication functionalities), which areout of the scope of this paper. Nevertheless, we can performa comparative analysis for the end-to-end delay, comparingevery two mechanisms while considering a certain trafficload for each NSI. Specifically, assume that the traffic loadfor each NSI is set to of the (minimum) capacity thatcan be provisioned under the DRP and every other resourceprovisioning mechanism. The improvement ratio of the end-to-end delay for the DRP mechanism compared to other schemesis summarized in Table IV. Indeed, an improvement of 10to 15% in the capacity provisioned by the DRP mechanismresults in reducing the end-to-end queuing delay by a factorof 3 to 4 (compared to the per-domain or multi-domain DRF)under a high-loading condition.To better observe the efficiency of the DRP mechanism inutilizing different resources, the resource utilization that isachieved on average across all nodes under different mecha-nisms in a high loading condition is shown in Fig. 8. The av-erage resource utilization over different nodes in each domain TABLE IV: The average improvement ratio for the end-to-enddelay of the DRP mechanism compared to other schemes.
Delay Improvement Ratio
Uniform MD-DRF PD-DRFDRP over other schemes 19.94 4.08 3.24 is also shown in Fig. 9-11, respectively. All of the results areaveraged over 100 experiments. The 95% confidence intervalis shown on the top of each bar graph. Our observationsindicate that at least one of the resources (either CPU orcommunication bandwidth) is fully booked over each nodein Domain 1 under each of the DRP, multi-domain DRF,and per-domain DRF mechanisms. It means that there is abottleneck (imposed by the limited resource capacities) oneach routing path under a high loading condition. Despite thegreedy nature of the per-domain DRF and multi-domain DRFmechanisms, it is observed that the average resource utilizationthat is achieved by the DRP mechanism is increased by around10% for all of the resources compared to the per-domain DRFand multi-domain DRF mechanisms, and by 20% comparedto the uniform allocation (see Fig. 8).
Average Resource Utilization
DRPMulti-domain DRFPer-domain DRFUniform allocationCPU RAM Comm. BW Memory BW
Fig. 8: The resource utilization when averaged over differ-ent nodes and over 100 experiments for different allocationmechanisms (from left to the right: DRP, multi-domain DRF,per-domain DRF, and uniform allocation) in a high-loadingcondition.Another observation that we make here is on the operationalexpenditures that are imposed (on average) to the network forprovisioning each unit of traffic. The DRP mechanism maylimit the allocated resources from different nodes in a low-loading condition in a way that the operational expendituresare covered by the payment from different NSIs. Hence, itmay not be fair to compare the absolute value of OPEX(which is considerably reduced by the DRP in a low loadingcondition), against greedy mechanisms such as multi-domainDRF or per-domain DRF. To make a fair comparison, wefind the average operational expenditures for each unit oftraffic under different mechanisms in low-loading and highloading conditions. As shown in Fig. 12, the per unit OPEX isreduced by the DRP mechanism under both low-loading andhigh-loading conditions. While the DRP mechanism makesa better utilization of different resources in a high-loading
Resource Utilization - Domain 1 (RAN)
DRPMulti-domain DRFPer-domain DRFUniform allocationCPU RAM Comm. BW Memory BW
Fig. 9: The resource utilization that is achieved on averageover different nodes in Domain 1 (i.e., RAN) under differentallocation mechanisms.
Resource Utilization - Domain 2
DRPMulti-domain DRFPer-domain DRFUniform allocationCPU RAM Comm. BW Memory BW
Fig. 10: The resource utilization that is achieved on averageover Domain 2 data centers under different allocation mecha-nisms.
Resource Utilization - Domain 3
DRPMulti-domain DRFPer-domain DRFUniform allocationCPU RAM Comm. BW Memory BW
Fig. 11: The average resource utilization for the core networkdata center under different allocation mechanisms.condition, yet it results in less OPEX for each unit of traffic,owing to the optimal routing decisions. The DRP mechanismresults in a more considerable reduction in per unit OPEX in a low loading condition. Intuitively, the DRP mechanismmay throttle the allocated resources of costly nodes in a low-loading condition, while allocating more resources from nodeswith low operational costs. Making jointly optimal routing andresource provisioning decisions, the DRP mechanism reducesthe per unit OPEX by 12% compared to the per-domainDRF and multi-domain DRF in a low loading condition (seeFig. 12). It is worth noting that OPEX for other mechanismsdoes not change much with respect to loading conditions.
Low-loading condition
DRPMulti-domain DRFPer-domain DRFUniform allocation
High-loading condition O P E X p er t r a ff i c un i t ( $ ) DRPMulti-domain DRFPer-domain DRFUniform allocation
Fig. 12: The average operational expenditures to provision oneunit of traffic under different allocation mechanisms.VII. C
ONCLUSION
We proposed an agile and distributed mechanism for end-to-end resource provisioning to NSIs in a multi-domain mobilenetwork environment. In the proposed solution, each networkslice tenant finds the optimal traffic volumes for different paths(comprising different chains-of-VNFs) so as to maximize alocal payoff function. Based on the solution to the networkslice optimization problem, each network slice tenant decideson the amount of resources which should be acquired fromthe service provider(s) in each domain, and accordingly bidsfor the required resources. Given the payments from differentNSIs, each service provider decides on the resource prices,and then allocates resources to different NSIs. We showed thatsuch an auction game has a unique NE (in terms of NSI trafficvolumes), which maximizes social welfare among networkslice tenants, while minimizing OPEX for service providers.Making optimal routing and resource provisioning decisionswhile employing a cost-aware resource pricing scheme, theDRP mechanism is shown to reduce the OPEX for provision-ing each unit of traffic, while enhancing the resource utilizationof the infrastructure network at the same time. The proposedDRP mechanism is distinguished from the existing works inthe literature, owing to generality of the model, agility of thesolution, and the possibility for a distributed implementationwithout sharing any private information among different par-ties. The DRP mechanism is superior (in terms of resourceutilization and OPEX) not only to the existing solutions,but also their enhanced versions proposed in this study. Anextension wherein the network slice manager integrates theachievable QoS to the NSI’s utility function can be part ofthe future work. Also a thorough analysis on the convergenceehavior of the DRP mechanism for a variety of practicalutility functions can be addressed in a future study.R
EFERENCES[1] A. Checko, H. L. Christiansen, Y. Yan, L. Scolari, G. Kardaras, M. S.Berger, and L. Dittmann, “Cloud RAN for mobile networks – A tech-nology overview,”
IEEE Communications Surveys & Tutorials , vol. 17,no. 1, pp. 405–426, Thirdquarter 2015.[2] T. Taleb, K. Samdanis, B. Mada, H. Flinck, S. Dutta, and D. Sabella,“On multi-access edge computing: A survey of the emerging 5G net-work edge cloud architecture and orchestration,”
IEEE CommunicationsSurveys & Tutorials , vol. 19, no. 3, pp. 1657–1681, Firstquarter 2017.[3] “5G network slicing whitepaper,” FCC Technological Advisory Council,5G IoT Working Group, 2018.[4] “Network slicing use case requirements,” GSM MoU Association(GSMA), Apr. 2018.[5] “Description of network slicing concept,” NGMN Alliance, Sep. 2016.[6] “TS 28.530 management and orchestration; concepts, use cases andrequirements; v15.1.0,” 3GPP System and Architecture Working Group5, Dec. 2018.[7] X. Foukas, G. Patounas, A. Elmokashfi, and M. K. Marina, “Networkslicing in 5G: Survey and challenges,”
IEEE Communications Magazine ,vol. 55, no. 5, pp. 94–100, May 2017.[8] H. Zhang, N. Liu, X. Chu, K. Long, A.-H. Aghvami, and V. C. Leung,“Network slicing based 5G and future mobile networks: Mobility, re-source management, and challenges,”
IEEE Communications Magazine ,vol. 55, no. 8, pp. 138–145, Aug. 2017.[9] X. Zhou, R. Li, T. Chen, and H. Zhang, “Network slicing as a service:Enabling enterprises’ own software-defined cellular networks,”
IEEECommunications Magazine , vol. 54, no. 7, pp. 146–153, Jul. 2016.[10] C. Liang and F. R. Yu, “Wireless virtualization for next generationmobile cellular networks,”
IEEE Wireless Communications , vol. 22,no. 1, pp. 61–69, Mar. 2015.[11] N. Nikaein, E. Schiller, R. Favraud, K. Katsalis, D. Stavropoulos,I. Alyafawi, Z. Zhao, T. Braun, and T. Korakis, “Network store:Exploring slicing in future 5G networks,” in
Proc. ACM InternationalWorkshop on Mobility in the Evolving Internet Architecture , 2015, pp.8–13.[12] X. Costa-P´erez, J. Swetina, T. Guo, R. Mahindra, and S. Rangarajan,“Radio access network virtualization for future mobile carrier networks,”
IEEE Communications Magazine , vol. 51, no. 7, pp. 27–35, Jul. 2013.[13] P. Rost, A. Banchs, I. Berberana, M. Breitbach, M. Doll, H. Droste,C. Mannweiler, M. A. Puente, K. Samdanis, and B. Sayadi, “Mobilenetwork architecture evolution toward 5G,”
IEEE Communications Mag-azine , vol. 54, no. 5, pp. 84–91, May 2016.[14] K. Samdanis, X. Costa-Perez, and V. Sciancalepore, “From networksharing to multi-tenancy: The 5G network slice broker,”
IEEE Com-munications Magazine , vol. 54, no. 7, pp. 32–39, Jul. 2016.[15] S. Clayman, E. Maini, A. Galis, A. Manzalini, and N. Mazzocca, “Thedynamic placement of virtual network functions,” in
Proc. IEEE NetworkOperations and Management Symposium (NOMS) , 2014, pp. 1–9.[16] A. Fischer, J. F. Botero, M. T. Beck, H. De Meer, and X. Hesselbach,“Virtual network embedding: A survey,”
IEEE Communications Surveys& Tutorials , vol. 15, no. 4, pp. 1888–1906, Firstquarter 2013.[17] E. Amaldi, S. Coniglio, A. M. Koster, and M. Tieves, “On the computa-tional complexity of the virtual network embedding problem,”
ElsevierElectronic Notes in Discrete Mathematics , vol. 52, pp. 213–220, Mar.2016.[18] M. Ghaznavi, A. Khan, N. Shahriar, K. Alsubhi, R. Ahmed, andR. Boutaba, “Elastic virtual network function placement,” in
Proc. IEEEInternational Conference on Cloud Networking (CloudNet) , 2015, pp.255–260.[19] B. Addis, D. Belabed, M. Bouet, and S. Secci, “Virtual network func-tions placement and routing optimization,” in
Proc. IEEE InternationalConference on Cloud Networking (CloudNet) , 2015, pp. 171–177.[20] Q. Zhang, Y. Xiao, F. Liu, J. C. Lui, J. Guo, and T. Wang, “Jointoptimization of chain placement and request scheduling for networkfunction virtualization,” in
Proc. 37th International Conference onDistributed Computing Systems (ICDCS) , 2017, pp. 731–741.[21] M. F. Bari, S. R. Chowdhury, R. Ahmed, and R. Boutaba, “Onorchestrating virtual network functions,” in
Proc. IEEE InternationalConference on Network and Service Management (CNSM) , 2015, pp.50–56. [22] R. A. Addad, T. Taleb, M. Bagaa, D. L. C. Dutra, and H. Flinck,“Towards modeling cross-domain network slices for 5G,” in
Proc. IEEEGlobal Communications Conference (GLOBECOM) , 2018, pp. 1–7.[23] M. Alzenad and H. Yanikomeroglu, “Coverage and rate analysis forvertical heterogeneous networks (VHetNets),”
IEEE Transactions onWireless Communications , vol. 18, no. 12, pp. 5643–5657, Dec. 2019.[24] N. C. Luong, P. Wang, D. Niyato, Y. Wen, and Z. Han, “Resourcemanagement in cloud networking using economic analysis and pricingmodels: A survey,”
IEEE Communications Surveys & Tutorials , vol. 19,no. 2, pp. 954–1001, Firstquarter 2017.[25] D. T. Nguyen, L. B. Le, and V. Bhargava, “Price-based resourceallocation for edge computing: A market equilibrium approach,”
IEEETransactions on Cloud Computing (Early Access) , pp. 1–15, Jun. 2018.[26] H. Halabian, “Distributed resource allocation optimization in 5G virtu-alized networks,”
IEEE Journal on Selected Areas in Communications ,vol. 37, no. 3, pp. 627–642, Feb. 2019.[27] W. Zhang, J. Hwang, S. Rajagopalan, K. Ramakrishnan, and T. Wood,“Flurries: Countless fine-grained nfs for flexible per-flow customization,”in
Proc. ACM 12th International Conference on Emerging NetworkingExperiments and Technologies , 2016, pp. 3–17.[28] J. Martins, M. Ahmed, C. Raiciu, V. Olteanu, M. Honda, R. Bifulco,and F. Huici, “ClickOS and the art of network function virtualization,”in
Proc. 11th USENIX Symposium on Networked Systems Design andImplementation , 2014, pp. 459–473.[29] S. Palkar, C. Lan, S. Han, K. Jang, A. Panda, S. Ratnasamy, L. Rizzo,and S. Shenker, “E2: A framework for NFV applications,” in
Proc. ACM25th Symposium on Operating Systems Principles , 2015, pp. 121–136.[30] W. Zhang, G. Liu, W. Zhang, N. Shah, P. Lopreiato, G. Todeschi,K. Ramakrishnan, and T. Wood, “OpenNetVM: A platform for highperformance network service chains,” in
Proc. ACM Hot Topics inMiddleboxes and Network Function Virtualization , 2016, pp. 26–31.[31] A. Bremler-Barr, Y. Harchol, and D. Hay, “OpenBox: A software-definedframework for developing, deploying, and managing network functions,”in
Proc. ACM SIGCOMM Conference , 2016, pp. 511–524.[32] B. Han, V. Gopalakrishnan, L. Ji, and S. Lee, “Network functionvirtualization: Challenges and opportunities for innovations,”
IEEECommunications Magazine , vol. 53, no. 2, pp. 90–97, Feb. 2015.[33] J. G. Herrera and J. F. Botero, “Resource allocation in NFV: Acomprehensive survey,”
IEEE Transactions on Network and ServiceManagement , vol. 13, no. 3, pp. 518–532, Aug. 2016.[34] P. Caballero, A. Banchs, G. de Veciana, and X. Costa-P´erez, “Multi-tenant radio access network slicing: Statistical multiplexing of spatialloads,”
IEEE Trans. on Networking , vol. 25, no. 5, pp. 3044–3058, Jul.2017.[35] M. Bagaa, T. Taleb, and A. Ksentini, “Service-aware network functionplacement for efficient traffic handling in carrier cloud,” in
Proc. IEEEWireless Communications and Networking Conference (WCNC) , 2014,pp. 2402–2407.[36] M. Bagaa, T. Taleb, A. Laghrissi, and A. Ksentini, “Efficient virtualevolved packet core deployment across multiple cloud domains,” in
Proc.IEEE Wireless Communications and Networking Conference (WCNC) ,2018, pp. 1–6.[37] W. Guan, X. Wen, L. Wang, Z. Lu, and Y. Shen, “A service-orienteddeployment policy of end-to-end network slicing based on complexnetwork theory,”
IEEE Access , vol. 6, pp. 19 691–19 701, Apr. 2018.[38] P. Rost, A. Banchs, I. Berberana, M. Breitbach, M. Doll, H. Droste,C. Mannweiler, M. A. Puente, K. Samdanis, and B. Sayadi, “Mobilenetwork architecture evolution toward 5G,”
IEEE Communications Mag-azine , vol. 54, no. 5, pp. 84–91, Mar. 2016.[39] A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, andI. Stoica, “Dominant resource fairness: Fair allocation of multipleresource types,” in
Proc. USENIX Symposium on Networked SystemsDesign and Implementation (NSDI) , 2011.[40] F. Fossati, S. Moretti, P. Perny, and S. Secci, “Multi-resource allocationfor network slicing,”
IEEE/ACM Transactions on Networking , pp. 1–14,Feb. 2020.[41] D. T. Nguyen, L. B. Le, and V. K. Bhargava, “A market-based frameworkfor multi-resource allocation in fog computing,”
IEEE/ACM Transac-tions on Networking , vol. 27, no. 3, pp. 1151–1164, Jun. 2019.[42] M. Leconte, G. S. Paschos, P. Mertikopoulos, and U. C. Kozat, “Aresource allocation framework for network slicing,” in
Proc. IEEEInfocom Conference , 2018, pp. 2177–2185.[43] D. Bertsekas and R. Gallager,
Data Networks, 2nd Ed.
Prentice Hall,1992.[44] F. P. Kelly, A. K. Maulloo, and D. K. Tan, “Rate control for commu-nication networks: Shadow prices, proportional fairness and stability,” pringer Journal of the Operational Research Society , vol. 49, no. 3,pp. 237–252, Apr. 1998.[45] Q. Ye, W. Zhuang, X. Li, and J. Rao, “End-to-end delay modeling forembedded VNF chains in 5G core networks,”
IEEE Internet of ThingsJournal , vol. 6, no. 1, pp. 692–704, Feb. 2019.[46] J. Khamse-Ashari, I. Lambadaris, G. Kesidis, B. Urgaonkar, and Y. Zhao,“Per-server dominant-share fairness (PS-DSF): A multi-resource fairallocation mechanism for heterogeneous servers,” in
Proc. IEEE Inter-national Conference on Communications (ICC) , 2017, pp. 1–7.[47] ——, “A cost-efficient and fair multi-resource allocation mechanismfor self-organizing servers,” in
Proc. IEEE Global CommunicationsConference (GLOBECOM) , 2018, pp. 1–7.[48] I. Bor-Yaliniz, M. Salem, G. Senerath, and H. Yanikomeroglu, “Is5G Ready for Drones: A Look into Contemporary and ProspectiveWireless Networks from a Standardization Perspective,”
IEEE WirelessCommunications , vol. 26, no. 1, pp. 18–27, Feb. 2019.[49] J. Khamse-Ashari, I. Lambadaris, G. Kesidis, B. Urgaonkar, and Y. Zhao,“An efficient and fair multi-resource allocation mechanism for heteroge-neous servers,”
IEEE Transactions on Parallel and Distributed Systems ,vol. 29, no. 12, pp. 2686–2699, Dec. 2018.[50] W. Wang, B. Liang, and B. Li, “Multi-resource fair allocation inheterogeneous cloud computing systems,”
IEEE Transaction on Parelleland Distributed Systems , vol. 26, pp. 2822–2835, Oct. 2015.[51] J. Khamse-Ashari, I. Lambadaris, G. Kesidis, B. Urgaonkar, and Y. Zhao,“A cost-aware fair allocation mechanism for multi-resource servers,”
IEEE Networking Letters , vol. 1, no. 1, pp. 34–37, Mar. 2019.[52] S. Boyd and L. Vandenberghe,
Convex Optimization . CambridgeUniversity Press, 2004. A PPENDIX
Proof of Theorem 1.
Problem 1 is a convex optimizationproblem. An allocation is an optimal solution to this problemif and only if there exists a set of multipliers λ i,r , and ν pn (corresponding to the constraints in (5) and (6), respectively)so that KKT conditions are satisfied [52] ∂U n,l ( x n,l ) ∂x n,l = (cid:88) i ∈ p (cid:88) r ( λ i,r + q i,r ) d pn,i,r − ν pn , ∀ n, p, (29) ≤ C i,r − (cid:88) n (cid:88) p : i ∈ p x pn d pn,i,r ⊥ λ i,r ≥ , ∀ i, r, (30) ≤ x pn ⊥ ν pn ≥ , ∀ n, p. (31)When x is an NE for DRP mechanism in conjunction withsome resource prices { µ i,r } , it will be an optimal solution toProblem 2 for every NSI n . Problem 2 is a convex optimizationproblem in terms of x n for each NSI n . It follows that x n isan optimal solution to Problem 2 if (and only if) there existsa set of multipliers { v pn } so that ∂U n,l ( x n,l ) ∂x n,l = (cid:88) i ∈ p (cid:88) r µ i,r d pn,i,r − v pn , ∀ p, (32) ≤ x pn ⊥ v pn ≥ , ∀ p, (33)where µ i,r is set according to (14), so µ i,r ≥ q i,r , ∀ i, r .We show that the conditions in (29)-(31) are satisfied whenchoosing λ i,r := µ i,r − q i,r , and ν pn = v pn . It means thatwe may find a set of multipliers in conjunction with eachNE of DRP mechanism, so that the KKT conditions in(29)-(31) are satisfied. This, in turn, implies that each NEof DRP mechanism is an optimal solution to Problem 1.It is straightforward to reach (29) and (31) when choosing λ i,r := µ i,r − q i,r ≥ , and ν pn = v pn . To observe (30), one The notation x ⊥ y means xy = 0 . may substitute for w n,i,r from (13) into (14), which results inan updated resource price, ˆ µ i,r = max { q i,r , (cid:80) n (cid:80) p : i ∈ p x pn d pn,i,r C i,r µ i,r } . (34)At the NE we should have ˆ µ i,r = µ i,r , ∀ i, r . This is estab-lished only if (cid:80) n (cid:80) p : i ∈ p x pn d pn,i,r ≤ C i,r for every node i andresource r with µ i,r = q i,r , and (cid:80) n (cid:80) p : i ∈ p x pn d pn,i,r = C i,r for every node i and resource r with µ i,r > q i,r . This isexactly equivalent to the condition in (30) when choosing λ i,r = µ i,r − q i,r .Now, consider an allocation x := { x pn | n ∈ N , p ∈ P l , l =1 , , .., A } , along with a set of multipliers which satisfy theconditions in (29)-(31). We show that x is an NE for DRPmechanism in conjunction with the resource prices chosen as µ i,r := q i,r + λ i,r . To have an NE, x n should be an optimalsolution to Problem 2 for every NSI n . On the other hand, x n is an optimal solution to Problem 2, if it satisfies conditions in(32)-(33), which is the case when choosing µ i,r := q i,r + λ i,r ,and v pn = ν pn (c.f. (29) and (31)). Finally, the condition in(30) implies that the resource prices remain steady in DRPmechanism (i.e., ˆ µ i,r = µ i,r ), when choosing µ i,r = q i,r + λ i,r (c.f. (34)). Proof of Theorem 2.
Problem 3 describes a convex optimiza-tion problem. An allocation x is a solution to this problemif and only if there exists a set of multipliers λ i,r , and ν pn (corresponding to the constraints in (26) and (27), respectively)so that KKT conditions are satisfied [52] w ∗ n,l x n,l = (cid:88) i ∈ p (cid:88) r ( λ i,r + q i,r ) d pn,i,r − ν pn , ∀ n, p, (35) ≤ C i,r − (cid:88) n (cid:88) p : i ∈ p x pn d pn,i,r ⊥ λ i,r ≥ , ∀ i, r, (36) ≤ x pn ⊥ ν pn ≥ , ∀ n, p. (37)It should be noted that w ∗ n,l is the payment made by NSI n in area l , when DRP mechanism is in an NE equilibrium. Itmeans that (c.f. (24)) w ∗ n,l = x ∗ n,l (cid:88) i ∈ p (cid:88) r ( λ ∗ i,r + q i,r ) d pn,i,r , p ∈ P ∗ n,l , (38)where x ∗ n,l is the solution at the NE, and P ∗ n,l is the set ofpaths with minimum transmission cost for NSI n in area l .According to Theorem 1, x ∗ is a solution to (29)-(31) inconjunction with { λ ∗ i,r } and { ν p ∗ n } . It follows from (38) and(29)-(31) that x ∗ is also a solution to (35)-(37) when choosing λ i,r = λ ∗ i,r , ∀ i, r and ν pn = ν p ∗ n , ∀ n, p . In the same way, itcan be observed that every solution to Problem 3 (satisfyingthe conditions in (29)-(31)) is a solution to Problem 1. Proof of Theorem 3.
According to Theorem 2, an allocation isan NE for the DRP mechanism if and only if it is a solution toProblem 3. Let λ i,r , and ν pn , respectively, denote the Lagrangemultipliers corresponding to the constraints in (26) and (27).For every path p ∈ P l and NSI n it follows that w ∗ n,l x n,l = (cid:88) i ∈ p (cid:88) r ( λ i,r + q i,r ) d pn,i,r − ν pn , (39)here ν pn = 0 , when x pn > .To show envy-freeness , we show that each NSI n would notprefer the allocated resources to another NSI m over any path p , when adjusted according to their payments. The resourcesallocated from data center i to NSI m for x pm unit of traffic isgiven by [ x pm d pm,i,r ] , r = 1 , , ..., M . The payment of NSI m for this amount of traffic is given by w p ∗ m := x pm w ∗ m,l /x m,l .Such resources are preferred by NSI n if x pn d pn,i,r w p ∗ n < x pm d pm,i,r w p ∗ m , ∀ r, i ∈ p, (40)or equivalently (by substituting for w p ∗ n and w p ∗ m ), x n,l d pn,i,r w ∗ n,l < x m,l d pm,i,r w ∗ m,l , ∀ r, i ∈ p. (41)Consider some path p for which x pm > , so that ν pm = 0 .It follows from (39) that w ∗ m,l x m,l = (cid:88) i ∈ p (cid:88) r µ i,r d pm,i,r , (42) w ∗ n,l x n,l ≤ (cid:88) i ∈ p (cid:88) r µ i,r d pn,i,r , (43)where µ i,r := λ i,r + q i,r . Multiplying both sides of (43) by x n,l /w ∗ n,l , and using the inequality in (41), result in ≤ (cid:88) i ∈ p (cid:88) r µ i,r x n,l w ∗ n,l d pn,i,r (44) < (cid:88) i ∈ p (cid:88) r µ i,r x m,l w ∗ m,l d pm,i,r = 1 , (45)which is a contradiction.Consider an NE resulting from the DRP mechanism. Toprove the sharing incentive property we need to show thateach NSI is provided with more traffic volume (under theNE) compared to the uniform allocation. To characterize theuniform allocation, let define γ pn,i as the (maximum) volumeof traffic which can be processed for NSI n through path p when monopolizing the whole resources allocated from node i ∈ p under the NE. That is, γ pn,i := min r η i,r C i,r d pn,i,r , (46)where η i,r is the portion of resource r that is utilized at theNE. The (maximum) volume of traffic which can be processedfor NSI n through path p is given by x p, uni n := min i ∈ p w p ∗ n,i W ∗ i γ pn,i , (47)where W ∗ i := (cid:88) m,p (cid:48) : i ∈ p (cid:48) w p ∗ n,i . (48)In the following we show that x pn ≥ x p, uni n for every path p ∈ P l with w p ∗ n > . In particular, for every path p with w p ∗ n > , if we multiply both sides of (39) by w p ∗ n,i γ pn,i /W ∗ i ,for every node i ∈ p , it follows that γ pn,i w p ∗ n,i W ∗ i w ∗ n,l x n,l ≤ w p ∗ n,i + γ pn,i w p ∗ n,i W ∗ i (cid:88) i (cid:48) ∈ p,i (cid:48) (cid:54) = i (cid:88) r µ i (cid:48) ,r d pn,i (cid:48) ,r , (49) where the inequality follows from the fact that γ pn,i d pn,i,r ≤ η i,r C i,r , ∀ r (c.f. (46)), and (cid:80) r µ i,r η i,r C i,r = W ∗ i . In thesame way (multiplying both sides of (49) by w p ∗ n,i (cid:48) γ pn,i (cid:48) /W ∗ i (cid:48) ),it can be observed that min i ∈ p { γ pn,i w p ∗ n,i W ∗ i } w ∗ n,l x n,l ≤ (cid:88) i ∈ p w p ∗ n,i = w p ∗ n . (50)On the other hand, w ∗ n,l x n,l = w p ∗ n x pn for every path p ∈ P l with w p ∗ n > . Hence, w p ∗ n x pn x p, uni n ≤ w p ∗ n , (51)or, x pn ≥ x p, uni nn