A Survey on the Use of Preferences for Virtual Machine Placement in Cloud Data Centers
AA Survey on the Use of Preferences for VirtualMachine Placement in Cloud Data Centers
Abdulaziz Alashaikh Eisa Alanazi Ala Al-Fuqaha ∗ Abstract
With the rapid development of virtualization techniques, cloud data centersallow for cost effective, flexible, and customizable deployments of applicationson virtualized infrastructure. Virtual machine (VM) placement aims to assigneach virtual machine to a server in the cloud environment. VM Placement is ofparamount importance to the design of cloud data centers. Typically, VM place-ment involves complex relations and multiple design factors as well as local poli-cies that govern the assignment decisions. It also involves different constituentsincluding cloud administrators and customers that might have disparate prefer-ences while opting for a placement solution. Thus, it is often valuable to not onlyreturn an optimized solution to the VM placement problem but also a solution thatreflects the given preferences of the constituents. In this paper, we provide a de-tailed review on the role of preferences in the recent literature on VM placement.We further discuss key challenges and identify possible research opportunities tobetter incorporate preferences within the context of VM placement.
Today, virtualization technology has facilitated the hosting of millions of services andapplications on the cloud and simplified their deployments and accesses. Many small,medium, and large organizations have been moving their applications or even their en-tire data centers to the cloud. This is mainly motivated to their economical advantage.in terms of real-estate, equipment costs and manpower compared to private data cen-ters. Cloud data centers also maintain high degree of flexibility where services and ap-plications can be deployed, terminated, migrated, and replicated on-demand [91, 114].In addition, cloud data centers offer a wide range of customizable settings and add-onfeatures to suit the requirements and non-functional needs of different customers.Essentially, virtualization enables dynamic sharing of physical resources such asprocessors, memory, and storage of the cloud infrastructure. An application is installedon a logically isolated Virtual Machine (VM) that has limited and pre-configured ac-cess to the shared resources of Physical Machines (PM) and networking devices. The ∗ Abdulaziz Alashaikh is with the Computer& Networks Engineering Department, University of Jeddah,Saudi Arabia ([email protected]). Eisa Alanazi with Computer Science Department, Umm Al-QuraUniversity, Saudi Arabia ([email protected]). Ala Al-Fuqaha is with the Computer Science Department,Western Michigan University, USA ([email protected]). a r X i v : . [ c s . D C ] J a n rocess of mapping the VMs to the existing PMs is commonly known as the VirtualMachine Placement (VMP) problem. The VMP problem is of paramount importance tothe design and operation of cloud data centers. Throughout the last decade, there havebeen many works addressing the VMP from different perspectives. The majority ofthese works are concerned about the profitability of data centers and the energy bill aswell as other operational aspects. While many proposals share similar objectives, theymay have taken different approaches, yielding a wide range of frameworks, models,and algorithms.A pre-requisite to the success of existing approaches is handling customer and ad-ministrator preferences when looking for a placement solution. This is due to the factthat cloud administrators and customers are often not satisfied with any placement so-lution. Rather, they are interested in the best, or at least good, solution that meets theirpreferences. As a motivating example, consider Bob who wants to place a VM thatrequires one unit of CPU and two units of RAM. Alice, on the other hand, has a VMthat requires two units of CPU and one unit of RAM. The cloud data center, which isadministrated by Charlie, has two physical machines P with 3 CPU units and 4 RAMunits and P with 4 CPU units and 3 RAM units. Assume the objective is to minimizethe total wastage of RAM and CPU units. Obviously, the four possible placements ( P , P ) , ( P , P ) , ( P , P ) , and ( P , P ) are all Pareto optimal with total wastage of 8.Now, assume both Alice and Bob prefer placing their VMs in a PM with fasterCPU. In such case,the solution ( P , P ) is preferred to all other solutions. Furthermore,in case this solution is unattainable ( e.g., violating a constraint), ( P , P ) and ( P , P ) are both preferred to ( P , P ) . These placements represent the most preferred feasibleoptions given both Bob and Alice preferences. Finally, Charlie prefers Bob’s VM beplaced on P to better meet Bob’s service level agreement (SLA) and therefore, ( P , P ) is the optimal placement given the three preferences. This example demonstrates thatcloud customers and administrators may have preferences that need to be consideredwhile consolidating customer VMs ( e.g., customers prefer servers with faster CPUs)and solutions are no longer indifferent but some are preferred to others.There has been a number of attempts trying to capture preferences during the VMPprocess. For instance, statements of the form “I prefer X and Y to be placed on adifferent geographic location", “I prefer X not to be co-located with other VMs", or "Iprefer X and Y to be placed on the same PM" are particularly popular in the literatureas customer preferences [13, 40, 42, 43, 120, 121, 138].Furthermore, the cloud administrator is usually overwhelmed with a large set ofpotential placements and has several preferences on the deployment process. Whileany solution in the set is optimal to the underlying constraints and objectives, somesolutions are expected to be preferred (by the administrator or customer) compared toothers [59, 61, 121, 123, 127]. Neglecting the preferences while finding a solutionwould discourage the customers from subscribing to the cloud and administrators fromusing the proposed approach. Hence, proper handling of preferences is a critical issueto address in the design of a cloud.The relevant literature is rich with survey papers on the VMP problem. There aremultiple VMP survey papers that interested readers may refer to, such as [19, 80, 81,85, 91, 93, 110, 114, 116, 128, 139]. Most of these surveys provide multiple classifica-tions and reviews of approaches to handle the VMP problem including modeling, the2 onstraintsObjectivesPreferences M u s t b e s a t i s fi e d Figure 1: The satisfaction level expected for different types of information: Constraintsare requirements that every solution needs to satisfy; Objectives are the deriving forcein searching and choosing one feasible solution over another in the feasible region;Preferences represent wishes and desires. Objectives and constraints are part of theproblem representation and thus need to be taken into account in any algorithm whilepreferences are the key concept in personalizing an approach to the customers andadministrators.objectives and constraints representation and formulation, the techniques used to solvethe problems, the scope and the context in which the placement is taking place andmore profoundly to its variants. The focus on this paper is different as it aims to lookdeeper into the notion of preferences in the VMP context. We begin with a clarificationof what is meant by preferences in the context of an optimization problem. Figure 1shows a conceptual difference between the three types of knowledge. Basically, theobjectives and constraints of the problem instantiate its optimality and feasibility re-gions, whereas satisfaction is commonly attributed to preferences [112]. Indeed, onecan promote a preference to an objective or constraint. Many attempts in the litera-ture adopt this methodology and add preferences as an objective to the problem. Webelieve such transformation albeit perhaps efficient, has its own drawbacks as it is notalways possible to construct objectives that reflect the preferences without losing someinformation. Especially when the customer or administrator pose complex preferencestatements over different aspects of the network.In this paper, we provide a detailed survey of VMP proposals that incorporate softconstraints and preferences highlighting their types, implications, and the algorithmsused to handle them. We also discuss some challenges and identify possible researchopportunities to better incorporate preferences in the context of VMP. A classificationof the proposed approaches is given in Figure 2. Some of the important aspects thatneed to be addressed when proposing an approach include:
Source identifies who initiates the preferences, the customer who owns the VMs, thecloud administrator who owns the PMs, or both.
Type depicts how the preferences are represented. ( i.e., via utility function, binaryrelation, etc.) 3
MP with PreferencesSource Type Approach ProblemDMCustomerCloud AdminVMPM UtilityImportanceBinary Rela-tionSoftConstraints MetaheuristicsMatchingMCDMHeuristics PlacementRe-PlacementMigration
Figure 2: General Classification of preferences based approaches on Virtualization
Approach classifies the algorithm family used to develop an algorithm. In a broadsense, they can be classified to heuristic, metaheuristic, and exact algorithms.
Problem describes the VMP problem context as there are different variations of theproblem, such as migration and re-placement.It is important to realize that the different considerations of these aspects impose somelimitations on how the preferences are represented and processed. More details aboutthese aspects and their implications will be discussed throughout the paper.The rest of the paper is organized as follows: Section 2 presents background re-garding the VMP problem and preferences. We then review the recent literature on theVMP problem with preferences in Section 3. Section 4 provides our insights on the po-tential benefits for the use of preferences in the context of the VMP problem, followedby a discussion over current limitations and future research opportunities. Section 5concludes this work.
VMP is an essential process for the design and operation of cloud data centers. VMPis concerned with the assignment of virtual machines to physical machines (servers)in the data center. The decision of such placement highly impacts the performanceof the data center in terms of utilization levels, traffic congestion level, and energyconsumption, all of which the Decision Makers (DMs) continuously try to optimize.A wide range for data center network topologies have been studied extensively inthe literature. Basically, the architecture tells us how routers, switches, and servers are4 nternet
CorelayerAgglayerEdgelayerServers(PMs)
V M V M V M V M CPU MEMCPU MEMCPU MEMCPU MEM
A server (PM)
Figure 3: A 3-layer tree topology of a data center.connected, and what is the number of ports required on these devices to implement thenetwork. The choice of a topology has a great impact on the cost, scalability, resiliency(in terms of alternative paths), complexity in terms of number wires, and bandwidthand traffic distribution within the data center. As an example, consider the tree topol-ogy shown in Figure 3. It is comprised of three layers; namely, Core, Aggregation,and Edge. The core layer is the data center gateway to the public Internet, whereasthe switches in the edge layer interconnect the servers (called physical machines) toeach other and to the gateway through the middle layer switches. This architectureis a classic one that is commonly used as a reference or baseline for evaluating otherproposals. It is widely adopted due to its simplicity. Despite that, it raises a number oftechnical concerns regarding its scalability and resiliency. Hence, modern data centershave evolved to utilize more sophisticated architectures such as Clos tree, recursive andrandom topologies. More about alternative architectures and their feature can be foundin [33, 94, 109, 137].Many virtualization tool developers have realized the importance of the customiz-ability of the virtual services and also enabled VMP based on multiple attributes thatalso consider non-technical ones, e.g.,
Profile-Driven Storage in vSphere 5.0 [9, 11],HP Moab [hpm], Server affinity policies in Oracle VM server [8], and many others[15].In realization to consumers varying needs, cloud service providers have designedmultiple options for VMs owners to choose from, in which an owner can choose froma set of pre-defined VM types ( i.e., a combination of disk space, allocated memory,bandwidth, etc.) [goo, ec2]. On top of that, many providers went to offer added-valueor policy-based options, such as colocation, geo-redundancy, availability, etc (see forinstance, placement policies and proximity placement group of Microsoft Azure [2, 5],phoenixNAP colocation and security [pho], IBM High availability [ibm] , and mostrecent mobile core network virtualization on AWS [10]).A basic VMP problem can be defined as follows: Given PM = { P , P , ..., P m } , aset of m physical machines ( P k ) and VM = { V , V , ..., V n } , a set of n virtual machines( V i ), how to place each virtual machine into a physical machine while optimizing a setof objectives and satisfying a set of constraints?5 .1.1 Objectives Basically, the utmost goal when solving a VMP problem is to optimize some metrics ofinterest. These metrics are identified as objectives. From a cloud administrator’s per-spective, the main objectives considered in the VMP are highly related to maintainingprofitable operation of the cloud. On top of the list of these objectives is energy con-sumption. Typically, energy consumption is modeled in terms of CPU utilization, and apossible minimization approach is to distribute the load among existing PMs to reduceCPU usage on each PM. Alternatively, turning off underutilized PMs and move theirVMs to other PMs appeared to be more a effective strategy. However, this approachbrings the issue of wear-and-tear to the surface as PMs maintaining higher resourceutilization are expected to fail sooner than others. In addition to energy consumption,efficient packing of VMs with low defragmentation (resource wastage index) increasesthe ability to accommodate more VMs in the cloud which is an important feature fromthe profitability perspective. Moreover, VMs may need to communicate with each otherto perform a certain task. If two communicating VMs are hosted on the same PM, thenno traffic will go through the switches. However, if they reside on two different ma-chines, traffic has to go through intermediate switches. This kind of traffic is calledintra-data center traffic as opposed to external traffic from/to the Internet. Excessiveintra-data center traffic may disrupt traffic between VMs and Internet as intermediateswitches may become congested. In order to prevent this from happening, the trafficbetween peer VMs needs to be localized as much as possible. This can be achievedby placing VMs that are highly interdependent close to each other. However, this ismainly constrained by the capacities of the PMs.The objectives may also involve other metrics that contribute to the profitability in-dex. For example, traffic minimization to avoid unpredicted/undesired congestion thatmay degrade performance, QoS satisfaction, infrastructure reliability, VM availabil-ity, energy consumption of switches, routers, and auxiliary equipment ( e.g., coolingsystems), etc. For more examples on alternative objectives the reader is referred to[19, 80, 85, 93, 110, 139].Generally, a single objective or multiple objectives can be involved in making theplacement decision. However, due to the nature of the VMP problem, involving multi-ple dimensions, factors, stakeholders, and influential metrics, researchers usually con-sider multiple objectives. As discussed above, these objectives are highly interdepen-dent and likely conflicting with each other.
Mostly, any VMP problem can be seen as a pin packing problem. Hence, the con-straints are more or less similar to the pin packing problem: the assignment constraintensures that each VM is assigned to one PM, and one PM can host multiple VMs, andthe capacity constraint ensures the availability of enough resources on the PM to utilizeand fulfill VM requirements. Typically, each VM requires resources described in termsof CPU cycles, RAM space, disk space, network bandwidth, etc. While a server P k ,leveraging virtualization techniques, can host more than a single VM, the resource uti-lization ( i.e., load) should not exceed its resource capacities. In addition, other design6onsiderations can be imposed as constraints. For example, a cloud administrator maybe interested in setting a limit on the number of VMs migrations to avoid instabilityand interruption of the cloud operation, or impose a constraint on security levels forsome VMs. There are multiple variants of the basic VMP problem that appeared in the literature.The basic VMP problem may assume that there are some VMs hosted in a DC andnew VMs are to be placed on PMs based on the current utilization and operationalstatus of the DC. On the long-run, this might lead to defragmentation of the resources.Therefore, rearrangement of the VM-PM mapping is desired in which multiple VMsare consolidated into a PM to enhance the mapping efficiency [115]. Migration is thetechnique used to move a running VM from its current PM and re-assign it to a new PM.It is a powerful technique that enables the cloud operator to reallocate resources andreconfigure assignment based on current status of the machines [139]. However, thereare critical considerations when making the migration decisions, particularly, related tothe timing, frequency, and the cost of migrations [124]. Hence, these factors exacerbatethe complexity of problem in which adopting soft rules or preferences to handle themand introduce a meaningful trade-off is advisable. In our study, we refer to these VMPproblem variants when classifying the research efforts that appeared in the literature.VMP can also be classified based on the number of the clouds involved [80]. Asingle-cloud (private or public) VMP problem is concerned with the placement of VMsinto PMs within the same data center. Multiple clouds VMP, typically, entails solvingtwo assignment problems in which the system has to distribute incoming VM requestsover the different clouds and then optimize each cloud individually. There are multi-ple architectures and models for multi-cloud VMP. For example, hybrid cloud [122],multiple cloud with multiple administrators [103], and others [49, 56, 58, 82]. Anotherrelated but different problem is the cloud administrators selection problem where it isassumed that customers express preferences and intermediate brokers negotiate for thebest offer [14, 65, 79, 83].
Preferences are ubiquitous. Whether we want to choose a restaurant for lunch, watcha movie or purchase an airline ticket, we often have preferences over the set of avail-able alternatives. Thus, preferences play a key role in the decision making process andhave been an active topic of research in many fields such as artificial intelligence, eco-nomics, and operations research [64]. Preferences can be broadly classified based ontheir representation as quantitative and qualitative. In quantitative preferences, usersexpress preferences with numerical values representing the utility of having one alter-native. On the other hand, qualitative preferences are expressed based on comparativeassessments of alternatives. Figure 4 shows a general classification of preferences.The graphical models refer to a class of models that have been introduced lately inthe literature to represent the preferences in a compact way. They rely on the notion of(conditional) preferential independency among attributes and can be described in terms7 referencesQualitative Graphical Models QuantitativeBinary relationLexicographic GAI-netsCP-nets Utility functionWeightsSoft constraints
Figure 4: A General classification of Preferences from representation point of view.of vertices and edges of a graph [16]. In what follows, we review the basic methodsused to represent preferences (interested readers can refer to [45] for a comprehensiveintroduction to the field). Informally, preferences are any additional knowledge thatis not part of the problem requirements and objectives, but represent the subjectivejudgments of the users in terms of liking or disliking an alternative.In decision theory, preferences provide a way to evaluate alternatives based onsubjective judgments. It is typically assumed that every alternative x is a vector oversome k attributes x = ( x , x , . . . , x k ) that represent the world. For instance, in thecontext of VMP, an alternative can be represented by a possible placement of the VMsto PMs and the set of alternatives represent the placements that are possible giventhe problem constraints and objectives. Given a set of alternatives S , there are twoimportant questions that need to be answered by the preference model: • (optimality) What is the best alternative given the preferences? • (dominance) Given two possible alternatives x and y , which one is more pre-ferred?One of the simplest models for preferences is the weighted sum model obtained byassigning an importance weight w i to every attribute i . Then alternatives are assessedbased on their value i.e., w ( x ) = (cid:205) ki = w i x i where w i ≥ and (cid:205) ki = w i = . Forinstance, consider three possible alternatives x = ( , , ) , x = ( , , ) , and x = ( , , ) defined on a set of three attributes a , a , a with importance weights . for a and . for both a and a . Using the weighted sum model, w ( x ) = . × + . × + . × = , similarly w ( x ) = and w ( x ) = . Given these values,one can conclude that x is the best alternative, as it has the highest value among otheralternatives and also x is preferred x since its value is higher.Another representation is the value (or utility) function where alternatives are mappedto real numbers µ : S → R . Here, every alternative x is mapped to a real number µ ( x ) that represents the desirability of the user for such alternative. Considering the aboveexample in the weighted sum model, we can have S = { x , x , x } and the alternativeutility is simply its sum, i.e., µ ( x ) = , µ ( x ) = , and µ ( x ) = . In general, anarbitrary numerical value can be associated with every possible alternative expressingits preference to the decision maker. 8enerally, a preference over a set of alternatives S , can be defined as a binaryrelation on S [64]. Several properties arise to different relations. We start by generaldefinition of preference relations. Definition 1
A preference relation (cid:23) is a preorder if (cid:23) is reflexive and transitive.
A preorder is called a partial order if it is also antisymmetric. A statement of the form x (cid:23) x (cid:48) is interpreted as x is at least as good as x (cid:48) . Given two alternatives x , x (cid:48) , one canderive some useful conclusions between the two as follows1. x is strictly preferred to x (cid:48) iff x (cid:23) x (cid:48) and x (cid:48) (cid:15) x .2. x is indifferent from x (cid:48) iff x (cid:23) x (cid:48) and x (cid:48) (cid:23) x .3. x is incomparable with x (cid:48) iff x (cid:15) x (cid:48) and x (cid:48) (cid:15) x .Incomparability captures the scenario where there is no possible information in (cid:23) toassess the two alternatives. However, being indifferent means the user actually prefersboth alternatives to the same level. One important property that is usually assumed inpractice is the completeness of the partial order. Definition 2
A partial order (cid:23) is said to be complete iff x (cid:23) x (cid:48) or x (cid:48) (cid:23) x for anyx , x (cid:48) ∈ S. A complete partial order is also known as a linear order or a ranking over a set S .In such case, any two alternatives are comparable and one is preferred to the other.Lexicographic preference relations are also common in practice where an importanceorder over attributes are assumed.Another model of preferences that has gained much attention in recent years is thesoft constraint model. In such model, constraints are associated with numerical valuesrepresenting the cost, importance, or preference and unlike hard constraints, alterna-tives are not required to satisfy all of them. A formal representation for soft constraintsis Valued Constraint Satisfaction Problem (VCSP) [105]. Lastly, the concept of ref-erence point is widely used in the evolutionary algorithms to capture users’ idealizedvision of the solution [37] and define a distance to relatively evaluate the closeness ofa given alternative to the reference point. Given the rise of multiple aspects involved in the design of communications systemthat are of interest to decision makers ( e.g., delay, availability, utilization level, hop-count etc.) as well as concepts or design features that are desired to achieve ( e.g., loadbalancing, fairness, human-centric design, etc.), many communications problems havebeen regarded today as multi-dimensional or multi-criteria problems. Hence, incorpo-rating the preference or soft constraint concept into the problem can be useful to manyapplications in communications beyond the VMP problem.There are many problems in the communications and networking field where itis advisable to consider such preferences and/or soft constraints. Such problems in-clude multiple decision makers in which each entity has its own requirements (solid9onstraints) and preferences, or a problem where multiple metrics are involved in thedesign with some sort of priority or importance ( e.g., delay is more important thanreliability) and require insightful handling of preferences in the design [30].Consider for example, a multi-criteria routing problem. Typically, one metric is re-garded as an objective while others are given as constraints, or alternatively the multiplemetrics are aggregated into a single objective function with some defined weights. In amulti-criteria approach, the problem is approached from a different perspective wheremetrics to be optimized are handled as objectives. This promotes optimization in all di-mensions of the problem which results in multiple solutions scattered in hybrid-planes.Using preferences ( e.g., bounds, sequences, etc.), solution space can be partitioned intosmaller regions each has a subset of the solutions with different desirability level [31].Similarly, a telecom network planning with multiple cost types ( e.g., investment,maintenance, management, extension, etc.) and different design criteria ( e.g., capa-cities, availabilities, diameter, edge degree, etc.) is another example where the adop-tion of preferences can help achieve more desirable outcome [38]. This also can beextended to multilayer network design to perform mapping between different layers ordesign overlay networks [100], traffic grooming [102], or resource provisioning con-sidering the cost of each layer independently [133].Another example in communication networks is the design of wireless sensor net-works which entails multiple design sub-problems such as sensor placement, data ag-gregation, power assignment, etc. Typically, each considers a set of objectives to beoptimized ( e.g., detection probability, latency, deployment cost), and a proper defini-tion of objective preferences or layout preferences would improve our perception to thedesign [67, 86, 97].Furthermore, the rapid evolution in many technological fields in communications aswell as the complexity associated with design decisions stems from the nature of theseproblems, being multi-criteria, multi-attribute, multi-stakeholders, make the adoptionof the preference concept a more viable and promising option. There are many prob-lems where the application of the preference concept can be seen advantageous in mod-eling and solving the problem. For example, in the multiobjective SDN controllersplacement problem, trade-offs between objective metrics are needed [104] [57]. An-other example is the resource scheduling problem in mobile networks where multipleusers of different requirements and classes of service are being served with limitedradio resources and changing conditions.
Pietri and Sakellariou [91] surveyed the literature addressing the VMP problem, andprovided a set of common optimization techniques used to achieve the desired ob-jectives. In this section, we survey existing techniques that incorporate the notion ofpreference and its variants in the context of the VMP problem.In the context of the VMP problem, preferences have been tackled by means ofhints, soft constraints, soft SLAs, relaxation bounds, weights on the objectives, weightson the constraints, and affinity. One natural classification is to distinguish betweenpreferences that are issued directly by the user ( i.e., cloud administrator or customer)10 unctional51.0%Non-Functional 30.6% Mixed18.4%
Figure 5: The distribution of the preference types in the literature from functional andnon-functional perspectives. 51% of the proposed approaches deal only with functionalpreferences neglecting users’ genuine preferences on the problem.and preferences that are provided implicitly from a certain measurement ( e.g., rankingVMs based on current communication cost). The later are related to measurable systemattributes. For example, CPU usage index, memory usage index, and inter-VM trafficare operational factors that can be translated into preference relations or rankings. Acommonly proposed approach of this type is the affinity-based VMP problem. It adoptsthe notion of affinity to evaluate which VM is preferred to be hosted on a PM, andwhich PM a VM prefers [25, 112].The former preferences are derived from people interacting with the system includ-ing cloud administrators and customers. For example, a cloud admin may prefer toassign some VMs to newer PMs due to cost considerations, or a customer prefers tohave his VMs located on the same PM or without sharing other VMs [13, 98, 121].Figure 5 shows the distribution of the approaches from functional and non-functionalperspectives.Section 3 lists some examples of preference uses reported in the recent literature.Clearly, the literature shows a wide variety of applications. For instance, preferenceshave been utilized to maintain higher degree of non-functional aspects such as SLAsatisfaction and availability. Also, they have utilized to represent the desire to assignsub-systems to servers with higher resources, or to servers in certain geographic lo-cations. One notable application is ranking PMs based on their available resourcesand ranking VMs based on their required resources. Such ranking is often used in thematching algorithms.In the remainder of this section, we review the literature based on the proposedapproaches. We classify existing works into five broad classes of algorithms: Meta-heuristics, Matching, MCDM, Heuristics, and others. Table 2 shows an overview ofthe existing works based on this classification. At the end of this section, we compara-ratively discuss the advantages and shortcomings of the five classes.11 reference References Usage exampleResources [14, 18, 20, 25,29, 34, 35, 47,62, 65, 68, 69,72, 89, 118, 119,126, 135, 136] • A VM ranks PMs based on available resources or re-sources utilization ratio. • A PM ranks VMs based on required resource/s. • A VM prefers a PM with faster CPU.Location [74, 92, 113,117] • A customer prefers a geographical location over anotherfor security or reliability concerns. • A cloud administrator prefers to place a group of VMs ina specific location for organizational purposes.Inter-VMdependency [75, 76, 89, 99,112, 136] • Preference information is derived from ex-pected/measured traffic between VMs. • VMs collaborating in performing specific tasks or ser-vices.Colocation [26, 41–43, 112,117, 122] • VMs belonging to the same customer are preferred to beplaced close to each other. • Flexible split preferences i.e., allow group VMs to beplaced on different clouds or in the same cloud.Anti-colocation [26, 42, 47] • Place VMs disjointly for security, reliability, or adminis-trative purposes.Sharability [79, 115] • A customer prefers to not share a PM with others.Migrationcost/overhead [65, 72, 134,142] • VMs with lower migration overhead are more preferredto migrate. • VMs with migration cost ( e.g., susceptible to SLA viola-tion due to migration) are less preferred to migrate.Availability/Reliability [17, 142] • Include the degree of PM wear-and-tear ( e.g., due to fre-quent PM ON/OFF cycles) in weighted function. • Replicate a VM to improve its availability.Cost [17, 20, 56] • A customer may prefer a lower cost PM. • A cloud admin may prefer to place VMs based on theirpayment scale and the cost of the infrastructure. • Typically, the cost computed as utility function compos-ing multiple attributes or just monetary value.Ordering [13, 52, 54] • Preference order of preferred PM to host a VM or a set ofVM.Distance [29, 66, 134] • An application prefers computing and storage PMs pairwith smaller distance. • An application prefers to be placed near a specific PM.Objectives [12, 22, 39, 54,77, 103, 125] • The extent to which an objective is prioritized over otherobjectives.SLA [108, 142] • Soft SLA tolerates some violations.Security [20, 141] • Security preferences can be modeled as a function of thesystem diversity. • Place VMs according to their security preferences andPMs capabilities.Other/Unspecified [44, 74, 87, 98,106, 120, 121,130] • a generic model in which preferences information arecustomized/defined by the user/DM. • Place a VM in a specific PM [74, 106].
Table 1: Existing application scenarios for preferences12able 2: Existing Approaches that handle preferences in VMP
Method References
Metaheuristics Simulated annealing [12, 120, 121], NSGA-II [13, 22], Improved grouping ge-netic algorithm [39], Memetic Algorithm [54, 77] Ant-colony [75, 76], heuristics[79] Hybrid algorithm [103]Matching Stable matching [34, 35, 41, 56, 66, 68, 69, 72, 126, 135], Matching game [20, 99],Three-sided stable matching [29], Markov approximation & stable matching [89],College admission problem [65, 136], Resource Balancing Placement matchmak-ing [74] Uncertainty principle of game theory [92], Egalitarian stable Matching[134], Game theory [141]MCDM TOPSIS [18, 44, 118, 119], PROMETHEE [62], General MCDM [98], FuzzyMCDM [17], AHP [111]Heuristic First-Fit [108], Next-Fit and First-Fit Decreasing [42], Opportunistic-Fit [47],Multiple Heuristics [115]Other Fair proportional scheduler [25] Bin packing [26, 52], ILP repacking algorithm[106], Distributed bartering algorithm [112] Random search (biased statisticalsampling) [117] Random search (biased statistical sampling) [122] Brute-force[125], Constraints-based model [130], NA [142], A framework [82, 87], cloudbroker [14]
Evolutionary Algorithms (EAs) [32] have proven to be a successful approach in solvingmany optimization problems including the VMP problem. EAs search for solutions byproducing a population of candidates in every run. The population evolves by applyingcrossover and mutation operators. In the past decade, several preference-based EAshave been proposed (see for example [96] for a survey on preference-based EAs) wherethe goal is to incorporate users’ preferences and desires when solving the problem.Thus, returning solutions that are not only optimized but also preferred by the user.For VMP, preferences have been expressed as a utility function of the administratorfor different combinations of SLA satisfaction, reliability and energy [39] and as tiebreaking mechanism for final solutions of the MOP [76]. Also, preferences have beenused to correlate two VMs to be placed on the same PM based on historical experi-ence; thus, minimizing number of active PMs [75]. Indeed, one of the most well-suitedroles of preferences in EA is to reduce the final Pareto set. Thus, the work of Iharaet al. [54], Blagodurov et al. [22] and López-Pires and Barán [77] considered admin-istrator preferences to reduce the final solutions of the VMP problem. A decentralizeddata center over different local departments have been considered in Saber et al. [103]where it assumes every department administrator has a (possibly conflicting) prefer-ence. Alashaikh and Alanazi [13] considered the case where some VMs have a rankingover which PM to be placed and used a variant of NSGA-II to return preferred solutionsbased on the notion of ceteris paribus preferences.
Another common technique to solve the VMP problem is Simulated Annealing, asingle-based metaheuristic technique to solve optimization problems [60]. A numberof proposals have developed simulated annealing-based heuristics to solve the VMPproblem. Tsakalozos et al. [120] developed a two-phase VM-to-PM placement scheme13able 3: Detailed overview of the existing literature (Metaheuristics)
Reference Perspective ProblemType Objectives Algorithm Preferences Comments
Addya et al.[12] cloud provider placement Energy – Revenue Simulated an-nealing Objectives A bi-objective optimization model to max-imize revenue and minimize energy con-sumption. A trade-off between the two ob-jectives is maintained based on DM’s pref-erence.Alashaikh andAlanazi [13] cloud provider placement Traffic – Energy –Resource wastage MOEA Ordering Placement decision preferences are given apriori as a total order.Blagodurovet al. [22] cloud provider placement Traffic – Energy –Resource consump-tion NSGA-II Objectives Posteriori preferences on objective space.Deng et al.[39] cloud provider consolidation Energy – Resourceconsumption – Avail-ability/Reliability MOEA - im-proved group-ing genetic al-gorithm Objectives Defines a weighted utility functions as aweighted combination of SLA satisfaction,reliability and energy impacts, where theweights on each metric represents the DMpreferences.Ihara et al.[54] cloud provider placement mi-gration Traffic Energy – Rev-enue – No. of mi-grations – Migrationcost MetaheuristicsMemetic Al-gorithm Ordering SetofParetosolutionswithpreferencesap-plied on objective space. Multiple selec-tion strategies such as distance to originand Lexicographic Order (Each objectivefunction is given in an order of evaluation)Liu et al. [76] cloud provider consolidation Energy – No. of mi-grations Ant-colony Inter-VM de-pendency Obtains Pareto-based solutions and thenthe trade-off between the two objectives isadjusted based on the DM’s preferences.Liu et al. [75] cloud provider consolidation Resource consump-tion – min. activePMs Ant-colony Inter-VM de-pendency The ant-colony algorithm deposits thepheromone between VMs rather than be-tween a VM and a PM and represents thepreference between VM and PM. The pref-erence is based on the historical experienceof packing the VM together with thoseVMs that have already been placed on thePM.López-Piresand Barán[77] cloud provider placement Traffic – Energy– load Balancing– Revenue – QoSpriority MetaheuristicsMemetic Al-gorithm Objectives Preference knowledge is utilized to reducesearch space by imposing upper and lowerbounds on objectives.Mahalingamand Sen-gottaiyan[79] user – multi-cloud placement satisfaction SimulatedAnnealing-based Sharing The model allows customers to specifytheir preferences (along with other QoS re-quirements) as hints based on whether theirVMs can be located with other users? VMsor alone. Further, it considers soft con-straints that only contribute to the satis-faction score and not enforced as the hardones. (Uses Nefeli [121])Saber et al.[103] multi-cloud re-placement Energy – Cost – Mi-gration cost – Avail-ability Metaheuristics Objectives Hybrid cloud managed by multiple enti-ties, each of which has different require-ments and preferences in maintaining itsvirtual cloud. Preferences are incorporatedas weights on the objectives. Three algo-rithms are utilized greedy algorithm to findinitial solutions, NSGA to diversify, and alocal search to obtain more non-dominatedsolutions.Tsakalozoset al. [120] user migration other/unspecified Simulatedannealing-based &2-phaseheuristics other/unspecifiedAdopts hints to represents user prefer-ences, internal physical cloud infrastruc-ture, and high-level management goals thatthe DM seek. Fine-grained constraints re-fer to a) all specifics regarding the hostingcapacity including features and resourceavailability in PMs, b) user-provided de-ployment hints and c) cloud administrativegoals. A ranking algorithm finds differ-ent sets (Cohorts) of PMs that satisfy VMrequests in terms of resources. SimulatedAnnealing algorithm to select a single co-hort that satisfy preferences.Tsakalozoset al. [121] user – cloud provider placement other/unspecified Simulated an-nealing other/unspecifiedproposes Nefeli, a hint-based VM place-ment algorithm. In Nefeli, a predefinedset of preferences are mapped onto a util-ity function with some weight (a score) oneach preference criterion. The model alsoutilizes soft and hard constraints for bothconsumer and provider.
14n which a group of PMs, termed cohort, are first identified as potential host basedon some properties, then the final constrained mapping is resolved taking into accountcustomers’ special requirements and preferences. The goal of selecting a subset of allavailable PMs in the first phase is to reduce the number of constraints and the searchspace during the second phase. In addition to resource and cloud specifications con-straints, the scheme considers customers’ deployment hints such as colocation of mul-tiple VMs in a single PM and reserving a whole PM for a single VM. The authorsproposed two heuristics for filtering cohort PMs and placement based on simulatedannealing.Moreover, Tsakalozos et al. [121] developed a cloud gateway, Nefeli, that takescustomers’ deployment hints when mapping incoming VMs to PMs. The hints realizeseveral preferences related to VMs deployment such as colocation, availability, prox-imity and other information.Their objective is to optimize cloud energy efficiency and load balancing whilemeeting cloud admin constraints. At the same time, Nefeli tries to satisfy customers’preferences but it may only satisfy parts of them and ignore the rest based on the avail-able resources. For evaluating multiple placement solutions, Nefeli adopts a weightedscore function on constraints, where a weight derived from customers’ hints is asso-ciated with each constraint. The set of constraints is further decoupled into hard andsoft constraints. While the first is strict and has to be met, the second only contributesto the score function. Eventually, the optimal solution is defined as the one with thehighest score. The algorithm adopts a simulated annealing-based algorithm to solvethe placement problem and a search space reduction method was used for large scaleproblems.Mahalingam and Sengottaiyan [79] adopted Nefeli to develop a cloud service selec-tion mechanism that takes into account VMs’ preferences for price and QoS and searchfor acceptable offers. Lastly, Addya et al. [12] developed a bi-objective optimizationmodel to maximize revenue and minimize energy consumption and solve it using asimulated annealing technique. The proposed model attempts to maximize the revenueand minimize the power budget. A trade-off between the two objectives is maintainedbased on administrator’s preference.Table 3 provides a summary to the works presented here.
VMP can be naturally described as a matching problem. The set of participants inthis matching are often the available VMs and PMs where every participant expressestheir preferences over who to partner with. Such preference relations are due to somemetrics such as transmission cost or processing capabilities and requirements whichcan be measured or estimated. The result of the matching is that every VM is placedon exactly one PM. Stability is an important property where a matching is considered stable when no machine (VM or PM) would prefer another machine to its currentpartner [90].The literature shows different roles of preferences when matching VMs to PMs.VMs could rank PMs based on some properties such as transmission cost [134], mi-gration time and cost [65] , processing capabilities [136] , and security [20] while15Ms rank VMs based on their migration overhead [65] , CPU and resource utilization[29, 136], and risk [20, 89]. Kim and Lee [66] also considered the case where prefer-ences are not necessarily a ranking and ties are allowed. The goal is often to minimizethe overall dissatisfaction score which is defined as sum of ranks. This score can beintuitively seen as the average happiness of VMs and PMs.Preferences have been utilized to place high throughput applications near compu-tation nodes to minimize the cost of communication [69]. Dhillon et al. [41] addressedconsolidating multiple VMs on the same PM while minimizing degradation perfor-mance. It assumed that every VM acts as a selfish agent and has a preference overwhich VMs to partner with on the same PM. The preference is based on some hid-den cost function related to the VM. Kella and Belalem [65] considered the case ofmultiple cloud providers with intermediate brokers where customers express prefer-ences. The live migration problem has been formulated as a matching problem withthe goal of reducing the overall energy consumption without degrading QoS. Cui andTso [35] considered networks where the policies, which reside on middle boxes suchas firewalls and load balancers, etc., are migrated first then a matching between PMsand VMs based on the communication cost and resource usage is applied. In [74], adescriptive language based on XML for the preferences of data center managers wasproposed. The goal is to automate the matching procedure while classifying prefer-ences into two classes; namely, flexible and inflexible. Wang et al. [126] addressed theproblem when VMs and PMs cooperate on a mutual objective (reducing the overall en-ergy consumption). Chu et al. [29] added storage PMs as a third set into the matchingwhere storage PMs prefer smaller VMs. Pillai and Rao [92] considered the case whereIaaS request is beyond the capability of a single PM and therefore developed a mech-anism to form coalition with the goal of minimizing the total execution time based onuncertainty principle in game theory. The work in [141] addressed the problem of mi-grating VMs for security preferences to make it harder for attackers to locate targetedVMs. Lastly, Korasidis et al. [68] enhanced the scalability of the stable matching tech-nique for the VMP problem by considering only the top-k alternatives in the preferencelist. Table 4 summarizes the presented works of this method.
Multi Criteria Decision Making (MCDM) [64] is a field of study to find and analyzepreferred options in complex problems under conflicting criteria. Two well-knowntechniques in MCDM are AHP and TOPSIS [128]. Araujo et al. [17] presented anMCDM method to rank the best cloud infrastructure based on a customized depend-ability metric and the cost. The customized metric takes customer service constraintssuch as reliability, downtime, and cost into consideration. The model allows users toexpress their preferences on cost versus availability, through a planning and analysistool, to obtain the best configuration for them. Tarighi et al. [118, 119] presented amethod for VM migration between cluster nodes (a group of PMs) using TOPSIS al-gorithm The aim of their algorithm is to achieve load balancing among PMs by movingload from the most overloaded PMs to the least-overloaded PMs and attempting to min-imize migration cost defined in terms of amount of data copied and transferred. TheTOPSIS method is a multi criteria decision making technique. It stands for order pref-16able 4: Detailed overview of the existing literature (Matching)
Reference Perspective ProblemType Objectives Algorithm Preferences Comments
Azougagheet al. [20] cloud provider migration Energy – Resourceconsumption Matchinggame Resource– Cost –Security VMs preferences and PMs are representedas utility functions that take into consider-ations multiple attributes such as resourcesand security level for VMs and risk, en-ergy, and revenue for PMs.Chu et al. [29] cloud provider placement Cost Three-sidedStable match-ing Resource –Distance PM nodes are decoupled as computingand storage nodes. three-sided matchingamong computing, storage resources andapplications, each with its preference listPham et al.[89] cloud provider placement Traffic Energy Markov ap-prox. andStable match-ing Resource –Inter-VMdependency vNFs of the same service chain maintaina preference list on PMs, while each PMmaintains a preference list on vNF in-stances.Cui and Tso[35] cloud provider migrationconsolidation Resource consump-tion – Policies Stable match-ing Resource preference knowledge is the cost of pathsand links for VMs traffic.Cui et al. [34] cloud provider migrationconsolidation Resource consump-tion – Policies Stable match-ing Resource same as above.Dhillon et al.[41] cloud provider placement Satisfaction Stable match-ing Colocation Every VM acts as selfish agent and hasa preference (strict total order) over otherVMs to partner with. The system tries tominimize the degradation of VMs place-ment problem and find a solution that mini-mizes the global performance degradation.Ranjana et al.[99] cloud provider placement Energy –comm. cost– min. active PMs Stable match-ing VM depen-dency Affinity-based algorithm.Jain et al. [56] multi-cloud placement Stable match-ing Cost multiple cloud providers, intermediate bro-kers and end usersKella and Be-lalem [65] multi-cloud migration Energy Stable match-ing /collegeadmissionproblem Resource –Migrationcost VMs preference list contain PMs put in anascending order according to the transfertime. PMs preference list contain VMs putin an ascending order according to theirimpact to cpu utilization.Kim and Lee[66] cloud provider migration Stable match-ing Distance the model allows ties in the preference listordering ( i.e., preferences are not given ina strict order)Korasidiset al. [68] user – cloud provider placement other/unspecified Stable match-ing other/unspecifiedAddresses algorithm scalability.Korupoluet al. [69] cloud provider placement Cost Stable match-ing Resource One of the first proposals that applied sta-ble matching to the VM placement prob-lem. It considers compute and storagePMs, and applications of different compu-tation and data needs. Requirements andother applications preferences can be inte-grated into a weighted utility function.Li et al. [74] user consolidation min. active PMs –Resource wastage Stable match-ing (resourcebalancing andmatchmak-ing) Location –Specific PM Defined normal and categorized types ofpreferences and flexible and inflexiblepreferences.Li et al. [72] cloud provider consolidation Energy Stable match-ing Migrationcost – Re-source Modeled as a matching problem on aweighted bipartite graph, with VM migra-tion cost used as a weight on edge betweenVM and PM. Each VM candidate for mi-gration has a preference list on a subsetof PMs chosen to accommodate migratedVMs. The list is ordered based on weights.Pillai and Rao[92] cloud provider placement Revenue Stable match-ing and uncer-tainty princi-ple of gametheory Location The proposal considers forming coalitionbetween PMs to satisfy requirements thatare beyond single machine capabilities.The PM has preference over which coali-tion of machines to go with based on ex-pected payoff.Wang et al.[126] cloud provider migration Energy stable match-ing Resource PMs and VMs have mutual objective offreeing overloaded PMs and utilizing un-derutilized PMs in order to reduce overallenergy consumption. The proposed algo-rithm tries to achieve a reasonable trade-off between power consumption and SLAviolation. Preference list is computed ac-cordingly.Xu and Li[134] cloud provider migration other/unspecified Egalitarianstable Match-ing other/unspecifiedA VM or a server ranks the agents on theother side based on some criteria, which isits preference list. An algorithm then com-putes a matching that cannot be improvedby any pair of VM and server. VMs rankservers in an ascending order of the trans-mission cost. Servers rank VMs in an as-cending order of the migration overhead.Xu and Li[135] user cloud provider placement Resource con-sumption –other/unspecified Stable match-ing other/unspecifiedA VM preference list ranks servers accord-ing to their resources. PMs may preferpacking larger or smaller VMs recognizingDM’s preference of consolidation or loadbalancing, respectively. Proposed a re-vised DA algorithm that finds weakly sta-ble matching which can be an effective ap-proach for handling large-scale problems.Xu et al. [136] user cloud provider placement Resource consump-tion Stable match-ing/ Collegeadmissionproblem Resource –Inter-VMdependency The stable matching theory is applied togenerate an optimal mapping from contain-ers to physical servers.Zhang et al.[141] cloud provider migration Cost game theory Security Users security preferences are modeled asa linear function of the system diversity(using weighted function of the total num-ber possible VM-placements). The cost ac-counts for migration time and number ofmigrations. i.e., given by the user).The alternatives are ranked according to their relative closeness to the ideal solu-tion. The closer the alternative, the better.PM nodes are sorted from the highest overloaded to the idle ones based on some cri-teria such as CPU, RAM, NET, etc. Each attribute is given a weight. Then after findingthe most saturated PM, the algorithm tries to find the best VM candidate for migrationbased on similar criteria w.r.t. VMs. The proposed TOPSIS algorithm can receive threetypes of information including deterministic, linguistic, and fuzzy information.Arianyan et al. [18] proposed a TOPSIS algorithm aiming to optimize energy, SLA,and number of migrations in a cloud data center. The algorithm selects VMs to bemigrated when overloaded PMs are predicted. It considers a decision matrix of theresource utilizations and capacities of VMs. Each VM is assigned a score for eachresource criterion based on the ideal points concept. The total score of a VM is theweighted combination of the scores of each criterion. The weights for each VM are setindependently. Similarly, Feng et al. [44] developed a TOPSIS algorithm that incorpo-rates AHP to find weights of coefficients of the resources decision matrix. Rane andSrivastava [98] introduced a cloud broker architecture to help application customers se-lect their cloud provider considering both functional and non-functional requirementson the service. While functional requirements must be satisfied, non-functional re-quirements are associated with weights to specify their priority on these requirements.Further, an MCDM constraint significance factor realizing the preference of each con-straint is utilized. Then, the method tries to maximize customers’ satisfaction. Kabiret al. [62] proposed a two phase VMP algorithm that first selects a cluster of PMsand then a PM inside the cluster to place a VM. The proposed algorithm is based onthe MCDM method PROMETHEE. Overall, the method is based on outranking alter-natives. Preferences on each criterion are quantified as real numbers and aggregatedonto a weighted preference index. Then, each alternative is evaluated by the numberof alternatives that outrank it and the number of alternatives it outranks based on thepreference index. The alternative with the largest net outranking is the most preferredone.Table 5 summarizes the works presented here.
Shi and Hong [108] proposed a First-Fit algorithm aiming to maximize the profit with-out violating the SLA and the power budget. The algorithm considers two kinds ofSLAs: the hard SLA and the soft SLA. A hard SLA is not tolerable and violatingany of its terms yields no revenue. For a soft SLA, application customers can tolerateperformance degradation to some extent and provider revenue suffers certain loss ac-cordingly. The more elastic the VMs are, the more VMs can be consolidated onto aPM. Thus, this elastic model allows providers to manage revenue based on SLA. More-18able 5: Detailed overview of the existing literature (MCDM)
Reference Perspective ProblemType Objectives Algorithm Preferences Comments
Araujo et al.[17] user placement Availability – Cost Stochasticmodel &MCDM Objectives The authors use stochastic models to assessand evaluate different clouds from user’sperspective. Assessment results are thenused to rank solutions, taking into accountthe user’s preference of each criterion (setas a weight).Arianyanet al. [18] cloud provider consolidation other/unspecified TOPSIS other/unspecifiedPreferences are expressed as weights foreach criteria. The weights are computedbased the average utilization of all sys-tem resources in the data center ( i.e., func-tional). The utmost objective is to removehotspots.Feng et al.[44] cloud provider consolidation Cost TOPSIS/AHP Resources The algorithm is mainly concerned aboutthe migration decision.Kabir et al.[62] multi-cloud placement Resource consump-tion PROMETHEE other/unspecifieda preference function is used to denotethe preference of solution over another foreach criterion. Then, each solution is eval-uated by the number of alternatives outrankit and the number of alternatives it outranksbased on the preference index. The solu-tion with the highest net outranking flow isconsidered as the most preferred.Rane and Sri-vastava [98] user re-placement Satisfaction MCDM other/unspecifiedDefines a constraint significance factor thatsignifies the preference of customer corre-sponding to each constraint.Son and Huh[111] user – cloud provider placement/migration Resource utilization– QoS – no. of mi-grations – Energy MCDM/ AHP Objectives Proposes to use Analytical Hierarchy pro-cess (AHP) coupled with a fuzzy interferersystem to solve the VMP problem. Bymonitoring resources and analyzing ser-vice requirements, a placement decisioncan be obtained from the fuzzy system.Then through AHP algorithm, a placementaction is performed. The AHP is managedto provide analysis of the weights for eachmetrics involved in the decision.Tarighi et al.[119] cloud provider migration load Balancing Mi-gration cost Fuzzy TOP-SIS Resource deterministic, linguistic, and fuzzy prefer-ence information.Tarighi et al.[118] cloud provider migration load Balancing Mi-gration cost TOPSIS Resource same as other one
In this part, we review several works that adopt exact methods or general customizedheuristics.Viswanathan et al. [125] addressed the placement of High-Performance Computing(HPC) applications in clouds. They proposed a VM consolidation strategy that aimsat optimizing resource utilization and energy efficiency while satisfying application’sQoS. A brute-force search algorithm is used to optimize energy efficiency and per-formance with a parameter utilized to adjust the possible trade-off between them. Heet al. [52] proposed a matching algorithm for VMs migration addressing IaaS manage-ment in clouds. For incoming IaaS request, the algorithm finds a suitable PM based onavailable resources. If there are multiple suitable PMs, the system gives preferentialattention to more important customers. A PM is recognized by the different propertiesit has, and a customer may express different preferences over these properties. Thepreferences are given in values indicating their importance to the customer. Then, thealgorithm chooses the most suitable PM that satisfies the preference order the most.Wright et al. [130] developed a two phase resource selection model that can handle acombination of hard and soft constraints. The model intends to enable users to matchtheir application requirements to infrastructure resources. A direct search algorithm isproposed to solve the problem.Chen et al. [25] considered the problem of mapping virtual CPUs of VMs to phys-ical ones. A proportional scheduler (PS) is proposed to provide fair allocation of CPUresources among virtual CPUs. It assigns a weight to every VM based on its com-putational requirements. CPU-affinity is defined as a property of a virtual CPU thatdescribes which physical CPU it can run on. In their study, they illustrated that theinappropriate assignment of weights may produce unfair proportional sharing of CPU.Sonnek et al. [112] presented a decentralized affinity-aware migration technique thatdynamically migrates VMs based on network affinity between pairs of VMs in order20o minimize intra-cloud traffic. Affinity here is defined as VMs interdependency ( i.e., volume of traffic monitored between pairs of VMs). The authors developed a dis-tributed bartering algorithm that strives to place VMs on a PM that is closer to thedata they need. VMs have preference ordering on which PM they migrate to basedon the monitored intra-traffic. The VM is responsible for negotiating the placementwith PMs which improves the scalability of the algorithm and minimizes the overhead.In a similar vein, Ranjana et al. [99] classify PMs based on CPU and communicationbandwidth and then apply affinity-based matching to minimize energy consumption,number of active PMs, and communication cost. Zhou et al. [142] developed three liveVM storage migration mechanisms. The mechanisms introduce a trade-off between theefficiency and cost of VM migration. They also consider user experience (IO penalty),cluster management (migration time) and device usage (degree of wear) in the migra-tion decision by utilizing a migration cost (utility function) in which weights reflectdifferent design preferences.Tantawi [117] proposed a biased random search method based on biased statisticalsampling. It maps VMs and other workloads into PMs and other physical entities sothat an objective function is optimized. The algorithm takes into account both admin-istrators’ and users’ requirements and allows users to specify their preferences overthe constraints (hard/soft constraints) over some criteria such as location. The biasingpushes the solution towards satisfying constraints and preferences. Unuvar et al. [122]adopted the same method to address VMP in hybrid clouds. The placement decisiontakes into account whether private or public cloud is preferred for a customer and a cus-tomer preference on splitting VMs over private and public clouds. Sotiriadis et al. [113]presented VMs migration within OpenStack cloud platforms considering the FIWAREplatform . The migration tool is developed following the FI-WARE conceptual modeland offers a set of software modules, called Specific Enablers (SEs), that allow flexi-ble development of healthcare applications. Within the deployment process of VMs,the system stores and retrieves users’ preferences including geographic location pref-erence. This is done utilizing a graphical interface to configure customer preferences.Chen et al. [26] proposed a Joint Affinity Aware Grouping and Bin Packing (JAGBP)method that aims to minimize application performance cost and maximize resourceutilization. The affinity relationships of VMs define colocation and disperse place-ment preference based on VMs interdependency. They define three affinity types andaffinity relations; namely, colocation placement affinity/relation (co-affinity/relation),no-colocation placement affinity/relation (no co-affinity/relation) and fixed placementaffinity/relation (fixed-affinity/relation). JAGBP was empirically found to improve theabove objectives compared to non-affinity aware placement.Table 6 summarizes the works presented in sections 3.4 and 3.5. It is clear from the five classes of approaches mentioned above that there exists differ-ent ways to incorporate preferences within the problem with different goals as some Reference Perspective ProblemType Objectives Algorithm Preferences Comments
Chen et al.[25] cloud provider placement Resource consump-tion Heuristics Resource VMs are classified into domains based ontheir CPU-affinity.Chen et al.[26] cloud provider placement min. active PMs Bin packing Colocation– Anti-colocation Affinity-aware placement – hard constraintDow [42] cloud provider consolidation min. active PMs Heuristics(pin-packingusing next-fitand ffd) Colocation– Anti-colocation decomposed into two phases; resourcescomputation then colocation.Georgiouet al. [47] cloud provider placement other/unspecified Heuristics(Opportunistic-fit greedyalgorithm) Resource– Anti-colocation IaaS requests. Used the notion of hintsto allow users specify their bandwidthneeds for specific pairs of VMs and anti-collocation constraints for pairs of VMs.aim to faster solution time. Flexibility ofconstraints.He et al. [52] user – cloud provider migration Resource consump-tion – Cost Heuristics Ordering Each PM machine has a set of propertiesassociated with its resources and attributes( e.g.,
RAM, type of OS, etc). If multiplePMs are suitable for hosting a VM ( i.e., satisfy the requirements), then the systemlooks for user’s preferences.Ooi et al. [87] cloud provider placement –replication Availability Team forma-tion other/unspecifiedA framework for resource evaluation toidentify preferable resources and preparefor service placement.Sedaghatet al. [106] cloud provider repacking Resource utilization ILP other/unspecifiedRepacking by changing the configurationof VMs (either number or sizes) that re-side in a server, as the load changes. Thegoal is minimizes resource utilization costaccounting for both price and performanceissues. A decision of repacking or not canbe based on the calculated cost and otherfactors e.g., expected time to next change,interruption time, etc.Shi and Hong[108] cloud provider placement Energy – Revenue First-fitHeuristic SLA Hard SLA and soft SLA.Sonnek et al.[112] cloud provider migration Traffic Heuristics Inter-VMdependency –Colocation affinity; satisfy it as much as possibleSotiriadiset al. [113] user – multi-cloud migration other/unspecified Migrationtool location –other/unspecifiedmigration of VMs in FIWARE systemsSunil Raoand SanthiThilagam[115] migration min. active PMs –Resource wastage Heuristics Sharing not implemented/discussion only.Tantawi [117] user – cloud provider placement Load Balancing Heuristics/randomsearch (biasedstatisticalsampling) Location –Colocation location constraints can be hard or soft.Unuvar et al.[122] user – multi-cloud placement Traffic load Balanc-ing Cost Heuristics- BiasedImportanceSampling Resource– Cost –other/unspecifiedIntroduced a hybrid cloud placement algo-rithm that incorporates two policy param-eter: reservation and cost weight and ad-dresses the trade-off between deploymentcost and request rejection rate.Viswanathanet al. [125] cloud provider placement Energy – Completiontime Brute-force +set partition-ing algorithm Objectives Addresses HPC applications placement inDC.Wright et al.[130] user placement other/unspecified Heuristics other/unspecifieduser perspective. measured/evaluated incostZhou et al.[142] cloud provider migration Cost – Migration cost Heuristics other/unspecifiedIncorporates user experience (IO penalty),cluster management (migration time) anddevice usage (degree of wear). Utilizesa weighted cost function in which theweights on each metric reflects differentdesign preferences. i.e., they represent provider or user, there is a need for interactivemetaheuristics approaches that consult the user during the search process [77].Indeed, the heuristic approaches require domain knowledge to construct an effec-tive heuristic function with which promising search space is discovered first. Suchdomain knowledge is sometimes prohibitive when multiple stakeholders’ preferencesare involved. One needs to assume preferences follow certain structure and often con-struct a heuristic function for a simplified version of the original problem. Hence, it isobvious that the proposed heuristic approaches are general and usually domain inde-pendent [12].Most of the matching methods consider resource as the attribute to base their strat-egy on. This is a reasonable consideration as physical machines have limited resourcesand utilizing them efficiently when looking for a stable matching between VMs andPMs is of great importance. Stability of the match guarantee the average happinessamong all participants, i.e., that there exists no pair ( X , Y ) of virtual machine and phys-ical machine that prefer to be with each other than their current partner ( e.g., the phys-ical machine hosting X , and the VMs placed on Y ).MCDM approaches such as AHP and TOPSIS are known to require large numberof pairwise comparisons and do not show attribute interdependencies. They are alsotime consuming and have scalability issues when the number of attributes is large. Insuch case, preferential dependencies among attributes can lower the time needed toassess preference between alternatives [63]. After the detailed review of preference applications, here we draw insights regardingthe benefits of incorporating preferences into the VMP problem.
User Satisfaction
Indeed one of the main benefits of considering preferences is to findsolutions that are not only optimized but also preferred by the user or decisionmaker. Apart from setting firm and solid requirements, a customer or a cloud ad-ministrator can express additional soft requirements through preferences. Con-sidering such preferences would increase the satisfaction of both parties and usu-ally a graphical interface is utilized to specify the preferences [113, 120, 121].
Reducing the Complexity
VMP is a complex problem given all the possible conflict-ing objectives, constraints from multiple entities, functional and non-functional23roperties, random nature of workload changes, and managerial and geographi-cal policies. It is remarkably noticed in the literature that considering preferencesnot only handles users’ and provider’s fine details but also introduces some sortof flexibility to the problem. In addition, the problem of optimizing VMP isknown to be NP-complete and even sub-optimal solutions needs to be obtainedin a timely manner. Preferences can effectively not only provide but also bethe tool to control the margin of flexibility considering performance, efficiency,solution time [101], and violation rate [28].For example, Ye et al. [138] reported in their study that the number of PMsrequired to host VMs with colocation constraints decreases when the constraintsare softened. In addition, the gain of incorporating preferences improves thescalability to large-scale problems as in [68].Moreover, preference information can be utilized to set criteria for managingcomplex processes. As an example, consider the dynamic PM consolidationprocess. Typically, there exists a trade-off between the goodness of the processand completion time due the complexity of the process [85]. Preferences canbe utilized to resolve this trade-off in an efficient manner. In addition, cloudproviders typically offer predefined instances of virtual resources with differentcapabilities as well as value-added features and services ( e.g., choice of OS,traffic on a private VLAN, etc). The customer choices impact the pricing ofsuch leasing. Wright et al. [130] defined application soft and hard constraints toreduce the complexity of choosing suitable deployment.Besides, there are cases that inspire to use predefined soft constraints. Hao et al.[51] considered bounded constraint violation to maintain and fix the feasibilityof the LP problem under study. The selection of constraints and the extent towhich they are softened is critical to the quality of the solution and should be setbased on administrator’s preferences rather than arbitrarily.
Reducing Search Space
Multiobjective problems typically obtain multiple Pareto so-lutions, and the number of these solutions grows significantly as the number ofobjectives grows. Thus, preferences have been utilized to help mitigate this bur-den [13, 39, 54, 75, 77, 103]. The same can be said about other algorithms ( e.g., alternative based algorithms). For example, the work in [120] considered a twophase approach and applied non-functional preferences in the second phase toreduce the number of candidate PMs.Another example, there can be the case where two or more solutions are available( e.g., ties in preference list [90]), and favoring such an option might have someadvantages that the DM was not aware of.
Adaptation of Specialized Services
Preferences and soft constraints can be utilizedwithin VMP schemes to enable the hosting and QoS guarantees of specializedservices ( e.g., mapping special services to a PM or a set of PMs). Securityconcerns [141], NFV chaining [89], HPC placement [125], mobile applications[140], smart TV services [66], and Web services [82] are examples of VMPfor specialized services that incorporate the notion of preferences in their ap-proaches. 24 mproving Performance
Preferences can be used as a guideline for optimizing theperformance of a system. Such preferences are usually inspired from meta-analysis of the problem and identified as preferred settings or configurations tothe machine. The work in [124] for instance, provides a detailed experimentalstudy and identifies the parameters that influence the cost of migrating a VMand provides some recommended rules to improve VM migration efficiency. Weexpect that these rules ( e.g., migrate applications with small active memory first,migrate low priority applications first) can suitably be identified as preferencesrather than strict constraints or objectives.Furthermore, constraints and objectives are considered known, hard and strict.preferences are soft, and can be given on top of the problem in a way that doesnot impact constraints and objectives. However, preferences can be known inadvance, known a posteriori, or learned throughout the system’s life. The latercan be useful in the VMP context as the system preferences can be learned andadapted to the operational status of the cloud [21]. This gives a wider perspectiveto administrators on getting to know how different unknown options arise fromthe problem and to what extent they can change the output [21, 87]
Software-defined Networking (SDN) and Network Function Virtualization (NFV) aretwo of the key technologies in the cloud and 5G networks. The SDN architectureprovides flexible and programmable network control by decoupling the control planefrom the data plane. This enables the use of simple network devices ( i.e., switches) tocreate and manage networks for different purposes on the fly via a logically centralizedSDN controller.NFV enables deploying network functions ( e.g., routing, firewalling, load balanc-ing, etc.) as software applications ( i.e.,
VM) stored and run on generic servers ( i.e.,
PM). It also introduces flexibility in terms of location, in which functions can be placedanywhere in the network satisfying users and provider requirements. However for aspecific task that requires more than function, there is an order sequence that must befollowed [71].The advent of SDN/NFV technologies has facilitated the cloudification of manycommunication systems [36, 50, 71, 84]. Here we shed light on a number of trends thatleverage virtualization and cloud computing technologies in order to improve overallsystem performance.
Internet of Spectrum Devices aims to expand the radio spectrum of wireless mobilenetworks in which devices/elements of the network ( e.g., base stations) haveno restriction on spectrum use and can utilize multiple shared resources of thespectrum dynamically. A cloud is responsible for gathering spectrum status/info.from monitoring devices and distributing them to spectrum utilizing devices.There is a virtual DM for each entity in the spectrum cloud that is responsible formaking spectrum decisions. Various cloud computing operations can be appliedbased on DM request and users’ preferences [132].25 loud-Radio Access Networks (C-RANs) offer cloud-based architecture in which allcontrol algorithms are placed and executed by data centers [131].Specifically, in a C-RAN, traditional base stations functionalities are decoupledinto radio functions that are stored in the remote radio unit (RRU) and digitaland higher layer functions that are stored in the baseband unit (BBU). The RRUremains at the base station site, while the BBU is moved to the cloud. C-RANsextend the mobile radio access capacity and coverage by pooling BBU resourcesfrom multiple cells. These cells can be heterogenous and may belong to multi-ple collaborating and competing carriers. Following NFV standards, each BBUfunctionality is represented by one or more VMs which can be enabled or dis-abled for each base station. Further, these VMs have to be placed in PMs thatsatisfy their requirements and should be dynamically migrated to where they areneeded. The sequence and order of execution of network functions as well as thecolocation requirements have to be taken into account when making a placementdecision.
Content delivery and Caching
In modern ISP networks, content (videos and cachedpages) can be stored in VMs, and the CDN is configured adaptively by SDNcontrollers to account for traffic loads on the ISP network [129].
Small Cells
One of the challenges to small cell technologies ( e.g.,
WiFi and Fem-tocells) is high-level of interference between cells due to the lack of channelcoordination. Cloud-based control is a promising method to manage channelassignment between different, multi-vendor, small cells. There are many othertechnologies that rely on cloud computing concepts such as mobile cloud, edgeComputing, cognitive IoT, etc. In general, cloud-based architectures can providemore flexibility and centralized control and management of the participating en-tities/devices, in which, coordination and performance can be optimized. Fur-thermore, proper understanding and translation of the requirements of each VM( e.g., application) will lead to desired placement decisions.
VMP with Preferences on the Edge
The problem of solving VMP with preferenceson the edge brings several challenges on top of the VMP challenges. One needscompact models to represent and reason about preferences. Such models needto be compact and succinct in their preference language to handle the possiblylarge number of attributes and limited resources on the edge. Even in the caseof successfully stored, preferences on the edge may be insecure for the gen-uine user’s preferences and selection criteria and, therefore, privacy-preservingtechniques for storing and dealing with preferences on the edge are required. Re-cently, Ouyang et al. [88] proposed an adaptive algorithm for service placementon the edge, which jointly optimizes latency and migration cost, weighted by theon-line learned user’s preferences. Qian et al. [95] developed an privacy-awarealgorithm for service placement in edge cloud, and adopting a user preferencemodel to learn about user preferences toward services. In addition, Chiti et al.[27] tackled the problem of virtual function placement in fog-cloud by applyingmatching game algorithm to fog nodes and virtual functions.26 utomation of Cloud Operations
The scale and the frequent changes of cloud datacenter networks and configurations necessitate faster handling of cloud opera-tions ( i.e., service management, deployment, and configuration). This includesthe provisioning of resources and instantiation, configuration, and delivery ofservices at acceptable service level and without human intervention. Thus, or-chestration and automation of cloud services have been of high interest to bothacademia and industry which gave rise to new concepts addressing these con-cerns. For instance, Zero-Touch networking is an emerging concept that lever-ages SDN and NFV capabilities and aims to automate cloud-based/virtual net-works management and operation. A related emerging concept is intent-drivennetworking. It is an end-to-end network configuration automation solution thataims to optimize performance and user experience. It leverages big data and AItechnologies to achieve such a goal. Preferences and non-functional parameterscan be a helpful tool in the design of automated control algorithms. [106]
Compactness and Cognitive Burden
As the number of alternatives becomes large,direct assessment of preferences becomes problematic. VMP indeed has hugespace of possible alternatives and directly evaluating them would be a demand-ing cognitive task. For example, consider a customer expressing her preferencesfor the placement over 10 PMs. She needs to express her preferences in ordinalway through pairwise comparisons of every pair of PMs. This is indeed problem-atic as the users would lose focus let alone being cognitively able to accuratelyspecify their preferences.The vast majority of VMP approaches assume the existence of a utility function.Accurately specifying utility functions by end-users has been shown to be a dif-ficult and tedious task [46]. In many situations, users cannot afford to specifymore than qualitative statements of the form “ x is better than y ". The literaturelacks concrete attempts that address user’s policies and preferences qualitatively.Furthermore, it is usually assumed that a utility function of one attribute is inde-pendent from other attributes. This assumption is too strong to be met in practiceas the utility of one attribute may depend on other attributes’ values. General-ized Additive Independence Networks (GAI-net) [48] weaken this assumptionby allowing the utility function of one attribute to depend on other attributes.On the qualitative side, Conditional Preference Networks (CP-nets) [23] are aprominent graphical model to represent conditional preferences.Such compact models have been proposed in the last decade to avoid the cog-nitive burden of specifying preferences in multiattribute domains. They requirereasonable effort from the user and adopt the conditional preferential indepen-dency notion. This allows for the decomposition of the preferences into smallerbut compact components while being expressive enough to capture many scenar-ios in practice [16, 64]. Uncertainty, Vagueness, and Risk
Recently, there have been several works on dy-namic and uncertain VMP [24, 53, 78, 80] where workloads and operational27tatuses may change frequently over time. However, it is important to tacklethe problem from the decision maker perspective and consider preferences inuncertain situations. Current approaches usually assume preferences occur withcertainty where, in practice, preferences occur with uncertainty over the set ofpossible alternatives. This is evident in VMP as customers are unaware of possi-ble complexities on the cloud. In such cases, preferences are no longer determin-istic choices but rather a lottery over the alternatives. A lottery is a probabilitydistribution over alternatives [64]. Preferences are then represented over lotteriesand known concepts such as expected utility can be used to rank the alternatives.
Learning and Revision
Machine learning has been recognized as an enabler for build-ing and designing cloud data centers [70, 73, 107]. The VMP problem is dynamicin nature and preferences often change over time. This urges for approaches tonot only handle uncertain preferences but also learn to revise them in light of newinformation. Instead of insisting on having the preferences upfront, it is betterto elicit them during the solving process. Such elicitation holds the promise ofproposing interactive learning strategies that consider humans in the loop whensolving the VMP; thus, lowering the cognitive burden in specifying preferences.Another approach is to learn customers’ and administrators’ preferences fromhistorical data ( i.e., learning preferences passively from past deployments).
Group Decision Making
The VMP problem is inherently a group decision makingproblem. A group of customers and cloud administrators issue their preferenceson different aspects of the problem. It is likely that the issued preferences areconflicting with each other and further aggregation and reconciliation methodsare required. For instance, consider the scenario of Ilkhechi et al. [55] whereeach group of VMs relies heavily on some PMs (named sinks). These sinks( e.g., supercomputers) are a subset of the PMs and it is likely that customers’preferences would be conflicting on how to utilize them. Thus, the aggregationof preferences needs to ensure certain properties like fairness of the deploymentamong customers.
Virtualization technology has enabled cloud data centers to become cost effective, flex-ible and customizable to host diverse enterprise services and applications at differentscales. The VMP is a problem of significant importance to the cloud design for sustain-ing its profit and popularity. The literature shows a wide variety of approaches to solvethe problem and its variants. The problem is also full of preferences of the involvedparties including cloud administrators and customers. Therefore, the proper handlingof preferences is crucial to the success of proposed approaches.In this paper, we discussed the current approaches for handling preferences in thecontext of VMP. We highlighted certain limitations and identified research opportuni-ties. Analyzing the VMP problem from a decision making point of view gives rise toseveral interesting research challenges. Adopting compact models of preferences suchas CP-nets and GAI-net is particularly promising given the large number of attributes28nvolved in typical VMP problems. This has the potential of reducing the cognitiveoverhead of the problem; thus, increase its applicability. Another important researchto be explored is interactively learning the constraints and preferences of the cloud ad-min and/or customers as well as passively learning the soft constraints among VMsand PMs from historical data. Lastly, proposals based on group decision making tech-niques hold the promise of solving the problem taking everyone’s preferences in a fairand transparent manner. Addressing such issues may lead to the development of user-centric VMP that is widely adopted by the stakeholders.
References [ec2] Amazon ec2 instance types. https://aws.amazon.com/ec2/instance-types/ . Accessed: 2019-12-11.[2] Introducing proximity placement groups. https://azure.microsoft.com/en-us/blog/introducing-proximity-placement-groups/ .Accessed: 2019-12-11.[goo] Machine types. https://cloud.google.com/compute/docs/machine-types . Accessed: 2019-12-09.[pho] Phoenixnap. https://phoenixnap.com . Accessed: 2019-12-12.[5] Placement policies for service fabric services. https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-resource-manager-advanced-placement\-rules-placement-policies . Accessed: 2019-12-11.[hpm] Policy-based optimization. Technical report.[ibm] Solving the ha challenge: Placement groups for virtual servers. . Accessed:2019-12-10.[8] (2011). Oracle optimized solution for enterprise cloud infrastructure. Technicalreport.[9] (2012). Cloud infrastructure architecture case study. Technical report.[10] (2019). Carrier-grade mobile packet core network on aws. Technical report.[11] (2019). Performance of vsphere 6.7 scheduling options. Technical report.[12] Addya, S. K., Turuk, A. K., Sahoo, B., Sarkar, M., and Biswash, S. K. (2017).Simulated annealing based vm placement strategy to maximize the profit for cloudservice providers.
Engineering Science and Technology, an International Journal ,20(4):1249 – 1259. 2913] Alashaikh, A. S. and Alanazi, E. A. (2019). Incorporating ceteris paribus prefer-ences in multiobjective virtual machine placement.
IEEE Access , 7:59984–59998.[14] Amato, A. and Venticinque, S. (2013). Multi-objective decision support for bro-kering of cloud SLA.
Proceedings - 27th International Conference on AdvancedInformation Networking and Applications Workshops, WAINA 2013 , pages 1241–1246.[15] Amazon Web Services, I. (2018). Architecting for the Cloud. Technical ReportNovember.[16] Amor, N. B., Dubois, D., Gouider, H., and Prade, H. (2016). Graphical models forpreference representation: An overview. In
International Conference on ScalableUncertainty Management , pages 96–111. Springer.[17] Araujo, J., Maciel, P., Andrade, E., Callou, G., Alves, V., and Cunha, P. (2018).Decision making in cloud environments: an approach based on multiple-criteriadecision analysis and stochastic models.
Journal of Cloud Computing , 7(1).[18] Arianyan, E., Taheri, H., and Sharifian, S. (2016). Novel heuristics for consolida-tion of virtual machines in cloud data centers using multi-criteria resource manage-ment solutions.
Journal of Supercomputing , 72(2):688–717.[19] Attaoui, W. and Sabir, E. (2018). Multi-criteria virtual machine placement incloud computing environments: A literature review.
CoRR .[20] Azougaghe, A., Oualhaj, O. A., Hedabou, M., Belkasmi, M., and Kobbane, A.(2017). Many-to-one matching game towards secure virtual machines migrationin cloud computing. , pages 1–7.[21] Battiti, R. and Passerini, A. (2010). Brain-computer evolutionary multiobjectiveoptimization: A genetic algorithm adapting to the decision maker.
IEEE Transac-tions on Evolutionary Computation , 14(5):671–687.[22] Blagodurov, S., Fedorova, A., Vinnik, E., Dwyer, T., and Hermenier, F. (2015).Multi-objective job placement in clusters. In
SC ’15: Proceedings of the Inter-national Conference for High Performance Computing, Networking, Storage andAnalysis , pages 1–12.[23] Boutilier, C., Brafman, R. I., Domshlak, C., Hoos, H. H., and Poole, D. (2004).Cp-nets: A tool for representing and reasoning with conditional ceteris paribus pref-erence statements.
J. Artif. Intell. Res. (JAIR) , 21:135–191.[24] Chamas, N., López-Pires, F., and Baran, B. (2017). Two-phase virtual machineplacement algorithms for cloud computing: An experimental evaluation under un-certainty. In , pages 1–10.IEEE. 3025] Chen, H., Jin, H., and Hu, K. (2010). Affinity-aware proportional share schedul-ing for virtual machine system.
Proceedings - 9th International Conference on Gridand Cloud Computing, GCC 2010 , pages 75–80.[26] Chen, J., He, Q., Ye, D., Chen, W., Xiang, Y., and Chiew, K. (2017). Joint affinityaware grouping and virtual machine placement.
Microprocessors and Microsystems ,52:365–380.[27] Chiti, F., Fantacci, R., Paganelli, F., and Picano, B. (2019). Virtual FunctionsPlacement With Time Constraints in Fog Computing: A Matching Theory Perspec-tive.
IEEE Transactions on Network and Service Management , 16(3):980–989.[28] Chowdhury, M. R., Mahmud, M. R., and Rahman, R. M. (2015). Implementationand performance analysis of various VM placement strategies in CloudSim.
Journalof Cloud Computing , 4(1):20.[29] Chu, Q., Cui, L., and Zhang, Y. (2017). Joint Computing and Storage ResourceAllocation Based on Stable Matching in Data Centers.
Proceedings - 3rd IEEE In-ternational Conference on Big Data Security on Cloud, BigDataSecurity 2017, 3rdIEEE International Conference on High Performance and Smart Computing, HPSC2017 and 2nd IEEE International Conference on Intelligent Data and Security, IDS2017 , pages 207–212.[30] Clímaco, J., Craveirinha, J., and Girão-Silva, R. (2016). Multicriteria Analysis inTelecommunication Network Planning and Design: A Survey. In Greco, S., Ehrgott,M., and Figueira, J. R., editors,
Multiple Criteria Decision Analysis , volume 233 of
International Series in Operations Research & Management Science , pages 1167–1233. Springer New York, New York, NY.[31] Clímaco, J., Craveirinha, J., and Martins, L. (2019). Comparison of RoutingMethods in Telecommunication Networks—An Overview and a New Proposal Us-ing aMulti-criteria Approach Dealing with Imprecise Information. In Doumpos, M.,Figueira, J. R., Greco, S., and Zopounidis, C., editors,
New Perspectives in Multi-ple Criteria Decision Making , Multiple Criteria Decision Making, pages 397–427.Springer International Publishing, Cham.[32] Coello, C. A. C., Lamont, G. B., Van Veldhuizen, D. A., et al. (2007).
Evolution-ary algorithms for solving multi-objective problems , volume 5. Springer.[33] Colman-meixner, C., Develder, C., Member, S., Tornatore, M., Member, S., andMukherjee, B. (2016). A Survey on Resiliency Techniques in Cloud Comput-ing Infrastructures and Applications.
IEEE Communications Surveys & Tutorials ,18(3):2244–2281.[34] Cui, L., Cziva, R., Tso, F. P., and Pezaros, D. P. (2016). Synergistic policy andvirtual machine consolidation in cloud data centers.
Proceedings - IEEE INFOCOM ,2016-July(May):10–15. 3135] Cui, L. and Tso, F. P. (2015). Joint virtual machine and network policy consol-idation in cloud data centers. , pages 153–158.[36] Dai, B., Xu, G., Huang, B., Qin, P., and Xu, Y. (2017). Enabling network inno-vation in data center networks with software defined networking: A survey.
Journalof Network and Computer Applications , 94:33–49.[37] Deb, K. and Sundar, J. (2006). Reference point based multi-objective optimiza-tion using evolutionary algorithms. In
Proceedings of the 8th annual conference onGenetic and evolutionary computation , pages 635–642. ACM.[38] Decouchon-Brochet, C. (2004). Case study 2: study of the extension of a telecom-munication network. In Collette, Y. and Siarry, P., editors,
Multiobjective Optimiza-tion Principles and Case Studies , Decision Engineering, chapter 11, pages 237–248.Springer Berlin Heidelberg, Berlin, Heidelberg.[39] Deng, W., Liu, F., Jin, H., Liao, X., and Liu, H. (2014). Reliability-aware serverconsolidation for balancing energy-lifetime tradeoff in virtualized cloud datacenters.
International Journal of Communication Systems , 27(4):623–642.[40] Deshpande, U. and Keahey, K. (2017). Traffic-sensitive Live Migration of VirtualMachines.
Future Generation Computer Systems , 72:118–128.[41] Dhillon, J. S., Purini, S., and Kashyap, S. (2013). Virtual machine coscheduling:A game theoretic approach. In
Proceedings - 2013 IEEE/ACM 6th InternationalConference on Utility and Cloud Computing, UCC 2013 , pages 227–234. IEEE.[42] Dow, E. M. (2016). Decomposed multi-objective bin-packing for virtual machineconsolidation.
PeerJ Computer Science , 2:e47.[43] Espling, D., Larsson, L., Li, W., Tordsson, J., and Elmroth, E. (2016). Modelingand Placement of Cloud Services with Internal Structure.
IEEE Transactions onCloud Computing , 4(4):429–439.[44] Feng, D., Wu, Z., Zuo, D., and Zhang, Z. (2019). A multiobjective migrationalgorithm as a resource consolidation strategy in cloud computing.
PLOS ONE ,14(2):e0211729.[45] Fishburn, P. C. (1970). Utility theory for decision making. Technical report,Research analysis corp McLean VA.[46] French, S., editor (1986).
Decision Theory: An Introduction to the Mathematicsof Rationality . Halsted Press, New York, NY, USA.[47] Georgiou, S., Tsakalozos, K., and Delis, A. (2013). Exploiting network-topologyawareness for vm placement in iaas clouds. In , pages 151–158.[48] Gonzales, C. and Perny, P. (2004). Gai networks for utility elicitation. KR , 4:224–234. 3249] Grozev, N. and Buyya, R. (2014). Inter-Cloud architectures and application bro-kering: Taxonomy and survey. Software - Practice and Experience , 44(3):369–390.[50] Han, B., Gopalakrishnan, V., Ji, L., and Lee, S. (2015). Network function vir-tualization: Challenges and opportunities for innovations.
IEEE CommunicationsMagazine , 53(2):90–97.[51] Hao, F., Kodialam, M., Lakshman, T. V., and Mukherjee, S. (2017). Online al-location of virtual machines in a distributed cloud.
IEEE/ACM Transactions onNetworking , 25(1):238–249.[52] He, S., Guo, L., and Guo, Y. (2011). Real time elastic cloud management forlimited resources. In ,pages 622–629.[53] Ihara, D., López-Pires, F., and Baran, B. (2015a). Many-objective virtual ma-chine placement for dynamic environments. In , pages 75–79. IEEE.[54] Ihara, D., Pirez, F. L., and Barán, B. (2015b). Many-Objective Virtual Ma-chine Placement for Dynamic Environments.
Proceedings - 2015 IEEE/ACM 8thInternational Conference on Utility and Cloud Computing, UCC 2015 , (September2017):75–79.[55] Ilkhechi, A. R., Korpeoglu, I., and Ulusoy, Ö. (2015). Network-aware virtualmachine placement in cloud data centers with multiple traffic-intensive components.
Computer Networks , 91:508–527.[56] Jain, S., Purini, S., and Reddy, P. V. (2018). A multi-cloud marketplacemodel with multiple brokers for iaas layer and generalized stable matching. ,pages 257–266.[57] Jalili, A., Keshtgari, M., Akbari, R., and Javidan, R. (2019). Multi criteria anal-ysis of Controller Placement Problem in Software Defined Networks.
ComputerCommunications , 133(August 2018):115–128.[58] Jamshidi, P., Pahl, C., and Mendonça, N. C. (2017). Pattern-based multi-cloudarchitecture migration.
Software - Practice and Experience , 47(9):1159–1184.[59] Jayasinghe, D., Pu, C., Eilam, T., Steinder, M., Whally, I., and Snible, E. (2011).Improving performance and availability of services hosted on iaas clouds with struc-tural constraint-aware virtual machine placement. In , pages 72–79. IEEE.[60] Johnson, D. S., Aragon, C. R., McGeoch, L. A., and Schevon, C. (1989). Opti-mization by simulated annealing: An experimental evaluation; part i, graph parti-tioning.
Operations research , 37(6):865–892.3361] Jung, G., Joshi, K. R., Hiltunen, M. A., Schlichting, R. D., and Pu, C. (2010). Per-formance and availability aware regeneration for cloud based multitier applications.In , pages 497–506. IEEE.[62] Kabir, M. H., Shoja, G. C., and Ganti, S. (2015). VM placement algo-rithms for hierarchical cloud infrastructure.
Proceedings of the InternationalConference on Cloud Computing Technology and Science, CloudCom , 2015-February(February):656–659.[63] Keeney, R. and Raiffa, H. (1976).
Decisions with multiple objectives: Preferencesand value tradeoffs . J. Wiley, New York.[64] Keeney, R. L. and Raiffa, H. (1993).
Decisions with multiple objectives: prefer-ences and value trade-offs . Cambridge university press.[65] Kella, A. and Belalem, G. (2014). VM live migration algorithm based on stablematching model to improve energy consumption and quality of service.
CLOSER2014 - Proceedings of the 4th International Conference on Cloud Computing andServices Science , pages 118–128.[66] Kim, G. and Lee, W. (2014). Stable matching with ties for cloud-assisted smartTV services.
Digest of Technical Papers - IEEE International Conference on Con-sumer Electronics , pages 558–559.[67] Konstantinidis, A., Yang, K., Zhang, Q., and Zeinalipour-Yazti, D. (2010). Amulti-objective evolutionary algorithm for the deployment and power assignmentproblem in wireless sensor networks.
Computer Networks , 54(6):960–976.[68] Korasidis, N., Giannakopoulos, I., Doka, K., Tsoumakos, D., and Koziris, N.(2017). Fair, fast and frugal large-scale matchmaking for vm placement. In Sellis,T. and Oikonomou, K., editors,
Algorithmic Aspects of Cloud Computing , pages131–145, Cham. Springer International Publishing.[69] Korupolu, M., Singh, A., and Bamba, B. (2009). Coupled placement in moderndata centers.
IPDPS 2009 - Proceedings of the 2009 IEEE International Paralleland Distributed Processing Symposium .[70] Kundu, S., Rangaswami, R., Gulati, A., Zhao, M., and Dutta, K. (2012). Mod-eling virtualized applications using machine learning techniques. In
ACM SigplanNotices , volume 47, pages 3–14. ACM.[71] Laghrissi, A. (2019). A Survey on the Placement of Virtual Resources and VirtualNetwork Functions.
IEEE Communications Surveys & Tutorials , 21(2):1409–1434.[72] Li, W., Wang, Y., Wang, Y., Xia, Y., Luo, X., and Wu, Q. (2017). An Energy-Aware and Under-SLA-Constraints VM Consolidation Strategy Based on the Opti-mal Matching Method.
Int. J. Web Serv. Res. , 14(4):75–89.3473] Li, X., Qian, Z., Lu, S., and Wu, J. (2013). Energy efficient virtual machineplacement algorithm with balanced and improved resource utilization in a data cen-ter.
Mathematical and Computer Modelling , 58(5-6):1222–1235.[74] Li, X., Ventresque, A., Murphy, J., and Thorburn, J. (2016). SOC: Satisfaction-Oriented Virtual Machine Consolidation in Enterprise Data Centers.
InternationalJournal of Parallel Programming , 44(1):130–150.[75] Liu, X. F., Zhan, Z. H., Deng, J. D., Li, Y., Gu, T., and Zhang, J. (2018). AnEnergy Efficient Ant Colony System for Virtual Machine Placement in Cloud Com-puting.
IEEE Transactions on Evolutionary Computation , 22(1):113–128.[76] Liu, X. F., Zhan, Z. H., and Zhang, J. (2017). An energy aware unified ant colonysystem for dynamic virtual machine placement in cloud computing.
Energies , 10(5).[77] López-Pires, F. and Barán, B. (2017). Many-Objective Virtual Machine Place-ment.
Journal of Grid Computing , 15(2):161–176.[78] López-Pires, F., Barán, B., Benítez, L., Zalimben, S., and Amarilla, A. (2018).Virtual machine placement for elastic infrastructures in overbooked cloud comput-ing datacenters under uncertainty.
Future Generation Computer Systems , 79:830–848.[79] Mahalingam, S. K. and Sengottaiyan, N. (2015). A QoS Guaranteed Selection ofEfficient Cloud Services.
Indian Journal of Science and Technology , 8(S9):103.[80] Mann, Z. Á. (2015). Allocation of Virtual Machines in Cloud Data Centers—ASurvey of Problem Models and Optimization Algorithms.
ACM Computing Surveys ,48(1):1–34.[81] Masdari, M., Nabavi, S. S., and Ahmadi, V. (2016). An overview of virtual ma-chine placement schemes in cloud computing.
J. Netw. Comput. Appl. , 66(C):106–127.[82] Menzel, M. and Ranjan, R. (2012). Cloudgenius: Decision support for web servercloud migration.
CoRR , abs/1203.3997.[83] Mezni, H. and Abdeljaoued, T. (2018). A cloud services recommendation sys-tem based on Fuzzy Formal Concept Analysis.
Data and Knowledge Engineering ,116(May):100–123.[84] Mijumbi, R., Serrat, J., Gorricho, J.-L., Bouten, N., De Turck, F., and Boutaba,R. (2016). Network Function Virtualization: State-of-the-Art and Research Chal-lenges.
IEEE Communications Surveys & Tutorials , 18(1):236–262.[85] Mohamadi Bahram Abadi, R., Rahmani, A. M., and Alizadeh, S. H. (2018).Server consolidation techniques in virtualized data centers of cloud environments: Asystematic literature review.
Software - Practice and Experience , 48(9):1688–1726.3586] Murugeswari, R., Radhakrishnan, S., and Devaraj, D. (2016). A multi-objectiveevolutionary algorithm based QoS routing in wireless mesh networks.
Applied SoftComputing Journal , 40:517–525.[87] Ooi, B. Y., Chan, H. Y., and Cheah, Y. N. (2012). Dynamic service placementand replication framework to enhance service availability using team formation al-gorithm.
Journal of Systems and Software , 85(9):2048–2062.[88] Ouyang, T., Li, R., Chen, X., Zhou, Z., and Tang, X. (2019). Adaptive User-managed Service Placement for Mobile Edge Computing: An Online Learning Ap-proach. In
IEEE INFOCOM 2019 - IEEE Conference on Computer Communica-tions , pages 1468–1476. IEEE.[89] Pham, C., Tran, N. H., Ren, S., Saad, W., and Hong, C. S. (2017). Traffic-aware and Energy-efficient vNF Placement for Service Chaining: Joint Samplingand Matching Approach.
IEEE Transactions on Services Computing , pages 1–1.[90] Phuke, N., Gharote, M., Patil, R., and Lodha, S. (2018). Multi-objective stablematching with ties.
IEEE International Conference on Industrial Engineering andEngineering Management , 2017-December:964–968.[91] Pietri, I. and Sakellariou, R. (2016). Mapping virtual machines onto physicalmachines in cloud computing: A survey.
ACM Comput. Surv. , 49(3):49:1–49:30.[92] Pillai, P. S. and Rao, S. (2016). Resource Allocation in Cloud Computing Usingthe Uncertainty Principle of Game Theory.
IEEE Systems Journal , 10(2):637–648.[93] Pires, F. L. and Barán, B. (2015). Virtual machine placement literature review.
CoRR , abs/1506.01509.[94] Qi, H., Shiraz, M., Liu, J.-y., Gani, A., Abdul Rahman, Z., and Altameem, T. A.(2014). Data center network architecture in cloud computing: review, taxonomy,and open research issues.
Journal of Zhejiang University SCIENCE C , 15(9):776–793.[95] Qian, Y., Hu, L., Chen, J., Guan, X., Hassan, M. M., and Alelaiwi, A. (2019).Privacy-aware service placement for mobile edge computing via federated learning.
Information Sciences , 505:562–570.[96] Rachmawati, L. and Srinivasan, D. (2006). Preference Incorporation in Multi-objective Evolutionary Algorithms: A Survey. , pages 962–968.[97] Rajagopalan, R., Mohan, C. K., Mehrotra, K. G., and Varshney, P. K. (2011).Multi-Objective Evolutionary Algorithms for Sensor Network Design.
Multi-Objective Optimization in Computational Intelligence , pages 208–238.[98] Rane, D. and Srivastava, A. (2015). Cloud Brokering Architecture for DynamicPlacement of Virtual Machines.
Proceedings - 2015 IEEE 8th International Confer-ence on Cloud Computing, CLOUD 2015 , pages 661–668.3699] Ranjana, R., Radha, S., and Raja, J. (2019). Network affinity aware energy effi-cient virtual machine placement algorithm.
International Journal of Business Intel-ligence and Data Mining , 14(1/2):40.[100] Risso, C., Nesmachnow, S., and Robledo, F. (2018). Metaheuristic approachesfor IP/MPLS network design.
International Transactions in Operational Research ,25(2):599–625.[101] Roytman, A., Kansal, A., Govindan, S., Liu, J., and Nath, S. (2013). AlgorithmDesign for Performance Aware VM Consolidation. Technical Report MSR-TR-2013-28, Microsoft Research.[102] Rubio-Largo, A., Vega-Rodriguez, M. A., Gomez-Pulido, J. A., and Sanchez-Perez, J. M. (2013). Multiobjective Metaheuristics for Traffic Grooming in OpticalNetworks.
IEEE Transactions on Evolutionary Computation , 17(4):457–473.[103] Saber, T., Thorburn, J., Murphy, L., and Ventresque, A. (2018). VM reassign-ment in hybrid clouds for large decentralised companies: A multi-objective chal-lenge.
Future Generation Computer Systems , 79:751–764.[104] Sahoo, K. S., Puthal, D., Obaidat, M. S., Sarkar, A., Mishra, S. K., and Sahoo,B. (2018). On the placement of controllers in software-Defined-WAN using meta-heuristic approach.
Journal of Systems and Software , 145(January):180–194.[105] Schiex, T., Fargier, H., Verfaillie, G., et al. (1995). Valued constraint satisfactionproblems: Hard and easy problems.
IJCAI (1) , 95:631–639.[106] Sedaghat, M., Hernandez-Rodriguez, F., and Elmroth, E. (2013). A virtual ma-chine re-packing approach to the horizontal vs. vertical elasticity trade-off for cloudautoscaling. In
Proceedings of the 2013 ACM Cloud and Autonomic ComputingConference , CAC ’13, pages 6:1–6:10, New York, NY, USA. ACM.[107] Shi, R., Zhang, J., Chu, W., Bao, Q., Jin, X., Gong, C., Zhu, Q., Yu, C., andRosenberg, S. (2015). Mdp and machine learning-based cost-optimization of dy-namic resource allocation for network function virtualization. In , pages 65–73. IEEE.[108] Shi, W. and Hong, B. (2011). Towards profitable virtual machine placement inthe data center.
Proceedings - 2011 4th IEEE International Conference on Utilityand Cloud Computing, UCC 2011 , pages 138–145.[109] Shooshtarian, L., Safaei, F., and Tizghadam, A. (2018). Scaling-up versusscaling-out networking in data centers : a comparative robustness analysis.
TheJournal of Supercomputing , 74(8):3950–3974.[110] Silva Filho, M. C., Monteiro, C. C., Inácio, P. R., and Freire, M. M. (2018).Approaches for optimizing virtual machine placement and migration in cloud envi-ronments: A survey.
Journal of Parallel and Distributed Computing , 111:222–250.37111] Son, A.-Y. and Huh, E.-N. (2019). Multi-Objective Service Placement SchemeBased on Fuzzy-AHP System for Distributed Cloud Computing.
Applied Sciences ,9(17):3550.[112] Sonnek, J., Greensky, J., Reutiman, R., and Chandra, A. (2010). Starling: Min-imizing communication overhead in virtualized computing platforms using decen-tralized affinity-aware migration.
Proceedings of the International Conference onParallel Processing , pages 228–237.[113] Sotiriadis, S., Vakanas, L., Petrakis, E., Zampognaro, P., and Bessis, N. (2016).Automatic migration and deployment of cloud services for healthcare applicationdevelopment in FIWARE.
Proceedings - IEEE 30th International Conference on Ad-vanced Information Networking and Applications Workshops, WAINA 2016 , pages416–419.[114] Sun, X., Ansari, N., and Wang, R. (2016). Optimizing resource utilization of adata center.
IEEE Communications Surveys Tutorials , 18(4):2822–2846.[115] Sunil Rao, K. and Santhi Thilagam, P. (2015). Heuristics based server consoli-dation with residual resource defragmentation in cloud data centers.
Future Gener-ation Computer Systems , 50:87–98.[116] Talebian, H., Gani, A., Sookhak, M., Abdelatif, A. A., Yousafzai, A., Vasilakos,A. V., and Yu, F. R. (2019). Optimizing virtual machine placement in IaaS datacenters: taxonomy, review and open issues.
Cluster Computing , 0123456789.[117] Tantawi, A. N. (2016). Solution biasing for optimized cloud workload place-ment.
Proceedings - 2016 IEEE International Conference on Autonomic Comput-ing, ICAC 2016 , pages 105–110.[118] Tarighi, M., Motamedi, S. A., and Arianyan, E. (2010a). Performance improve-ment of virtualized cluster computing system using TOPSIS algorithm. , pages1–6.[119] Tarighi, M., Motamedi, S. A. A. A., and Sharifian, S. (2010b). A new modelfor virtual machine migration in virtualized cluster server based on Fuzzy DecisionMaking.
Journal of Telecommunications , 1(1):40–51.[120] Tsakalozos, K., Roussopoulos, M., and Delis, A. (2011). VM placement in non-homogeneous IaaS-clouds.
Lecture Notes in Computer Science (including subseriesLecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 7084LNCS:172–187.[121] Tsakalozos, K., Roussopoulos, M., and Delis, A. (2013). Hint-based executionof workloads in clouds with Nefeli.
IEEE Transactions on Parallel and DistributedSystems , 24(7):1331–1340. 38122] Unuvar, M., Steinder, M., and Tantawi, A. N. (2014). Hybrid cloud placementalgorithm. In , pages 197–206.[123] Van, H. N., Tran, F. D., and Menaud, J.-M. (2009). Autonomic virtual resourcemanagement for service hosting platforms. In , pages 1–8. IEEE.[124] Verma, A., Kumar, G., and Koller, R. (2010). The cost of reconfiguration in acloud. In
Proceedings of the 11th International Middleware Conference IndustrialTrack , Middleware Industrial Track ’10, pages 11–16, New York, NY, USA. ACM.[125] Viswanathan, H., Lee, E. K., Rodero, I., Pompili, D., Parashar, M., and Gamell,M. (2011). Energy-aware application-centric vm allocation for hpc workloads. In , pages 890–897.[126] Wang, J. V., Fok, K. Y., Cheng, C. T., and Tse, C. K. (2016). A Stable Matching-Based Virtual Machine Allocation Mechanism for Cloud Data Centers.
Proceedings- 2016 IEEE World Congress on Services, SERVICES 2016 , pages 103–106.[127] Wang, X., Lan, D., Wang, G., Fang, X., Ye, M., Chen, Y., and Wang, Q.(2007). Appliance-based autonomic provisioning framework for virtualized out-sourcing data center. In
Fourth International Conference on Autonomic Computing(ICAC’07) , pages 29–29. IEEE.[128] Whaiduzzaman, M., Gani, A., Anuar, N. B., Shiraz, M., Haque, M. N., andHaque, I. T. (2014). Cloud Service Selection Using Multicriteria Decision Analysis.
The Scientific World Journal , 2014:1–10.[129] Wichtlhuber, M., Reinecke, R., and Hausheer, D. (2015). An SDN-basedCDN/ISP collaboration architecture for managing high-volume flows.
IEEE Trans-actions on Network and Service Management , 12(1):48–60.[130] Wright, P., Sun, Y. L., Harmer, T., Keenan, A., Stewart, A., and Perrott, R.(2012). A constraints-based resource discovery model for multi-provider cloud en-vironments.
Journal of Cloud Computing , 1(1):1–14.[131] Wu, J., Zhang, Z., Hong, Y., and Wen, Y. (2015). Cloud radio access network(C-RAN): a primer.
IEEE Network , 29(1):35–41.[132] Wu, Q., Ding, G., Du, Z., Sun, Y., Jo, M., and Vasilakos, A. V. (2016). ACloud-Based Architecture for the Internet of Spectrum Devices Over Future Wire-less Networks.
IEEE Access , 4:2854–2862.[133] Xing, C., Addie, R. G., Peng, Y., Lin, R., Li, F., Hu, W., Abramov, V. M., andZukerman, M. (2019). Resource Provisioning for a Multi-Layered Network.
IEEEAccess , 7:16226–16245. 39134] Xu, H. and Li, B. (2011). Egalitarian stable matching for VM migration in cloudcomputing. , (May 2011):631–636.[135] Xu, H. and Li, B. (2013). Anchor: A versatile and efficient framework forresource management in the cloud.
IEEE Transactions on Parallel and DistributedSystems , 24(6):1066–1076.[136] Xu, X., Yu, H., and Pei, X. (2014). A novel resource scheduling approach incontainer based clouds. In , pages 257–264. IEEE.[137] Yao, F., Wu, J., Venkataramani, G., and Subramaniam, S. (2014). A comparativeanalysis of data center network architectures. In , pages 3106–3111.[138] Ye, K., Wu, Z., Wang, C., Zhou, B. B., Si, W., Jiang, X., and Zomaya, A. Y.(2015). Profiling-based workload consolidation and migration in virtualized datacenters.
IEEE Transactions on Parallel and Distributed Systems , 26(3):878–890.[139] Zhang, F., Liu, G., Fu, X., and Yahyapour, R. (2018). A survey on virtual ma-chine migration: Challenges, techniques, and open issues.
IEEE CommunicationsSurveys Tutorials , 20(2):1206–1243.[140] Zhang, W., Tan, S., Xia, F., Chen, X., Li, Z., Lu, Q., and Yang, S. (2016). A sur-vey on decision making for task migration in mobile cloud environments.
Personaland Ubiquitous Computing , 20(3):295–309.[141] Zhang, Y., Li, M., Bai, K., Yu, M., and Zang, W. (2012). Incentive compatiblemoving target defense against vm-colocation attacks in clouds. In Gritzalis, D., Fur-nell, S., and Theoharidou, M., editors,
Information Security and Privacy Research ,pages 388–399, Berlin, Heidelberg. Springer Berlin Heidelberg.[142] Zhou, R., Liu, F., Li, C., and Li, T. (2013). Optimizing virtual machine livestorage migration in heterogeneous storage environment.