Mahyar Movahed Nejad

Energy-Aware Scheduling of MapReduce Jobs for Big Data Applications

The majority of large-scale data intensive applications executed by data centers are based on MapReduce or its open-source implementation, Hadoop. Such applications are executed on large clusters requiring large amounts of energy, making the energy costs a considerable fraction of the data centers overall costs. Therefore minimizing the energy consumption when executing each MapReduce job is a critical concern for data centers. In this paper, we propose a framework for improving the energy efficiency of MapReduce applications, while satisfying the service level agreement (SLA). We first model the problem of energy-aware scheduling of a single MapReduce job as an Integer Program. We then propose two heuristic algorithms, called energy-aware MapReduce scheduling algorithms (EMRSA-I and EMRSA-II), that find the assignments of map and reduce tasks to the machine slots in orderto minimize the energy consumed when executing the application. We perform extensive experiments on a Hadoop cluster to determine the energy consumption and execution time for several workloads from the HiBench benchmark suite including TeraSort, PageRank, and K-means clustering, and then use this data in an extensive simulation study to analyze the performance of the proposed algorithms. The results show that EMRSA-I and EMRSA-II are able to find near optimal job schedules consuming approximately 40 percent less energy on average than the schedules obtained by a common practice scheduler that minimizes the makespan.

An Online Mechanism for Resource Allocation and Pricing in Clouds

Cloud providers provision their various resources such as CPUs, memory, and storage in the form of virtual machine (VM) instances which are then allocated to the users. The users are charged based on a pay-as-you-go model, and their payments should be determined by considering both their incentives and the incentives of the cloud providers. Auction markets can capture such incentives, where users name their own prices for their requested VMs. We design an auction-based online mechanism for VM provisioning, allocation, and pricing in clouds that considers several types of resources. Our proposed online mechanism makes no assumptions about future demand of VMs, which is the case in real cloud settings. The proposed online mechanism is invoked as soon as a user places a request or some of the allocated resources are released and become available. The mechanism allocates VM instances to selected users for the period they are requested for, and ensures that the users will continue using their VM instances for the entire requested period. In addition, the mechanism determines the payment the users have to pay for using the allocated resources. We prove that the mechanism is incentive-compatible, that is, it gives incentives to the users to reveal their actual requests. We investigate the performance of our proposed mechanism through extensive experiments.

Incentive-Compatible Online Mechanisms for Resource Provisioning and Allocation in Clouds

Cloud providers provision their various resources such as CPUs, memory, and storage in the form of Virtual Machine (VM) instances which are then allocated to the users. We design online mechanisms for VM provisioning and allocation in clouds that consider several types of available resources. Our proposed online mechanisms make no assumptions about future demand of VMs, which is the case in real cloud settings. The proposed mechanisms are invoked as soon as a user places a request or some of the allocated resources are released and become available. The mechanisms allocate VM instances to selected users for the period they are requested for, and ensure that the users will continue using their VM instances for the entire requested period. In addition, the mechanisms determine the payment the users have to pay for using the allocated resources. We prove that the mechanisms are incentive-compatible, that is, they give incentives to the users to reveal their true valuations for their requested bundles of VM instances. We investigate the performance of our proposed mechanisms through extensive experiments.

A PTAS Mechanism for Provisioning and Allocation of Heterogeneous Cloud Resources

Cloud providers provision their heterogeneous resources such as CPUs, memory, and storage in the form of virtual machine (VM) instances which are then allocated to the users. One of the major challenges faced by the cloud providers is to allocate and provision these resources such that their profit is maximized, and the resources are utilized efficiently. Recently, cloud providers have introduced auction-based models which allow users to submit bids for their requested VMs. We address the problem of autonomic VM provisioning and allocation for the auction-based model considering multiple types of resources by designing an approximation mechanism. In addition, the mechanism determines the payment the users have to pay for using the allocated resources. This problem is computationally intractable, and our proposed mechanism is by far the strongest approximation result that can be achieved for this problem. We show that the proposed approximation mechanism is a polynomial-time approximation scheme (PTAS). Furthermore, our proposed mechanism drives the system into an equilibrium in which the users do not have incentives to manipulate the system by untruthfully reporting their VM bundle requests and valuations. We perform extensive experiments using real workload traces in order to investigate the performance of the proposed mechanism.

Energy-Aware Scheduling of MapReduce Jobs

The majority of large-scale data intensive applications executed by data centers are based on MapReduce or its open-source implementation, Hadoop. Such applications are executed on large clusters requiring large amounts of energy, making the energy costs a large fraction of the data centers overall costs. Therefore minimizing the energy consumption when executing MapReduce jobs is a critical concern for data centers. In this paper, we propose a framework for improving the energy efficiency of MapReduce applications, while satisfying the service level agreement (SLA). We first model the problem of energy-aware scheduling of MapReduce jobs as an Integer Program. We then propose a greedy algorithm, called Energy-aware MapReduce Scheduling Algorithm (EMRSA), that finds the assignments of map and reduce tasks to the machine slots in order to minimize the energy consumed when executing the application. We perform experiments on a large Hadoop cluster to determine the energy consumption of several MapReduce benchmark applications, and then use this data in an extensive simulation study to characterize the performance of the proposed algorithm. The results show that EMRSA is able to find job schedules consuming 40% less energy on average than the schedules obtained by a common practice scheduler that minimizes the makespan.

A truthful approximation mechanism for autonomic virtual machine provisioning and allocation in clouds

One of the major challenges faced by the cloud providers is to allocate and provision the resources such that their profit is maximized and the resources are utilized efficiently. We address this challenge by designing an autonomic VM (Virtual Machine) provisioning and allocation mechanism that adapts to the changing user demands. We show that the proposed mechanism is a PTAS (Polynomial-Time Approximation Scheme) and that it is truthful, that is, the users do not have incentives to lie about their requested bundles of VM instances and their valuations. We perform extensive experiments in order to investigate the properties of the mechanism.

Effects of traffic network dynamics on hierarchical community-based representations of large road networks

Efficient representation of traffic networks, including congestion states, plays an important role in the effectiveness of routing algorithms incorporating Intelligent Transportation Systems (ITS) data. We employ an emerging concept in analyzing complex networks called “community structure detection” to capture traffic network dynamics in the form of hierarchical community-based representations of road networks. A key strength of these community (structure) detection methods is their computational efficiency. We investigate the impact of traffic dynamics on the hierarchical community-based representations of large road networks. The resulting hierarchical community representations and their evolution over varying traffic conditions with time can aid the computational performance of real-time routing algorithms. We analyze the performance of hierarchical community detection methods on the metropolitan road networks of New York City, Detroit, and San Francisco Bay area.

State space reduction in modeling traffic network dynamics for dynamic routing under ITS

Effective en route guidance for vehicles can play an important role in alleviating the negative impacts of evergrowing congestion. As network traffic conditions change due to recurrent and non-recurrent congestion, the optimal route can change, and updated directions should be given to the driver in real-time. However, the task of exploiting real-time traffic information for optimal routing is computationally challenging. On the other hand, simplistic schemes (e.g., assuming constant speeds for different network arcs across all hours of the day) lead only to poor travel time performance and driver dissatisfaction. Hence, there is need for compact yet effective representations of traffic network dynamics for supporting routing algorithms. In this paper, we propose two state space reduction approaches employing knowledge discovery and data mining (KDD) methods and mathematical programming (MP) to strike an effective balance between accuracy and state space reduction (i.e., compactness). In doing so, they exploit historical data from ITS systems. We demonstrate the performance of the proposed approaches using actual road network data from Southeast Michigan.

A two-sided market mechanism for trading big data computing commodities

The big data trend is generating compute-intensive and data-intensive applications requiring unique services that are different from conventional computing services. Therefore, there is a need to fundamentally address such requirements by developing market mechanisms for managing, trading, and pricing big data computing services. The cloud computing platforms have a great potential to meet the economic requirements of market mechanisms for big data applications due to their technological advances, cost benefit ratios, and easy to use services. We design a two-sided mechanism for trading computing resources for big data applications. Our proposed mechanism is universally strategy-proof, providing incentives for both cloud providers and users to voluntarily reveal their true private information. We perform extensive experiments to evaluate our proposed mechanism.

Optimal Routing for Plug-In Hybrid Electric Vehicles

We propose routing algorithms for plug-in hybrid electric vehicles (PHEVs) that account for the significant energy efficiency differences of vehicle operating modes and recommend the predominant mode of operation for each road segment during route planning. This is to enhance fuel economy and reduce emissions. We introduce the energy-efficient routing problem (EERP) for PHEVs and formulate this problem as a new class of the shortest path problem. The objective of the EERP is to not only find a path to any given destination but also to identify the predominant operating mode for each segment of the path to minimize fuel consumption. EERP can be generalized to a new class of problems in the context of network optimization, where for each arc we need to choose which resources to use to minimize the consumption of one of the resources subject to a constraint on the other resource. In this problem, the resource selection is mutually exclusive, which means we cannot choose both resources together for an arc. We...