Nidhi Tiwari

Classification Framework of MapReduce Scheduling Algorithms

A MapReduce scheduling algorithm plays a critical role in managing large clusters of hardware nodes and meeting multiple quality requirements by controlling the order and distribution of users, jobs, and tasks execution. A comprehensive and structured survey of the scheduling algorithms proposed so far is presented here using a novel multidimensional classification framework. These dimensions are (i) meeting quality requirements, (ii) scheduling entities, and (iii) adapting to dynamic environments; each dimension has its own taxonomy. An empirical evaluation framework for these algorithms is recommended. This survey identifies various open issues and directions for future research.

An Empirical Study of Hadoop's Energy Efficiency on a HPC Cluster☆

Abstract Map-Reduce programming model is commonly used for efficient scientific computations, as it executes tasks in parallel and distributed manner on large data volumes. The HPC infrastructure can effectively increase the parallelism of map-reduce tasks. However such an execution will incur high energy and data transmission costs. Here we empirically study how the energy efficiency of a map-reduce job varies with increase in parallelism and network bandwidth on a HPC cluster. We also investigate the effectiveness of power-aware systems in managing the energy consumption of different types of map-reduce jobs. We comprehend that for some jobs the energy efficiency degrades at high degree of parallelism, and for some it improves at low CPU frequency. Consequently we suggest strategies for configuring the degree of parallelism, network bandwidth and power management features in a HPC cluster for energy efficient execution of map-reduce jobs.

Identification of critical parameters for MapReduce energy efficiency using statistical Design of Experiments

Energy efficiency is an important concern for data centers today. Most of these data centers use MapReduce frameworks for big data processing. These frameworks and modern hardware provide the flexibility in form of parameters to manage the performance and energy consumption of system. However tuning these parameters such that it reduces energy consumption without impacting performance is challenging since - 1) there are a large number of parameters across the layers of frameworks, 2) impact of the parameters differ based on the workload characteristics, 3) the same parameter may have conflicting impacts on performance and energy and 4) parameters may have interaction effects. To streamline the parameter tuning, we present the systematic design of experiments to study the effects of different parameters on performance and energy consumption with a view to identify the most influential ones quickly and efficiently. The final goal is to use the identified parameters to build predictive models for tuning the environment. We perform a detailed analysis of the main and interaction effects of rationally selected parameters on performance and energy consumption for typical MapReduce workloads. Based on a relatively small number of experiments, we ascertain that replication-factor has highest impact and, surprisingly compression has least impact on the energy efficiency of MapReduce systems. Furthermore, from the results of factorial design we infer that the two-way interactions between block-size, Map-slots, and CPU-frequency, parameters of Hadoop platform have a high impact on energy efficiency of all types of workloads due to the distributed, parallel, pipe-lined design.

CPU Frequency Tuning to Improve Energy Efficiency of MapReduce Systems

Energy efficiency is a major concern in todays data centers that house large scale distributed processing systems such as data parallel MapReduce clusters. Modern power aware systems utilize the dynamic voltage and frequency scaling mechanism available in processors to manage the energy consumption. In this paper, we initially characterize the energy efficiency of MapReduce jobs with respect to built-in power governors. Our analysis indicates that while a built-in power governor provides the best energy efficiency for a job that is CPU as well as IO intensive, a common CPU-frequency across the cluster provides best the energy efficiency for other types of jobs. In order to identify this optimal frequency setting, we derive energy and performance models for MapReduce jobs on a HPC cluster and validate these models experimentally on different platforms. We demonstrate how these models can be used to improve energy efficiency of the machine learning MapReduce applications running on the Yarn platform. The execution of jobs at their optimal frequencies improves the energy efficiency by average 25% over the default governor setting. In case of mixed workloads, the energy efficiency improves by up to 10% when we use an optimal CPU-frequency across the cluster.

Improving Energy Efficiency of IO-Intensive MapReduce Jobs

Map-Reduce is a popular data-parallel programming model for varied analysis of huge volumes of data. While a multicore and many CPU HPC infrastructure can be used to improve parallelism of map-reduce tasks, IO-bandwidth limitations may make them ineffective. IO-intensive activities are essential in any MapReduce cluster. In HPC nodes, IO-intensive jobs get queued at the IO-resources while the CPU remain underutilized, resulting in a poor performance, high power consumption and thus, energy inefficiency. In this paper, we investigate which power management setting can be used to improve the energy efficiency of IO-intensive MapReduce jobs by performing a thorough empirical study. Our analysis indicates that a constant CPU frequency can reduce the energy consumption of an IO-intensive job, while improving its performance. Consequently, we build a set of regression models to predict the energy consumption of IO-intensive jobs at a CPU frequency for a given input data volume. We obtained same set of models, with different coefficients, for two different types of IO-intensive jobs, which substantiates the suitability of identified models. These models predict respective outcomes with 80% accuracy for 80% of the new test cases.

Optimizing MapReduce for energy efficiency: Optimizing MapReduce for Energy Efficiency

The efficient use of energy is essential to address concerns of cost and sustainability. Many data centers contain MapReduce clusters to process Big Data applications. A large number of machines and fault tolerance capabilities make MapReduce clusters energy inefficient. In this paper, we present a Configurator based on performance and energy models to improve the energy efficiency of MapReduce systems. Our solution is novel as it takes into account the dependence of the performance and energy consumption of a cluster on MapReduce parameters. While this dependence is known, we are the first to model it and design a Configurator to optimize these parameter settings for maximizing the energy efficiency of MapReduce systems. Our empirical evaluations show that the Configurator can result in up to 50% improvement in the energy efficiency of typical MapReduce applications in two architecturally different clusters.

Towards end-to-end multi-dimensional quality evaluation of business processes

Business Process Models (BPMs) are commonly used in complex business operations. These BPMs are used not only for functional definition of processes but also for their conformance and performance quality evaluations. However, different business domains involve varied operations having multiple stakeholders with different quality requirements. A holistic Quality Evaluation Framework (QEF) is presented to capture and validate these varied quality requirements of such processes using their respective BPMs. The flexible framework consists of a context specific quality model, generic methods and elements to capture quality characteristics in form of profiles in BPMs. The implementation approach of QEF includes use of quality analysis for process quality validation. The case study of a claims settlement process demonstrates the benefits of using proposed QEF.