Mingming Sun

HPSO: Prefetching Based Scheduling to Improve Data Locality for MapReduce Clusters

Due to cluster resource competition and task scheduling policy, some map tasks are assigned to nodes without input data, which causes significant data access delay. Data locality is becoming one of the most critical factors to affect performance of MapReduce clusters. As machines in MapReduce clusters have large memory capacities, which are often underutilized, in-memory prefetching input data is an effective way to improve data locality. However, it is still posing serious challenges to cluster designers on what and when to prefetch. To effectively use prefetching, we have built HPSO (High Performance Scheduling Optimizer), a prefetching service based task scheduler to improve data locality for MapReduce jobs. The basic idea is to predict the most appropriate nodes to which future map tasks should be assigned and then preload the input data to memory without any delaying on launching new tasks. To this end, we have implemented HPSO in Hadoop-1.1.2. The experiment results have shown that the method can reduce the map tasks causing remote data delay, and improves the performance of Hadoop clusters.

Scheduling algorithm based on prefetching in MapReduce clusters

HighlightsWe explain in detail the architecture of prefetching module in Section 4.4.We detail the framework of HPSO by example in Section 4.1.We modify the scheduling algorithm based on prefetching to fully exploit the potential map tasks with data locality in Section 4.3.1. This method has the advantages of reducing network transmission. Furthermore, we consider part of nodes, whose remaining time is less then threshold Tunder to avoid invalid data prefetching.We conduct a serial of experiments to evaluate performance of the proposed system using different 5 applications (Section 5).A survey on the state-of-the-art method for improving data locality is conducted in Section 6. Due to cluster resource competition and task scheduling policy, some map tasks are assigned to nodes without input data, which causes significant data access delay. Data locality is becoming one of the most critical factors to affect performance of MapReduce clusters. As machines in MapReduce clusters have large memory capacities, which are often underutilized, in-memory prefetching input data is an effective way to improve data locality. However, it is still posing serious challenges to cluster designers on what and when to prefetch. To effectively use prefetching, we have built HPSO (High Performance Scheduling Optimizer), a prefetching service based task scheduler to improve data locality for MapReduce jobs. The basic idea is to predict the most appropriate nodes for future map tasks based on current pending tasks and then preload the needed data to memory without any delaying on launching new tasks. To this end, we have implemented HPSO in Hadoop-1.1.2. The experiment results have shown that the method can reduce the map tasks causing remote data delay, and improves the performance of Hadoop clusters.

Bwasw-Cloud: Efficient sequence alignment algorithm for two big data with MapReduce

The recent next-generation sequencing machines generate sequences at an unprecedented rate, and a sequence is not short any more called read. The reference sequences which are aligned reads against are also increasingly large. Efficiently mapping large number of long sequences with big reference sequences poses a new challenge to sequence alignment. Sequence alignment algorithms become to match on two big data. To address the above problem, we propose a new parallel sequence alignment algorithm called Bwasw-Cloud, optimized for aligning long reads against a large sequence data (e.g. the human genome). It is modeled after the widely used BWA-SW algorithm and uses the open-source Hadoop implementation of MapReduce. The results show that Bwasw-Cloud can effectively and quickly match two big data in common cluster.

Design of a More Scalable Database System

With the development of cloud computing and internet, e-Commerce, e-Business and corporate world revenue are increasing with high rate. These areas require scalable and consistent databases. NoSQL databases such as HBase has been proven to scalability and well performance on cloud computing platforms. However, the inevitable special data with few increment and frequent access leads to hotspot data and unbalanced accessing distribution between data storage servers. Due to their properties, these data often cannot be stored in multiple tables. Some storage nodes become the bottleneck of the distributed storage system, therefore, it becomes difficult to improve the performance by increasing the number of nodes which severely limits the scalability of the storage system. In order to make the performance of the cluster increases with the size of the cluster simultaneously, we devise a new distributed database storage framework to solve those issues mentioned above by changing the storage and read-write mode of the hotspot data. This structure guarantees that the hotspot data will not aggregate in the same storage node, as it guarantees that the data is not too hot in a single storage node. We implement the scalable database based on Apache HBase, which achieve almost double performance of throughput considering heavy read-write pressure situation only with double reading substites. Besides, heavy load node owing to hotspot data will no longer present in the new distributed database.

Wave: Trigger Based Synchronous Data Process System

With the rapid development of cloud computing, more and more applications need to process large amount of data on clusters. Different types of data processing frameworks in cloud have appeared, such as MapReduce, Spark and Percolator. These frameworks are used to a certain type of data processing. In this paper, we provide processing framework called Wave, which is designed for bulk data processing, incremental computing and iterative processing with a uniform application interface. Wave is an event driven data process model for semi-structured data of distributed systems. Programmers use events and trigger reactions to process the data. Wave provides simplified API for users to implements parallel programs on cluster. Programs running in Wave are automatically parallelized and executed on cluster synchronously. Wave uses an implicit mechanism to synchronize the parallel programs execution without any user specification.

HDFS+: Concurrent Writes Improvements for HDFS

HDFS is a popular distributed file system which provides high scalability and throughput. It lacks built-in support for multi-source data generating, which arise naturally in many applications including log mining, data analysis etc. There needs a data collection step before analysis in basic HDFS environment because of many data are in local disk, such as log. We proposed a solution which can compose many existent files to a single file and it is suitable for concurrent writes by many data producers. Programs only have to implements data processing against one single file without a data collection step when data analysis. We implemented HDFS+ by modifying existent HDFS, and evaluated with applications including log analysis. Our results show great throughput improvements in data concurrent writes. HDFS+ vastly simplifies the data collecting steps in data analysis procedure.

Enumeration System on HBase for Low-Latency

HBase is a popular distributed Key/Value storage system based on the idea of BigTable. It is being used in many data-centers, such as Facebook and Twitter, for their portability and scalability. For the system, low-latency and large storage is expected when used in industry. However, it is time consuming when retrieving one column via another one. Many technologies were considered to solve the problem. One approach is to add secondary index for HBase such as h index, which achieves high performance in retrieving. Unfortunately, when one column is of limited kinds of data, secondary index cannot reduce storage consumption when accelerating the retrieval. In this paper, we present a novel design of HBase to reduce storage consumption as well as accelerating the retrieval in the above situation. We design an enumeration system for HBase and provide an interface to create enumeration for specific column in tables. Our performance evaluation reveals that it achieves 2.27x improvement in retrieval and 12x reduction in storage compared with non-enumeration in HBase.

DLBer: A Dynamic Load Balancing Algorithm for the Event-Driven Clusters

The event-driven programming model has been proposed to efficiently process iterative applications and incremental applications. In clusters based the event-driven model, applications are structured as a series of triggers, each of which will be invoked when associate events are trigged. And framework assigns a newly submitted trigger to a node where the relevant datasets set. Unfortunately it may lead to load imbalance because associate events occur by chance. Numerous triggers in a node may be simultaneously invoked but other nodes have no triggers running. Jobs composed of short, sub-second triggers present a difficult balancing challenge. To the end, we design DLBer, a new dynamic load balancing algorithm for the event-driven clusters to maximize improve the utilization of node resources.

DLBS: Decentralized load balancing scheme for event-driven cloud frameworks

With the development of cloud computing, more and more applications are moving to a distributed fashion to solve problems. These applications usually contain complex iterative or incremental procedures and have a more urgent requirement on low-latency. Thus many event-driven cloud frameworks are proposed. To optimize this kind of frameworks, an efficient strategy to minimize the execution time by redistributing work- loads is needed. Nowadays, load balance is a critical issue for the efficient operation of cloud platforms and many centralized schemes have already been proposed. However, few of them have been designed to support event-driven frameworks. Besides, as the cluster size and volume of tasks increases, centralized scheme will lead to a bottleneck of master node. In this paper, we demonstrate a decentralized load balancing scheme named DLBS for event-driven cloud frameworks and present two technologies to optimize it. In our design, schedulers are placed in every node for independently load-monitoring, autonomous decision-making and parallel task-scheduling. With the help of DLBS, master frees from the burden and tasks are executed with lower latency. We analyze the excellence of DLBS theoretically and proof it through simulation. At last, we implement and deploy it on a 64-machine cluster and demonstrate that it performs within 20% of an ideal scheme, which are consistent with simulation results.