Hang Zhuang

HPSO: Prefetching Based Scheduling to Improve Data Locality for MapReduce Clusters

Due to cluster resource competition and task scheduling policy, some map tasks are assigned to nodes without input data, which causes significant data access delay. Data locality is becoming one of the most critical factors to affect performance of MapReduce clusters. As machines in MapReduce clusters have large memory capacities, which are often underutilized, in-memory prefetching input data is an effective way to improve data locality. However, it is still posing serious challenges to cluster designers on what and when to prefetch. To effectively use prefetching, we have built HPSO (High Performance Scheduling Optimizer), a prefetching service based task scheduler to improve data locality for MapReduce jobs. The basic idea is to predict the most appropriate nodes to which future map tasks should be assigned and then preload the input data to memory without any delaying on launching new tasks. To this end, we have implemented HPSO in Hadoop-1.1.2. The experiment results have shown that the method can reduce the map tasks causing remote data delay, and improves the performance of Hadoop clusters.

Scheduling algorithm based on prefetching in MapReduce clusters

HighlightsWe explain in detail the architecture of prefetching module in Section 4.4.We detail the framework of HPSO by example in Section 4.1.We modify the scheduling algorithm based on prefetching to fully exploit the potential map tasks with data locality in Section 4.3.1. This method has the advantages of reducing network transmission. Furthermore, we consider part of nodes, whose remaining time is less then threshold Tunder to avoid invalid data prefetching.We conduct a serial of experiments to evaluate performance of the proposed system using different 5 applications (Section 5).A survey on the state-of-the-art method for improving data locality is conducted in Section 6. Due to cluster resource competition and task scheduling policy, some map tasks are assigned to nodes without input data, which causes significant data access delay. Data locality is becoming one of the most critical factors to affect performance of MapReduce clusters. As machines in MapReduce clusters have large memory capacities, which are often underutilized, in-memory prefetching input data is an effective way to improve data locality. However, it is still posing serious challenges to cluster designers on what and when to prefetch. To effectively use prefetching, we have built HPSO (High Performance Scheduling Optimizer), a prefetching service based task scheduler to improve data locality for MapReduce jobs. The basic idea is to predict the most appropriate nodes for future map tasks based on current pending tasks and then preload the needed data to memory without any delaying on launching new tasks. To this end, we have implemented HPSO in Hadoop-1.1.2. The experiment results have shown that the method can reduce the map tasks causing remote data delay, and improves the performance of Hadoop clusters.

Efficient Distributed Smith-Waterman Algorithm Based on Apache Spark

The Smith-Waterman algorithm, which produces the optimal local alignment between pairwise sequences, is universally used as a key component in bioinformatics fields. It is more sensitive than heuristic approaches, but also more time-consuming. To speed up the algorithm, Single-Instruction Multiple-Data (SIMD) instructions have been used to parallelize the algorithm by leveraging data parallel strategy. However, SIMD-based Smith-Waterman (SW) algorithms show limited scalability. Moreover, the recent next-generation sequencing machines generate sequences at an unprecedented rate, so faster implementations of the sequence alignment algorithms are needed to keep pace. In this paper, we present CloudSW, an efficient distributed Smith-Waterman algorithm which leverages Apache Spark and SIMD instructions to accelerate the algorithm. To facilitate easy integration of distributed Smith-Waterman algorithm into third-party software, we provide application programming interfaces (APIs) service in cloud. The experimental results demonstrate that 1) CloudSW has outstanding performance and achieves up to 3.29 times speedup over DSW and 621 times speedup over SparkSW. 2) CloudSW has excellent scalability and achieves up to 529 giga cell updates per second (GCUPS) in protein database search with 50 nodes in Aliyun Cloud, which is the highest performance that has been reported as far as we know.

DSA: Scalable Distributed Sequence Alignment System Using SIMD Instructions

Sequence alignment algorithms are a basic and critical component of many bioinformatics fields. With rapid development of sequencing technology, the fast growing reference database volumes and longer length of query sequence become new challenges for sequence alignment. However, the algorithms have prohibitively high time and space complexity. In this paper, we present DSA, a scalable distributed sequence alignment system that employs Apache Spark to process sequences data in a horizontally scalable distributed environment, and leverages data parallel strategy based on Single Instruction Multiple Data (SIMD) instruction to parallelize the algorithms in each core of worker node. The experimental results demonstrate that 1) DSA has outstanding performance and achieves up to 201x speedup over SparkSW. 2) DSA has excellent scalability and achieves near linear speedup when increasing the number of nodes in cluster.

Design of a More Scalable Database System

With the development of cloud computing and internet, e-Commerce, e-Business and corporate world revenue are increasing with high rate. These areas require scalable and consistent databases. NoSQL databases such as HBase has been proven to scalability and well performance on cloud computing platforms. However, the inevitable special data with few increment and frequent access leads to hotspot data and unbalanced accessing distribution between data storage servers. Due to their properties, these data often cannot be stored in multiple tables. Some storage nodes become the bottleneck of the distributed storage system, therefore, it becomes difficult to improve the performance by increasing the number of nodes which severely limits the scalability of the storage system. In order to make the performance of the cluster increases with the size of the cluster simultaneously, we devise a new distributed database storage framework to solve those issues mentioned above by changing the storage and read-write mode of the hotspot data. This structure guarantees that the hotspot data will not aggregate in the same storage node, as it guarantees that the data is not too hot in a single storage node. We implement the scalable database based on Apache HBase, which achieve almost double performance of throughput considering heavy read-write pressure situation only with double reading substites. Besides, heavy load node owing to hotspot data will no longer present in the new distributed database.

Wave: Trigger Based Synchronous Data Process System

With the rapid development of cloud computing, more and more applications need to process large amount of data on clusters. Different types of data processing frameworks in cloud have appeared, such as MapReduce, Spark and Percolator. These frameworks are used to a certain type of data processing. In this paper, we provide processing framework called Wave, which is designed for bulk data processing, incremental computing and iterative processing with a uniform application interface. Wave is an event driven data process model for semi-structured data of distributed systems. Programmers use events and trigger reactions to process the data. Wave provides simplified API for users to implements parallel programs on cluster. Programs running in Wave are automatically parallelized and executed on cluster synchronously. Wave uses an implicit mechanism to synchronize the parallel programs execution without any user specification.

Light Weight Key-Value Store for Efficient Services on Local Distributed Mobile Devices

With the development of mobile network and corresponding techniques, more and more works focus on providing efficient services based on mobile devices. Furthermore, motivated by IoT, studies of local distributed mobile devices attract attentions of both industry and academia in recent years. However, existing storage systems cannot manage data and support the QoS of mobile services well. This paper presents LKSM, a light weight key-value storage system, which can be deployed on either one node or multiple nodes. To the best of our knowledge, it is the first attempt to propose key-value store in this scenario. We carefully analyze the challenges when designing the system on mobile cluster, and further propose RDS for addressing. With the help of RDS, LKSM achieves the goal of lower latency, better scalability, and higher availability. We organize LKSM using a log-structured merge-tree, and implement it based on LevelDB, an open source key-value storage system proposed by Google. Experiments on physical smartphones demonstrate that LKSM presents much higher performance compared with the ported LevelDB on mobile devices.

Natural Language Processing Service Based on Stroke-Level Convolutional Networks for Chinese Text Classification

With the development of deep learning and artificial intelligence, more and more research apply neural networks to natural language processing tasks. However, while the majority of these research take English corpus as the dataset, few studies have been done using Chinese corpus. Meanwhile, Existing Chinese processing algorithms typically regard Chinese word or Chinese character as the basic unit but ignore the deeper information into the Chinese character. In Chinese linguistic, strokes are the basic unit of Chinese character who are similar to letters of the English word. Inspired by the recent success of deep learning at character-level, we delve deeper to Chinese stroke level for Chinese language processing and developed it into service for Chinese text classification. In this paper, we dig the basic feature of the strokes considering the similar Chinese character components and propose a new method to leverage Chinese stroke for learning the continuous representation of Chinese character and develop it into a service for Chinese text classification. We develop a dedicated neural architecture based on the convolutional neural network to effectively learn character embedding and apply it to Chinese word similarity judgment and Chinese text classification. Both experiments results show that the stroke level method is effective for Chinese language processing.

A Fast and Better Hybrid Recommender System Based on Spark

With the rapid development of information technology, recommender systems have become critical components to solve information overload. As an important branch, weighted hybrid recommender systems are widely used in electronic commerce sites, social networks and video websites such as Amazon, Facebook and Netflix. In practice, developers typically set a weight for each recommendation algorithm by repeating experiments until obtaining better accuracy. Despite the method could improve accuracy, it overly depends on experience of developers and the improvements are poor. What worse, workload will be heavy if the number of algorithms rises. To further improve performance of recommender systems, we design an optimal hybrid recommender system on Spark. Experimental results show that the system can improve accuracy, reduce execution time and handle large-scale datasets. Accordingly, the hybrid recommender system balances accuracy and execution time.

Enumeration System on HBase for Low-Latency

HBase is a popular distributed Key/Value storage system based on the idea of BigTable. It is being used in many data-centers, such as Facebook and Twitter, for their portability and scalability. For the system, low-latency and large storage is expected when used in industry. However, it is time consuming when retrieving one column via another one. Many technologies were considered to solve the problem. One approach is to add secondary index for HBase such as h index, which achieves high performance in retrieving. Unfortunately, when one column is of limited kinds of data, secondary index cannot reduce storage consumption when accelerating the retrieval. In this paper, we present a novel design of HBase to reduce storage consumption as well as accelerating the retrieval in the above situation. We design an enumeration system for HBase and provide an interface to create enumeration for specific column in tables. Our performance evaluation reveals that it achieves 2.27x improvement in retrieval and 12x reduction in storage compared with non-enumeration in HBase.