Bibo Tu

Improving Data Locality of MapReduce by Scheduling in Homogeneous Computing Environments

Data Locality is one of the critical factors to affect performance. This paper proposes a next-k-node scheduling (NKS) method to improve the data locality of map tasks. The method first calculates the probabilities of each map task, and then preferentially schedules the one with the highest probability. It generates low probabilities for the tasks which satisfy node locality with the nodes to issue requests, so it can reserve these tasks to these nodes. We have implemented the NKS method in hadoop-0.20.2. The experiment results have shown that the NKS method reduced 78% of the map tasks processed without node locality, reduced 77%of the network load caused by the tasks, and improved the performance of Hadoop MapReduce when comparing with the default task scheduling method in Hadoop. Obviously, the NKS method is very suitable for the homogeneous environment with network overload.

Accelerating Spatial Data Processing with MapReduce

Map Reduce is a key-value based programming model and an associated implementation for processing large data sets. It has been adopted in various scenarios and seems promising. However, when spatial computation is expressed straightforward by this key-value based model, difficulties arise due to unfit features and performance degradation. In this paper, we present methods as follows: 1) a splitting method for balancing workload, 2) pending file structure and redundant data partition dealing with relation between spatial objects, 3) a strip-based two-direction plane sweeping algorithm for computation accelerating. Based on these methods, ANN(All nearest neighbors) query and astronomical cross-certification are developed. Performance evaluation shows that the Map Reduce-based spatial applications outperform the traditional one on DBMS.

Performance analysis and optimization of MPI collective operations on multi-core clusters

Memory hierarchy on multi-core clusters has twofold characteristics: vertical memory hierarchy and horizontal memory hierarchy. This paper proposes new parallel computation model to unitedly abstract memory hierarchy on multi-core clusters in vertical and horizontal levels. Experimental results show that new model can predict communication costs for message passing on multi-core clusters more accurately than previous models, only incorporated vertical memory hierarchy. The new model provides the theoretical underpinning for the optimal design of MPI collective operations. Aimed at horizontal memory hierarchy, our methodology for optimizing collective operations on multi-core clusters focuses on hierarchical virtual topology and cache-aware intra-node communication, incorporated into existing collective algorithms in MPICH2. As a case study, multi-core aware broadcast algorithm has been implemented and evaluated. The results of performance evaluation show that the above methodology for optimizing collective operations on multi-core clusters is efficient.

Transformer: A New Paradigm for Building Data-Parallel Programming Models

Cloud computing drives the design and development of diverse programming models for massive data processing. the transformer programming framework aims to facilitate the building of diverse data-parallel programming models. transformer has two layers: a common runtime system and a model-specific system. using transformer, the authors show how to implement three programming models: dryad-like data flow, MapReduce, and All-Pairs.

Multi-core aware optimization for MPI collectives

MPI collective operations on multi-core clusters should be multi-core aware. In this paper, collective algorithms with hierarchical virtual topology focus on the performance difference among different communication levels on multi-core clusters, simply for intra-node and inter-node communication; Furthermore, to select befitting segment sizes for intra-node collective communication can cater to cache hierarchy in multi-core processors. Based on existing collective algorithms in MPICH2, above two techniques construct portable optimization methodology over MPICH2 for collective operations on multi-core clusters. Conforming to above optimization methodology, multi-core aware broadcast algorithm has been implemented and evaluated as a case study. The results of performance evaluation show that the multi-core aware optimization methodology over MPICH2 is efficient.

Accurate Analytical Models for Message Passing on Multi-core Clusters

Memory hierarchy on multi-core clusters has two-fold characteristics: vertical memory hierarchy and horizontal memory hierarchy. Vertical memory hierarchy has been modeled by previous work (e.g. memory logP, lognP, log3P etc.) to analyze middleware’s effects on point-to-point communication with different message sizes and message strides; Horizontal memory hierarchy has become more prominent due to distinct performance among three levels of communication in a multi-core cluster: intra-CMP, inter-CMP and inter-node, which should adequately be considered. Derived from lognP and log3P models, new analytical models mlognP and its reduction 2log{2,3}P are proposed to unitedly abstract memory hierarchy on multi-core clusters in vertical and horizontal levels. The results of performance evaluation show that it is indispensable to incorporate horizontal memory hierarchy into new models suitable for multi-core clusters, and 2log{2,3}P model can predict communication costs for message passing on multi-core clusters more accurately than log3P model.

The Failure-rate Aware Scheduling Policies for Large-scale Cluster Systems

With the scale expanding, node failures become one of the important obstacles when using large-scale cluster systems. The traditional scheduling policies of cluster only took into account the factors such as jobs priority and node load with the node failure rate omitted. The function of job scheduling in cluster system can be divided into two sub-processes: job selection process and node allocation process. In this paper, we introduce several scheduling policies considering the node failure rate with which the more dependable nodes are selected during the node allocation process. In the end, we use the discrete event-driven simulation method to evaluate the policies and the simulation results show that the failure-rate aware scheduling policies do better than random node allocation policy for the system performance

The design methodology of Phoenix cluster system software stack

Though many research groups have explored the design methodology of cluster system software stack, few works discuss what constitutes a good one. In this paper, we choose four criteria throughout the lifecycle of cluster system software stack to evaluate its design methodology, including code reusability, evolveability, adaptability and manageability. According to the four criteria, we have proposed a management service-based layered design methodology and built a complete cluster system software stack for both scientific and business computing. Our practices and evaluations show our design methodology has advantages over others in terms of the proposed criteria.

Design Patterns of Scalable Cluster System Software

The design pattern of cluster system software has an important influence on scalability of massive cluster system. The paper presents design patterns of scalable cluster system software, including scalable software topologies and optimized communication modes. These design patterns have been widely applied in Dawning series of supercomputers and some results of performance evaluation show their good scalability

PhoenixG: a unified management framework for industrial information grid

The industrial information grid is a special kind of system, the users of which exclusively own geographically distributed computing resources for business service, and try to maintain the lowest total cost of ownership while guaranteeing quality of service. In this paper, we classify the industrial information grid as an extension to grid problem; develop a unified management framework for new management paradigm, which supports the distribution of administration labor and collaboration of system administrator at different locations; propose a self-organizing algorithm, which supports the initial establishment, daily management and exception processing of industrial information grid. Finally, we evaluate the performance of system management, and analyze the management overhead with this new management paradigm.