Hung Chang Hsiao

Load Rebalancing for Distributed File Systems in Clouds

Distributed file systems are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions; a file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance in a distributed file system; that is, the file chunks are not distributed as uniformly as possible among the nodes. Emerging distributed file systems in production systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, a fully distributed load rebalancing algorithm is presented to cope with the load imbalance problem. Our algorithm is compared against a centralized approach in a production system and a competing distributed solution presented in the literature. The simulation results indicate that our proposal is comparable with the existing centralized approach and considerably outperforms the prior distributed algorithm in terms of load imbalance factor, movement cost, and algorithmic overhead. The performance of our proposal implemented in the Hadoop distributed file system is further investigated in a cluster environment.

A novel code assignment scheme for W-CDMA systems

The third-generation wideband CDMA (W-CDMA) systems support the relative higher and variable bit rate transmissions for applications with various quality-of-service (QoS) requirements. Code assignment and reassignment schemes in W-CDMA are essential, with the aim of boosting utilization of codes. A dynamic code assignment scheme (DCA) is proposed for traffic with various QoS classes introduced by Universal Mobile Telecommunication System (UMTS). From the simulation results, the proposed DCA can reduce the new call blocking probability and thus improve the system utilization for W-CDMA.

Load Balance with Imperfect Information in Structured Peer-to-Peer Systems

With the notion of virtual servers, peers participating in a heterogeneous, structured peer-to-peer (P2P) network may host different numbers of virtual servers, and by migrating virtual servers, peers can balance their loads proportional to their capacities. The existing and decentralized load balance algorithms designed for the heterogeneous, structured P2P networks either explicitly construct auxiliary networks to manipulate global information or implicitly demand the P2P substrates organized in a hierarchical fashion. Without relying on any auxiliary networks and independent of the geometry of the P2P substrates, we present, in this paper, a novel load balancing algorithm that is unique in that each participating peer is based on the partial knowledge of the system to estimate the probability distributions of the capacities of peers and the loads of virtual servers, resulting in imperfect knowledge of the system state. With the imperfect system state, peers can compute their expected loads and reallocate their loads in parallel. Through extensive simulations, we compare our proposal to prior load balancing algorithms.

Resolving the Topology Mismatch Problem in Unstructured Peer-to-Peer Networks

Prior studies show that more than 70 percent of communication paths in a popular unstructured peer-to-peer (P2P) system (i.e., Gnutella) do not exploit the physical network topology, leading to the topology mismatch problem, and thus, lengthen communication between participating peers. While previous efforts in solving overlay topology matching problems do not guarantee the bounds of performance metrics (e.g., the communication delay between any two overlay peers and the broadcasting scope of any participating peer), in this paper, we present a novel topology matching algorithm that has provable performance qualities. In our proposal, each participating node creates and manages a constant number of overlay connections to other peers in a distributed manner. In rigorous performance analysis, we show that 1) the expected overlay communication delay between any two nodes in our P2P network is a constant; 2) in addition, any joining node has the exponential broadcasting scope in expectation; 3) furthermore, a participating node takes a polylogarithmic overhead to exploit the physical network locality and maintain its flooding scope. Together with extensive simulations, we present our proposal that significantly outperforms two recent solutions, i.e., THANCS and mOverlay, in terms of overlay communication latency and/or broadcasting scope.

Similarity discovery in structured P2P overlays

Peer-to-peer (P2P) overlays are appealing, since they can aggregate resources of end systems without relying on sophisticated infrastructures. Services can thus be rapidly deployed over such overlays. Primitive P2P overlays only support searches with single keywords. For queries with multiple keywords, presently only unstructured P2P systems can support by extensively employing message flooding. We propose a similarity information retrieval system called Meteorograph for structured P2P overlays without relying on message flooding. Meteorograph is fault-resilient, scalable, responsive and self-administrative, which is particularly suitable for an environment with an explosion of information and a large number of dynamic entities. An information item stored in Meteorograph is represented as a vector. A small angle between two vectors means that the corresponding items are characterized by some identical keywords. Meteorograph further stores similar items at nearby locations in the P2P overlay. To retrieve similar items, only nodes in nearby locations are located and consulted. Meteorograph is evaluated with simulation. The results show that Meteorograph can effectively distribute loads to the nodes. Discovering a single item and a set (in size k) of similar items takes O(log N) and (k/c)middotO(log N) messages and hops respectively, where N is the number of nodes in the overlay and c is the storage capacity of anode

A Dynamic Data Placement Strategy for Hadoop in Heterogeneous Environments

Abstract Cloud computing is a type of parallel distributed computing system that has become a frequently used computer application. MapReduce is an effective programming model used in cloud computing and large-scale data-parallel applications. Hadoop is an open-source implementation of the MapReduce model, and is usually used for data-intensive applications such as data mining and web indexing. The current Hadoop implementation assumes that every node in a cluster has the same computing capacity and that the tasks are data-local, which may increase extra overhead and reduce MapReduce performance. This paper proposes a data placement algorithm to resolve the unbalanced node workload problem. The proposed method can dynamically adapt and balance data stored in each node based on the computing capacity of each node in a heterogeneous Hadoop cluster. The proposed method can reduce data transfer time to achieve improved Hadoop performance. The experimental results show that the dynamic data placement policy can decrease the time of execution and improve Hadoop performance in a heterogeneous cluster.

Bristle: a mobile structured peer-to-peer architecture

Most hash-based structured peer-to-peer (HS-P2P) architectures proposed previously focus on stationary peer nodes. When peers move to new locations in the network, these schemes become cumbersome and inefficient in delivering messages. This paper proposes a mobile HS-P2P architecture, called Bristle, which allows nodes to move freely. A node can disseminate its location information through the P2P network, and other nodes can learn its movement or query its location also through the P2P network. Consequently, an overlay infrastructure based on Bristle can be efficiently maintained over mobile nodes. The scalability, reliability and performance of the HS-P2P are thus leveraged. We believe that the concept proposed in Bristle can be applied to existing HS-P2P overlays.

A Symmetric Load Balancing Algorithm with Performance Guarantees for Distributed Hash Tables

Peers participating in a distributed hash table (DHT) may host different numbers of virtual servers and are enabled to balance their loads in the reallocation of virtual servers. Most decentralized load balance algorithms designed for DHTs based on virtual servers require the participating peers to be asymmetric, where some serve as the rendezvous nodes to pair virtual servers and participating peers, thereby introducing another load imbalance problem. While state-of-the-art studies intend to present symmetric load balancing algorithms, they introduce significant algorithmic overheads and guarantee no rigorous performance metrics. In this paper, a novel symmetric load balancing algorithm for DHTs is presented by having the participating peers approximate the system state with histograms and cooperatively implement a global index. Each peer independently reallocates in our proposal its locally hosted virtual servers by publishing and inquiring the global index based on their histograms. Unlike competitive algorithms, our proposal exhibits analytical performance guarantees in terms of the load balance factor and the algorithmic convergence rate, and introduces no load imbalance problem due to the algorithmic workload. Through computer simulations, we show that our proposal clearly outperforms existing distributed algorithms in terms of load balance factor with a comparable movement cost.

On Optimizing Overlay Topologies for Search in Unstructured Peer-to-Peer Networks

Unstructured peer-to-peer (P2P) file-sharing networks are popular in the mass market. As the peers participating in unstructured networks interconnect randomly, they rely on flooding query messages to discover objects of interest and thus introduce remarkable network traffic. Empirical measurement studies indicate that the peers in P2P networks have similar preferences, and have recently proposed unstructured P2P networks that organize participating peers by exploiting their similarity. The resultant networks may not perform searches efficiently and effectively because existing overlay topology construction algorithms often create unstructured P2P networks without performance guarantees. Thus, we propose a novel overlay formation algorithm for unstructured P2P networks. Based on the file sharing pattern exhibiting the power-law property, our proposal is unique in that it poses rigorous performance guarantees. Theoretical performance results conclude that in a constant probability, 1) searching an object in our proposed network efficiently takes O(lnc N) hops (where c is a small constant), and 2) the search progressively and effectively exploits the similarity of peers. In addition, the success ratio of discovering an object approximates 100 percent. We validate our theoretical analysis and compare our proposal to competing algorithms in simulations. Based on the simulation results, our proposal clearly outperforms the competing algorithms in terms of 1) the hop count of routing a query message, 2) the successful ratio of resolving a query, 3) the number of messages required for resolving a query, and 4) the message overhead for maintaining and formatting the overlay.

Mobility churn in DHTs

Most peer-to-peer (P2P) overlays based on distributed hash tables (DHTs) focus on stationary Internet hosts. However, when nodes in the last-mile wireless extension are allowed to join the overlay, we will face immediately the problem of peer mobility. That is, when a peer moves to a new location in the network, all the state information regarding the moving peer will become stale, resulting in creating mobility churn to the system in addition to ordinary churn due to peers joining, departure and failure. The paper extensively investigates the performance of a DHT that is operated under mobility churn. The DHT relies on rudimentary failure detection/recovery mechanisms and peer dynamic joining/departure algorithms to handle peer mobility. We show that when compared with an ideal design, rudimentary failure handling mechanisms and peer dynamics processing algorithms often take unhelpful maintenance bandwidth. The rudimentary implementation has fair performance results when ordinary churn rate and/or route request rate is tremendously high.