Xueshan Luo

Theory and Network Applications of Dynamic Bloom Filters

A bloom filter is a simple, space-efficient, randomized data structure for concisely representing a static data set, in order to support approximate membership queries. It has great potential for distributed applications where systems need to share information about what resources they have. The space efficiency is achieved at the cost of a small probability of false positive in membership queries. However, for many applications the space savings and short locating time consistently outweigh this drawback. In this paper, we introduce dynamic bloom filters (DBF) to support concise representation and approximate membership queries of dynamic sets, and study the false positive probability and union algebra operations. We prove that DBF can control the false positive probability at a low level by adjusting the number of standard bloom filters used according to the actual size of current dynamic set. The space complexity is also acceptable if the actual size of dynamic set does not deviate too much from the predefined threshold. Furthermore, we present multidimension dynamic bloom filters (MDDBF) to support concise representation and approximate membership queries of dynamic sets in multiple attribute dimensions, and study the false positive probability and union algebra operations through mathematic analysis and experimentation. We also explore the optimization approach and three network applications of bloom filters, namely bloom joins, informed search, and global index implementation. Our simulation shows that informed search based on bloom filters can obtain higher recall and success rate of query than the blind search protocol.

The Dynamic Bloom Filters

A Bloom filter is an effective, space-efficient data structure for concisely representing a set, and supporting approximate membership queries. Traditionally, the Bloom filter and its variants just focus on how to represent a static set and decrease the false positive probability to a sufficiently low level. By investigating mainstream applications based on the Bloom filter, we reveal that dynamic data sets are more common and important than static sets. However, existing variants of the Bloom filter cannot support dynamic data sets well. To address this issue, we propose dynamic Bloom filters to represent dynamic sets, as well as static sets and design necessary item insertion, membership query, item deletion, and filter union algorithms. The dynamic Bloom filter can control the false positive probability at a low level by expanding its capacity as the set cardinality increases. Through comprehensive mathematical analysis, we show that the dynamic Bloom filter uses less expected memory than the Bloom filter when representing dynamic sets with an upper bound on set cardinality, and also that the dynamic Bloom filter is more stable than the Bloom filter due to infrequent reconstruction when addressing dynamic sets without an upper bound on set cardinality. Moreover, the analysis results hold in stand-alone applications, as well as distributed applications.

Moore: An Extendable Peer-to-Peer Network Based on Incomplete Kautz Digraph with Constant Degree

The topological properties of peer-to-peer overlay networks are critical factors that dominate the performance of these systems. Several non-constant and constant degree interconnection networks have been used as topologies of many peer-to-peer networks. One of these has many desirable properties: the Kautz digraph. Unlike interconnection networks, peer-to-peer networks need a topology with an arbitrary size and degree, but the complete Kautz digraph does not possess these properties. In this paper, we propose Moore: the first effective and practical peer-to-peer network based on the incomplete Kautz digraph with <i>O</i>(log<i> _d </i> <i>N</i>) diameter and constant degree under a dynamic environment. The diameter and average routing path length are [log<i> _d </i>(<i>N</i>) - log<i> _d </i>(1 + 1/<i>d</i>)] and log<i> _d </i> <i>N</i>, respectively, and are shorter than that of CAN, butterfly, and cube-connected-cycle. They are close to that of complete de Bruijn and Kautz digraphs. The message cost of node joining and departing operations are at most 2.5 <i>d</i> log<i> _d </i> <i>N</i> and (2.5 <i>d</i> + 1) log<i> _d </i> <i>N</i>, and only <i>d</i> and 2<i>d</i> nodes need to update their routing tables. Moore can achieve optimal diameter, high performance, good connectivity and low congestion evaluated by formal proofs and simulations.

A Bloom filters based dissemination protocol in wireless sensor networks

There is a growing need for enabling reprogramming a working sensor network in unattended area. We prefer to meet the requirements remotely by disseminating parameters instead of collecting all deployed sensors. Identifying the version differences of parameters on different sensor nodes can significantly reduce the communication overhead, because only those out-of-date ones need to be updated. In this paper, we propose BDP, a Bloom filter based data dissemination protocol for wireless sensor networks. Using Bloom filters as compact storage of the version information of data items, BDP efficiently identifies the version differences among data items with the same key and guarantees network-wide consistency with high reliability. Testbed experiment and simulation results demonstrate that BDP outperforms the existing schemes with low energy cost, short propagation delay of updating new items, and high reliability.

The architecture of SIG computing environment and its application to image processing

Spatial Information Grid (SIG) is a project of applying grid technology to share and integrate spatial data resources, information processing resources, equipment resources, and knowledge resources. SIG computing environment aims to apply the concept of SIG to share hybrid computing resources for processing remote sensing (RS) images. RS image processing is a data-intensive computing problem, and it adapts to be processed according data parallel computing model. In this paper, we discuss the architecture of SIG computing environment, which can provide a powerful computing infrastructure used to process RS image cooperatively. In order to achieve high performance, we propose a model of the image division. From the relation among the processing time, the communication latency, and the transferring ratio, we can achieve some useful conclusions to determine the strategy of the image division. Furthermore, we can discover two optimal division strategies through comparing the experimental results with those useful conclusions.

Localization-Oriented Network Adjustment in Wireless Ad Hoc and Sensor Networks

Localization is an enabling technique for many sensor network applications. Real-world deployments demonstrate that, in practice, a network is not always entirely localizable, leaving a certain number of theoretically nonlocalizable nodes. Previous studies mainly focus on how to tune network settings to make a network localizable. However, the existing methods are considered to be coarse-grained, since they equally deal with localizable and nonlocalizable nodes. Ignoring localizability induces unnecessary adjustments and accompanying costs. In this study, we propose a fine-grained approach, localizability-aided localization (LAL), which basically consists of three phases: node localizability testing, structure analysis, and network adjustment. LAL triggers a single round adjustment, after which some popular localization methods can be successfully carried out. Being aware of node localizability, all network adjustments made by LAL are purposefully selected. Experiment and simulation results show that LAL effectively guides the adjustment while makes it efficient in terms of the number of added edges and affected nodes.

Robust Backup Path Selection in Overlay Routing with Bloom Filters

Routing overlay offers an ideal methodology to improve the end-to-end communication performance by deriving a backup path for any node pair. This paper focuses on a challenging issue of selecting a proper backup path to bypass the failures on the default path with high probability for any node pair. For existing backup path selection approaches, our trace-driven evaluation results demonstrate that the backup and default paths for any node pair overlap with high probability and hence usually fail simultaneously. Consequently, such approaches fail to derive a robust backup path that can take over in the presence of failure on the default path. In this paper, we propose a three-phase RBPS approach to identify a proper and robust backup path. It utilizes the traceroute probing approach to obtain the fine-grained topology information, and systematically employs the grid quorum system and the Bloom filter to reduce the resulting communication overhead. Two criteria, delay and fault-tolerant ability on average, of the backup path are proposed to evaluate the performance of our RBPS approach. Extensive trace-driven evaluations show that the fault-tolerant ability of the backup path can be improved by about 60%, while the delay gain ratio concentrated at 14% after replacing existing approaches with ours. Consequently, our approach can derive a more robust and available backup path for any node pair than existing approaches. This is more important than finding a backup path with the lowest delay compared to the default path for any node pair.

Exploiting Efficient and Scalable Shuffle Transfers in Future Data Center Networks

Distributed computing systems like MapReduce in data centers transfer massive amount of data across successive processing stages. Such shuffle transfers contribute most of the network traffic and make the network bandwidth become a bottleneck. In many commonly used workloads, data flows in such a transfer are highly correlated and aggregated at the receiver side. To lower down the network traffic and efficiently use the available network bandwidth, we propose to push the aggregation computation into the network and parallelize the shuffle and reduce phases. In this paper, we first examine the gain and feasibility of the in-network aggregation with BCube, a novel server-centric networking structure for future data centers. To exploit such a gain, we model the in-network aggregation problem that is NP-hard in BCube. We propose two approximate methods for building the efficient IRS-based incast aggregation tree and SRS-based shuffle aggregation subgraph, solely based on the labels of their members and the data center topology. We further design scalable forwarding schemes based on Bloom filters to implement in-network aggregation over massive concurrent shuffle transfers. Based on a prototype and large-scale simulations, we demonstrate that our approaches can significantly decrease the amount of network traffic and save the data center resources. Our approaches for BCube can be adapted to other servercentric network structures for future data centers after minimal modifications.

A MapReduce-supported network structure for data centers

Several novel data center network structures have been proposed to improve the topological properties of data centers. A common characteristic of these structures is that they are designed for supporting general applications and services. Consequently, these structures do not match well with the specific requirements of some dedicated applications. In this paper, we propose a hyper‐fat‐tree network (HFN): a novel data center structure for MapReduce, a well‐known distributed data processing application. HFN possesses the advanced characteristics of BCube as well as fat‐tree structures and naturally supports MapReduce. We then address several challenging issues that face HFN in supporting MapReduce. Mathematical analysis and comprehensive evaluation show that HFN possesses excellent properties and is indeed a viable structure for MapReduce in practice. Copyright

Localization in non-localizable sensor and ad-hoc networks: A Localizability-aided approach

Localization is an enabling technique for many sensor and ad-hoc network applications. Real-world deployments demonstrate that, in practice, a network is not always entirely localizable, leaving a certain number of theoretically non-localizable nodes. Previous studies mainly focus on how to tune network settings to make a network localizable; however, they are considered to be coarse-grained, since they equally deal with localizable and non-localizable nodes. Ignoring localizability induces unnecessary adjustments and accompanying costs. In this study, we propose a fine-grained approach, Localizability-aided Localization (LAL), which basically consists of three phases: node localizability testing, component tree construction, and network adjustment. LAL triggers a single round adjustment, after which some popular localization methods can be successfully carried out. Being aware of node localizability, all adjustments made by LAL are purposefully selected. Simulation results show that LAL effectively guides the adjustment.