Jing Xing

Dawning Nebulae: A PetaFLOPS Supercomputer with a Heterogeneous Structure

Dawning Nebulae is a heterogeneous system composed of 9280 multi-core x86 CPUs and 4640 NVIDIA Fermi GPUs. With a Linpack performance of 1.271 petaFLOPS, it was ranked the second in the TOP500 List released in June 2010. In this paper, key issues in the system design of Dawning Nebulae are introduced. System tuning methodologies aiming at petaFLOPS Linpack result are presented, including algorithmic optimization and communication improvement. The design of its file I/O subsystem, including HVFS and the underlying DCFS3, is also described. Performance evaluations show that the Linpack efficiency of each node reaches 69.89%, and 1024-node aggregate read and write bandwidths exceed 100 GB/s and 70GB/s respectively. The success of Dawning Nebulae has demonstrated the viability of CPU/GPU heterogeneous structure for future designs of supercomputers.

Memory Based Metadata Server for Cluster File Systems

In high performance computing environment, the metadata servers of distributed file system become critical to impact overall system performance. An approach of memory based metadata server is proposed, instead of the disk based approach. We present a metadata management system with matrix organization, non-overhead reliable mechanism and static scalability method, which is design to efficiently utilize large memory and provide high performance. We examine and demonstrate the performance, overhead of reliability and scalability in a test bed environment of 28 machines. The result shows that the performance of our system is higher than other traditional distributed file system, the reliability can be achieved with little overhead and the metadata servers can be linear scaling.

The Extraction and Evaluation of Skeleton in Sensor Networks

In sensor networks community, the skeleton (or medial axis), as an important infrastructure which can correctly capture the topological and geometrical features of the underlying network, has been widely used for facilitating routing, navigation, segmentation, etc. Even though there are a handful of skeleton extraction solutions, the measurement of the goodness of the derived skeleton is often application-oriented, and there is no quantitative metric for this task. In this paper, we study the problem of skeleton extraction and conduct the first work on quantitative evaluation of skeleton in sensor networks. Different from traditional schemes which assume complete or incomplete boundaries, the proposed skeleton extraction algorithm is based on mere connectivity information, without reliance on any boundary information. More specifically, for each node we compute its variability factor based on the neighborhood sizes of the node and its neighbors, which can reflect how central a sensor node is to the network, and a sensor node identifies itself as a skeleton node if its variability factor is locally maximal. Next, we present a light-weight scheme to connect these skeleton nodes. Finally, we proposed a metric, named visibility coefficient, to quantitatively evaluate the derived skeleton.

A General Framework of Skeleton Extraction in Sensor Networks

In sensor networks, skeleton (also known as medial axis) extraction is recognized as an appealing approach to support many applications, such as load-balanced routing and location-free segmentation. Existing solutions in the literature rely heavily on the identified boundaries, posing severe limitations on the applicability of the skeleton extraction algorithm. In this paper, we conduct the first work of a connectivity-based and boundary-free skeleton extraction scheme in sensor networks, and propose a centrality-and-connectivity-based boundary-free algorithm, which is simple, distributed, and scalable, and can correctly identify a few skeleton nodes and connects them into a meaningful representation of the network, without reliance on any constraint on communication radio model or nodal distribution. The key idea of our algorithm is to exploit the necessary (but not sufficient) condition of skeleton points: the intersection area of the disk centered at a skeleton point x should be the largest one as compared with the other points on the chord generated by x, where the chord is referred to as the line segment connecting x and the tangent point in the boundary. To that end, we present the concept of r-centrality of a point, quantitatively measuring how central a point is. Accordingly, a skeleton point should have the largest value of r-centrality, as compared with the other points on the chord generated by this point. We then propose a distributed algorithm to connect the identified skeleton nodes, while obtaining two by-products, i.e., the boundaries and the segmentation result of the network. We also design a light-weight scheme based on the Voronoi diagram, entitled Voronoi-based skeleton extraction algorithm, that yields a skeleton with less communication overhead while sacrificing slightly the skeleton accuracy, providing a tradeoff between skeleton accuracy and communication cost.

WiLocator: WiFi-Sensing Based Real-Time Bus Tracking and Arrival Time Prediction in Urban Environments

Offering the services of real-time tracking and arrival time prediction is a common welfare for bus riders and transit agencies, especially in urban environments. On the down side, the traditional GPS-based solutions work poorly in urban areas due to urban canyons, while the location systems based on cellular signal also suffer from inherent limitations. In this paper, we present a powerful tool named Signal Voronoi Diagram (SVD) to partition the radio-frequency (RF) signal space of WiFi Access Points (APs), distributed where a bus travels, into Signal Cells, and then into fine-grained Signal Tiles, tackling the problem of noisy received signal strength (RSS) readings and possible AP dynamics. On top of SVD, we present a novel framework so-called WiLocator, to track and predict the arrival time of an urban bus based on the surrounding WiFi information collected by the commodity off-the-shelf (COTS) smartphones of bus riders, the mobility constraint of a bus and the temporal consistency of travel time of buses on the overlapped road segments. We also show the WiLocators power of generating an accurate and real-time traffic map with the predicted travel time on each road segment. We implement the prototype of WiLocator and conduct the in-situ experiment to demonstrate its accuracy.

A Novel Skeleton Extraction Algorithm in Sensor Networks

This paper analyzes the problem of location-free skeleton extraction in sensor networks . Different from most of the previous solutions, the proposed algorithm has no dependency on boundary information. Our work is based on the proposed index of a node, named centrality, which can reflect the centeredness of the node. We first identify the node having the maximal centrality as the root skeleton node, based on whether a skeleton arc is obtained. Secondly, each node then computes its hop count distance to the skeleton arc, and the network is decomposed into a set of level sets. The node with the maximal centrality is identified as a skeleton node. Thirdly, these skeleton nodes are connected properly to form a coarse skeleton possibly with spurious branches. Finally, a pruning operation is conducted on the coarse skeleton, and the final skeleton is generated. Extensive simulations show the efficiency of the proposed algorithm.

A Load-Aware Data Placement Policy on Cluster File System

In a large-scale cluster system with many applications running on it, cluster-wide I/O access workload disparity and disk saturation on only some storage servers have been the severe performance bottleneck that deteriorates the system I/O performance. As a result, the system response time will increase and the throughput of the system will decrease drastically. In this paper, we present a load-aware data placement policy that will distribute data across the storage servers based on the load of each server and automatically migrate data from heavily-loaded servers to lightly-loaded servers. This policy is adaptive and self-managing. It operates without any prior knowledge of application access workload characteristics or the capabilities of storage servers. It can make full use of the aggregate disk bandwidth of all storage servers efficiently. Performance evaluation shows that our policy will improve the aggregate I/O bandwidth by 10%-20% compared with random data placement policy especially under mixed workloads.

Optimizing the Datapath for Key-value Middleware with NVMe SSDs over RDMA Interconnects

In-memory key-value store is a crucial building block of large-scale web architecture. Given the growth of the data volume and the need for low-latency responses, cost-effective storage expansion and fast large-message processing are the major challenges. In this paper, we explore the design of key-value middleware that takes advantage of modern NVMe SSDs and RDMA interconnects to achieve high performance without excessive DRAM deployment. We propose an all-in-userland approach to improve the data plane efficiency. Both NVMe and RDMA are interfaced directly from the user-space for effective data access and tailored data management. We present a low-latency storage extension framework based on NVMe and a new design of JVM-aware Memcache protocol based on RDMA. To further accelerate large-message transfer, we provide a hybrid communication protocol fusing Eager and Rendezvous schemas, and a united I/O staging approach to achieve maximum latency hiding through pipelining. As the benchmarking results indicate, with the non-negligible JVM overhead taken into account, our solution obtains comparable communication performance with the RDMA-Memcached released by the OSU. For SSD-involved operations, the latency decreases by up to 31% compared to the kernel-based I/O processing.

ESP: Evaluation-Based Skeleton Pruning in Sensor Networks

Skeleton has been successfully incorporated in the design of high-performance protocol in sensor networks. As there exists a tradeoff between skeleton simplicity and reconstruction error, it is important to quantitatively judge the skeleton and refine it properly. While some studies on skeleton extraction and refinement have been proposed, there still lacks deep understanding of quantitative evaluation and skeleton pruning. In this paper, we consider both the simplicity and reconstruction ability of skeleton, and present ESP, an evaluation-based skeleton pruning algorithm. By analyzing the skeleton properties, we first present the adjusted coverage index of skeleton to measure its goodness, and then provide a novel approach for skeleton pruning. We also analyze that the adjusted coverage index is closely related to the performance of skeleton-based applications, and show by simulations that the ESP can correctly judge the skeletons goodness and yield a refined skeleton.

Write bandwidth optimization of online Erasure Code based cluster file system

As the data volume is growing from big to huge in many science labs and data centers, more and more data owners are willing to choose Erasure Code based storage to reduce the storage cost. However, online Erasure Code based cluster file systems still have not been applied widely because of write bottlenecks in data encoding and data placement. We proposed two optimizations to address them respectively. We propose a Partition Encoding policy to accelerate the encoding arithmetic through SIMD extensions and to overlap data encoding with data committing. We devise Adaptive Placement policy to provide incremental expansion and high availability, as well as good scalability. The experimental results in our prototype ECFS show that the aggregate write bandwidth can be improved by 42%, while keeping the storage in a more balanced state.