Rongfeng Tang

Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems

Most supercomputers nowadays are based on large clusters, which call for sophisticated, scalable, and decentralized metadata processing techniques. From the perspective of maximizing metadata throughput, an ideal metadata distribution policy should automatically balance the namespace locality and even distribution without manual intervention. None of existing metadata distribution schemes is designed to make such a balance. We propose a novel metadata distribution policy, Dynamic Dir-Grain (DDG), which seeks to balance the requirements of keeping namespace locality and even distribution of the load by dynamic partitioning of the namespace into size-adjustable hierarchical units. Extensive simulation and measurement results show that DDG policies with a proper granularity significantly outperform traditional techniques such as the Random policy and the Subtree policy by 40 percent to 62 times. In addition, from the perspective of file system reliability, metadata consistency is an equally important issue. However, it is complicated by dynamic metadata distribution. Metadata consistency of cross-metadata server operations cannot be solved by traditional metadata journaling on each server. While traditional two-phase commit (2PC) algorithm can be used, it is too costly for distributed file systems. We proposed a consistent metadata processing protocol, S2PC-MP, which combines the two-phase commit algorithm with metadata processing to reduce overheads. Our measurement results show that S2PC-MP not only ensures fast recovery, but also greatly reduces fail-free execution overheads.

Improving data availability for a cluster file system through replication

Data availability is a challenging issue for large- scale cluster file systems built upon thousands of individual storage devices. Replication is a well-known solution used to improve data availability. However, how to efficiently guarantee replicas consistency under concurrent conflict mutations remains a challenge. Moreover, how to quickly recover replica consistency from a storage server crash or storage device failure is also a tough problem. In this paper, we present a replication-based data availability mechanism designed for a large-scale cluster file system prototype named LionFS. Unlike other replicated storage systems that serialize replica updates, LionFS introduces a relaxed consistency model to enable concurrent updating all replicas for a mutation operation, greatly reducing the latency of operations. LionFS ensures replica consistency if applications use file locks to synchronize the concurrent conflict mutations. Another novelty of this mechanism is its light-weight log, which only records failed mutations and imposes no overhead on failure-free execution and low overhead when some storage devices are unavailable. Furthermore, recovery of replica consistency needs not stop the file system services and running applications. Performance evaluation shows that our solution achieves 50-70% higher write performance than serial replica updates. The logging overhead is shown to be low, and the recovery time is proportional to the amount of data written during the failure.

IncFS: an integrated high-performance distributed file system based on NFS

Scientific computing applications running in the cluster environment require high performance distributed file system to store and share data. A new approach, the IncFS, of building a high performance distributed file system by integrating many NFS servers is presented in this paper. The IncFS is aimed at providing a simple and convenient way to achieve high aggregate I/O bandwidth for scientific computing applications that require intensive concurrent file access. The IncFS uses a hyper structure to integrate multiple NFS file systems. And it provides multiple data layouts to effectively distribute file data among those NFS servers. Performance evaluations demonstrate that the IncFS has very good data access bandwidth with near perfect scalability, while still maintains an acceptable meta data throughput

An Efficient Metadata Distribution Policy for Cluster File Systems

How to distribute the items in the file system hierarchy across a group of metadata servers is an important issue that determines the holistic metadata processing performance (HMPP) of a cluster file system which manages its metadata by a group of metadata servers. The HMPP is affected by two factors: balance degree of metadata distribution and number of branch points. Two types of well-used metadata distribution policies are the dynamic subtree policy and the random policy. Both of them emphasize one factor and neglect the other factor. As a result, their HMPP is low. In order to make good use of processing capacity of all metadata servers, we present a novel metadata distribution policy, called dynamic dir-grain (DDG) policy, which takes both factors into account. Our performance results show that this policy is potentially more efficient than the other two types of policies under real environments, as well as the conditions of creation or removal of a large hierarchy

SuperNBD: An Efficient Network Storage Software for Cluster

Networked storage has become an increasingly common and essential component for cluster. In this environment, the network storage software through which the client nodes can directly access remote network attached storage is an important and critical requisite. There are many implementations exist with this function, such as iSCSI. However, they are not tailored for the large-scale cluster environment and cannot well satisfy its high efficiency and scalability requirements. In this paper, we present a more efficient technology for network storage in cluster and also give detailed evaluation for it through our implementation – SuperNBD.The results indicate that SuperNBD is more efficient, more scalable, and better fit for cluster environment.

HMF: High-available Message-passing Framework for Cluster File System

In large-scale cluster systems, the failure rate of network connection is non-negligibly high. A cluster file system must have the ability to handle network failures in order to provide high-available data accesses service. Traditionally, network failure handling is only guaranteed by network protocol, or implemented within the file system semantic layer. We present the high-available message-passing framework which is called HMF. Based on the operation hierarchy in cluster file system, HMF guarantees the availability of each pair of network transmissions and their interaction with the file system sub-operations. It separates the network fault-tolerance design from the file system and keeps a simple interface between them. HMF could handle a lot of network failures internally, which greatly simplifies the implementation of file system semantic layer. Performance results show that HMF can increase the availability of message passing and reduce the cost of recovery from network failures. When there are two network channels, HMF also improves aggregate I/O bandwidth by 80% in normal condition while the performance degradation due to recovery is below 10%.

Research on Key Technologies of Load Balancing for NFS Server with Multiple Network Paths

NFS server is designed to run on a single node. Even if NFS server is configured with multiple network interfaces, each client can only access NFS server through one network interface of the server. In this paper, we design and implement the multi-path load distribution mechanism in the SunRPC layer of NFS. This mechanism allows the NFS clients to use the aggregated network bandwidth of the server which has multiple network channels and balances the load among the multiple network channels. The result shows that the extended version of NFS can make good use of multi-path of the server and can tolerate network failure

A New Way to High Performance NFS for Clusters

For its simplicity, reliability and maturity, NFS is widely-used in clusters. However, due to its high overheads and implementation limitations, the standard NFS cannot fully exert the potential abilities provided by multiple network channels and multiple SCSI channels on the server. In this paper, we present a new efficient way to high performance NFS implementation for cluster applications. By adding mechanisms to make good use of NFS servers multiple communication channels and multiple I/O channels, CluserNFS can potentially provide better I/O performance than standard NFS, as illustrated by our simulation experiment results.

A storage space management policy for a cluster file system

The major challenge in designing cluster file systems is to provide high aggregate I/O bandwidth and high metadata processing throughput for applications running on large-scale cluster systems. And with the rapid increase of required data storage, how to improve access performance of a PB-scale cluster file system is also a challenging issue. In this paper, we introduce the storage space management policy for a PB-scale cluster file system. Our performance results showed that compared with Lustre and GFS, DCFS2 is able to provide comparable or even better aggregate I/O bandwidth

Optimized implementation of extendible hashing to support large file system directory

Extendible hashing is a kind of fast indexing technology; it provides with a way of storing structural data records so that each of them can be gotten very quickly. In this paper, we present our own specially optimized implementation of using extendible hashing in cluster file system is order to support large directory.