Hiroya Matsuba

Improving Parallel Write by Node-Level Request Scheduling

In a cluster of multiple processors or cpu-cores, many processes may run on each compute node. Each process tends to issue contiguous I/O requests for snapshot, checkpointing or so, however, if large number of processes enter the I/O phase at the same time, the requests from the same process may be interrupted by the requests of other processes. Then, the I/O nodes receive these requests as non-contiguous way. This interleaved access pattern causes performance degradation in parallel file systems. In order to overcome the problem, we have designed the Gather-Arrange-Scatter (GAS) I/O architecture, for optimizing the parallel write performance. The GAS is an architecture for capturing write operations, buffering them in the memory, and scheduling them to reduce I/O cost at I/O nodes. The scheduling is done per compute node, and the requests are sent to the remote disks in parallel. In this paper, after introducing the GAS architecture in detail, its efficiency and scalability are evaluated using the NAS Parallel Benchmark BTIO. GAS is 5.2%faster than ROMIO collective I/O on PVFS2 in BTIO with 16 nodes/64 processes, and 34.9% faster than MPI noncollective I/O in the same configuration.

Single IP Address Cluster for Internet Servers

Operating a cluster on a single IP address is required when the cluster is used to provide certain Internet services. This paper proposes SAPS, a new method to assign a single IP address to a cluster. The TCP/IP protocol is handled at a single node called the I/O server. The other nodes, called application nodes, provide the socket interface to applications. The I/O server and applications nodes are connected using a cluster-dedicated network, such as the Myrinet network. The key benefit of the proposed method is that the TCP/IP protocol does not care about congestion and packet loss in the cluster, which often happens if multiple nodes send packets to the bottleneck router. Instead, the cluster-dedicated network manages the packet congestion more efficiently than the TCP/IP protocol. The result of the bandwidth benchmark shows SAPS fully utilizes the bandwidth of the Gigabit Ethernet. The result of the SPEC Web benchmark shows SAPS handles 7.9% more requests than the existing method.

TCP Connection Scheduler in Single IP Address Cluster

A broadcast-based single IP cluster aims at being both scalable and available. However, existing systems can only employ static traffic assignment based on incoming packets. In this paper we propose FTCS, a new TCP connection dispatching mechanism that enables a single IP cluster to use more flexible load balancing algorithms. In this mechanism, one of the cluster nodes acts as a master node. A centralized connection scheduler runs on the master node in order to dispatch TCP connections to nodes of the clusters. Since connections are scheduled by a single scheduler, the master node is able to employ arbitrary scheduling algorithms. Once a TCP connection is established on a node, succeeding communication is handled without involving the master node. When the master node fails, one of the nodes takes over the role of the master node. Therefore the master node does not become a single point of failure. Benchmark results using SPECweb2005 Support benchmark show that a four-node Linux cluster using FTCS balances workloads well and successfully handles 13% more requests than the existing method, on average.

Gather-arrange-scatter: Node-level request reordering for parallel file systems on multi-core clusters

Multiple processors or multi-core CPUs are now in common, and the number of processes running concurrently is increasing in a cluster. Each process issues contiguous I/O requests individually, but they can be interrupted by the requests of other processes if all the processes enter the I/O phase together. Then, I/O nodes handle these requests as non-contiguous. This increases the disk seek time, and causes performance degradation. To overcome this problem, a node-level request reordering architecture, called gather-arrange-scatter (GAS) architecture, is proposed. In GAS, the I/O requests in the same node are gathered and buffered locally. Then, those are arranged and combined to reduce the I/O cost at I/O nodes, and finally they are scattered to the remote I/O nodes in parallel. A prototype is implemented and evaluated using the BTIO benchmark. This system reduces up to 84.3% of the lseekO calls and reduces up to 93.6% of the number of requests at I/O nodes. This results in up to a 12.7% performance improvement compared to the non-arranged case.

On-demand file staging system for Linux clusters

An on-demand file staging system, Catwalk, is proposed. Catwalk is designed so that it can run on any Linux clusters without any special or additional hardware. By having hook functions on the system calls of file operations, a file staging system can be transparent from the view of users, and users can be free from having wrong file staging scripts. In Catwalk, the file copying is done via normal TCP protocol so that Catwalk can run over ordinary, widely-used Ethernet. The stage-in file copy is pipelined to maximize the bandwidth from single file server. The performance of Catwalk is evaluated and compared with NFS using synthetic but realistic workloads. The evaluations show the stage-in performance with the pipeline technique is much better than the performance of NFS. The stage-out performance is comparable with the NFS performance despite the extra copying of files, and the file server is lightly loaded with the Catwalk stage-out while NFS entails much heavier server loads. The biggest problems of NFS are its centralized design and lack of scheduling for the parallel workloads. The performance of Catwalk shows that remote file access performance can be improved much better if file accesses are scheduled in a proper way. Thus the proposed file staging system can be a strong complement to NFS, especially for small clusters often having no dedicated parallel file system.

Network performance model for TCP/IP based cluster computing

A new communication model, called the PlogPT model, is proposed to predict communication performance in a commodity cluster where computing nodes communicate using TCP/IP. This model extends the PlogP model in order to consider the variation of bandwidth brought about by bottleneck links in network switches and the delay of packet retransmission by TCP/IP handling. Network switches are modeled as binary tree connections. To demonstrate the PlogPT modelpsilas modeling capability, the execution time of two all-to-all communication algorithms are estimated and compared with the actual execution time and that of the PlogP model. The PlogPT model predicts the execution time of those two algorithms more precisely than the existing model.

The Design of Seamless MPI Computing Environment for Commodity-Based Clusters

This paper describes the design and implementation of a seamless MPI runtime environment, called MPI-Adapter, that realizes MPI program binary portability in different MPI runtime environments. MPI-Adapter enables an MPI binary program to run on different MPI implementations. It is implemented as a dynamic loadable module so that the module dynamically captures all MPI function calls and invokes functions defined in a different MPI implementation using the data type translation techniques. A prototype system was implemented for Linux PC clusters to evaluate the effectiveness of MPI-Adapter. The results of an evaluation on a Xeon Processor (3.8GHz) based cluster show that the MPI translation overhead of MPI sending (receiving) is around 0.028μs , and the performance degradation of MPI-Adapter is negligibly small on the NAS parallel benchmark IS.

Fault Detection System Activated by Failure Information

Intrusion tolerance has been proposed as a new paradigm for computer systems security. The idea is to apply the fault tolerance paradigm in the domain of systems security accepting that malicious faults (attacks, intrusions) can never be entirely prevented, and that highly resilient systems have to tolerate these faults. Research in this area has produced a set of clever intrusion-tolerant protocols and systems (I/T protocols and I/T systems for short). However, we believe that an issue has been overlooked: that servers have, finite memory, so the number of messages that can be stored in their buffers is limited. Intuitively, this can be a problem in systems in which there are many messages being exchanged. Moreover, all of these systems assume that the environment is essentially asynchronous, i.e., that there are no bounds on communication and processing delays. Assuming this kind of model is very important in order to prevent the success of attacks against time.We propose a fault detection system activated by an application when the application recognizes the occurrence of a failure, in order to realize self managing systems that automatically find the source of a failure. In existing detection systems, there are three issues for constructing self managing applications: i) the detection results are not sent to the applications, ii) they can not identify the source failure from all of the detected failures, and iii) configuring the detection system for networked system is hard work. For overcoming these issues, the proposed system takes three approaches: i) the system receives failure information from an application and returns a result set to the application, ii) the system identifies the source failure using relationships among errors, and Hi) the system obtains information of the monitored system from a database. The relationship is expressed by a tree. This tree is called error relationship tree. The database provides information which are system entities such as hardware devices, software object, and network topology. When the proposed system starts looking for the source of a failure, causal relations from an error relation tree are referred to, and the correspondence of error definitions and actual objects is derived using the database. We show the design of the detection operation activated by the failure information and the architecture of the proposed system.