A software bridged data transfer on a FPGA cluster by using pipelining and InfiniBand verbs

Abstract

A heterogeneous system with Field Programmable Gate Array (FPGA) is gathering attention in High-Performance Computing (HPC) area. When FPGA is used as an accelerator attached to the host CPU, there can be many configurations such as network topology to construct FPGA cluster. Sustained data transfer bandwidth between FPGA memory and CPU memory on a distant node is one of the most important factors to decide a topology of FPGA cluster. In order to explore the best topology, a quantitative evaluation of bandwidth is required. We conducted bandwidth measurement on two host nodes; both nodes are connected via 100Gbps InfiniBand cable and one host node has PCIe Gen3 x8-based FPGA accelerator card. We implemented a Direct Memory Access (DMA) function on an FPGA-attached node and a software bridged data transfer function to transfer data between two nodes. The result shows that DMA function and software bridged data transfer function achieve 82.2% and 69.6% of the theoretical bandwidth of PCIe Gen3 x8, a bottleneck of data transfer path, respectively.