Xinwei Hong

Experiences in Design and Implementation of a High Performance Transport Protocol

This paper describes our experiences in the development of the UDP-based Data Transport (UDT) protocol, an application level transport protocol used in distributed data intensive applications. The new protocol is motivated by the emergence of wide area high-speed optical networks, in which TCP is often found to fail to utilize the abundant bandwidth. UDT demonstrates good efficiency and fairness (including RTT fairness and TCP friendliness) characteristics in high performance computing applications where a small number of bulk sources share the abundant bandwidth. It combines both rate and window control and uses bandwidth estimation to determine the control parameters automatically. This paper presents the rationale behind UDT: how UDT integrates these schemes to support high performance data transfer, why these schemes are used, and what the main issues are in the design and implementation of this high performance transport protocol.

Experimental studies using photonic data services at IGrid 2002

We describe an architecture for remote and distributed data intensive applications that integrates optical path services, network protocol services for high performance data transport, and data services for remote data analysis and distributed data mining. We also present experimental evidence using geoscience data that this architecture scales to long haul, high performance networks.

Data webs for earth science data

We describe high performance data webs for earth science data, which are designed for interactively analyzing small to moderate size remote data sets, as well as mining distributed data sets. Achieving high performance required developing specialized high performance transport services as well as specialized high performance middleware services for merging multiple data streams. Data webs complement data grids, which are grid based infrastructures designed to support arbitrary distributed computation over distributed data using a trusted computing model.

Experimental studies of data transport and data access of earth-science data over networks with high bandwidth delay products

Although the amount of earth-science data is growing rapidly, as is the availability of high-performance networks, our ability to access large remote earth-science data sets is still very limited. This is particularly true of networks with high bandwidth-delay products (BDP), such as those between the US and Europe. Recently, several network protocols have emerged that improve the situation and hold the promise of being much more effective than striped TCP (wherein multiple, parallel connections are used to increase data throughput). In this paper, we report on experimental studies using one of these new protocols called UDT, and compare UDT to other approaches. In addition, we consider the effectiveness of these new protocols when reading and writing data to and from disk over high BDP networks. We also consider the problem of accessing remote data by attribute over these same networks. We show that with the appropriate protocol, the speed of accessing data across the Atlantic can be improved significantly. We note that the UDT protocol used here can be deployed as an application library for earth-science applications and neither requires upgrades to existing network infrastructure, such as routers, nor to the Linux kernels on the servers involved.