Xukang Lu

Automation and management of scientific workflows in distributed network environments

Large-scale computation-intensive applications in various science fields feature complex DAG-structured workflows comprised of distributed computing modules with intricate inter-module dependencies. Supporting such workflows in heterogeneous network environments and optimizing their end-to-end performance are crucial to the success of large-scale collaborative scientific applications. We design and develop a generic Scientific Workflow Automation and Management Platform (SWAMP), which contains a set of easy-to-use computing and networking toolkits for application scientists to conveniently assemble, execute, monitor, and control complex computing workflows in distributed network environments. The current version of SWAMP integrates the graphical user interface of Kepler to compose abstract workflows and employs Condor DAGMan for workflow dispatch and execution. SWAMP provides a web-based user interface to automate and manage workflow executions and uses a special workflow mapper to optimize the end-to-end workflow performance. A case study of the workflow for Spallation Neutron Source datasets in real networks is presented to show the efficacy of the proposed platform.

A Distributed Workflow Management System with Case Study of Real-life Scientific Applications on Grids

Next-generation scientific applications feature complex workflows comprised of many computing modules with intricate inter-module dependencies. Supporting such scientific workflows in wide-area networks especially Grids and optimizing their performance are crucial to the success of collaborative scientific discovery. We develop a Scientific Workflow Automation and Management Platform (SWAMP), which enables scientists to conveniently assemble, execute, monitor, control, and steer computing workflows in distributed environments via a unified web-based user interface. The SWAMP architecture is built entirely on a seamless composition of web services: the functionalities of its own are provided and its interactions with other tools or systems are enabled through web services for easy access over standard Internet protocols while being independent of different platforms and programming languages. SWAMP also incorporates a class of efficient workflow mapping schemes to achieve optimal end-to-end performance based on rigorous performance modeling and algorithm design. The performance superiority of SWAMP over existing workflow mapping schemes is justified by extensive simulations, and the system efficacy is illustrated by large-scale experiments on real-life scientific workflows for climate modeling through effective system implementation, deployment, and testing on the Open Science Grid.

On Tree Construction of Super Peers for Hybrid P2P Live Media Streaming

This paper considers a hybrid hierarchical P2P overlay network structure that consists of both super and normal peers. The media streaming architecture is built upon a tree-structured network of super peers and the tree construction process has a significant impact on the overall system performance. We build network cost models and formulate a specific type of problem to maximize the minimum node throughput in Tree Construction (max-minTC), which aims at optimizing the systems stream rate by constructing an efficient spanning tree among super peers. We consider two scenarios: (i) When the overlay network has an arbitrary topology, we prove max-minTC to be NP-complete by reducing from the Degree Constrained Spanning Tree problem and propose an efficient heuristic algorithm. The performance superiority of the proposed algorithm is justified by experimental results collected by a live media streaming system deployed in real networks and is also illustrated by extensive simulations performed on a large set of simulated networks of various sizes from small to large scales in comparison with other methods, (ii) When the topology of the overlay network is complete, we rigorously prove that the same heuristic algorithm yields an optimal solution.

Network-aware data movement advisor

Next-generation eScience applications often generate large amounts of simulation or experimental data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate the massive data transport necessary for such data sharing and collaboration. However, these technologies and services have not been fully utilized by application users mainly because their use typically requires significant domain knowledge and in many cases even their existence is not made aware to the public. We design and develop a Network-aware Data Movement Advisor (NADMA) utility to enable automated discovery of network and system resources and advise the user of efficient strategies for fast and successful data transfer. NADMA is primarily a client-end program that interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP, and network resource provisioning systems such as TeraPaths and OSCARS. NADMA acts as a route planner in a typical vehicle navigation system to provide the user a set of feasible route options along with performance estimations as well as specific steps and commands to authorize and execute data transfer. We demonstrate the efficacy of NADMA in several use cases based on its implementation and deployment in wide-area networks.

On transport methods for peak utilization of dedicated connections

Several research and production networks now provide multiple Gbps dedicated connections to meet the demands of large data transfers over wide-area networks. Application throughputs, however, were not able to match these rates because the traditional transport methods have not been optimized for such connections. We propose a transport method based on stochastic approximation methods that: (a) stabilizes the source rate for peak utilization of connection bandwidth, and (b) adapts the acknowledgment interval to maximize the goodput at the receiver. We show the asymptotic stability and convergence of this method in maximizing the throughput over dedicated connections under fairly general conditions. Extensive experimental results indicate the effectiveness of this transport method in achieving file transfer throughputs that closely match iperf bandwidth measurements on dedicated connections of several thousand miles over UltraScience Net and ESnet, and also illustrate its superior performance on a local dedicated connection in comparison with existing methods.

On topology construction in layered P2P live streaming networks

Peer-to-peer (P2P) overlay networks provide a highly effective and scalable solution to live media streaming systems that require the collective use of massively distributed network resources. A P2P media streaming architecture is typically built completely or partially upon a tree-structured network topology and the process of tree construction has a significant impact on the overall system performance. We build network cost models and formulate a specific type of topology construction problem, Maximum Average Bandwidth Spanning Tree (MABST), which aims at optimizing the systems average stream rate. We prove that MABST is NP-complete by reducing from Hamiltonian Path problem and propose an efficient heuristic algorithm. The performance superiority of the proposed algorithm is justified by experimental results using a live media streaming system deployed in real networks and is also illustrated by an extensive set of simulations on simulated networks of various sizes in comparison with other methods based on a degree constraint or a greedy strategy.

On Parallel UDP-Based Transport Control over Dedicated Connections

Several research and production high-performance networks now provision multi-Gbps dedicated channels to support large data transfers in network-intensive applications. However, end users have not seen a corresponding increase in application throughput mainly because traditional end-to-end transport methods are not optimized for such connections. New congestion or flow control mechanisms are desirable to meet the challenges brought by dedicated connections to transport protocol design. The advent and proliferation of multi-core processors make it now possible to improve application throughput by providing multiple processing and networking resources to a single data transfer. Based on the existing PLUT method, we propose a new transport method, Para-PLUT, which utilizes multiple parallel UDP connections to take advantage of the full power of multicore processors for maximum aggregate goodput. We implement and test Para-PLUT in a local dedicated network testbed and the experimental results illustrate its superior performance over several existing methods.

On Performance-Adaptive flow control for large data transfer in high speed networks

Several research and production high-performance networks now provision multi-Gbps dedicated channels to meet the demands of large data transfers in network-intensive applications. However, end users have not seen corresponding increase in application throughput mainly because (i) the existence of high-bandwidth links has shifted the congestion from the network to end hosts, and (ii) such congestion is not well handled by TCPs Additive Increase and Multiplicative Decrease algorithm. Particularly, due to the sharing with unknown background workloads, the data receiver oftentimes lacks sufficient system resources to process the arriving packets, hence leading to significant packet drops at the end system. This paper proposes a UDP-based transport method that incorporates a performance-adaptive flow control mechanism to regulate the activities of both the sender and receiver in response to system dynamics to achieve high throughput. We construct a mathematical model for the socket receive buffer and data receiving process, and employ a profiling-based method to estimate the initial receiving bottleneck rate, which is dynamically adjusted and sent back to the sender for source rate control. The sending rate is stabilized at the estimated bottleneck rate based on a stochastic approximation algorithm. We test the proposed method on a local dedicated connection and the experimental results illustrate its superior performance over existing methods.

On optimization of scientific workflows to support streaming applications in distributed network environments

Large-scale data-intensive streaming applications in various science fields feature complex DAG-structured workflows comprised of distributed computing modules with intricate inter-module dependencies. Supporting such workflows in high-performance network environments and optimizing their throughput are crucial to collaborative scientific exploration and discovery. We formulate workflow mapping as a frame rate optimization problem and propose an efficient heuristic solution, which is integrated into the Condor-based Scientific Workflow Automation and Management Platform (SWAMP) in place of Condors default mapping scheme. The SWAMP system is also augmented with several new components to improve the workflow management process. The performance superiority of the proposed solution is verified using both simulations and a real-life scientific workflow for climate modeling deployed in a distributed heterogeneous network environment.

Stabilizing transport dynamics of control channels over wide-area networks

The next generation large-scale computing applications require network support for interactive visualization, computational steering and instrument control over wide-area networks. In particular, these applications require stable transport streams over wide-area networks, which are not adequately supported by current transport methods. We propose a new class of protocols capable of stabilizing a transport channel at a specified throughput level in the presence of random network dynamics based on the classical Robbins-Monro stochastic approximation approach. These protocols dynamically adjust the window size or sleep time at the source to achieve stable throughput at the destination. The target throughput typically corresponds to a small fraction of the available connection bandwidth. This approach yields provably probabilistically stable protocols as a consequence of carefully adjusted step sizes. The superior and robust stabilization performance of the proposed approach is extensively evaluated in a simulated environment and further verified through real-life implementations and deployments over both Internet and dedicated connections under disparate network conditions in comparison with existing transport methods.