Rubing Duan

ASKALON: a Grid application development and computing environment

We present the ASKALON environment whose goal is to simplify the development and execution of workflow applications on the Grid. ASKALON is centered around a set of high-level services for transparent and effective Grid access, including a Scheduler for optimized mapping of workflows onto the Grid, an Enactment Engine for reliable application execution, a Resource Manager covering both computers and application components, and a Performance Prediction service based on training phase and statistical methods. A sophisticated XML-based programming interface that shields the user from the Grid middleware details allows the high-level composition of workflow applications. ASKALON is used to develop and port scientific applications as workflows in the Austrian Grid project. We present experimental results using two real-world scientific applications to demonstrate the effectiveness of our approach.

ASKALON: A Development and Grid Computing Environment for Scientific Workflows

Most existing Grid application development environments provide the application developer with a nontransparent Grid. Commonly, application developers are explicitly involved in tedious tasks such as selecting software components deployed on specific sites, mapping applications onto the Grid, or selecting appropriate computers for their applications. Moreover, many programming interfaces are either implementation-technology-specific (e.g., based on Web services [24]) or force the application developer to program at a low-level middleware abstraction (e.g., start task, transfer data [22, 153]). While a variety of graphical workflow composition tools are currently being proposed, none of them is based on standard modeling techniques such as Unified Modeling Language (UML).

Performance and cost optimization for multiple large-scale grid workflow applications

Scheduling large-scale applications on the Grid is a fundamental challenge and is critical to application performance and cost. Large-scale applications typically contain a large number of homogeneous and concurrent activities which are main bottlenecks, but open great potentials for optimization. This paper presents a new formulation of the well-known NP-complete problems and two novel algorithms that addresses the problems. The optimization problems are formulated as sequential cooperative games among workflow managers. Experimental results indicate that we have successfully devised and implemented one group of effective, efficient, and feasible approaches. They can produce soultuins of significantly better performance and cost than traditional algorithms. Our algorithms have considerably low time complexity and can assign 1,000,000 activities to 10,000 processors within 0.4 second on one Opteron processor. Moreover, the solutions can be practically performed by workflow managers, and the violation of QoS can be easily detected, which are critical to fault tolerance.

A Hybrid Intelligent Method for Performance Modeling and Prediction of Workflow Activities in Grids

Grid schedulers require individual activity performance predictions to map workflow activities on different Grid sites. The effectiveness of the scheduling systems is hampered by inaccurate predictions due to the inability of existing predictors to effectively model the dynamic and heterogeneous nature of Grid resources, or the wide range of problem sizes and runtime arguments. To address this deficiency, we propose a hybrid Bayesian-neural network approach to dynamically model and predict the execution time of activities in real workflow applications. Bayesian network is used for a high-level representation of activities performance probability distribution against different factors affecting the performance. The important attributes are dynamically selected by the Bayesian network and fed into a radial basis function neural network to make further predictions. Our approach is generic to any type of scientific applications, and flexible to import expert knowledge to further improve accuracies. Experimental results for activities from three realworld workflow applications are presented to show effectivenessof our approach.

Run-time Optimisation of Grid Workflow Applications

The execution of workflow applications on the grid is a complex issue because of its dynamic and heterogeneous nature. While the grid provides good potential for achieving high performance, it also introduces a broad set of unpredictable overheads and possible failures. In this paper we present new methods for scalable and fault tolerant coordination of workflows in dynamic grid environments, including partitioning, static and dynamic optimisation, as well as virtual single execution environment, incorporated into the ASKALON distributed workflow enactment engine. We demonstrate the effectiveness of our methods on a material science workflow application executed in a real-world grid environment

Multi-Objective Game Theoretic Schedulingof Bag-of-Tasks Workflows on Hybrid Clouds

Scheduling multiple large-scale parallel workflow applications on heterogeneous computing systems like hybrid clouds is a fundamental NP-complete problem that is critical to meeting various types of QoS (Quality of Service) requirements. This paper addresses the scheduling problem of large-scale applications inspired from real-world, characterized by a huge number of homogeneous and concurrent bags-of-tasks that are the main sources of bottlenecks but open great potential for optimization. The scheduling problem is formulated as a new sequential cooperative game and propose a communication and storage-aware multi-objective algorithm that optimizes two user objectives (execution time and economic cost) while fulfilling two constraints (network bandwidth and storage requirements). We present comprehensive experiments using both simulation and real-world applications that demonstrate the efficiency and effectiveness of our approach in terms of algorithm complexity, makespan, cost, system-level efficiency, fairness, and other aspects compared with other related algorithms.

DEE: a distributed fault tolerant workflow enactment engine for grid computing

It is a complex task to design and implement a workflow management system that supports scalable executions of large-scale scientific workflows for dynamic and heterogeneous Grid environments. In this paper we describe the Distributed workflow Enactment Engine (DEE) of the ASKALON Grid application development environment for Grid computing. DEE proposes a de-centralized architecture that simplifies and reduces the overhead for managing large workflows through partitioning, improved data locality, and reduced workflow-level checkpointing overhead. We report experimental results for a real-world material science workflow application.

Short Paper: Data Mining-based Fault Prediction and Detection on the Grid

This paper describes a novel approach to fault detection and prediction on the grid based on data mining techniques. Data mining techniques are here applied as a mean to effectively process the significant amount of captured data from grid sites, services, workflows and activities. The paper provides a first approach of proposed techniques in terms of its ability of utilizing relevant information and the fault tolerance requirements. Such approach is one intelligent, distributed framework of fault detection and prediction for anomaly and failed activity by using resource- and workflow-based information. We use fault predictions to improve the performance of the workflow execution by avoiding potential faults of activities

Real world workflow applications in the askalon grid environment

The workflow paradigm is widely considered as an important class of truly distributed Grid applications which poses many challenges for a Grid computing environment. Still to this time, rather few real-world applications have been successfully ported as Grid-enabled workflows. We present the Askalon programming and computing environment for the Grid which comprises a high-level abstract workflow language and a sophisticated service-oriented runtime environment including meta-scheduling, performance monitoring and analysis, and resource brokerage services.We demonstrate the development of a real-world river modelling distributed workflow system in the Askalon environment that harnesses the computational power of multiple Grid sites to optimise the overall execution time.

A sequential cooperative game theoretic approach to scheduling multiple large-scale applications in grids

Scheduling large-scale applications in heterogeneous distributed computing systems is a fundamental NP-complete problem that is critical to obtaining good performance and execution cost. In this paper, we address the scheduling problem of an important class of large-scale Grid applications inspired by the real world, characterized by a huge number of homogeneous, concurrent, and computationally intensive tasks that are the main sources of performance, cost, and storage bottlenecks. We propose a new formulation of this problem based on a cooperative distributed game-theory-based method applied using three algorithms with low time complexity for optimizing three important metrics in scientific computing: execution time, economic cost, and storage requirements. We present comprehensive experiments using simulation and real-world applications that demonstrate the effectiveness of our approach in terms of time and fairness compared to other related algorithms. We schedule large-scale Grid applications with a huge identical parallel tasks.We formulate the problem using a cooperative distributed game-theory-based method.We design three algorithms for optimizing time, cost, and storage requirements.We evaluate our method using simulation and real-world applications.We compare our results with related algorithms.