Xubo Fei

A Reference Architecture for Scientific Workflow Management Systems and the VIEW SOA Solution

Scientific workflows have recently emerged as a new paradigm for scientists to formalize and structure complex and distributed scientific processes to enable and accelerate many scientific discoveries. In contrast to business workflows, which are typically control flow oriented, scientific workflows tend to be dataflow oriented, introducing a new set of requirements for system development. These requirements demand a new architectural design for scientific workflow management systems (SWFMSs). Although several SWFMSs have been developed that provide much experience for future research and development, a study from an architectural perspective is still missing. The main contributions of this paper are: 1) based on a comprehensive survey of the literature and identification of key requirements for SWFMSs, we propose the first reference architecture for SWFMSs; 2) according to the reference architecture, we further propose a service-oriented architecture for View (a VIsual sciEntific Workflow management system); 3) we implemented View to validate the feasibility of the proposed architectures; and 4) we present a View-based scientific workflow application system (SWFAS), called FiberFlow, to showcase the application of our View system.

Service-Oriented Architecture for VIEW: A Visual Scientific Workflow Management System

Scientific workflows have recently emerged as a new paradigm for scientists to formalize and structure complex and distributed scientific processes to enable and accelerate many scientific discoveries. In contrast to business workflows, which are typically control flow oriented, scientific workflows tend to be dataflow oriented, introducing a new set of requirements for system development. These requirements demand a new architectural design for scientific workflow management systems (SWFMSs). Although several SWFMSs have been developed that provide much experience for future research and development, a study from an architectural perspective is still missing. The main contributions of this paper are: (i) based on a comprehensive survey of the literature and identification of key requirements for SWFMSs, we propose the first reference architecture for SWFMSs, (ii) in compliance with the reference architecture, we further propose a service-oriented architecture for VIEW (a VIsual sciEntificWorkflow management system), (iii) we implement VIEW to validate the feasibility of the proposed architectures, and (iv) we present two case studies to showcase the applications of our VIEW system.

RDFProv: A relational RDF store for querying and managing scientific workflow provenance

Provenance metadata has become increasingly important to support scientific discovery reproducibility, result interpretation, and problem diagnosis in scientific workflow environments. The provenance management problem concerns the efficiency and effectiveness of the modeling, recording, representation, integration, storage, and querying of provenance metadata. Our approach to provenance management seamlessly integrates the interoperability, extensibility, and inference advantages of Semantic Web technologies with the storage and querying power of an RDBMS to meet the emerging requirements of scientific workflow provenance management. In this paper, we elaborate on the design of a relational RDF store, called RDFProv, which is optimized for scientific workflow provenance querying and management. Specifically, we propose: i) two schema mapping algorithms to map an OWL provenance ontology to a relational database schema that is optimized for common provenance queries; ii) three efficient data mapping algorithms to map provenance RDF metadata to relational data according to the generated relational database schema, and iii) a schema-independent SPARQL-to-SQL translation algorithm that is optimized on-the-fly by using the type information of an instance available from the input provenance ontology and the statistics of the sizes of the tables in the database. Experimental results are presented to show that our algorithms are efficient and scalable. The comparison with two popular relational RDF stores, Jena and Sesame, and two commercial native RDF stores, AllegroGraph and BigOWLIM, showed that our optimizations result in improved performance and scalability for provenance metadata management. Finally, our case study for provenance management in a real-life biological simulation workflow showed the production quality and capability of the RDFProv system. Although presented in the context of scientific workflow provenance management, many of our proposed techniques apply to general RDF data management as well.

A MapReduce-Enabled Scientific Workflow Composition Framework

MapReduce has recently gained a lot of attention as a parallel programming model for scalable data-intensive business and scientific analysis. In order to benefit from this powerful programming model in a scientific workflow environment, we propose a MapReduce-enabled scientific workflow composition framework consisting of: i) a dataflow based scientific workflow model that separates the declaration of the workflow interface from the definition of its functional body; ii) a set of dataflow constructs, including Map, Reduce, Loop, and Conditional, and their composition semantics to enable MapReduce-style scientific workflows; iii) an XML-based scientific workflow specification language, called WSL, in which both Map and Reduce are fully composable with other dataflow constructs in both flat and hierarchical manners. Besides leveraging the power of MapReduce to the workflow level, our workflow composition framework is unique in that workflows are the only operands for composition; in this way, our approach elegantly solves the two-world problem of existing composition frameworks, in which composition needs to deal with both the world of tasks and the world of workflows. The proposed framework is implemented and a case study is conducted to validate our techniques.

Storing and Querying Scientific Workflow Provenance Metadata Using an RDBMS

Provenance management has become increasingly important to support scientific discovery reproducibility, result interpretation, and problem diagnosis in scientific workflow environments. This paper proposes an approach to provenance management that seamlessly integrates the interoperability, extensibility, and reasoning advantages of semantic Web technologies with the storage and querying power of an RDBMS. Specifically, we propose: i) two schema mapping algorithms to map an arbitrary OWL provenance ontology to a relational database schema that is optimized for common provenance queries; ii) two efficient data mapping algorithms to map provenance RDF metadata to relational data according to the generated relational database schema, and iii) a schema-independent SPARQL-to-SQL translation algorithm that is optimized on-the-fly by using the type information of an instance available from the input provenance ontology and the statistics of the sizes of the tables in the database. Experimental results are presented to show that our algorithms are efficient and scalable.

A Task Abstraction and Mapping Approach to the Shimming Problem in Scientific Workflows

Recently, there has been an increasing need in scientific workflows to solve the shimming problem, the use of a special kind of adaptors, called shims, to link related but incompatible workflow tasks. However, existing techniques produce scientific workflows that are cluttered with many visible shims, which distract a scientist’s focus on functional components. Moreover, these techniques do not address a new type of shimming problem that occurs due to the incompatibility between the ports of a task and the inputs/outputs of its internal task component. To address these issues, 1) we propose a task template model which encapsulates the composition and mapping of shims and functional task component within a task interface; 2) we design an XML based task specification language, called TSL, to realize the proposed task template model; 3) we propose a service oriented architecture for task management to enable the distributed execution of shims and functional components; and 4) we implement the proposed model, language and architecture and present a case study to validate them. Our technique uniquely addresses both types of shimming problems. To our best knowledge, this is the first shimming technique that makes shims invisible at the workflow level, resulting in scientific workflows that are more elegant and readable.

A Dataflow-Based Scientific Workflow Composition Framework

Scientific workflow has recently become an enabling technology to automate and speed up the scientific discovery process. Although several scientific workflow management systems (SWFMSs) have been developed, a formal scientific workflow composition model in which workflow constructs are fully compositional one with another is still missing. In this paper, we propose a dataflow-based scientific workflow composition framework consisting of (1) a dataflow-based scientific workflow model that separates the declaration of the workflow interface from the definition of its functional body; (2) a set of workflow constructs, including Map, Reduce, Tree, Loop, Conditional, and Curry, which are fully compositional one with another; (3) a dataflow-based exception handling approach to support hierarchical exception propagation and user-defined exception handling. Our workflow composition framework is unique in that workflows are the only operands for composition; in this way, our approach elegantly solves the two-world problem in existing composition frameworks, in which composition needs to deal with both the world of tasks and the world of workflows. The proposed framework is implemented and several case studies are conducted to validate our techniques.

VIEW: a VIsual sciEntificWorkflow management system

In this demo, we present the current status of our visual scientific workflow management system called VIEW, highlighting the following two features: (i) the use of Semantic Web technology to represent, store, and query provenance metadata, leading to an interoperable and extensible provenance system, and (ii) the support of visualization of various provenance graphs and intermediate or final data products of a workflow run in the form of medical images or 3-D graphical models. VIEW seamlessly integrates the interoperability, extensibility, and reasoning advantages of Semantic Web technology, the querying and storage power of a RDBMS, and the appealing visual features of visualization techniques.

Finding females: pheromone-guided reproductive tracking behavior by male Nereis succinea in the marine environment.

SUMMARY Pheromones trigger reproductive responses of many marine organisms, but little is known about how pheromones mediate mate-finding behavior in the marine environment. This paper investigates whether the tetrapeptide nereithione (cysteine-glutathione disulfide), known to be released by females of the polychaete Nereis succinea to trigger spawning in male N. succinea, can also be used at lower concentrations to guide males to the females. Low concentrations of pheromone elicited increased swim speed and turning left or right 84% of the time. Animals sometimes weaved back and forth, or in other cases swam straight along the trails an average of 8.1±1.2 cm before veering off. At higher concentrations, the males circled frequently, often encountering 10–20 cm of pheromone trail before swimming away. Male responses to nereithione were modeled by computer simulation, taking into account arousal of swim speed, activation of turning, speed of response and its decay, etc. In the model, low concentrations (<10–8 mol l–1) of pheromone significantly increased the number of encounters with the pheromone trail, an average following of simulated trails of 10.5±3.6 cm, and a significant increase in the frequency of encountering a virtual female on the trail (ANOVA, P<0.001). The model supports the hypothesis that a pheromone can have a dual function, with low concentration pheromone trails being used by male N. succinea to find females and increase their likelihood of mating whereas high concentrations of the same pheromone trigger the spawning behavior itself.

Atomicity and provenance support for pipelined scientific workflows

Today many significant scientific discoveries are achieved through complex and distributed scientific computations that are structured and represented as scientific workflows. Although atomicity is a well studied topic in transaction processing and business workflows, such an important capability needs to be revisited in a scientific workflow environment. Firstly, the semantics of atomicity needs to be defined in a dataflow-oriented scientific workflow model, particularly for pipelined execution of hierarchical scientific workflows. Secondly, in a scientific workflow environment, atomic regions are specified or inferred dynamically as needed and are committed implicitly, which are in contrast to a priori well-defined transaction boundaries and explicit commits in transaction processing and business workflows. Finally, although atomicity and provenance are related to each other, their interactions and relationships have never been explored in the literature. In this paper, we propose: (i) an architecture for scientific workflow management systems that supports both provenance and atomicity; (ii) a dataflow-oriented atomicity model that supports the notions of commit and abort; and (iii) a dataflow-oriented provenance model that supports querying and visualizing provenance.