Martin Koehler

Design of an Adaptive Framework for Utility-Based Optimization of Scientific Applications in the Cloud

Cloud computing plays an increasingly important role in realizing scientific applications by offering virtualized compute and storage infrastructures that can scale on demand. In this paper we report on the design of a self-configuring adaptive framework for developing and optimizing scientific applications on top of Cloud technologies. Our framework relies on a MAPE-K loop, known from autonomic computing, for optimizing the configuration of scientific applications taking into account the three abstraction layers of the Cloud stack: the application layer, the execution environment layer, and the resource layer. By evaluating monitored resources, the framework configures the layers and allocates resources on a per job basis. The evaluation of configurations relies on historic data and a utility function that ranks different configurations regarding to the arising costs. The adaptive framework has been integrated into the Vienna Cloud Environment (VCE) and has been evaluated with a MapReduce application.

A grid services cloud for molecular modelling workflows

Scientific workflows require seamless access to HPC applications, deployed on remote, globally distributed computing resources. Typically, scientific workflows are both compute- and data-intensive, and often require dynamic execution control mechanisms. We present a service-oriented infrastructure that addresses these challenges by seamlessly integrating grid computing technologies with a Cloud infrastructure to support the scheduling of dynamic scientific workflows. A case study implementing a complex scientific workflow for computing photodynamics of biologically relevant molecules, a simulation of the non-adiabatic dynamics of 2,4-pentadieneiminum-cation (Protonated Schiff Base 3 (PSB3)) solvated in Water, is realised via the presented infrastructure.

An Adaptive Framework for the Execution of Data-Intensive MapReduce Applications in the Cloud

Cloud computing technologies play an increasingly important role in realizing data-intensive applications by offering a virtualized compute and storage infrastructure that can scale on demand. A programming model that has gained a lot of interest in this context is MapReduce, which simplifies processing of large-scale distributed data volumes, usually on top of a distributed file system layer. In this paper we report on a self-configuring adaptive framework for developing and optimizing data-intensive scientific applications on top of Cloud and Grid computing technologies and the Hadoop framework. Our framework relies on a MAPE-K loop, known from autonomic computing, for optimizing the configuration of data-intensive applications at three abstraction layers: the application layer, the MapReduce layer, and the resource layer. By evaluating monitored resources, the framework configures the layers and allocates the resources on a per job basis. The evaluation of configurations relies on historic data and a utility function that ranks different configurations regarding to the arising costs. The optimization framework has been integrated in the Vienna Grid Environment (VGE), a service-oriented application development environment for providing applications on HPC systems, clusters and Clouds as services. An experimental evaluation of our framework has been undertaken with a data-analysis application from the field of molecular systems biology.

A Service Oriented Approach for Distributed Data Mediation on the Grid

Seamless integrated access to data stored in globally distributed databases has become a major challenge in many scientific disciplines. In this paper, we describe a Grid-based service-oriented infrastructure that tackles this challenge through the provisioning of virtual data sources realized by means of distributed data mediation services. Virtual data sources offer to the user a single integrated view of distributed heterogeneous databases and information sources while hiding the details of data location, data formats, and access mechanisms of the underlying physical data sources. Personalized views of data sources, tailor-made for specific usage scenarios, may be offered by different virtual data sources. Virtual data sources rely on flexible mediation techniques and utilize distributed query processing to optimize complex data integration scenarios. Distributed data mediation services have been realized on top of standard Grid and Web Services technologies including OGSA-DAI and OGSA-DQP. The generic data service infrastructure described in this paper is being utilized in the context of the European @neurIST project, which develops an advanced service-oriented Grid infrastructure for the management and treatment of multi-factorial diseases. Details of distributed data mediation and query processing are presented in the context of an experimental scenario integrating clinical data bases.

Towards collaborative data management in the VPH-Share project

The goal of the Virtual Physiological Human Initiative is to provide a systematic framework for understanding physiological processes in the human body in terms of anatomical structure and biophysical mechanisms across multiple length and time scales. In the long term it will transform the delivery of European healthcare into a more personalised, predictive, and integrative process, with significant impact on healthcare and on disease prevention. This paper outlines how the recently funded project VPH-Share contributes to this vision. The project is motivated by the needs of the whole VPH community to harness ICT technology to improve health services for the individual. VPH-Share will provide the organisational fabric (the infostructure), realised as a series of services, offered in an integrated framework, to expose and to manage data, information and tools, to enable the composition and operation of new VPH workflows and to facilitate collaborations between the members of the VPH community.

A Cloud-Based Framework for Collaborative Data Management in the VPH-Share Project

The VPH-Share project objective is to store, share, integrate, and link data, information, knowledge, and wisdom about the physiopathology of the human body to enable their reuse within the virtual physiological human community. Therefore, the projects develops a modular and generic data management platform on top of a distributed Cloud infrastructure. The data management platform enables the ontological annotation of VPH-relevant datasets, their provisioning in the Cloud, and supports different data integration approaches. In this paper we present the architecture and implementation of this VPH-Share Cloud and data management platform and we go into detail about two different data integration approaches: relational data mediation, which has been realized on top of a distributed data mediation engine, and semantic data integration, which is supported on the basis of the SPARQL federation extension. Both approaches are examined on top of a project-specific scenario executed in the VPH-Share Cloud environment.

The VADA Architecture for Cost-Effective Data Wrangling

Data wrangling, the multi-faceted process by which the data required by an application is identified, extracted, cleaned and integrated, is often cumbersome and labor intensive. In this paper, we present an architecture that supports a complete data wrangling lifecycle, orchestrates components dynamically, builds on automation wherever possible, is informed by whatever data is available, refines automatically produced results in the light of feedback, takes into account the users priorities, and supports data scientists with diverse skill sets. The architecture is demonstrated in practice for wrangling property sales and open government data.

Supporting molecular modeling workflows within a grid services cloud

Seamless integrated support for scientific workflows accessing HPC applications, deployed on globally distributed computing resources, has become a major challenge in scientific computing. Scientific workflows in the domain of theoretical chemistry are typically long running, deal with huge files, and have a need for dynamic execution control mechanisms. In this paper, we describe a service-oriented approach based on the Vienna Grid Environment (VGE) that tackles these challenges by seamlessly integrating the Ubuntu Cloud infrastructure supporting the scheduling of dynamic and partitioned workflows. The VGE service environment, which enables the provisioning of HPC applications and data sources as Web services, has been enhanced with support for virtualized workflows. The generic scientific workflow infrastructure is utilized in the context of the CPAMMS project, an interdisciplinary research initiative in the area of computational molecular modeling and simulation. A case study implementing a complex scientific workflow for computing photodynamics of biologically relevant molecules, a simulation of the nonadiabatic dynamics of 2,4-pentadieneiminum-cation (Protonated Schiff Base 3, PSB3) solvated in water, is realized via the presented infrastructure.

The VPH-Share Data Management Platform: Enabling Collaborative Data Management for the Virtual Physiological Human Community

The objective of the Virtual Physiological Human Initiative is to provide a systematic framework for understanding physiological processes in the human body in terms of anatomical structure and biophysical mechanisms across multiple length and time scales. The VPH-Share project, which has been funded in the context of the initiative, contributes to this vision, especially in terms of data management. In this paper we present the VPH-Share data management platform enabling sharing VPH-relevant datasets within the community on the basis of Cloud technologies. The data management platform aims at supporting the data management life-cycle but starting from already available data. The life cycle covers all processes from data selection, semantic data annotation, data integration, data publishing, and data access. Herein we describe the infrastructure supporting the data management life-cycle towards collaborative data management, consisting of the data publication suite and the dataset service environment.

User Driven Multi-Criteria Source Selection

Abstract Source selection is the problem of identifying a subset of available data sources that best meet a user’s needs. In this paper we propose a user-driven approach to source selection that seeks to identify sources that are most fit for purpose. The approach employs a decision support methodology to take account of a user’s context, to allow end users to tune their preferences by specifying the relative importance between different criteria, looking to find a trade-off solution aligned with his/her preferences. The approach is extensible to incorporate diverse criteria, not drawn from a fixed set, and solutions can use a subset of the data from each selected source, rather than require that sources are used in their entirety or not at all. The paper describes and motivates the approach, presenting a methodology for modelling a user’s context, and its collection of optimisation algorithms for exploring the space of solutions, and compares and evaluates the resulting algorithms using multiple real world data sets. The experiments show how source selection results are produced that are attuned to each user’s preferences, both with respect to overall weighted utility and through faithful representation of a user’s preferences within a result, while scaling to potentially thousands of sources.