Amrey Krause

Introduction to OGSA-DAI services

In todays large collaborative environments, potentially composed of multiple distinct organisations, uniform controlled access to data has become a key requirement if these organisations are to work together as Virtual Organisations. We refer to such an integrated set of data resources as a virtual data warehouse. The Open Grid Services Architecture – Data Access and Integration (OGSA-DAI) project was established to produce a common middleware solution, aligned with the Global Grid Forums (GGF) OGSA vision [OGSA] to allow uniform access to data resources using a service based architecture. In this paper the service infrastructure provided by OGSA-DAI is presented providing a snapshot of its current state, in an evolutionary process, which is attempting to build infrastructure to allow easy integration and access to distributed data using grids or web services. More information about OGSA-DAI is available from the project web site: www.ogsadai.org.

The WS-DAI family of specifications for web service data access and integration

This month, we are pleased to provide to our readers a column that addresses an important aspect of grid computing: data access.

Integrating distributed data sources with OGSA–DAI DQP and Views

OGSA-DAI (Open Grid Services Architecture Data Access and Integration) is a framework for building distributed data access and integration systems. Until recently, it lacked the built-in functionality that would allow easy creation of federations of distributed data sources. The latest release of the OGSA-DAI framework introduced the OGSA-DAI DQP (Distributed Query Processing) resource. The new resource encapsulates a distributed query processor, that is able to orchestrate distributed data sources when answering declarative user queries. The query processor has many extensibility points, making it easy to customize. We have also introduced a new OGSA-DAI Views resource that provides a flexible method for defining views over relational data. The interoperability of the two new resources, together with the flexibility of the OGSA-DAI framework, allows the building of highly customized data integration solutions.

Seasonal variations in human seminal parameters

OBJECTIVES To obtain data on seasonal variations of sperm parameters in an andrology laboratory. STUDY DESIGN Semen parameter values and hormone values of 2454 patients attending our infertility clinic between 1990 and 1997 were analysed. Seasonal trends were calculated using the method of Edwards [Ann. Hum. Genet. 25 (1961) 83]. RESULTS The total group of mean sperm count did not show a significant variation, but a significant circannual trend occurred of patients born in the groups 1950-1954, 1955-1959 and 1965-1969. A significant variation of acrosin activity occurred with a maximum in March. The other parameters did no show significant variations. CONCLUSIONS The knowledge on circannual variation of semen parameters and hormone values may be of value in diagnostic and therapeutic decisions in reproductive medicine.

Data-intensive architecture for scientific knowledge discovery

This paper presents a data-intensive architecture that demonstrates the ability to support applications from a wide range of application domains, and support the different types of users involved in defining, designing and executing data-intensive processing tasks. The prototype architecture is introduced, and the pivotal role of DISPEL as a canonical language is explained. The architecture promotes the exploration and exploitation of distributed and heterogeneous data and spans the complete knowledge discovery process, from data preparation, to analysis, to evaluation and reiteration. The architecture evaluation included large-scale applications from astronomy, cosmology, hydrology, functional genetics, imaging processing and seismology.

dispel4py: a Python framework for data-intensive scientific computing

This paper presents dispel4py, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. The main aim of dispel4py is to enable scientists to focus on their computation instead of being distracted by details of the computing infrastructure they use. Therefore, special care has been taken to provide dispel4py with the ability to map abstract workflows to different enactment platforms dynamically, at run time. In this work we present four dispel4py mappings: Apache Storm, MPI, multi-threading and sequential. The results show that dispel4py is successful in enacting on different platforms, while also providing scalable performance.

VERCE Delivers a Productive E-science Environment for Seismology Research

The VERCE project has pioneered an e-Infrastructure to support researchers using established simulation codes on high-performance computers in conjunction with multiple sources of observational data. This is accessed and organised via the VERCE science gateway that makes it convenient for seismologists to use these resources from any location via the Internet. Their data handling is made flexible and scalable by two Python libraries, ObsPy and dispel4py and by data services delivered by ORFEUS and EUDAT. Provenance driven tools enable rapid exploration of results and of the relationships between data, which accelerates understanding and method improvement. These powerful facilities are integrated and draw on many other e-Infrastructures. This paper presents the motivation for building such systems, it reviews how solid-Earth scientists can make significant research progress using them and explains the architecture and mechanisms that make their construction and operation achievable. We conclude with a summary of the achievements to date and identify the crucial steps needed to extend the capabilities for seismologists, for solid-Earth scientists and for similar disciplines.

An outline of the global grid forum data access and integration service specifications

Grid computing concerns itself with building the infrastructure to facilitate the sharing of computational and data resources to enable collaboration within virtual organisations. The Global Grid Forum (GGF) provides a framework for users, developers and vendors to come together to develop standards to ensure interoperability between middleware from different service providers. Central to this effort is the Open Grid Services Architecture (OGSA), and its associated specifications. These define consistent interfaces, generally couched as web services, and the components required to construct grid infrastructures. Both the web service and grid communities stand to benefit from the provision of consistent and agreed web service interfaces for data resources and the systems that manage them. This paper describes, motivates and presents the context for the work that has been undertaken by the GGF Data Access and Integration Services Working Group (DAIS-WG). The group has defined a set of data access and integration interfaces that are consistent with the OGSA vision. A brief overview of the current family of DAIS specifications is given: WS-DAI specifies a collection of generic data resource properties and messages that are specialised by WS-DAIR and WS-DAIX for use with relational and XML data resources, respectively. The WS-DAI specifications can be applied in regular web services environments or as part of a grid fabric.

dispel4py: An Agile Framework for Data-Intensive eScience

We present dispel4py a versatile data-intensive kit presented as a standard Python library. It empowers scientists to experiment and test ideas using their familiar rapid-prototyping environment. It delivers mappings to diverse computing infrastructures, including cloud technologies, HPC architectures and specialised data-intensive machines, to move seamlessly into production with large-scale data loads. The mappings are fully automated, so that the encoded data analyses and data handling are completely unchanged. The underpinning model is lightweight composition of fine-grained operations on data, coupled together by data streams that use the lowest cost technology available. These fine-grained workflows are locally interpreted during development and mapped to multiple nodes and systems such as MPI and Storm for production. We explain why such an approach is becoming more essential in order that data-driven research can innovate rapidly and exploit the growing wealth of data while adapting to current technical trends. We show how provenance management is provided to improve understanding and reproducibility, and how a registry supports consistency and sharing. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved. Finally we present the next steps to achieve scalability and performance.

Managing and Analysing Genomic Data Using HPC and Clouds

Database management techniques using distributed processing services have evolved to address the issues of distributed, heterogeneous data collections held across dynamic, virtual organisations [1-3]. These techniques, originally developed for data grids in domains such as high-energy particle physics [4], have been adapted to make use of the emerging cloud infrastructures [5].