Ognjen Prnjat

Development of Grid e-Infrastructure in South-Eastern Europe

Over the period of six years and three phases, the SEE-GRID programme has established a strong regional human network in the area of distributed scientific computing and has set up a powerful regional Grid infrastructure. It attracted a number of user communities and applications from diverse fields from countries throughout the South-Eastern Europe. From the infrastructure point view, the first project phase has established a pilot Grid infrastructure with more than 20 resource centers in 11 countries. During the subsequent two phases of the project, the infrastructure has grown to currently 55 resource centers with more than 6,600 CPUs and 750 TBs of disk storage, distributed in 16 participating countries. Inclusion of new resource centers to the existing infrastructure, as well as a support to new user communities, has demanded setup of regionally distributed core services, development of new monitoring and operational tools, and close collaboration of all partner institution in managing such a complex infrastructure. In this paper we give an overview of the development and current status of SEE-GRID regional infrastructure and describe its transition to the NGI-based Grid model in EGI, with the strong SEE regional collaboration.

A Resilient Methodology for Accessing and Exploiting Data and Scientific Codes on Distributed Environments

The execution of data-based applications on distributed environments is prone to different failures in the different steps of the process. These steps range from the search on available references and data to the adaptation of the applications of interest to the different platforms where they are going to be efficiently run, which requires a deep understanding of their specific characteristics. In this work, a workflow to efficiently develop, maintain and execute highly portable distributed applications on dynamic environments, performing experiments based on Data Repositories, is presented. With this approach, the development, execution and maintenance of distributed applications is significantly simplified with respect to previous solutions, increasing their robustness and allowing running them on different computational platforms unattendedly. Data search and usage is also significantly simplified and can be automatically retrieved as input data into a code already integrated in the proposed workflow. Such a search is based on metadata standards and relies on Persistent Identifiers (PID) to assign specific repositories to the new produced output.

SEE-GRID eInfrastructure for Regional eScience

In the past 6 years, a number of targeted initiatives, funded by the European Commission via its information society and RTD programmes and Greek infrastructure development actions, have articulated a successful regional development actions in South East Europe that can be used as a role model for other international developments. The SEEREN (South-East European Research and Education Networking initiative) project, through its two phases, established the SEE segment of the pan-European G ´EANT network and successfully connected the research and scientific communities in the region. Currently, the SEE-LIGHT project is working towards establishing a dark-fiber backbone that will interconnect most national Research and Education networks in the region. On the distributed computing and storage provisioning i.e. Grid plane, the SEE-GRID (South-East European GRID e-Infrastructure Development) project, similarly through its two phases, has established a strong human network in the area of scientific computing and has set up a powerful regional Grid infrastructure, and attracted a number of applications from different fields from countries throughout the South-East Europe. The current SEEGRID-SCI project, ending in April 2010, empowers the regional user communities from fields of meteorology, seismology and environmental protection in common use and sharing of the regional e-Infrastructure. Current technical initiatives in formulation are focusing on a set of coordinated actions in the area of HPC and application fields making use of HPC initiatives. Finally, the current SEERA-EI project brings together policy makers – programme managers from 10 countries in the region. The project aims to establish a communication platform between programme managers, pave the way towards common e-Infrastructure strategy and vision, and implement concrete actions for common funding of electronic infrastructures on the regional level. The regional vision on establishing an e-Infrastructure compatible with European developments, and empowering the scientists in the region in equal participation in the use of pan- European infrastructures, is materializing through the above initiatives. This model has a number of concrete operational and organizational guidelines which can be adapted to help e-Infrastructure developments in other world regions. In this paper we review the most important developments and contributions by the SEEGRID- SCI project.

A Fault Tolerant Workflow for Reproducible Research

In this work, the authors present a set of tools to overcome the problem of creating and executing distributed applications on dynamic environments in a resilient way, also ensuring the reproducibility of the performed experiments. The objective is to provide a portable, unattended and fault-tolerant set of tools, encapsulating the infrastructure-dependent operations away from the application developers and users, allowing to perform experiments based on open access data repositories. In this way, users can seamlessly search and lately access datasets that can be automatically retrieved as input data into a code already integrated in the proposed workflow. Such a search is based on metadata standards and relies on Persistent Identifiers (PID) to assign specific repositories. The applications profit from Distributed Toolbox, a newly created framework devoted to the creation and execution of distributed applications and includes tools for unattended Cluster and Grid execution, where a total fault tolerance is provided. By decoupling the definition of the remote tasks from its execution and control, the development, execution and maintenance of distributed applications is significantly simplified with respect to previous solutions, increasing their robustness and allowing running them on different computational platforms with little effort. The integration with open access databases and employment of PIDs for long-lasting references ensures that the data related to the experiments will persist, closing a complete research circle of data access / processing/ storage / dissemination of results.

Improving Service Management for Federated Resources to Support Virtual Research Environments

Virtual research environments provide an easy access to e-Infrastructures for researchers by creating an abstracted service-oriented layer on top of the available resources. Using the portal, researchers can focus on the research workflow and data analysis while being provided with a consolidated unified view of all tools necessary for their activities. The sustainable lifecycle of a virtual research environment can only be achieved if it is going to be used with high quality of experience by a large body of users. Aiming for this goal, in this paper we analyse the requirements and implementation of a cross-community virtual research environment that brings together researchers from three different domains. Promoting interdisciplinary research and cooperation, the federated virtual research environment is based on the service orientation paradigm, offering anything as a service solutions. Thus, the main pillar for a successful implementation of this solution is the careful design and management of the underlying elementary services and service compositions. The rest of the paper discusses the challenges of the service management implementation focusing on interoperability by design and service management standards.

Adapting Reproducible Research Capabilities to Resilient Distributed Calculations

Nowadays, computing calculations are becoming more and more demanding due to the huge pool of resources available. This demand must be satisfied in terms of computational efficiency and resilience, which is compromised in distributed and heterogeneous platforms. Not only this, data obtained are often either reused by other researchers or recalculated. In this work, a set of tools to overcome the problem of creating and executing fault tolerant distributed applications on dynamic environments is presented. Such a set also ensures the reproducibility of the performed experiments providing a portable, unattended and resilient framework that encapsulates the infrastructure-dependent operations away from the application developers and users, allowing performing experiments based on Open Access data repositories. In this way, users can seamlessly search and lately access datasets that can be automatically retrieved as input data into a code already integrated in the proposed workflow. Such a search is based on metadata standards and relies on Persistent Identifiers PID to assign specific repositories. The applications profit from Distributed Toolbox, a framework devoted to the creation and execution of distributed applications and includes tools for unattended cluster and grid execution, where a total fault tolerance is provided. By decoupling the definition of the remote tasks from its execution and control, the development, execution and maintenance of distributed applications is significantly simplified with respect to previous solutions, increasing their robustness and allowing running them on different computational platforms with little effort. The integration with Open Access databases and employment of PIDs for long-lasting references ensures that the data related to the experiments will persist, closing a complete research circle of data access/processing/storage/dissemination of results.

Virtual Research Environment for value-added services in national and regional NRENs: case studies

Case studies are presented for the role of national and regional NRENs beyond connectivity, encompassing computing (grid, cloud and High-Performance computing) services and big data management services. The case studies are the recently started VI-SEEM and MAGIC projects. The VI-SEEM project, started in October 2015, unifies the existing e-Infrastructures in South-East Europe (SEE) and the Eastern Mediterranean (EM), including Grid, cloud, and High-Performance Computing resources. It does so in order to better utilise synergies, for an improved service provision within a unified Virtual Research Environment (VRE) to be provided to the scientific user communities in this large region. The overall objective is to provide a user-friendly integrated e-Infrastructure platform for scientific communities in Climatology, Life Sciences, and Cultural Heritage for the SEE and EM regions; by linking networking, computing, data, and visualization resources, as well as services, software and tools. The Virtual Research Environment provides the scientists and researchers with the support in the full lifecycle of scientific research: accessing relevant data necessary for their research, using it with provided codes and tools to carry out new experiments and simulations on large-scale e-Infrastructures, and producing and integrating new knowledge and data which is stored and shared within the same VRE. The project is founded on the serviceoriented data-driven approach, where a specific set of activities deals not just with simple eInfrastructure data storage (live, dropbox-like), but includes the support for the full data lifecycle for the 3 target communities. The value-added computing services provided to the researchers include grid and cloud computing, as well as large-scale High-Performance Computing platforms, with a set of management tools provided for the unified management of computing resources. All project services will be provided through a service catalogue. Similarly the MAGIC project, started in May 2015, aims to adopt a service-oriented approach to advertising specifically the NREN cloud services, and this approach will be implemented through a compatible service catalogue, based on the GEANT cloud catalogue: thus the envisaged MAGIC cloud catalogue is also briefly discussed.

Surveying clouds in a global environment

This paper reports on the results of the survey regarding the Research and Education (R&E) clouds in a number of world regions covered by the CHAIN-REDS project, including the sub-Saharan Africa. The survey includes the technical aspects of R&E clouds, as well as issues related to interoperation, interoperability, compatibility, orchestration and federation. Results obtained show a high interest in standards, thus pointing to the fact that the cloud solutions offered by CHAIN-REDS, which include a global cloud federation test-bed and the use of OCCI and CDMI standards, provide a building block for global cloud federated environment.

Enabling intercontinental e-Infrastructures - a case for Africa

CHAIN-REDS, an EU co-funded project, focuses on promoting and supporting technological and scientific collaboration across different e-Infrastructures established and operated in various continents. The project implemented a Regional Operations Centre (ROC) model for enabling Grid computing interoperation across continents, and an operational ROC has been set up for Africa. Moreover, the project operates a global Cloud federation test-bed, where also nascent African Cloud sites contribute. Finally, the project is supporting a real-life use-case from Africa that of APHRC, using its data e-Infrastructure services.

Guest Editors' Message

This issue of the Journal of Grid Computing is focused on production grids. Grid computing has now reached the phase where 24/7 production-level computing and storage resources are available for users to access large distributed data collections and run highthroughput compute-intensive applications. On the other hand, petascale-class supercomputing systems are becoming increasingly interlinked and available to wider international user communities. The governments and scientific institutions around the world have strongly invested over the past decade in not just deploying nationallevel resources but also linking those with their international peers to form large federated structures spanning whole continents. As one of the most notable examples, the European Commission has co-invested to first set up, and then extend the European Electronic Infrastructure technology (particularly Grid) and its operational and organizational principles to a number of regions, supporting global scientific endeavors. A number of different collaboration models have thus been established between Europe and the rest of the world.