Charles Loomis

The Grid Observatory

The goal of the Grid Observatory project (GO) is to contribute to an experimental theory of large grid systems by integrating the collection of data on the behaviour of the flagship European Grid Infrastructure (EGI) and its users, the development of models, and an ontology for the domain knowledge. The GO gives access to a database of grid usage traces available to the wider computer science community without the need of grid credentials. The paper presents the architecture of the digital curation process enacted by the GO and examples of their exploitation.

CYCLONE: A Platform for Data Intensive Scientific Applications in Heterogeneous Multi-cloud/Multi-provider Environment

This paper presents results of the ongoing development of the CYCLONE as a platform for scientific applications in heterogeneous multi-cloud/multi-provider environment. The paper explains the general use case that provides a general motivation for the CYCLONE architecture and provides detailed analysis of the bioinformatics use cases that define specific requirements to the CYCLONE infrastructure components. Special attention is given to the federated access control and security infrastructure that must provide consistent security and data protection for distributed bioinformatics data processing infrastructure and distributed cross-organisations collaborating teams of scientists. The paper provides information about selected use cases implementation using SlipStream cloud automation and management platform with application recipe example. The paper also addresses requirements for providing dedicated intercloud network infrastructure which is currently not addressed by cloud providers (both public and scientific/community).

Cloud based big data infrastructure: Architectural components and automated provisioning

This paper describes the general architecture and functional components of the cloud based Big Data Infrastructure (BDI). The proposed BDI architecture is based on the analysis of the emerging Big Data and data intensive technologies and supported by the definition of the Big Data Architecture Framework (BDAF) that defines the following components of the Big Data technologies: Big Data definition, Data Management including data lifecycle and data structures, Big Data Infrastructure (generically cloud based), Data Analytics technologies and platforms, and Big Data security, compliance and privacy. The paper provides example of requirements analysis and implementation of two bioinformatics use cases on cloud and using SlipStream based cloud applications deployment and management automation platform being developed in the CYCLONE project. The paper also refers to importance of standardisation of all components of BDAF and BDI and provides short overview of the NIST Big Data Interoperability Framework (BDIF). The paper discusses importance of automation of all stages of the Big Data applications developments, deployment and management and refers to existing cloud automation tools and new developments in the SlipStream cloud automation platform that allows multi-cloud applications deployment and management.

Cloud Computing Infrastructure for Data Intensive Applications

Abstract This chapter describes the general architecture and functional components of the cloud-based big data infrastructure (BDI). The chapter starts with the analysis of emerging Big Data and data intensive technologies and provides the general definition of the Big Data Architecture Framework (BDAF) that includes the following components: Big Data definition, Data Management including data lifecycle and data structures, generically cloud based BDI, Data Analytics technologies and platforms, and Big Data security, compliance, and privacy. The chapter refers to NIST Big Data Reference Architecture (BDRA) and summarizes general requirements to Big Data systems described in NIST documents. The proposed BDI and its cloud-based components are defined in accordance with the NIST BDRA and BDAF. This chapter provides detailed analysis of the two bioinformatics use cases as typical example of the Big Data applications that have being developed by the authors in the framework of the CYCLONE project. The effective use of cloud for bioinformatics applications requires maximum automation of the applications deployment and management that may include resources from multiple clouds and providers. The proposed CYCLONE platform for multicloud multiprovider applications deployment and management is based on the SlipStream cloud automation platform and includes all necessary components to build and operate complex scientific applications. The chapter discusses existing platforms for cloud powered applications development and deployment automation, in particularly referring to the SlipStream cloud automation platform, which allows multicloud applications deployment and management. The chapter also includes a short overview of the existing Big Data platforms and services provided by the major cloud services providers which can be used for fast deployment of customer Big Data applications using the benefits of cloud technologies and global cloud infrastructure.

Customisable Data Science Educational Environment: From Competences Management and Curriculum Design to Virtual Labs On-Demand

Data Science is an emerging field of science, which requires a multi-disciplinary approach and is based on the Big Data and data intensive technologies that both provide a basis for effective use of the data driven research and economy models. Modern data driven research and industry require new types of specialists that are capable to support all stages of the data lifecycle from data production and input to data processing and actionable results delivery, visualisation and reporting, which can be jointly defined as the Data Science professions family. The education and training of Data Scientists currently lacks a commonly accepted, harmonized instructional model that reflects all multi-disciplinary knowledge and competences that are required from the Data Science practitioners in modern, data driven research and the digital economy. The educational model and approach should also solve different aspects of the future professionals that includes both theoretical knowledge and practical skills that must be supported by corresponding education infrastructure and educational labs environment. In modern conditions with the fast technology change and strong skills demand, the Data Science education and training should be customizable and delivered in multiple form, also providing sufficient data labs facilities for practical training. This paper discussed both aspects: building customizable Data Science curriculum for different types of learners and proposing a hybrid model for virtual labs that can combine local university facility and use cloud based Big Data and Data analytics facilities and services on demand. The proposed approach is based on using the EDISON Data Science Framework (EDSF) developed in the EU funded Project EDISON and CYCLONE cloud automation systems being developed in another EU funded project CYCLONE.

An Autonomic Cloud Application Placement Tool Based on Cost Criteria

The cloud computing ecosystem comprises hundreds of providers, offering diverse computing services, incompatible APIs, and significantly different pricing models. Cloud application management platforms hide the heterogeneity of the services and APIs, allowing, to varying degrees, portability between providers. These tools remove technical barriers to switching providers, but they do not provide a mechanism for evaluating the cost effectiveness of switching.This paper presents a decision support system, working within cloud application management platforms, that evaluates the costs of a customers applications using resources from different cloud service providers. The system (1) generalizes and normalizes multiple cloud pricing models and (2) gathers pricing data from cloud providers. These, in conjunction with the application resource consumption model, allow the cloud pricing module to estimate a price for the application for each cloud provider.To demonstrate this, our cloud pricing module has been integrated with the SlipStream multi-cloud application management platform, allowing its users to optimize their choice of provider(s).

Runtime Application Performance Management for Multi-Cloud Cyclone Environment

This paper presents results of the ongoing development of CYCLONE as a platform for scientific applications in heterogeneous multi-cloud/multi-provider environment. In particular, we focus on QoS management of the multi-cloud applications within CYCLONE. A challenging factor for application deployment and exploitation within the CYCLONE infrastructure is its highly dynamic nature, which raises the need for control methods that quickly adapt to, or even anticipate, changing circumstances. In this paper we present the solution that allows for performance optimization of the running applications within CYCLONE. This solution is envisioned as an eternal software solution which can be integrated with SlipStream, a multi-cloud application management platform. Based on the analysis of the measurement data, a number of actions may be taken. These actions may include re-scaling of the cloud resources, re-deployment of the application, or choosing of the alternative deployment of the same application. The core of our solution is the analytics engine, and, for a given scenario, we illustrate the some of the engine algorithms.

Towards a Cost-Optimized Cloud Application Placement Tool

The cloud computing ecosystem comprises hundreds of providers, offering diverse computing services, incompatible APIs, and significantly different pricing models. Cloud application management platforms hide the heterogeneity of the services and APIs, allowing, to varying degrees, portability between providers. These tools remove technical barriers to switching providers, but they do not provide a mechanism for evaluating the cost effectiveness of switching. This paper presents a decision support system, working within cloud application management platforms, that evaluates the costs of a customers applications using resources from different cloud service providers. The system (1) generalizes and normalizes multiple cloud pricing models and (2) gathers pricing data from cloud providers. These, in conjunction with the application resource consumption model, allow the cloud pricing module to estimate a price for the application for each cloud provider. To demonstrate this, our cloud pricing module has been inte-grated with the SlipStream multi-cloud application management platform, allowing its users to optimize their choice of provider(s).

Appliance Management for Federated Cloud Environments

Cloud infrastructures provide compelling features for scientific and engineering applications. Federated clouds additionally promise improved scalability via access to a larger pool of resources and improved service availability through geographically distributed redundant servers. Effective use of federated clouds requires the creation of portable appliances and consistent appliance management techniques. The Stratus Lab Marketplace, a platform-agnostic appliance registry, facilitates appliance management in a federated environment. This paper describes the Marketplace design goals, implementation, and security concerns. It also covers the planned improvements based on our experience of running this service in production for more than two years.