Víctor Méndez Muñoz

Rafhyc: an Architecture for Constructing Resilient Services on Federated Hybrid Clouds

Federated hybrid clouds is a model of service access and delivery to community cloud infrastructures. This model opens an opportunity window to allow the integration of the enhanced science (eScience) with the Cloud paradigm. The eScience is computationally intensive science that is carried out in highly distributed computing infrastructures. Nowadays, the eScience big issue on Cloud Computing is how to leverage on-demand computing in scientific research. This requires innovation at multiple levels, from architectural design to software platforms. This paper characterizes the requirements of a federated hybrid cloud model of Infrastructure as a Service (IaaS) to provide eScience. Additionally, an architecture is defined for constructing Platform as a Service (PaaS) and Software as a Service (SaaS) in a resilient manner over federated resources. This architecture is named Rafhyc (for Resilient Architecture of Federated HYbrid Clouds). This paper also describes a prototype implementation of the Rafhyc architecture, which integrates an interoperable community middleware, named DIRAC, with federated hybrid clouds. In this way DIRAC is providing SaaS for scientific computing purposes, demonstrating that Rafhyc architecture can bring together eScience and federated hybrid clouds.Federated hybrid clouds is a model of service access and delivery to community cloud infrastructures. This model opens an opportunity window to allow the integration of the enhanced science (eScience) with the Cloud paradigm. The eScience is computationally intensive science that is carried out in highly distributed computing infrastructures. Nowadays, the eScience big issue on Cloud Computing is how to leverage on-demand computing in scientific research. This requires innovation at multiple levels, from architectural design to software platforms. This paper characterizes the requirements of a federated hybrid cloud model of Infrastructure as a Service (IaaS) to provide eScience. Additionally, an architecture is defined for constructing Platform as a Service (PaaS) and Software as a Service (SaaS) in a resilient manner over federated resources. This architecture is named Rafhyc (for Resilient Architecture of Federated HYbrid Clouds). This paper also describes a prototype implementation of the Rafhyc architecture, which integrates an interoperable community middleware, named DIRAC, with federated hybrid clouds. In this way DIRAC is providing SaaS for scientific computing purposes, demonstrating that Rafhyc architecture can bring together eScience and federated hybrid clouds.

Outlook on moving of computing services towards the data sources

We propose a big data architecture to suit the internet of things in data-information and information-knowledge layers.We explore technological possibilities of using nano data centres to move data-information layer towards the source of data.We describe a cloud computing model to integrate the big data analytics in this new internet of things.The proposed architecture alleviates transfers and storage to cope the increasing sizes of data.We present the result of a poll to 300 IT professionals to validate the internet of things adoption in big data. The internet of things (IoT) is potentially interconnecting unprecedented amounts of raw data, opening countless possibilities by two main logical layers: become data into information, then turn information into knowledge. The former is about filtering the significance in the appropriate format, while the latter provides emerging categories of the whole domain. This path of the data is a bottom-up flow. On the other hand, the path of the process is a top-down flow, starting at the strategic level of business and scientific institutions. Today, the path of the process treasures a sizeable amount of well-known methods, architectures and technologies: the so called Big Data. On the top, Big Data analytics aims variable association (e-commerce), data mining (predictive behaviour) or clustering (marketing segmentation). Digging the Big Data architecture there are a myriad of enabling technologies for data taking, storage and management. However the strategic aim is to enhance knowledge with the appropriate information, which does need of data, but not vice versa. In the way, the magnitude of upcoming data from the IoT will disrupt the data centres. To cope with the extreme scale is a matter of moving the computing services towards the data sources. This paper explores the possibilities of providing many of the IoT services which are currently hosted in monolithic cloud centres, moving these computing services into nano data centres (NaDa). Particularly, data-information processes, which usually are performing at sub-problem domains. NaDa distributes computing power over the already present machines of the IP provides, like gateways or wireless routers to overcome latency, storage cost and alleviate transmissions. Large scale questionnaires have been taken for 300 IT professionals to validate the points of view for IoT adoption. Considering IoT is by definition connected to the Internet, NaDa may be used to implement the logical low layer architecture of the services. Obviously, such distributed NaDa send results on a logical high layer in charge of the information-knowledge turn. This layer requires the whole picture of the domain to enable those processes of Big Data analytics on the top.

Integration of Cloud resources in the LHCb Distributed Computing

This contribution describes how Cloud resources have been integrated in the LHCb Distributed Computing. LHCb is using its specific Dirac extension (LHCbDirac) as an interware for its Distributed Computing. So far, it was seamlessly integrating Grid resources and Computer clusters. The cloud extension of DIRAC (VMDIRAC) allows the integration of Cloud computing infrastructures. It is able to interact with multiple types of infrastructures in commercial and institutional clouds, supported by multiple interfaces (Amazon EC2, OpenNebula, OpenStack and CloudStack) – instantiates, monitors and manages Virtual Machines running on this aggregation of Cloud resources. Moreover, specifications for institutional Cloud resources proposed by Worldwide LHC Computing Grid (WLCG), mainly by the High Energy Physics Unix Information Exchange (HEPiX) group, have been taken into account. Several initiatives and computing resource providers in the eScience environment have already deployed IaaS in production during 2013. Keeping this on mind, pros and cons of a cloud based infrasctructure have been studied in contrast with the current setup. As a result, this work addresses four different use cases which represent a major improvement on several levels of our infrastructure. We describe the solution implemented by LHCb for the contextualisation of the VMs based on the idea of Cloud Site. We report on operational experience of using in production several institutional Cloud resources that are thus becoming integral part of the LHCb Distributed Computing resources. Furthermore, we describe as well the gradual migration of our Service Infrastructure towards a fully distributed architecture following the Service as a Service (SaaS) model.

The Integration of CloudStack and OCCI/OpenNebula with DIRAC

The increasing availability of Cloud resources is arising as a realistic alternative to the Grid as a paradigm for enabling scientific communities to access large distributed computing resources. The DIRAC framework for distributed computing is an easy way to efficiently access to resources from both systems. This paper explains the integration of DIRAC with two open-source Cloud Managers: OpenNebula (taking advantage of the OCCI standard) and CloudStack. These are computing tools to manage the complexity and heterogeneity of distributed data center infrastructures, allowing to create virtual clusters on demand, including public, private and hybrid clouds. This approach has required to develop an extension to the previous DIRAC Virtual Machine engine, which was developed for Amazon EC2, allowing the connection with these new cloud managers. In the OpenNebula case, the development has been based on the CernVM Virtual Software Appliance with appropriate contextualization, while in the case of CloudStack, the infrastructure has been kept more general, which permits other Virtual Machine sources and operating systems being used. In both cases, CernVM File System has been used to facilitate software distribution to the computing nodes. With the resulting infrastructure, the cloud resources are transparent to the users through a friendly interface, like the DIRAC Web Portal. The main purpose of this integration is to get a system that can manage cloud and grid resources at the same time. This particular feature pushes DIRAC to a new conceptual denomination as interware, integrating different middleware. Users from different communities do not need to care about the installation of the standard software that is available at the nodes, nor the operating system of the host machine which is transparent to the user. This paper presents an analysis of the overhead of the virtual layer, doing some tests to compare the proposed approach with the existing Grid solution. License Notice: Published under licence in Journal of Physics: Conference Series by IOP Publishing Ltd.

Converging Future Internet, “Things”, and Big Data: A Specification Following NovaGenesis Model

The convergence of Internet of “things” (IoT) with big data platforms and cloud computing is already happening. However, the vast majority, if not all the proposals are based on the current Internet technologies. The convergence of IoT, big data and cloud in “clean slate” architectures is an unexplored topic. In this article, we discuss this convergence considering the viewpoint of a “clean slate” proposal called NovaGenesis. We specify a set of NovaGenesis services to publish sensor device’s data in distributed hash tables employing selfverifying addresses and contract-based trust network formation. IoT devices capabilities and configurations are exposed to software-controllers, which control their operational parameters. The specification covers how the “things” sensed information are subscribed by a big data service and injected in Spark big data platform, allowing NovaGenesis services to subscribe data analytics from Spark. Future work include implementation of the proposed specifications and further investigation of NovaGenesis services performance and scalability.

Cloud flexibility using DIRAC interware

Communities of different locations are running their computing jobs on dedicated infrastructures without the need to worry about software, hardware or even the site where their programs are going to be executed. Nevertheless, this usually implies that they are restricted to use certain types or versions of an Operating System because either their software needs an definite version of a system library or a specific platform is required by the collaboration to which they belong. On this scenario, if a data center wants to service software to incompatible communities, it has to split its physical resources among those communities. This splitting will inevitably lead to an underuse of resources because the data centers are bound to have periods where one or more of its subclusters are idle. It is, in this situation, where Cloud Computing provides the flexibility and reduction in computational cost that data centers are searching for. This paper describes a set of realistic tests that we ran on one of such implementations. The test comprise software from three different HEP communities (Auger, LHCb and QCD phenomelogists) and the Parsec Benchmark Suite running on one or more of three Linux flavors (SL5, Ubuntu 10.04 and Fedora 13). The implemented infrastructure has, at the cloud level, CloudStack that manages the virtual machines (VM) and the hosts on which they run, and, at the user level, the DIRAC framework along with a VM extension that will submit, monitorize and keep track of the user jobs and also requests CloudStack to start or stop the necessary VMs. In this infrastructure, the community software is distributed via the CernVM-FS, which has been proven to be a reliable and scalable software distribution system. With the resulting infrastructure, users are allowed to send their jobs transparently to the Data Center. The main purpose of this system is the creation of flexible cluster, multiplatform with an scalable method for software distribution for several VOs. Users from different communities do not need to care about the installation of the standard software that is available at the nodes, nor the operating system of the host machine, which is transparent to the user.

Towards Combining Reactive and Proactive Cloud Elasticity on Running HPC Applications.

The elasticity feature of cloud computing has been proved as pertinent for parallel applications, since users do not need to take care about the best choice for the number of processes/resources beforehand. To accomplish this, the most common approaches use threshold-based reactive elasticity or time-consuming proactive elasticity. However, both present at least one problem related to: the need of a previous user experience, lack on handling load peaks, completion of parameters or design for a specific infrastructure and workload setting. In this regard, we developed a hybrid elasticity service for parallel applications named SelfElastic. As parameterless model, SelfElastic presents a closed control loop elasticity architecture that adapts at runtime the values of lower and upper thresholds. Besides presenting SelfElastic, our purpose is to provide a comparison with our previous work on reactive elasticity called AutoElastic. The results present the SelfElastic’s lightweight feature, besides highlighting its performance competitiveness in terms of application time and cost metrics.

Journal of Grid Computing, Special Issue of Cloud Computing and Services Science

The stated mission of The Journal of Grid Computing is exploring “an emerging technology that enables large-scale resource sharing problem solving within distributed, loosely coordinated groups.” The technologies in the journal’s scope are extremely broad and deep, and fundamental to modern computing infrastructure and solutions. Cloud Computing focuses on delivering physical hardware, Infrastructure-as-a-Service (IaaS), software infrastructure for solutions (Platform-as-a-Service) and complete applications or application APIs as a service (SaaS). Advances in cloud computing support and are critical to advances in grid computing. Equally important is the concept of services. At a technical level, Service Oriented Architecture (SOA) is core to how clouds implement use of and management of the resources they provide. More generally, the cloud is a network of business services providing technical, administrative, regulatory and financial capabilities necessary for resource sharing and problem solving. The importance and interrelatedness of grid computing, cloud computing and services sciences motivates a special issue on these topics.

Easing Scientific Computing and Federated Management in the Cloud with OCCI

One of the benefits of OCCI stems from simplifying the life of developers aiming to integrate multiple cloud managers. It provides them with a single protocol to abstract the differences between cloud service implementations used on sites run by different providers. This comes particularly handy in federated clouds, such as the EGI Federated Cloud Platform, which bring together providers who run different cloud management platforms on their sites: most notably OpenNebula, OpenStack, or Synnefo. Thanks to the wealth of approaches and tools now available to developers of virtual resource management solutions, different paths may be chosen, ranging from a small-scale use of an existing command line client or single-user graphical interface, to libraries ready for integration with large workload management frameworks and job submission portals relied on by large science communities across Europe. From lone wolves in the long-tail of science to virtual organizations counting thousands of users, OCCI simplifies their life through standardization, unification, and simplification. Hence cloud applications based on OCCI can focus on user specifications, saving cost and reaching a robust development life-cycle. To demonstrate this, the paper shows several EGI Federated Cloud experiences, demonstrating the possible approaches and design principles.

Multivariate Analysis of Variance for High Energy Physics Software in Virtualized Environments

Victor Fernandez Albor1, Marcos Seco1, Victor Mendez Munoz2, Tomas Fernandez Pena3, Juan Saborido Silva1 and Ricardo Graciani Diaz4 1 Physics department, Santiago de Compostela University Av Ciencias sn, Santiago de Compostela, Spain E-mail: {victormanuel.fernandez,marcos.seco,juan.saborido}@usc.es 2 Computer Architecture and Operating Systems (CAOS),Universidad Autonoma de Barcelona E-mail: victor.mendez@uab.es 3 Research Center in Information Technologies (CiTIUS), Santiago de Compostela University Av Ciencias sn, Santiago de Compostela, Spain E-mail: tf.pena@usc.es 4 Departamento de Estructura y Constituyentes de la Materia,Barcelona University E-mail: Ricardo.graciani@ecm.ub.es