Maurício O. Tsugawa

CloudBLAST: Combining MapReduce and Virtualization on Distributed Resources for Bioinformatics Applications

This paper proposes and evaluates an approach to the parallelization, deployment and management of bioinformatics applications that integrates several emerging technologies for distributed computing. The proposed approach uses the MapReduce paradigm to parallelize tools and manage their execution, machine virtualization to encapsulate their execution environments and commonly used data sets into flexibly deployable virtual machines, and network virtualization to connect resources behind firewalls/NATs while preserving the necessary performance and the communication environment. An implementation of this approach is described and used to demonstrate and evaluate the proposed approach. The implementation integrates Hadoop, Virtual Workspaces, and ViNe as the MapReduce, virtual machine and virtual network technologies, respectively, to deploy the commonly used bioinformatics tool NCBI BLAST on a WAN-based test bed consisting of clusters at two distinct locations, the University of Florida and the University of Chicago. This WAN-based implementation, called CloudBLAST, was evaluated against both non-virtualized and LAN-based implementations in order to assess the overheads of machine and network virtualization, which were shown to be insignificant. To compare the proposed approach against an MPI-based solution, CloudBLAST performance was experimentally contrasted against the publicly available mpiBLAST on the same WAN-based test bed. Both versions demonstrated performance gains as the number of available processors increased, with CloudBLAST delivering speedups of 57 against 52.4 of MPI version, when 64 processors on 2 sites were used. The results encourage the use of the proposed approach for the execution of large-scale bioinformatics applications on emerging distributed environments that provide access to computing resources as a service.

Sky Computing

Infrastructure-as-a-service (IaaS) cloud computing is revolutionizing how we approach computing. Compute resource consumers can eliminate the expense inherent in acquiring, managing, and operating IT infrastructure and instead lease resources on a pay-as-you-go basis. IT infrastructure providers can exploit economies of scale to mitigate the cost of buying and operating resources and avoid the complexity required to manage multiple customer-specific environments and applications. The authors describe the context in which cloud computing arose, discuss its current strengths and shortcomings, and point to an emerging computing pattern it enables that they call sky computing.

From virtualized resources to virtual computing grids: the In-VIGO system

This paper describes the architecture of the first implementation of the In-VIGO grid-computing system. The architecture is designed to support computational tools for engineering and science research In Virtual Information Grid Organizations (as opposed to in vivo or in vitro experimental research). A novel aspect of In-VIGO is the extensive use of virtualization technology, emerging standards for grid-computing and other Internet middleware. In the context of In-VIGO, virtualization denotes the ability of resources to support multiplexing, manifolding and polymorphism (i.e. to simultaneously appear as multiple resources with possibly different functionalities). Virtualization technologies are available or emerging for all the resources needed to construct virtual grids which would ideally inherit the above mentioned properties. In particular, these technologies enable the creation of dynamic pools of virtual resources that can be aggregated on-demand for application-specific user-specific grid-computing. This change in paradigm from building grids out of physical resources to constructing virtual grids has many advantages but also requires new thinking on how to architect, manage and optimize the necessary middleware. This paper reviews the motivation for In-VIGO approach, discusses the technologies used, describes an early architecture for In-VIGO that represents a first step towards the end goal of building virtual information grids, and reports on first experiences with the In-VIGO software under development.

A virtual network (ViNe) architecture for grid computing

This paper describes a virtual networking approach for grids called ViNe. It enables symmetric connectivity among grid resources and allows existing applications to run unmodified. Novel features of the ViNe architecture include: easy virtual networking administration; support for physical private networks and support for multiple independent virtual networks in the same infrastructure. The requirements of an application-friendly virtual network environment are presented and it is shown how the proposed solution meets them. Qualitative arguments are provided to justify all design decisions. Also presented is an experimental evaluation of the round-trip latencies and bandwidths achieved by a reference implementation. Measurements are reported for WAN-scenarios involving three different institutions. Under favorable conditions, ViNe bandwidths are within 90 to 100% of the available physical network bandwidth

Transnational Information Sharing, Event Notification, Rule Enforcement and Process Coordination

Solutions to global problems such as disease detection and control, terrorism, immigration and border control, and illicit drug trafficking require sharing and coordinating information and collaboration among government agencies within a country and across national boundaries. This paper presents an approach to achieve information sharing, event notification, enforcement of policies, constraints, regulations, security and privacy rules, and process coordination. The proposed system, designed in collaboration with stakeholders and end users in two Latin American countries, achieves the desired capabilities by integrating a distributed query processor (DQP) that provides form-based and conversational user interfaces, a language translation system, an event server for event filtering and notification, and an event-trigger-rule server. The Web-services infrastructure is used to achieve the interoperation of these heterogeneous component systems. A prototype of the integrated transnational information system is described.

Single sign-on in In-VIGO: role-based access via delegation mechanisms using short-lived user identities

Summary form only given. Single sign-on (SSO) is an essential desired feature of computational grids. Its implementation is challenging because resources cross administrative domains and are managed by heterogeneous access schemes. We present an approach for single sign-on in a deployed functioning grid called In-VIGO. The approach relies on decoupling grid user accounts from local user accounts and making use of role-based access control lists. Role-based accesses via delegation mechanisms using short-lived user identities enable In-VIGO to handle interactive applications and application-specific authentication mechanisms. This capability is not present in existing grid architectures. SSO implementations for usage scenarios in In-VIGO are described to highlight the applicability of the proposed approach. In particular, access to interactive applications with their own security mechanisms, such as VNC, and access to remote data can be achieved using proxies that delegate In-VIGO user access via short-lived user identities.

User-level virtual networking mechanisms to support virtual machine migration over multiple clouds

Dynamic allocation of multiple cloud resources adapting to application needs over time can be achieved by taking advantage of wide-area VM live migration technologies. However, migration of VMs across different subnets, potentially in multiple clouds, requires networking support to keep the network state of moving VMs unchanged. Two problems make traditional solutions to machine mobility inefficient in this scenario: (1) administrative overheads due to coordination requirements between moving machines and the network infrastructure; and (2) degraded network performance of machines moved away from their “home” networks. New solutions are needed to efficiently support the migration of virtual machines over multiple cloud providers. The user-level virtual network architecture presented in this paper implements mechanisms to allow VM migration over clouds without requiring support from the physical network infrastructure, and automatically reconfiguring virtual networks to maximize the network performance of migrated virtual machines.

User-Level Virtual Network Support for Sky Computing

With the emergence of multiple cloud providers of Infrastructure-as-a-Service, it becomes possible to envision a near-future when high-performance computing users could combine services from different clouds to access huge numbers of resources. However, as more administrative privileges are exposed to end users, providers are required to deploy network security measures that present challenges to the network virtualization technologies that are needed to enable inter-cloud communication. This paper studies these challenges and proposes techniques to enable unmodified applications on resources across distinct clouds. The techniques are implemented in TinyViNe, an extension to ViNe, a virtual networking technology for distributed resources in different administrative domains. The results of evaluating TinyViNe on a WAN-based testbed across three sites are reported for a bioinformatics application (BLAST) and MPI benchmarks. The results confirm that TinyViNe enables cross-cloud computing while having little impact on application performance. TinyViNe also has auto-configuration and “download-and-run” capabilities for easy deployment by users who are not knowledgeable about networking.

Science gateways made easy: the In‐VIGO approach

Science gateways require the easy enabling of legacy scientific applications on computing Grids and the generation of user-friendly interfaces that hide the complexity of the Grid from the user. This paper presents the In-VIGO approach to the creation and management of science gateways. First, we discuss the virtualization of machines, networks and data to facilitate the dynamic creation of secure execution environments that meet application requirements. Then we discuss the virtualization of applications, i.e. the execution on shared resources of multiple isolated application instances with customized behavior, in the context of In-VIGO. A Virtual Application Service (VAS) architecture for automatically generating, customizing, deploying, and using virtual applications as Grid services is then described. Starting with a grammar-based description of the command-line syntax, the automated process generates the VAS description and the VAS implementation (code for application encapsulation and data binding) that is deployed and made available through a Web interface. A VAS can be customized on a per-user basis by restricting the capabilities of the original application or by adding to it features such as parameter sweeping. This is a scalable approach to the integration of scientific applications as services into Grids and can be applied to any tool with an arbitrarily complex command-line syntax. Copyright

On the use of virtualization and service technologies to enable grid-computing

The In-VIGO approach to Grid-computing relies on the dynamic establishment of virtual grids on which application services are instantiated. In-VIGO was conceived to enable computational science to take place In Virtual Information Grid Organizations. Having its first version deployed on July of 2003, In-VIGO middleware is currently used by scientists from various disciplines, a noteworthy example being the computational nanoelectronics research community (http://www.nanohub.org). All components of an In-VIGO-generated virtual grid – machines, networks, applications and data – are themselves virtual and services are provided for their dynamic creation. This article reviews the In-VIGO approach to Grid-computing and overviews the associated middleware techniques and architectures for virtualizing Grid components, using services for creation of virtual grids and automatically Grid-enabling unmodified applications. The In-VIGO approach to the implementation of virtual networks and virtual application services are discussed as examples of Grid-motivated approaches to resource virtualization and Web-service creation.