Adam K. L. Wong

A unified framework for the deployment, exposure and access of HPC applications as services in clouds

Clouds have provided on-demand, scalable and affordable High Performance Computing (HPC) resources to discipline (e.g., Biology, Medicine, Chemistry) scientists. However, the steep learning curve of preparing a HPC cloud and deploying HPC applications has hindered many scientists to achieve innovative discoveries for which HPC resources must be relied on. With the world moving to web-based tools, scientists are also seeking more web-based technologies to support their research. Unfortunately, the discipline problems of high-performance computational research are both unique and complex, which make the development of web-based tools for this research difficult. This paper presents our work on developing a unified cloud framework that allows discipline users to easily deploy and expose HPC applications in public clouds as services. To provide a proof of concept, we have implemented the cloud framework prototype by integrating three components: (i) Amazon EC2 public cloud for providing HPC infrastructure, (ii) a HPC service software library for accessing HPC resources, and (iii) the Galaxy web-based platform for exposing and accessing HPC application services. This new approach can reduce the time and money needed to deploy, expose and access discipline HPC applications in clouds.

Evaluating the EASY-backfill job scheduling of static workloads on clusters

This research aims at improving our understanding of backfilling job scheduling algorithms. The most frequently used algorithm, EASY-backfilling, was selected for a performance evaluation by scheduling static workloads of parallel jobs on a computer cluster. To achieve the aim, we have developed a batch job scheduler for Linux clusters, implemented several scheduling algorithms including ARCA and EASY-Backfilling, and carried out their performance evaluation by running well known MPI applications on a real cluster. Our performance evaluation carried out for EASY-Backfilling serves two purposes. First, the performance results obtained from our evaluation can be used to validate other researcherspsila results generated by simulation, and second, the methodology used in our evaluation has alleviated many problems existed in the simulations presented in the current literature.

Toward Exposing and Accessing HPC Applications in a SaaS Cloud

The cost and time of deploying HPC applications on clouds is a problem. Instead of conducting their research discipline specialists are forced to carry out activities for application deployment, publication and ease of access. In response, a new approach for HPC application deployment and access in clouds is proposed. The major innovations are a new approach to deploying and executing HPC applications on IaaS and PaaS clouds, and exposing HPC applications as services. Through three case studies this paper demonstrates the feasibility and effectiveness of the proposed approach that could lead to the building of a SaaS library of discipline-oriented services evocable through user friendly, discipline specific interfaces. The new approach will reduce the time and money needed to deploy and expose discipline HPC applications.

A VMD Plugin for NAMD Simulations on Amazon EC2

VMD and NAMD are two major molecular dynamics simulation software packages, which can work together for mining structural information of bio-molecules. Carrying out such molecular dynamics simulations can help researchers to understand the roles and functions of various bio-molecules in life science research. Recently, clouds have provided HPC clusters on demand that allow users to benefit from their flexibility, elasticity, and lower costs. Although cloud computing promises to provide seamless access to HPC clusters through the abstraction of services, which hide the details of the underlying software and hardware infrastructure, users without in depth computing knowledge are still forced to cope with many low level system and programming details. Therefore, we have designed and developed a software plugin of VMD, which can provide an integrated framework for NAMD to be executed on Amazon EC2. The proposed Amazon EC2 Plugin for VMD frees users from performing many tedious computing tasks such as launching, connecting and terminating Amazon EC2 compute instances; configuring a HPC cluster; and installing middleware and software applications before the system is readily available for any scientific investigation. This allows VMD/NAMD users to spend less time getting applications to work on HPC clusters but more time for bio-research.

Performance evaluation of the concurrent execution of NAS parallel benchmarks with BYTE sequential benchmarks on a cluster

Computers of a non-dedicated cluster are often idle (users attend meetings, have lunch or coffee breaks) or lightly loaded (users carry out simple computations). These underutilized computers can be employed to execute parallel applications not only during weekends and at nights but also during office hours. Thus, they have to be shared by parallel and sequential applications which could lead to the improvement of their execution performance. However, there is a lack of experimental study showing the behavior and performance of parallel and sequential applications executing concurrently on clusters. We present here the result of an experimental study into load balancing based scheduling of a mixture of parallel and sequential applications on a non-dedicated cluster.

Scheduling of a parallel computation-bound application and sequential applications executing concurrently on a cluster: a case study

Studies have shown that most of the computers in a non-dedicated cluster are often idle or lightly loaded. The underutilized computers in a non-dedicated cluster can be employed to execute parallel applications. The aim of this study is to learn how concurrent execution of a computation-bound and sequential applications influence their execution performance and cluster utilization. The result of the study has demonstrated that a computation-bound parallel application benefits from load balancing, and at the same time sequential applications suffer only an insignificant slowdown of execution. Overall, the utilization of a non-dedicated cluster is improved.

The Impact of Under-Estimated Length of Jobs on EASY-Backfill Scheduling

The issue of under-estimated length of jobs (parallel applications) on backfill-based scheduling is ignored in the current literature because users want to avoid their jobs to be killed when the requested time expires. Therefore, users prefer to over-estimate the length of their jobs. This paper shows the impact of underestimated length of jobs on their execution performance in an EASY-backfill scheduling-based system. We have developed a batch job scheduler for Linux clusters that implements an enhanced EASY- backfilling algorithm in such a way that a job with an under-estimated execution time would not be killed unless it would delay other jobs. We have carried out performance evaluation by scheduling static workloads of well known MPI parallel applications on a real cluster. Our results show that most of the jobs do not have to be aborted even though their job lengths are under-estimated whereas the slowdown of jobs and the throughput of the system are only slightly degraded.

Using an enterprise grid for execution of MPI parallel applications: a case study

An enterprise has not only a single cluster but a set of geographically distributed clusters – they could be used to form an enterprise grid. In this paper we show based on our case study that enterprise grids could be efficiently used as parallel computers to carry out high-performance computing.

Execution environments and benchmarks for the study of applications’ scheduling on clusters

In this paper, we have demonstrated how the existing programming environments, tools and middleware could be used for the study of execution performance of parallel and sequential applications on a non-dedicated cluster. A set of parallel and sequential benchmark applications selected for and used in the experiments were characterized, and experiment requirements shown.

Concurrent execution of multiple NAS parallel programs on a cluster

Currently, coordinated scheduling of multiple parallel applications across computers has been considered as the critical factor to achieve high execution performance. We claim in this report that the performance and costs of the execution of parallel applications could be improved if not only dedicated clusters but also non-dedicated clusters were used and several parallel applications were executed concurreontly. To support this claim we carried out experimental study into the performance of multiple NAS parallel programs executing concurrently on a non-dedicated cluster.