Stephen Lien Harrell

Cost-Effective HPC: The Community or the Cloud?

The increasing availability of commercial cloud computing resources in recent years has caught the attention of the high-performance computing (HPC) and scientific computing community. Many researchers have subsequently examined the relative computational performance of commercially available cloud computing offerings across a number of HPC application bench-marks and scientific workflows, but the analogous cost comparisons—i.e., comparisons between the cost of doing scientific computation in traditional HPC environments vs. cloud computing environments—are less frequently discussed and are difficult to make in meaning-ful ways. Such comparisons are of interest to traditional HPC resource providers as well as to members of the scientific research community who need access to HPC resources on a routine basis. This paper is a case study of costs incurred by faculty end-users of Purdue University’s HPC “community cluster” program. We develop and present a per node-hour cloud computing equivalent cost that is based upon actual usage patterns of the community cluster participants and is suitable for direct comparison to hourly costs charged by one commercial cloud computing provider. We find that the majority of community cluster participants incur substantially lower out-of-pocket costs in this community cluster program than in purchasing cloud computing HPC products.

Student cluster competition: a multi-disciplinary undergraduate HPC educational tool

National labs, academic institutions and industry have a strong need for scientists and staff that understand high performance computing (HPC) and the complex interconnections across individual topics in HPC. However, domain science and computer science undergraduate programs are not providing sufficient educational resources, and are far from conveying the interdisciplinary and collaborative nature of the HPC environment. The Student Cluster Competition (SCC) was created as an educational tool to immerse undergraduates in HPC. It is a microcosm of professional HPC centers that teaches and inspires students to pursue careers in the field. The SCCs impact is reflected in new undergraduate curricula and through the experience of the students themselves. The SCC can complement a strong parallel and distributed computing (PDC) curriculum through experiential learning and engagement with the HPC community as a whole, which will prepare graduates for the interdisciplinary nature of work in HPC fields.

Methods of creating student cluster competition teams

This paper aims to describe methods that can be used to create new Student Cluster Competition teams from the standpoint of the team advisors. The purpose is to share these methods in order to create an easier path for organizing a successful team. These methods were gleaned from a survey of advisors that have formed teams in the last four years. Four advisors responded to the survey and those responses fit into five categories: (1) early preparation, (2) coursework specific to the competition, (3) close relationships with the hardware vendors, (4) concentration on the applications over the hardware, and (5) the need to encourage the team members to write papers about their experiences. In addition to these commonalities which may be best practices there are a few divergent but intriguing techniques that may also prove useful for potential advisors. Both will be discussed here and these methods can serve as a primer for anyone looking to start a new Student Cluster Competition team.

Linux Clusters Institute Workshops: Building the HPC and Research Computing Systems Professionals Workforce

We discuss training workshops run by the Linux Clusters Institute (LCI), which provides education and advanced technical training for IT professionals who deploy and support High Performance Computing (HPC) Linux clusters, which have become the most ubiquitous tools for HPC worldwide. The LCI offers workshops that cover the basics of Linux HPC cluster system administration, including hardware (computing, storage, and networking); system-level software (e.g., provisioning systems, resource ma nagers, and job schedulers); system security; and user support. These workshops also aim to seed an HPC systems professional community of practice, by bringing together groups of research computing professionals to obtain essential training in cluster administration, while interacting with the experienced HPC systems professionals who serve as instructors and mentors.

Windows-based Workflows on Linux-based Beowulf Clusters

Scientists with non-traditional computational and workflow needs are a growing demographic in Research Computing. In order to serve these scientific communities we must create new ways to leverage existing resources. One set of problems that is not properly served revolves around Microsoft Windows-based software. These softwares can be both legacy software that have no modern counterpart or traditionally GUI software that have had batch components integrated into them. In this paper we describe a general solution and three successful use-cases using Microsoft Windows Virtual Machines in both interactive and non-interactive batch jobs on Linux-based Beowulf-style clusters to complete workflows based on Microsoft Windows software. With this general solution the utility of ubiquitous Beowulf clusters can be extended.

There and Back Again: A Case Study of Configuration Management of HPC

Configuration management is a critical tool in the management of large groups of computer systems which are vital to deployment of High Performance Computing (HPC). In this paper, we describe the history, architecture, overarching goals, and outcomes of various configuration management systems utilized in support of HPC at Purdue University. Additionally, we enumerate best practices of configuration management that have been discovered in the strongly iterative HPC environment at Purdue.

A Study of Failures in Community Clusters: The Case of Conte

Large community clusters are becoming increasingly common in universities and other organizations due to the benefits they provide to the researchers in terms of operational costs and resource availability. However, efficient administration, failure diagnosis, and performance debugging on community clusters are challenging tasks due to the sheer diversity of workloads and users. These clusters are typically shared by users coming from various scientific domains and experience levels. Many users have little experience in computing and, hence, often face performance issues—leading to resource wastage. In this paper, we study these dynamics in one of the largest university-wide community clusters (Conte at Purdue University). We perform in-depth analysis of library and application usage patterns, job failures and performance issues. Further, we introduce a set of novel analysis techniques that can be used to identify hidden trends and diagnose job failures in compute clusters in general. We provide concrete recommendations for the cluster administrators and present case studies highlighting how such information can be used to proactively solve many user issues, ultimately leading to better quality of service.