Khawaja S. Shams

Neptune: a domain specific language for deploying hpc software on cloud platforms

In this paper, we present the design and implementation of Neptune, a domain specific language (DSL) that automates configuration and deployment of existing HPC software via cloud computing platforms. We integrate Neptune into a popular, open-source cloud platform, and extend the platform with support for user-level and automated placement of cloud services and HPC components. Such platform integration of Neptune facilitates hybrid-cloud application execution as well as portability across disparate cloud fabrics. Supporting additional cloud fabrics through a single interface enables high throughput computing (HTC) to be achieved by users who do not necessarily own grid-level resources but do have access to otherwise independent cloud technologies. We evaluate Neptune using different applications that employ a wide range of popular HPC packages for their implementation including MPI, X10, MapReduce, DFSP, and dwSSA. In addition, we show how Neptune can be extended to support other HPC software and application domains, and thus be used as a mechanism for many task computing (MTC).

A Scalable Image Processing Framework for gigapixel Mars and other celestial body images

The Mars Reconnaissance Orbiters HiRISE (High Resolution Imaging Science Experiment) camera takes the largest images of the Martian surface. The image size is typically around 2.52 gigapixels. There is only a handful of software capable of doing a task as simple as reducing the size of the image by half and saving the result as a new image. The Scalable Image Processing Framework (SIPF) overcomes these issues by creating a generalized tile-based processing pipeline that loads only a small portion of the image into memory. This allows for the data in memory at any given time to become manageable. Image tiles are an intrinsic property that provides scalability and efficiency while processing images. Distributed computing technologies such as cloud computing can be applied naturally. A mathematical framework for scalable image operations is defined that provides insight into the scalable considerations needed with each class of operations. We also formalize the deferred execution design pattern and show how it is used as a basis for our implementation. The SIPF has the ability to perform a variety of Scalable Image Operations such as Cropping, Rotation, Scaling (Bilinear and Nearest Neighbor Interpolation), Edge Detection, Sharpening, Convolution (Filters), Brightness, Contrast, and Gaussian Blurring. The Scalable Image Processing Framework will be used to process incoming images from the Mars Exploration Rovers and eventually the Mars Science Laboratory. It will be integrated with the Maestro software (science visualization and planning tool). Maestro is used for the Mars Exploration Rover Mission and other celestial body exploratory missions. 1 2

Polyphony: A Workflow Orchestration Framework for Cloud Computing

Cloud Computing has delivered unprecedented compute capacity to NASA missions at affordable rates. Missions like the Mars Exploration Rovers (MER) and Mars Science Lab (MSL) are enjoying the elasticity that enables them to leverage hundreds, if not thousands, or machines for short durations without making any hardware procurements. In this paper, we describe Polyphony, a resilient, scalable, and modular framework that efficiently leverages a large set of computing resources to perform parallel computations. Polyphony can employ resources on the cloud, excess capacity on local machines, as well as spare resources on the supercomputing center, and it enables these resources to work in concert to accomplish a common goal. Polyphony is resilient to node failures, even if they occur in the middle of a transaction. We will conclude with an evaluation of a production-ready application built on top of Polyphony to perform image-processing operations of images from around the solar system, including Mars, Saturn, and Titan.

Language and Runtime Support for Automatic Configuration and Deployment of Scientific Computing Software over Cloud Fabrics

In this paper, we present the design and implementation of Neptune, a simple, domain-specific language based on the Ruby programming language. Neptune automates the configuration and deployment of scientific software frameworks over disparate cloud computing systems. Neptune integrates support for MPI, MapReduce, UPC, X10, StochKit, and others. We implement Neptune as a software overlay for the AppScale cloud platform and extend AppScale with support for elasticity and hybrid execution for scientific computing applications. Neptune imposes no overhead on application execution, yet significantly simplifies the application deployment process, enables portability across cloud systems, and promotes lock-in avoidance by specific cloud vendors.

Delivering Images for Mars Rover Science Planning

Mars rover images provide essential context for planning science activities. This work describes a method for delivering Mars rover images to operations planners that is highly efficient and scalable. Experimental results of various image compression strategies applied to rover images are given. Next, an adaptive level-of-detail tile-based delivery methodology for images is presented. With a tile-aware image browsing application, images of virtually limitless size may be distributed to participating scientists with great efficiency and thus provide a common collaborative context. This work also describes advances in mosaicking rover images in support of operations planning.

Evaluating the efficacy of the cloud for cluster computation

Computing requirements vary by industry, and it follows that NASA and other research organizations have computing demands that fall outside the mainstream. While cloud computing made rapid inroads for tasks such as powering web applications, performance issues on highly distributed tasks hindered early adoption for scientific computation. One venture to address this problem is Nebula, NASAs homegrown cloud project tasked with delivering science-quality cloud computing resources. However, another industry development is Amazons high-performance computing (HPC) instances on Elastic Cloud Compute (EC2) that promises improved performance for cluster computation. This paper presents results from a series of benchmarks run on Amazon EC2 and discusses the efficacy of current commercial cloud technology for running scientific applications across a cluster. In particular, a 240-core cluster of cloud instances achieved 2 TFLOPS on High-Performance Linpack (HPL) at 70% of theoretical computational performance. The clusters local network also demonstrated sub-100 μs inter-process latency with sustained inter-node throughput in excess of 8 Gbps. Beyond HPL, a real-world Hadoop image processing task from NASAs Lunar Mapping and Modeling Project (LMMP) was run on a 29 instance cluster to process lunar and Martian surface images with sizes on the order of tens of gigapixels. These results demonstrate that while not a rival of dedicated supercomputing clusters, commercial cloud technology is now a feasible option for moderately demanding scientific workloads.

Advances in Distributed Operations and Mission Activity Planning for Mars Surface Exploration

A centralized mission activity planning system for any long-term mission, such as the Mars Exploration Rover Mission (MER), is completely infeasible due to budget and geographic constraints. A distributed operations system is key to addressing these constraints; therefore, future system and software engineers must focus on the problem of how to provide a secure, reliable, and distributed mission activity planning system. We will explain how Maestro, the next generation mission activity planning system, with its heavy emphasis on portability and distributed operations has been able to meet these design challenges. MER has been an excellent proving ground for Maestros new approach to distributed operations. The backend that has been developed for Maestro could benefit many future missions by reducing the cost of centralized operations system architecture.

Enabling earth science through cloud computing

Cloud Computing holds tremendous potential for missions across the National Aeronautics and Space Administration. Several flight missions are already benefiting from an investment in cloud computing for mission critical pipelines and services through faster processing time, higher availability, and drastically lower costs available on cloud systems. However, these processes do not currently extend to general scientific algorithms relevant to earth science missions. The members of the Airborne Cloud Computing Environment task at the Jet Propulsion Laboratory have worked closely with the Carbon in Arctic Reservoirs Vulnerability Experiment (CARVE) mission to integrate cloud computing into their science data processing pipeline. This paper details the efforts involved in deploying a science data system for the CARVE mission, evaluating and integrating cloud computing solutions with the system and porting their science algorithms for execution in a cloud environment.

Mission critical cloud computing in a week

NASAs vision is to “reach for new heights and reveal the unknown so that what we do and learn will benefit all humankind.” While our missions provide large volumes of unique and invaluable data to the scientific community, they also serve to inspire and educate the next generation of engineers and scientists. One critical aspect of “benefiting all humankind” is to make our missions as visible and accessible as possible to facilitate the transfer of scientific knowledge to the public. The recent successful landing of the Curiosity rover on Mars exemplified this vision: we shared the landing event via live video streaming and web experiences with millions of people around the world. The video stream on Curiositys website was delivered by a highly scalable stack of computing resources in the cloud to cache and distribute the video stream to our viewers. While this work was done in the context of public outreach, it has extensive implications for the development of mission critical, highly available, and elastic applications in the cloud for a diverse set of use cases across NASA.

Cloud Sourcing Cycles: How Cloud Computing is Revolutionizing NASA Mission Operations

Developers across the Jet Propulsion Laboratory are leveraging cloud computing on a variety of problems on multiple missions. In this paper, we explore cloud computing and its potential to revolutionize spacecraft operations. We describe concrete applications that have been developed and integrated with ground data systems. We will evaluate the role cloud computing is playing in the development of MSLICE (Mars Science Laboratory Interface), the primary software tool to control the (MSL) Mars Science Laboratory Rover.