Kenneth Yoshimoto

Impact of reservations on production job scheduling

The TeraGrid is a closely linked community of diverse resources: computational, data, and experimental, e.g., the imminent very large computational system at the University of Texas, the extensive data facilities at SDSC, and the physics experiments at ORNL. As research efforts become more extensive in scope, the co-scheduling of multiple resources becomes an essential part of scientific progress. This can be at odds with the traditional management of the computational systems, where utilization, queue wait times, and expansion factors are considered paramount and anything that affects their performance is considered with suspicion. The only way to assuage concerns is with intensive investigation of the likely effects of allowing advance reservations on these performance metrics. To understand the impact, we developed a simulator that reads our actual production job log and reservation request data to investigate different scheduling scenarios. We explored the effect of reservations and policies using job log data from two different months within consecutive years and present our initial results. Results from the simulations suggest that utilization, expansion factor and queue wait time indeed can be affected negatively by significant numbers and size of reservations, but this effect can be mitigated with appropriate policies.

Co-scheduling with user-settable reservations

As grid computing becomes more commonplace, so does the importance of coscheduling these geographically distributed resources. Negotiating resource management and scheduling decisions for these resources is similar to making travel arrangements: guesses are made and then remade or confirmed depending on the availability of resources. This “Travel Agent Method” serves as the basis for a production scheduler and metascheduler suitable for making travel arrangements for a grid. This strategy is more easily implemented than centralized metascheduler because arrangements can be made without requiring control over the individual schedulers for each resource: the reservations are set by users or automatically by negotiating with each local schedulers user settable interface. The Generic Universal Remote is a working implementation of such a system and proves that a user-settable reservation facility on local schedulers in a grid is sufficient to enable automated metascheduling.

Implementations of Urgent Computing on Production HPC Systems

Abstract The requirements of urgent computing applications distinguish those applications from other jobs running within a production batch system. To accomodate urgent computing, specific strategies may be used to allow fast job start time, while mitigating adverse effects on the other parts of the workload. San Diego Supercomputer Center (SDSC) has provided high-performance computing resources in a production-oriented environment to urgent computing appli-cations using several techniques. Processing of data, triggered by real-world disaster events has been accomplished. An on-demand reservation capability to support urgent computing applications has been developed and is currently available on the SDSC Trestles machine.

Apache Airavata Sharing Service: A Tool for Enabling User Collaboration in Science Gateways

Science Gateways provide user environments and a set of supporting services that help researchers make effective and enhanced use of a diverse set of computing, storage, and related resources. Gateways provide the services and tools users require to enable their scientific exploration, which includes tasks such as running computer simulations or performing data analysis. Historically gateways have been constructed to support the workflow of individual users, but collaboration between users has become an increasingly important part of the discovery process. This trend has created a driving need for gateways to support data sharing between users. For example, a chemistry research group may want to run simulations collaboratively, analyze experimental data or tune parameter studies based on simulation output generated by peers, whether as a default capability, or through explicit creation of sharing privileges. As another example, students in a classroom setting may be required to share their simulation output or data analysis results with the instructor. However most existing gateways (including the popularly used XSEDE gateways SEAGrid, Ultrascan, CIPRES, and NSG), do not support direct data sharing, so users have to handle these collaborations outside the gateway environment. Given the importance of collaboration in current scientific practice, user collaboration should be a prime consideration in building science gateways. In this work, we present design considerations and implementation of a generic model that can be used to describe and handle a diverse set of user collaboration use cases that arise in gateways, based on general requirements gathered from the SEAGrid, CIPRES, and NSG gateways. We then describe the integration of this sharing service into these gateways. Though the model and the system were tested and used in the context of Science Gateways, the concepts are universally applicable to any domain, and the service can support data sharing in a wide variety of use cases.

Early experiences in developing and managing the neuroscience gateway

The last few decades have seen the emergence of computational neuroscience as a mature field where researchers are interested in modeling complex and large neuronal systems and require access to high performance computing machines and associated cyber infrastructure to manage computational workflow and data. The neuronal simulation tools, used in this research field, are also implemented for parallel computers and suitable for high performance computing machines. But using these tools on complex high performance computing machines remains a challenge because of issues with acquiring computer time on these machines located at national supercomputer centers, dealing with complex user interface of these machines, dealing with data management and retrieval. The Neuroscience Gateway is being developed to alleviate and/or hide these barriers to entry for computational neuroscientists. It hides or eliminates, from the point of view of the users, all the administrative and technical barriers and makes parallel neuronal simulation tools easily available and accessible on complex high performance computing machines. It handles the running of jobs and data management and retrieval. This paper shares the early experiences in bringing up this gateway and describes the software architecture it is based on, how it is implemented, and how users can use this for computational neuroscience research using high performance computing at the back end. We also look at parallel scaling of some publicly available neuronal models and analyze the recent usage data of the neuroscience gateway. Copyright

A neuroscience gateway: software and implementation

In this paper, we describe the neuroscience gateway (NSG), which facilitates access to high performance computing resources for computational neuroscientists. Through a simple web-based portal, the NSG provides a streamlined environment for uploading models, specifying HPC job parameters, querying running job status, receiving job completion notices, and storing and retrieving output data. The NSG architecture transparently distributes user jobs to appropriate HPC resources available through the XSEDE organization.

The CIPRES workbench: a flexible framework for creating science gateways

Here we describe the CIPRES Workbench (CW), an open source software framework for creating new science gateways with minimal overhead. The CW is a web application that can be deployed on a modest server, and can be configured to submit command line instructions to any resource where the application has submission privileges. It is designed to be highly configurable/customizable, and supports GUI-based access to HPC resources through a web browser interface as well as programmatic access via a ReSTful API. Using browser access, the CW architecture creates an environment with secure user accounts where user input data, job results, and job provenance are stored. The ReSTful API allows users with a registered a client application to deliver command lines to analytical codes and retrieve results from remote compute resources. A development effort is underway to allow the CW to submit jobs via the Science Gateways as a Platform (SciGaP) services hosted at Indiana University.

Using Gordon to accelerate LHC science

The discovery of the Higgs boson by the Large Hadron Collider (LHC) has garnered international attention. In addition to this singular result, the LHC may also uncover other fundamental particles, including dark matter. Much of this research is being done on data from one of the LHC experiments, the Compact Muon Solenoid (CMS). The CMS experiment was able to capture data at higher sampling frequencies than planned during the 2012 LHC operational period. The resulting data had been parked, waiting to be processed on CMS computers. While CMS has significant compute resources, by partnering with SDSC to incorporate Gordon into the CMS workflow, analysis of the parked data was completed months ahead of schedule. This allows scientists to review the results more quickly, and could guide future plans for the LHC.

TeraGrid resource selection tools: a road test

On a grid of computers, users often must decide between individual machines for job submission. Usually, the goal is to minimize time-to-completion. Several tools are available on TeraGrid to help users make this decision. In this paper, we use these tools to perform actual job submissions on TeraGrid machines. We evaluate the time-to-job-start effectiveness of these tools.

Understanding the Evolving Cyberinfrastructure Needs of the Neuroscience Community

In this paper, we first present a brief summary of the Neuroscience Gateway (NSG) which has been in operation since 2013. NSG is providing computational neuroscientists access to Extreme Science and Engineering Discovery Environment (XSEDE) high performance computing (HPC) resources. As a part of running the NSG we have interacted closely with the neuroscience community. This has given us the opportunity to receive input and feedback from the neuroscience researchers regarding their cyberinfrastructure needs. This is now more important given the context of the BRAIN (Brain Research through Advancing Innovative Neurotechnologies) Initiative which is a national initiative announced in 2013. Based on this interaction with the neuroscience community and the input we have received for the last three years, we analyze the comprehensive cyberinfrastructure needs of the neuroscience community in the second part of the paper.