D.L. Groep

Workload characteristics of a multi-cluster supercomputer

This paper presents a comprehensive characterization of a multi-cluster supercomputer workload using twelve-month scientific research traces. Metrics that we characterize include system utilization, job arrival rate and interarrival time, job cancellation rate, job size (degree of parallelism), job runtime, memory usage, and user/group behavior. Correlations between metrics (job runtime and memory usage, requested and actual runtime, etc) are identified and extensively studied. Differences with previously reported workloads are recognized and statistical distributions are fitted for generating synthetic workloads with the same characteristics. This study provides a realistic basis for experiments in resource management and evaluations of different scheduling strategies in a multi-cluster research environment.

VLAM-G: A Grid-based virtual laboratory

The Grid-based Virtual Laboratory AMsterdam (VLAM-G), provides a science portal for distributed analysis in applied scientific research. It offers scientists remote experiment control, data management facilities and access to distributed resources by providing cross-institutional integration of information and resources in a familiar environment. The main goal is to provide a unique integration of existing standards and software packages. This paper describes the design and prototype implementation of the VLAM-G platform. In this testbed we applied several recent technologies such as the Globus toolkit, enhanced federated database systems, and visualization and simulation techniques. Several domain specific case studies are described in some detail. Information management will be discussed separately in a forthcoming paper.

Predicting job start times on clusters

In a computational Grid which consists of many computer clusters, job start time predictions are useful to guide resource selections and balance the workload distribution. However, the basic Grid middleware available today either has no means of expressing the time that a site will take before starting a job or uses a simple linear scale. In this paper we introduce a system for predicting job start times on clusters. Our predictions are based on statistical analysis of historical job traces and simulation of site schedulers. We have deployed the system on the EDG (European Data-Grid) production cluster at NIKHEF. The experimental results show that acceptable prediction accuracy is achieved to reflect real site states and site-specific scheduling policies. We find that the average error of our job start time predictions is 18.9 percent of the average job queue wait time and this is around 20 times smaller than the average prediction error using the EDG solution.

Efficient response time predictions by exploiting application and resource state similarities

In large-scale grids with many possible resources (clusters of computing elements) to run applications, it is useful that the resources can provide predictions of job response times so users or resource brokers can make better scheduling decisions. Two metrics need to be estimated for response time predictions: one is how long a job executes on the resource (application run time), the other is how long the job waits in the queue before starting (queue wait time). In this paper we propose an instance based learning technique to predict these two metrics by mining historical workloads. The novelty of our approach is to introduce policy attributes in representing and comparing resource states, which is defined as the pool of running and queued jobs on the resource at the time to make a prediction. The policy attributes reflect the local resource scheduling policies and they can be automatically discovered using a genetic search algorithm. The main advantages of this approach compared with scheduler simulation are two-folds: Firstly, it has a better performance to meet the real time requirement of Grid resource brokering; secondly, it is more general because the scheduling policies are learned from past observations. Our experimental results on the NIKHEF LCG production cluster show that acceptable prediction accuracy can be obtained, where the relative prediction errors for response times are between 0.35 and 0.70.

Job Failure Analysis and Its Implications in a Large-Scale Production Grid

In this paper we present an initial analysis of job failures in a large-scale data-intensive Grid. Based on three representative periods in production, we characterize the interarrival times and life spans of failed jobs. Different failure types are distinguished and the analysis is carried out further at the Virtual Organization (VO) level. The spatial behavior, namely where job failures occur in the Grid, is also examined. Cross-correlation structures, including how arrivals correlate with life spans of job failures, are analyzed and illustrated. We further investigate statistical models to fit the failure data and propose several failureaware scheduling strategies at the Grid level. Our results show that the overall failure rates in the Grid are quite significant, ranging from 25% to 33% of all submitted jobs. However, only 5% to 8% of the jobs failed after running on a certain Computing Element (CE). The rest of failed jobs are aborted or cancelled without running. A majority of failed jobs come from several large production VOs and a large amount of these failures are centered around several main CEs. The interarrival time processes of failed jobs are shown to be bursty, and the life spans exhibit strong autocorrelations. Based on the failure patterns we argue that it is important for the Grid resource brokers to track historical failure and take it into account in decision making. Some proactive measures and accountability issues are also discussed.

Mining performance data for metascheduling decision support in the grid

Metaschedulers in the Grid need dynamic information to support their scheduling decisions. Job response time on computing resources, for instance, is such a performance metric. In this paper, we propose an Instance Based Learning technique to predict response times by mining historical performance data. The novelty of our approach is to introduce policy attributes in representing and comparing resource states, which are defined as the pools of running and queued jobs on the resources at the time of making predictions. The policy attributes reflect the local scheduling policies and they can be automatically discovered using genetic search. An extensive empirical evaluation is conducted to validate our technique using real workload traces, which are collected from the NIKHEF production cluster on the LHC Computing Grid and Blue Horizon in the San Diego Supercomputer Center (SDSC). The experimental results show that acceptable prediction accuracy can be achieved, where the normalized average prediction errors for response times are ranging from 0.57 to 0.79.

VLAM-G: a grid-based virtual laboratory

The Grid-based Virtual Laboratory AMsterdam (VLAM-G) provides a science portal for distributed analysis in applied scientific research. By facilitating access to distributed compute and information resources held by multiple organizations, and providing remote experiment control, data management and information retrieval capabilities, it allows scientists to better analyze their data. The ability to use data from multiple sources and correlating these data sets without in-depth domain expertise is a prime goal of the system. This paper describes the design and an implementation prototype of the VLAMG platform. The feasibility of the system is demonstrated by a generalized sample scenario from the chemo-physical analysis domain.

Evidence for short-range correlations in O-16

Abstract The reaction 16 O( e , e ′ pp ) 14 C has been investigated at three values of the transferred energy ω . The differential cross sections were determined as a function of the missing energy and the missing momentum. Evidence for short-range correlations in 16 O has been obtained from the transition to the ground state of 14 C. The cross sections for this transition are well reproduced by two independent parameter-free microscopic calculations. The results of both calculations show that the reaction is dominated by knockout of a proton pair in a 1 S 0 state, driven by short-range-correlations.

Definition and Implementation of a SAML-XACML Profile for Authorization Interoperability Across Grid Middleware in OSG and EGEE

In order to ensure interoperability between middleware and authorization infrastructures used in the Open Science Grid (OSG) and the Enabling Grids for E-science (EGEE) projects, an Authorization Interoperability activity was initiated in 2006. The interoperability goal was met in two phases: firstly, agreeing on a common authorization query interface and protocol with an associated profile that ensures standardized use of attributes and obligations; and secondly implementing, testing, and deploying on OSG and EGEE, middleware that supports the interoperability protocol and profile. The activity has involved people from OSG, EGEE, the Globus Toolkit project, and the Condor project. This paper presents a summary of the agreed-upon protocol, profile and the software components involved.

Authentication and Authorization Mechanisms for Multi-domain Grid Environments

This article discusses the authentication and the authorization aspects of security in grid environments spanning multiple administrative domains. Achievements in these areas are presented using the EU DataGrid project as an example implementation. It also gives an outlook on future directions of development.