Florina M. Ciorba

Dynamic multi phase scheduling for heterogeneous clusters

Distributed computing systems are a viable and less expensive alternative to parallel computers. However, concurrent programming methods in distributed systems have not been studied as extensively as for parallel computers. Some of the main research issues are how to deal with scheduling and load balancing of such a system, which may consist of heterogeneous computers. In the past, a variety of dynamic scheduling schemes suitable for parallel loops (with independent iterations) on heterogeneous computer clusters have been obtained and studied. However, no study of dynamic schemes for loops with iteration dependencies has been reported so far. In this work we study the problem of scheduling loops with iteration dependencies for heterogeneous (dedicated and non-dedicated) clusters. The presence of iteration dependencies incurs an extra degree of difficulty and makes the development of such schemes quite a challenge. We extend three well known dynamic schemes (CSS, TSS and DTSS) by introducing synchronization points at certain intervals so that processors compute in pipelined fashion. Our scheme is called dynamic multi-phase scheduling (DMPS) and we apply it to loops with iteration dependencies. We implemented our new scheme on a network of heterogeneous computers and studied its performance. Through extensive testing on two real-life applications (the heat equation and the Floyd-Steinberg algorithm), we show that the proposed method is efficient for parallelizing nested loops with dependencies on heterogeneous systems.

Towards the Robustness of Dynamic Loop Scheduling on Large-Scale Heterogeneous Distributed Systems

Dynamic loop scheduling (DLS) algorithms provide application-level load balancing of loop iterates, with the goal of maximizing application performance on the underlying system. These methods use run-time information regarding the performance of the applications execution (for which irregularities change over time). Many DLS methods are based on probabilistic analyses, and therefore account for unpredictable variations of application and system related parameters. Scheduling scientific and engineering applications in large-scale distributed systems (possibly shared with other users) makes the problem of DLS even more challenging. Moreover, the chances of failure, such as processor or link failure, are high in such large-scale systems. In this paper, we employ the hierarchical approach for three DLS methods, and propose metrics for quantifying their robustness with respect to variations of two parameters (load and processor failures), for scheduling irregular applications in large-scale heterogeneous distributed systems.

Towards the Scalability of Dynamic Loop Scheduling Techniques via Discrete Event Simulation

To improve their performance, scientific applications often use loop scheduling algorithms as techniques for load balancing data parallel computations. Over the years, a number of dynamic loop scheduling (DLS) techniques have been developed. These techniques are based on probabilistic analyses, and are effective in addressing unpredictable load imbalances in the system arising from various sources, such as, variations in application, algorithmic, and systemic characteristics. Modern, high-end computing facilities can now offer petascale performance (1015 flops), and several initiatives have already begun with the goal of achieving exascale performance (1018 flops) towards the end of the current decade. Efficient and scalable algorithms are therefore required to utilize the petascale and exascale resources. In this paper, a study of the scalability of DLS techniques via discrete event simulation is presented, both in terms of number of processors, and problem size. To facilitate the scalability study, a dynamic loop scheduler was designed and was implemented using the SimGrid simulation framework. The results of the study demonstrate the scalability of the DLS techniques and their effectiveness in addressing load imbalance in large scale computing systems.

Enhancing self-scheduling algorithms via synchronization and weighting

Existing dynamic self-scheduling algorithms, used to schedule independent tasks on heterogeneous clusters, cannot handle tasks with dependencies because they lack the support for internode communication. To compensate for this deficiency we introduce a synchronization mechanism that provides inter-processor communication, thus, enabling self-scheduling algorithms to handle efficiently nested loops with dependencies. We also present a weighting mechanism that significantly improves the performance of dynamic self-scheduling algorithms. These algorithms divide the total number of tasks into chunks and assign them to processors. The weighting mechanism adapts the chunk sizes to the computing power and current run-queue state of the processors. The synchronization and weighting mechanisms are orthogonal, in the sense that they can simultaneously be applied to loops with dependencies. Thus, they broaden the application spectrum of dynamic self-scheduling algorithms and improve their performance. Extensive testing confirms the efficiency of the synchronization and weighting mechanisms and the significant improvement of the synchronized-weighted versions of the algorithms over the synchronized-only versions.

Self-Adapting Scheduling for Tasks with Dependencies in Stochastic Environments

This paper addresses dynamic load balancing algorithms for non-dedicated heterogeneous clusters of workstations. We propose an algorithm called self-adapting scheduling (SAS), targeted at nested loops with dependencies in a stochastic environment. This means that the load entering the system, not belonging to the parallel application under execution, follows an unpredictable pattern which can be modeled by a stochastic process. SAS takes into account the history of previous timing results and the load patterns in order to make accurate load balancing predictions. We study the performance of SAS in comparison with DTSS. We established in previous work that DTSS is the most efficient self-scheduling algorithm for loops with dependencies on heterogeneous clusters. We test our algorithm under the assumption that the interarrival times and life-times of incoming jobs are exponentially distributed. The experimental results show that SAS significantly outperforms DTSS especially with rapidly varying loads

Analyzing the Robustness of Dynamic Loop Scheduling for Heterogeneous Computing Systems

Scheduling scientific applications in parallel on non-dedicated, heterogeneous systems, where the computing resources may differ in availability, is a challenging task, and requires efficient execution and robust scheduling methods. Dynamic loop scheduling methods provide means to achieve the desired robust performance. These methods are based on probabilistic analyses and are inherently robust. However, a methodology is required to measure the robustness of the dynamic loop scheduling methods that ensures their performance in unpredictably changing computing environments. In this paper, a methodology is proposed for performing robustness analysis of the dynamic loop scheduling techniques using a metric, formulated in earlier work, to measure their robustness in heterogeneous computing systems with uncertainties. The dynamic loop scheduling methods have been implemented in a simulation. The experimental results were used as an input to the proposed methodology, which in turn has been used to experimentally analyze the robustness of a number of dynamic loop scheduling methods on a heterogeneous system with variable availability.

Predicting the Flexibility of Dynamic Loop Scheduling Using an Artificial Neural Network

In this paper, an artificial neural network (ANN) model is proposed to predict the flexibility (or robustness against system load fluctuations in heterogeneous computing systems) of dynamic loop scheduling (DLS) methods. The multilayer perceptron (MLP) ANN model has been used to predict the degree of robustness of a DLS method, given specific values for the problem size, the system size, and the characteristics of the system load fluctuations as a compound effect of the variations in the applications iteration execution times and the processor availabilities. The developed MLP ANN model can be useful in an effective selection of the most robust DLS technique for scheduling a certain type of scientific application onto a given set of non-dedicated heterogeneous processors, when their system load is expected to fluctuate unpredictably during the applications runtime.

Distributed dynamic load balancing for pipelined computations on heterogeneous systems

One of the most significant causes for performance degradation of scientific and engineering applications on high performance computing systems is the uneven distribution of the computational work to the resources of the system. This effect, which is known as load imbalance, is even more noticeable in the case of irregular applications and heterogeneous distributed systems. This motivated the parallel and distributed computing research community to focus on methods that provide good load balancing for scientific and engineering applications running on (heterogeneous) distributed systems. Efficient load balancing and scheduling methods are employed for scientific applications from various fields, such as mechanics, materials, physics, chemistry, biology, applied mathematics, etc. Such applications typically employ a large number of computational methods in order to simulate complex phenomena, on very large scales of time and magnitude. These simulations consist of routines that perform repetitive computations (in the form of DO/FOR loops) over very large data sets, which, if not properly implemented and executed, may suffer from poor performance. The number of repetitive computations in the simulation codes is not always constant. Moreover, the computational nature of these simulations may be in fact irregular, leading to the case when one computation takes (unpredictably) more time than others. For successful and timely results, large scale simulations require the use of large scale computing systems, which often are widely distributed and highly heterogeneous. Moreover, large scale computing systems are usually shared among multiple users, which causes the quality and quantity of the available resources to be highly unpredictable. There are numerous load balancing methods in the literature for different parallel architectures. The most recent of these methods typically follow the master-worker paradigm, where a single coordinator (master) is responsible for making all the scheduling decisions based on information provided by the workers. Depending on the application requirements, the scheduling policy and the computational environment, the benefits of this paradigm may be limited as follows: (1) its efficiency may not scale as the number of processors increases, and (2) it is quite probable that the scheduling decisions are made based on outdated information, especially on systems where the workload changes rapidly. In an effort to address these limitations, we propose a distributed (master-less) load balancing scheme, in which the scheduling decisions are made by the workers in a distributed fashion. We implemented this method along with other two master-worker schemes (a previously existing one and a recently modified one) for three different scientific computational kernels. In order to validate the usefulness and efficiency of the proposed scheme, we conducted a series of comparative performance tests with the two master-worker schemes for each computational kernel. The target system is an SMP cluster, on which we simulated three different patterns of system load fluctuation. The experiments strongly support the belief that the distributed approach offers greater performance and better scalability on such systems, showing an overall improvement ranging from 13% to 24% over the master-worker approaches.

Evaluating the Flexibility of Dynamic Loop Scheduling on Heterogeneous Systems in the Presence of Fluctuating Load Using SimGrid

Scientific applications running on heterogeneous computing systems, which often have unpredictable behavior, enhance their performance by employing loop scheduling techniques as methods to avoid load imbalance through an optimized assignment of their parallel loops. With current computing platforms facilitating petascale performance and promising exascale performance towards the end of the present decade, efficient and robust algorithms are required to guarantee optimal performance of parallel applications in the presence of unpredictable perturbations. A number of dynamic loop scheduling (DLS) methods based on probabilistic analyses have been developed to achieve the desired robust performance. In earlier work, two metrics (flexibility and resilience) have been formulated to quantify the robustness of various DLS methods in heterogeneous computing systems with uncertainties. In this work, to ensure robust performance of the scientific applications on current (petascale) and future(exascale) high performance computing systems, a simulation model was designed and integrated into the SimGrid simulation toolkit, thus enabling a comprehensive study of the robustness of the DLS methods which uses results of experimental cases with various combinations of number of processors, problem sizes, and scheduling methods. The DLS methods have been implemented into the simulation model and analyzed for the purpose of exploring their flexibility (robustness against unpredictable variations in the system load), when involved in a range of case scenarios comprised of various distributions characterizing loop iteration execution times and system availability. The simulation results reported are used to compare the robustness of the DLS methods under the various environments considered, using the flexibility metric.

Towards the optimal synchronization granularity for dynamic scheduling of pipelined computations on heterogeneous computing systems

Loops are the richest source of parallelism in scientific applications. A large number of loop scheduling schemes have therefore been devised for loops with and without data dependencies (modeled as dependence distance vectors) on heterogeneous clusters. The loops with data dependencies require synchronization via cross‐node communication. Synchronization requires fine‐tuning to overcome the communication overhead and to yield the best possible overall performance. In this paper, a theoretical model is presented to determine the granularity of synchronization that minimizes the parallel execution time of loops with data dependencies when these are parallelized on heterogeneous systems using dynamic self‐scheduling algorithms. New formulas are proposed for estimating the total number of scheduling steps when a threshold for the minimum work assigned to a processor is assumed. The proposed model uses these formulas to determine the synchronization granularity that minimizes the estimated parallel execution time. The accuracy of the proposed model is verified and validated via extensive experiments on a heterogeneous computing system. The results show that the theoretically optimal synchronization granularity, as determined by the proposed model, is very close to the experimentally observed optimal synchronization granularity, with no deviation in the best case, and within 38.4% in the worst case. Copyright