Jan Broeckhove

Cost-Optimal Scheduling in Hybrid IaaS Clouds for Deadline Constrained Workloads

With the recent emergence of public cloud offerings, surge computing –outsourcing tasks from an internal data center to a cloud provider in times of heavy load– has become more accessible to a wide range of consumers. Deciding which workloads to outsource to what cloud provider in such a setting, however, is far from trivial. The objective of this decision is to maximize the utilization of the internal data center and to minimize the cost of running the outsourced tasks in the cloud, while fulfilling the applications’ quality of service constraints. We examine this optimization problem in a multi-provider hybrid cloud setting with deadline-constrained and preemptible but non-provider-migratable workloads that are characterized by memory, CPU and data transmission requirements. Linear programming is a general technique to tackle such an optimization problem. At present, it is however unclear whether this technique is suitable for the problem at hand and what the performance implications of its use are. We therefore analyze and propose a binary integer program formulation of the scheduling problem and evaluate the computational costs of this technique with respect to the problem’s key parameters. We found out that this approach results in a tractable solution for scheduling applications in the public cloud, but that the same method becomes much less feasible in a hybrid cloud setting due to very high solve time variances.

Online cost-efficient scheduling of deadline-constrained workloads on hybrid clouds

Cloud computing has found broad acceptance in both industry and research, with public cloud offerings now often used in conjunction with privately owned infrastructure. Technical aspects such as the impact of network latency, bandwidth constraints, data confidentiality and security, as well as economic aspects such as sunk costs and price uncertainty are key drivers towards the adoption of such a hybrid cloud model. The use of hybrid clouds introduces the need to determine which workloads are to be outsourced, and to what cloud provider. These decisions should minimize the cost of running a partition of the total workload on one or multiple public cloud providers while taking into account the application requirements such as deadline constraints and data requirements. The variety of cost factors, pricing models and cloud provider offerings to consider, further calls for an automated scheduling approach in hybrid clouds. In this work, we tackle this problem by proposing a set of algorithms to cost-efficiently schedule the deadline-constrained bag-of-tasks applications on both public cloud providers and private infrastructure. Our algorithms take into account both computational and data transfer costs as well as network bandwidth constraints. We evaluate their performance in a realistic setting with respect to cost savings, deadlines met and computational efficiency, and investigate the impact of errors in runtime estimates on these performance metrics.

Towards mechanistic models of plant organ growth

Modelling and simulation are increasingly used as tools in the study of plant growth and developmental processes. By formulating experimentally obtained knowledge as a system of interacting mathematical equations, it becomes feasible for biologists to gain a mechanistic understanding of the complex behaviour of biological systems. In this review, the modelling tools that are currently available and the progress that has been made to model plant development, based on experimental knowledge, are described. In terms of implementation, it is argued that, for the modelling of plant organ growth, the cellular level should form the cornerstone. It integrates the output of molecular regulatory networks to two processes, cell division and cell expansion, that drive growth and development of the organ. In turn, these cellular processes are controlled at the molecular level by hormone signalling. Therefore, combining a cellular modelling framework with regulatory modules for the regulation of cell division, expansion, and hormone signalling could form the basis of a functional organ growth simulation model. The current state of progress towards this aim is that the regulation of the cell cycle and hormone transport have been modelled extensively and these modules could be integrated. However, much less progress has been made on the modelling of cell expansion, which urgently needs to be addressed. A limitation of the current generation models is that they are largely qualitative. The possibilities to characterize existing and future models more quantitatively will be discussed. Together with experimental methods to measure crucial model parameters, these modelling techniques provide a basis to develop a Systems Biology approach to gain a fundamental insight into the relationship between gene function and whole organ behaviour.

Combining Futures and Spot Markets: A Hybrid Market Approach to Economic Grid Resource Management

Economic forms of resource management in which users can express their valuations for service, offer new possibilities for optimizing resource allocations in Grids. If users are to correctly express these valuations, quality of service guarantees need to be given with respect to the turnaround time of their workloads. Market mechanisms that support bidding and allocations in future time are crucial for delivering such guarantees. To deal with the significant delays that these mechanisms introduce in the allocation process, we present a hybrid market approach in which a low-latency spot market coexists with a higher latency futures market. Based on simulated market scenarios, we show how this combination can significantly increase the total value realized by the Grid infrastructure. We also demonstrate how providers can react to price dynamics in such a hybrid market setting.

Scalability of Grid Simulators: An Evaluation

Due to the distributed nature of resources in grids that cover multiple administrative domains, grid resource management cannot be optimally implemented using traditional approaches. In order to investigate new grid resource management systems, researchers utilize simulators which allows them to efficiently evaluate new algorithms on a large scale. We have developed the Grid Economics Simulator (GES) in support of research into grid resource management in general and economic grid resource management in particular. This paper compares GES to SimGrid and GridSim, two established grid simulation frameworks. We demonstrate that GES compares favourably to the other frameworks in terms of scalability, runtime performance and memory requirements. We explain how these differences are related to the simulation paradigm and the threading model used in each simulator.

Runtime Prediction Based Grid Scheduling of Parameter Sweep Jobs

This paper examines the problem of predicting job runtimes by exploiting the properties of parameter sweeps. A new parameter sweep prediction framework GIPSy (grid information prediction system) is introduced. Predictions are made based on prior runtime information and the parameters used to configure each job. The main objective is providing a tool combining development, simulation and application of prediction models within one framework. The different kinds of available sample selectors and models are discussed in detail. Results are presented for a quantum physics problem. A previously introduced scheduling technique and the implementation called PGS (prediction based grid scheduling) is improved and presented in combination with GIPSy to obtain a real-world grid implementation that optimizes the distribution of parameter sweeps.

PRICING SUBSTITUTABLE GRID RESOURCES USING COMMODITY MARKET MODELS

Enhancing Grid technology with market models for trading resources, is a promising step for Grids to become open systems that allow for user-centric service provisioning. This paper introduces a market model for trading substitutable Grid resources in a commodity market. We develop a pricing scheme and evaluate the market mechanisms through simulation. We show that the resource market achieves price stability and correctness, allocative eciency and fairness.

A Simulation Framework for Studying Economic Resource Management in Grids

Economic principles are increasingly being regarded as a way to address conflicting user requirements, to improve the effectiveness of grid resource management systems, and to deliver incentives for providers to join virtual organizations. Because economic resource management mechanisms can encourage grid participants to reveal the true valuations of their jobs and resources, the system becomes capable of making better scheduling decisions. A lot of exploratory research into different market mechanisms for grids is ongoing. Since it is impractical to conduct analysis of novel mechanisms on operational grids, most of this research is being carried out using simulation. This paper presents the Grid Economics Simulator (GES) in support of such research. The key design goals of the framework are enabling a wide variety of economic and non-economic forms of resource management while simultaneously supporting distributed execution of simulations and exhibiting good scalability properties.

Economic Grid Resource Management for CPU Bound Applications with Hard Deadlines

The introduction of economic principles in grid resource management provides an interesting avenue for efficiently addressing the problem of conflicting user requirements. In shared computing infrastructures such as grids, such conflicting requirements are prevalent and stem from the selfish actions users follow when formulating their service requests. We develop and analyze both a centralized and a decentralized algorithm for economic resource management in the context of consumer requests for CPU bound applications with deadline-based QoS requirements and non- migratable workloads. A comparison with an algorithm recently proposed in the literature is presented with a focus on performance in terms of realized consumer value. We establish that our algorithms perform well and that they compare favorably to existing approaches.

Putting theory to the test: which regulatory mechanisms can drive realistic growth of a root?

In recent years there has been a strong development of computational approaches to mechanistically understand organ growth regulation in plants. In this study, simulation methods were used to explore which regulatory mechanisms can lead to realistic output at the cell and whole organ scale and which other possibilities must be discarded as they result in cellular patterns and kinematic characteristics that are not consistent with experimental observations for the Arabidopsis thaliana primary root. To aid in this analysis, a ‘Uniform Longitudinal Strain Rule’ (ULSR) was formulated as a necessary condition for stable, unidirectional, symplastic growth. Our simulations indicate that symplastic structures are robust to differences in longitudinal strain rates along the growth axis only if these differences are small and short-lived. Whereas simple cell-autonomous regulatory rules based on counters and timers can produce stable growth, it was found that steady developmental zones and smooth transitions in cell lengths are not feasible. By introducing spatial cues into growth regulation, those inadequacies could be avoided and experimental data could be faithfully reproduced. Nevertheless, a root growth model based on previous polar auxin-transport mechanisms violates the proposed ULSR due to the presence of lateral gradients. Models with layer-specific regulation or layer-driven growth offer potential solutions. Alternatively, a model representing the known cross-talk between auxin, as the cell proliferation promoting factor, and cytokinin, as the cell differentiation promoting factor, predicts the effect of hormone-perturbations on meristem size. By down-regulating PIN-mediated transport through the transcription factor SHY2, cytokinin effectively flattens the lateral auxin gradient, at the basal boundary of the division zone, (thereby imposing the ULSR) to signal the exit of proliferation and start of elongation. This model exploration underlines the value of generating virtual root growth kinematics to dissect and understand the mechanisms controlling this biological system.