P. Balakrishnan

Workflow Scheduling Algorithms and Approaches

Cloud infrastructures typically offer access to boundless virtual resources dynamically provisioned on demand for hosting, running, and managing a variety of mission-critical applications like scientific workflows, big data processing application, business intelligence-based applications, high-performance computing (HTC), and high transaction computing (HTC). Due to the surging popularity of the irresistible cloud idea, there are cloud datacenters spreading across the globe comprising heterogeneous cloud platforms and infrastructures catering to fast-evolving demands of worldwide businesses. The pervasive connectivity has enabled for the unprecedented success of the cloud concept. However, intensive automation is the key to the originally intended success of the cloud paradigm. Researchers across the world are focusing on unearthing powerful and pioneering tools and techniques for automated infrastructure life-cycle management. Similarly there are pathbreaking work-around approaches, algorithms, and architectures for workload consolidation. In short, there are many cloud-related aspects yearning for technologically sound automation, acceleration, and augmentation capabilities.

Workload Consolidation Through Automated Workload Scheduling

Workload consolidation is an approach to enhance the server utilization by grouping the VMs that are executing workflow tasks over multiple servers based on their server utilization. The primary objective is to optimally allocate the number of servers for executing the workflows which in turn minimize the cost and energy of data centers. This chapter consolidates the cost- and energy-aware workload consolidation approaches along with the tools and methodologies used in modern cloud data centers.

Workflow Integration and Orchestration, Opportunities and the Challenges

Workflow orchestration is a method which smartly organizes the enterprise function with application, data, and infrastructure. The applications as well as their infrastructure can be dynamically scaled up or down using orchestration. On the contrary, integration enables the development of new applications with the capability to connect to any other application through specified interfaces. In this chapter, firstly, the opportunities and challenges in workflow orchestration and integration are explained. Following that, BioCloud, an architecture that demonstrates the task-based workflow orchestration using two bioinformatics workflows is explained in detail.

Automated Optimization Methods for Workflow Execution

Workflow optimization is an approach to enhance the speed, robustness, and compactness of workflows by exploiting their structure, runtime, and output. This chapter initially highlights the significance of workflow optimization along with different possible levels of optimization. Further, it outlines the Taverna optimization framework over single and distributed infrastructure together with the optimization plug-ins that are validated using two scientific workflow executions.

Workflow Predictions Through Operational Analytics and Machine Learning

Workflow execution employs predictive analytics to extract significant, unidentified as well as precious insights from several stages of execution. Further, the operational analytics integrates these valuable insights directly into decision engine which enables analytical as well as machine learning-driven decision-making for an efficient workflow execution. This chapter highlights several analytical and machine learning approaches that are practiced in workflow predictions. Additionally, it explains the significance of hybrid approach which includes both analytical and machine learning models for workflow prediction. Finally, it describes the hybrid approach employed in PANORAMA architecture using two workflow applications.

The Hybrid IT, the Characteristics and Capabilities

With the faster adoption of the cloud idea across industry verticals with all the elegance and the enthusiasm, the traditional IT is bound to enlarge its prospects and potentials. Thatis, the IT capabilities and capacities are being enhanced with the seamless and spontaneous association with the cloud paradigm in order to meet up fast-emerging and evolving business requirements. This is a kind of new IT getting enormous attention and garnering a lot of attraction among business executives and IT professionals lately. The systematic amalgamation of the cloud concepts with the time-tested and trusted enterprise IT environment is to deliver a bevy of significant advantages for business houses in the days ahead. This model of next-generation computing through the cognitive and collective leverage of enterprise and cloud IT environments is being touted as the hybrid IT. There are a variety of technologies and tools expressly enabling the faster realization of hybrid era. This chapter is specially crafted for digging deep and describing the various implications of the hybrid IT.

Workflow Management Systems

Today’s scientific application requires tremendous amount of computation-driven as well as data-driven supported resources. The scientific applications are represented as workflows. The workflow management systems are designed and developed to depict the workflows of complex nature. The workflow management systems are able to reliably and efficiently coordinate among various resources in a distributed environment. This chapter describes various workflow management software like Kepler, Taverna, Triana, Pegasus, and Askalon. The architecture and functionalities of these workflow management systems are explained in the following sections.

Stepping into the Digital Intelligence Era

There are several noteworthy implications of the digitization movement. A myriad of digitation-enabling processes, platforms, products, patterns, and practices are hitting the market. There is a kind of convergence blurring the boundaries of physical and virtual worlds. Further on, all kinds of physical, mechanical, electrical, and electronics systems are adequately empowered to be instrumented and interconnected to exhibit intelligent behavior. Further on, all kinds of connected devices and digitized objects in our everyday environments are systematically integrated with remotely held and cloud-hosted applications, services, and data sources in order to be adaptive in their deeds, decisions, and deliveries.

Workflow Modeling and Simulation Techniques

Modeling and simulation of scientific workflow play a vital role in resource allocation in a distributed environment. Simulation is one of the methods to solve the complex scientific workflows in distributed environment. There are many scientific workflow simulation software frameworks that are available for grid and cloud environment. WorkflowSim is an open-source simulator. WorkflowSim Simulator extends the existing CloudSim Simulator. The architecture, components, and scheduling algorithms used and also the simulation results are explained for CloudSim Simulator and WorkflowSim Simulator.

Execution of Workflow Scheduling in Cloud Middleware

Many scientific applications are often modeled as workflows. The data and computational resource requirements are high for such workflow applications. Cloud provides a better solution to this problem by offering the promising environment for the execution of these workflow. As it involves tremendous data computations and resources, there is a need to automate the entire process. Workflow management system serves this purpose by orchestrating workflow task and executing it on distributed resources. Pegasus is a well-known workflow management system that has been widely used in large-scale e-applications. This chapter provides an overview about the Pegasus Workflow Management System, describes the environmental setup with OpenStack and creation and execution of workflows in Pegasus, and discusses about the workflow scheduling in cloud with its issues.