Tevfik Kosar

Stork: making data placement a first class citizen in the grid

Todays scientific applications have huge data requirements which continue to increase drastically every year. These data are generally accessed by many users from all across the the globe. This implies a major necessity to move huge amounts of data around wide area networks to complete the computation cycle, which brings with it the problem of efficient and reliable data placement. The current approach to solve this problem of data placement is either doing it manually, or employing simple scripts which do not have any automation or fault tolerance capabilities. Our goal is to make data placement activities first class citizens in the Grid just like the computational jobs. They will be queued, scheduled, monitored, managed, and even check-pointed. More importantly, it will be made sure that they complete successfully and without any human interaction. We also believe that data placement jobs should be treated differently from computational jobs, since they may have different semantics and different characteristics. For this purpose, we have developed Stork, a scheduler for data placement activities in the grid.

A new paradigm: Data-aware scheduling in grid computing

Efficient and reliable access to large-scale data sources and archiving destinations in a widely distributed computing environment brings new challenges. The insufficiency of the traditional systems and existing CPU-oriented batch schedulers in addressing these challenges has yielded a new emerging era: data-aware schedulers. In this article, we discuss the limitations of the traditional CPU-oriented batch schedulers in handling the challenging data management problem of large-scale distributed applications; give our vision for the new paradigm in data-intensive scheduling; and elaborate on our case study: the Stork data placement scheduler.

Workflow Management in Condor

The Condor project began in 1988 and has evolved into a feature-rich batch system that targets high-throughput computing; that is, Condor ([262], [414]) focuses on providing reliable access to computing over long periods of time instead of highly tuned, high-performance computing for short periods of time or a small number of applications.

PhoneLab: A Large Programmable Smartphone Testbed

As smartphones have emerged as the most widely deployed mobile computing platform, the scale of smartphone experimentation has lagged behind. New facilities enabling large-scale experiments are needed to ensure that research discoveries translate to the billions of smartphones in use today. To meet this challenge, we introduce PhoneLab, a 288-device smartphone testbed deployed at the University at Buffalo. PhoneLab provides access to smartphone users incentivized to participate in experiments while simplifying experiment data collection. The testbed will open for public experimentation in October, 2013, and continue to expand in 2014. To demonstrate the power of PhoneLab, we present three selected results from a usage characterization experiment run on 115 phones for 21 days. We use each result to motivate a future PhoneLab experiment, demonstrating how PhoneLab will enable mobile systems research.

A data throughput prediction and optimization service for widely distributed many-task computing

In this paper, we present the design and implementation of an application-layer data throughput prediction and optimization service for many-task computing in widely distributed environments. This service uses multiple parallel TCP streams to improve the end-to-end throughput of data transfers. A novel mathematical model is developed to determine the number of parallel streams, required to achieve the best network performance. This model can predict the optimal number of parallel streams with as few as three prediction points. We implement this new service in the Stork Data Scheduler, where the prediction points can be obtained using Iperf and GridFTP samplings. Our results show that the prediction cost plus the optimized transfer time is much less than the nonoptimized transfer time in most cases. As a result, Stork data transfer jobs with optimization service can be completed much earlier, compared to nonoptimized data transfer jobs.

Which network measurement tool is right for you? a multidimensional comparison study

Network performance measurement and prediction is one of the most prominent and indispensable components in distributed computing environments. The selection of the most advantageous network measurement tool or system for specific needs can be very time consuming and may require detailed experimental analysis. The multi-dimensional aspects and properties of such systems or tools should be considered in parallel. In this paper, we take two of the most widely used and accepted network measurement tools as a case study: Iperf and network weather service. We compare these two prediction tools by listing the pros and cons based on accuracy, overhead, intrusiveness, system requirements, capabilities, reliability, scalability and response time. We present different methodologies used to measure their performance in previous experiments and run experiments for comparing them to actual FTP, GridFTP and SCP transfers based on different parameters.

Phoenix: making data-intensive grid applications fault-tolerant

A major hurdle facing data intensive grid applications is the appropriate handling of failures that occur in the grid-environment. Implementing the fault-tolerance transparently at the grid-middleware level would make different data intensive applications fault-tolerant without each having to pay a separate cost and reduce the time to grid-based solution for many scientific problems. We analyzed the failures encountered by four real-life production data intensive applications: NCSA image processing pipeline, WCER video processing pipeline, US-CMS pipeline and BMRB BLAST pipeline. Taking the result of the analysis into account, we have designed and implemented Phoenix, a transparent middleware-level fault-tolerance layer that detects failures early, classifies failures into transient and permanent and appropriately handIes the transient failures. We applied our fault-tolerance layer to a prototype of the NCSA image processing pipeline and considerably improved the failure handling and report on the insights gained in the process.

Data Intensive Distributed Computing: Challenges and Solutions for Large-scale Information Management

The trend in scientific, as well as commercial, applications from a diverse range of fields has been towards being more and more data-intensive over time. Data Intensive Distributed Computing: Challenges and Solutions for Large-scale Information Management focuses on the challenges of distributed systems imposed by data intensive applications and on the different state-of-the-art solutions proposed to overcome such challenges. Providing hints on how to manage low-level data handling issues when performing data intensive distributed computing, this publication is ideal for scientists, researchers, engineers, and application developers, alike. With the knowledge of the correct data management techniques for their applications, readers will be able to focus on their primary goal, assured that their data management needs are handled reliably and efficiently.

Dynamic Adaptation of Parallelism Level in Data Transfer Scheduling

We discuss dynamic parameter tuning in wide-area data transfers for efficient utilization of available network capacity and optimized end-to-end application performance.Impacts of parallel TCP streams as well as concurrent data transfer jobs running simultaneously have been studied.We present an adaptive approach for tuning parallelism level of data placement jobs in distributed environments.The adaptive data scheduling includes dynamically setting parameters of data placement jobs.The proposed methodology operates without depending on any external profiles to adapt to changing network conditions.

Conditional workflow management: A survey and analysis

Workflows form the essential part of the process execution both in a single machine and in distributed environments. Although providing conditional structures is not mandatory for a workflow management system, support for conditional workflows is very important in terms of error handling, flexibility and robustness. Several of the existing workflow management systems already support conditional structures via use of different constructs. In this paper, we study the most widely used workflow management systems and their support for conditional structures such as if, switch, and while. We compare implementation of common conditional structures using each of these workflow management systems via case studies, and discuss capabilities of each system.