Stuart Bailey

PSockets: The Case for Application-level Network Striping for Data Intensive Applications using High Speed Wide Area Networks

Transmission Control Protocol (TCP) is used by various applications to achieve reliable data transfer. TCP was originally designed for unreliable networks. With the emergence of high-speed wide area networks various improvements have been applied to TCP to reduce latency and achieve improved bandwidth. The improvement is achieved by having system administrators tune the network and can take a considerable amount of time. This paper introduces PSockets (Parallel Sockets), a library that achieves an equivalent performance without manual tuning. The basic idea behind PSockets is to exploit network striping. By network striping we mean striping partitioned data across several open sockets. We describe experimental studies using PSockets over the Abilene network. We show in particular that network striping using PSockets is effective for high performance data intensive computing applications using geographically distributed data.

The management and mining of multiple predictive models using the predictive modeling markup language

Abstract We introduce a markup language based upon XML for working with the predictive models produced by data mining systems. The language is called the predictive model markup language (PMML) and can be used to define predictive models and ensembles of predictive models. It provides a flexible mechanism for defining schema for predictive models and supports model selection and model averaging, involving multiple predictive models. It has proved useful for applications requiring ensemble learning, partitioned learning and distributed learning. In addition, it facilitates moving predictive models across applications and systems.

Hadoop Acceleration in an OpenFlow-Based Cluster

This paper presents details of our preliminary study of how Hadoop can control its network resources using OpenFlow in order to improve performance. Hadoops distributed compute framework called MapReduce, exploits the distributed storage architecture of Hadoops distributed file system HDFS to deliver scalable, reliable parallel processing services for arbitrary algorithms. The shuffle phase of Hadoops MapReduce computation involves movement of intermediate data from Mappers to Reducers. Reducers are often delayed due to inadequate bandwidth between them and the Mappers, and thereby lower the performance of the cluster. OpenFlow is a popular example of software-defined network (SDN) technology. Our study explores the use of OpenFlow to provide better link bandwidth for shuffle traffic, and thereby decrease the time that Reducers have to wait to gather data from Mappers. Our experiments show decrease in execution time for a Hadoop job, when the shuffle traffic can use more of the available bandwidth on a link. Our approach illustrates how high performance computing applications can improve performance by controlling their underlying network resources. The work presented in this paper is a starting point for some experiments being done as part of SC12 SCinet Research Sandbox which will quantify the performance advantages of a version of Hadoop that uses OpenFlow to dynamically adjust the network topology of local and wide area Hadoop clusters.

A methodology for supporting collaborative exploratory analysis of massive data sets in tele-immersive environments

This paper proposes a methodology for employing collaborative, immersive virtual environments as a high-end visualization interface for massive data-sets. The methodology employs feature detection, partitioning, summarization and decimation to significantly cull massive data-sets. These reduced data-sets are then distributed to the remote CAVEs, ImmersaDesks and desktop workstations for viewing. The paper also discusses novel techniques for collaborative visualization and meta-data creation.

A High Performance Implementation of the Data Space Transfer Protocol (DSTP)

With the emergence of high performance networks, clusters of workstations can now be connected by commodity networks (meta-clusters) or high speed networks (super-clusters) such as the very high speed Backbone Network Service (vBNS) or Internet2s Abilene. Distributed clusters are enabling a new class of data mining applications in which large amounts of data can be transferred using high performance networks and statistically and numerically intensive computations can be done using clusters of workstations. In this paper, we briefly describe a protocol called the Data Space Transfer Protocol (DSTP) for distributed data mining. With high performance networks, it becomes possible to move large amounts of data for certain queries when necessary. This paper describes the design of a high performance DSTP data server called Osiris which is designed to efficiently satisfy data requests for distributed data mining queries. In particular, we describe 1) Osiriss ability to lay out data by row or by column, 2) a scheduler intended to handle requests using standard network links and requests using network links enjoying some type of premium service, and 3) a mechanism designed to hide latency.

OpenFlow Configuration Protocol: Implementation for the of Management Plane

Separation of data and control plane offers benefits of having programmability of the forwarding tables according to the needs of the applications. The need for efficient and effective management of network resources is crucial in providing effective control plane functionality to the applications. OpenFlow standardization efforts at Open Networking Foundation resulted in an OpenFlow Configuration specification to address the management of resources in OpenFlow-enabled switches. We report the implementation of the OF-Config 1.1 standard [revision - 25th June 2012] as softconf.d to retrieve and update the controller IP of an OpenvSwitch.

OpenFlow-Based Network Management with Visualization of Managed Elements

The new software defined networking (SDN) paradigm advocates separating the data plane and the control plane, making network switches simple packet forwarding devices and leaving a logically-centralized software to control the behavior of the network. SDN introduces new possibilities for a centralized network management and configuration. The main benefit is having the programmability of the forwarding tables according to the needs of the applications. Therefore, efficient and effective management of network resources becomes even more crucial in providing effective control plane functionality to the applications. OpenFlow standardization efforts at the Open Networking Foundation resulted in an OpenFlow Configuration (OFConfig) specification to address the management of resources in networks with OpenFlow-enabled switches. We report the implementation of an intuitively easy to use interface for the OpenFlow-capable logical devices as managed resources in a SDN.

A data intensive computing approach to path planning and mode management for hybrid systems

We describe an approach to the design, analysis, and control of hybrid systems which is data intensive in contrast to more traditional approaches which tend to be compute-intensive. A key idea is to use a low overhead, high performance persistent object manager to trade space for time, even when the amount of data is very large. The main advantage is that near real time solutions to problems can be obtained which would be prohibitive with other approaches. To illustrate the effectiveness of this approach, we show how millions of trajectories segments can be stored and retrieved to solve path planning problems and how hundreds of modes can be stored to aid in the study of mode switching strategies.

Clusters, meta-clusters, and digital libraries: digital libraries for scientific, engineering and medical applications

We describe the architectural design and our early experience with scientific, engineering, and medical digital libraries that we have developed as part of the National Scalable Cluster Project (NSCP). Our goal is to develop an infrastructure for scalable, interactive digital libraries supporting high performance computing and high performance data management which interfaces to and interoperates with the World Wide Web (WWW).