António Manuel Silva Pina

Geographical partition for distributed web crawling

This paper evaluates scalable distributed crawling by means of the geographical partition of the Web. The approach is based on the existence of multiple distributed crawlers each one responsible for the pages belonging to one or more previously identified geographical zones. The work considers a distributed crawler where the assignment of pages to visit is based on page content geographical scope. For the initial assignment of a page to a partition we use a simple heuristic that marks a page within the same scope of the hosting web server geographical location. During download, if the analyze of a page contents recommends a different geographical scope, the page is forwarded to the well-located web server.A sample of the Portuguese Web pages, extracted during the year 2005, was used to evaluate: a) page download communication times and the b) overhead of pages exchange among servers. Evaluation results permit to compare our approach to conventional hash partitioning strategies.

cl OpenCL: supporting distributed heterogeneous computing in HPC clusters

Clusters that combine heterogeneous compute device architectures, coupled with novel programming models, have created a true alternative to traditional (homogeneous) cluster computing, allowing to leverage the performance of parallel applications. In this paper we introduce clOpenCL, a platform that supports the simple deployment and efficient running of OpenCL-based parallel applications that may span several cluster nodes, expanding the original single-node OpenCL model. clOpenCL is deployed through user level services, thus allowing OpenCL applications from different users to share the same cluster nodes and their compute devices. Data exchanges between distributed clOpenCL components rely on Open-MX, a high-performance communication library. We also present extensive experimental data and key conditions that must be addressed when exploiting clOpenCL with real applications.

PCor: A Prototype For Resource Oriented Computing

In this paper we present COR a resource oriented computing model that address the question of how to integrate user-level fine-grained multithreading, communication and coordination into a cluster of symmetrical multiprocessor computers. To support the design of complex distributed application using the proposed paradigm we built pCoR a run-time system which has new areas that represents extensions to the strict shared memory and message passing models supported by other platforms: remote operations, dynamic domains, communication ports, multithreading management, shared memory, replication and partition are some of its distinguished features. In addition, it provides a thread-safe transport communication layer to take advantage of modem high-performance commodity hardware/software like Myrinet network.

RoCL: A Resource Oriented Communication Library

RoCL is a communication library that aims to exploit the low-level communication facilities of today’s cluster networking hardware and to merge, via the resource oriented paradigm, those facilities and the high-level degree of parallelism achieved on SMP systems through multi-threading.

A cluster oriented model for dynamically balanced DHTs

Summary form only given. We refine previous work on a model for a distributed hash table (DHT) with support to dynamic balancement across a set of heterogeneous cluster nodes. We present new high-level entities, invariants and algorithms developed to increase the level of parallelism and globally reduce memory utilization. In opposition to a global distribution mechanism, that relies on complete knowledge about the current distribution of the hash table, we adopt a local approach, based on the division of the DHT into separated regions, that possess only partial knowledge of the global hash table. Simulation results confirm the hypothesis that the increasing of parallelism has as counterpart the degradation of the quality of the balancement achieved with the global approach. However, when compared with consistent hashing and our global approach, the same results clarify the relative merits of the extension, showing that, when properly parameterized, the model is still competitive, both in terms of the quality of the distribution and scalability.

Domus – an architecture for cluster-oriented distributed hash tables

This paper presents a high level description of Domus, an architecture for cluster-oriented Distributed Hash Tables. As a data management layer, Domus supports the concurrent execution of multiple and heterogeneous DHTs, that may be simultaneously accessed by different distributed/parallel client applications. At system level, a load balancement mechanism allows for the (re)distribution of each DHT over cluster nodes, based on the monitoring of their resources, including CPUs, memory, storage and network. Two basic units of balancement are supported: vnodes, a coarse-grain unit, and partitions, a fine-grain unit. The design also takes advantage of the strict separation of object lookup and storage, at each cluster node, and for each DHT. Lookup follows a distributed strategy that benefits from the joint analysis of multiple partition-specific routing information, to shorten routing paths. Storage is accomplished through different kinds of data repositories, according to the specificity and requirements of each DHT.

Genetic regulatory mechanisms by means of extended interactive Petri nets

We choose to integrate formalism for knowledge representation with formalism for process representation as a way to specify and regulate the overall activity of a multi cellular agent. The result of this approach is XP/sub i/N, another formalism, wherein a distributed system can be modeled as a collection of interrelated sub nets sharing a common explicit control structure. Each sub net represents a system of asynchronous concurrent threads modeled by a set of transitions. XP/sub i/N combines local state and control with interaction and hierarchy to achieve a high level abstraction and to model the complex relationships between all the components of a distributed system. Viewed as a tool, XP/sub i/N provides a carefully devised conflict resolution strategy that intentionally mimics the genetic regulatory mechanism used in an organic cell to select the next genes to process.

Running User-Provided Virtual Machines in Batch-Oriented Computing Clusters

The use of virtualization in HPC clusters can provide rich software environments, application isolation and efficient workload management mechanisms, but system-level virtualization introduces a software layer on the computing nodes that reduces performance and inhibits the direct use of hardware devices. We present an unobtrusive user-level platform to execute virtual machines inside batch jobs that does not handicap the computing clusters ability to execute the most demanding applications. A per-user platform uses a static mode in which the VMs run entirely within the resources of a single batch job and a dynamic mode in which the VMs navigate at runtime between the continuously allocated jobs node time-slots. In the dynamic mode fault-tolerant system agents are integrated using group communication to control the system, to execute user commands and to implement user-defined scheduling policies. In our tests compute intensive applications suffered negligible performance overhead compared to the native configuration, but the user-mode network overlay introduced a significant penalty on the more taxing networked applications.

pDomus: a prototype for Cluster-oriented Distributed Hash Tables

The Domus architecture for distributed hash tables (DHTs) is specially designed to support the concurrent deployment of multiple and heterogeneous DHTs, in a dynamic shared-all cluster environment. The execution model is compatible with the simultaneous access of several distributed/parallel client applications to the same or different running DHTs. Support to distributed routing and storage is dynamically configurable per node, as a function of applications requirements, node base resources and the overall cluster communication, memory and storage usage. pDomus is a prototype of Domus that creates an environment where to evaluate the model embedded concepts and planned features. In this paper, we present a series of experiments conduced to obtain figures of merit i) for the performance of basic dictionary operations, and ii) for the storage overhead resulting from several storage technologies. We also formulate a ranking formula that takes into account access patterns of clients to DHTs, to objectively select the most adequate storage technology, as a valuable metric for a wide range of application scenarios. Finally, we also evaluate client applications and services scalability, for a select dictionary operation. Results of the overall evaluation are promising and a motivation for further work

Two high-performance alternatives to ZLIB scientific-data compression

ZLIB is used in diverse frameworks by the scientific community, both to reduce disk storage and to alleviate pressure on I/O. As it becomes a bottleneck on multi-core systems, higher throughput alternatives must be considered, exploring parallelism and/or more effective compression schemes. This work provides a comparative study of the ZLIB, LZ4 and FPC compressors (serial and parallel implementations), focusing on CR, bandwidth and speedup. LZ4 provides very high throughput (decompressing over 1GB/s versus 120MB/s for ZLIB) but its CR suffers a degradation of 5-10%. FPC also provides higher throughputs than ZLIB, but the CR varies a lot with the data. ZLIB and LZ4 can achieve almost linear speedups for some datasets, while current implementation of parallel FPC provides little if any performance gain. For the ROOT dataset, LZ4 was found to provide higher CR, scalability and lower memory consumption than FPC, thus emerging as a better alternative to ZLIB.