Amos Brocco

SmartGRID: A Fully Decentralized Grid Scheduling Framework Supported by Swarm Intelligence

Resource management and scheduling has proven to be one of the key topics for grid computing. Nowadays, the resource management field is subdivided into low-level and high-level approaches. While low-level resource management systems normally concern the scheduling activities within a single virtual organization, high-level schedulers focus on the large scale resources utilization with unstable resource availability, low reliability networks, multi-policies, multi-administrative domains, etc. In this paper, we propose a decentralized framework named SmartGRID to tackle high-level grid resource management and scheduling. Within the SmartGRID framework, swarm intelligence algorithms are used for resource discovery and monitoring, standard protocols and schemes are adopted for scheduler interoperability, and an embedded plugin mechanism is provided to utilize multi-type external scheduling strategies. With a clearly decoupled layered architecture, SmartGRID has been designed to be a generic and modular environment to support intelligent and interoperable grid resource management upon a volatile, dynamics, and heterogeneous grid computing infrastructure.

Bounded diameter overlay construction: A self organized approach

This paper describes a distributed algorithm to construct and maintain a peer-to-peer network overlay with bounded diameter. The proposed approach merges a bio-inspired self-organized behavior with a pure peer-to-peer approach, in order to adapt the overlay to underlying changes in the network topology. Ant colonies are used to collect and spread information across all peers, whereas pheromone trails help detecting crashed nodes. Construction of the network favors balanced distribution of links across all peers, so that the resulting topology does not exhibit large hubs. Fault resilience and recovery mechanisms have also been implemented to prevent network partition in the event of node crashes. Validation has been conducted through simulations of different network scenarios.

Enabling efficient information discovery in a self-structured grid

One of the key success factors enabling the deployment of large scale grid systems is the existence of efficient resource discovery mechanisms. Accordingly, the main issues to be addressed by such a grid information system are those of scalability and minimal network overhead. In this respect, we propose a solution based on proactive information caching supported by a self-structured overlay topology. The proposed approach features a fully distributed ant-inspired self-organized overlay construction that maintains a bounded diameter overlay, and a selective flooding-based discovery algorithm that exploits local caches to reduce the number of visited nodes. To improve the caching scheme while retaining minimal bandwidth consumption, cache contents are periodically exchanged between neighboring nodes using an epidemic replication mechanism that is based on a gossiping algorithm, thus allowing nodes to have a more general view of the network and its resources. Extensive experimentation provides evidence that the average number of hops required to efficiently locate resources is limited and that our framework performs well with respect to hit rate and network overhead.

Towards an Integrated Vision across Inter-cooperative Grid Virtual Organizations

Much work has been done to exploit the benefit brought by allowing job execution on distributed computational resources. Nodes are typically able to share jobs only within the same virtual organization, which is inherently bounded by various reasons such as the adopted information system or other agreed constraints. The problem raised by such limitation is thus related to finding a way to enable interoperation between nodes from different virtual organizations. We introduce a novel technique for integrating visions from both resource users and providers, allowing to serve multiple virtual organizations as a whole. By means of snapshot data stored within each grid node, such as processing and interacting history, we propose a demand-centered heuristic scheduling approach named Critical Friend Community (CFC). To this end, a set of simplified community scheduling targeted algorithms and processing workflows are described. A prototype of our scheduling approach is being implemented within the SmartGRID project.

Using Metadata Snapshots for Extending Ant-Based Resource Discovery Service in Inter-cooperative Grid Communities

Much work is under way within the resource management community on issues associated with grid scheduling upon dynamically discovered information. In this paper we tackle the problem by exploiting a bio-inspired resource discovery mechanism, where information is provided by ant-based lightweight mobile agents traveling across a grid network and collecting data from each visited node. We start by providing the current state of the adopted grid scheduler, which is the result of an existing collaborative project named SmartGRID, and its underlying architecture constructed by ant-based mobile agents. We consider the problem of discovering resources in specific grid communities, which are bounded due to different shared community policies, such as diverse ant colonies, different resource discovery approaches, or other issues. Several issues have been raised during the design and implementation of such infrastructure. A notable issue, namely how grid schedulers from various bounded grid communities could be used in a manner which would extend current SmartGRID functionality is identified. Our shared view is that by utilizing already discovered and stored grid nodes’s metadata snapshots in the first instance we can facilitate a more convenient and efficient resource discovery operation next time. With this in mind, our paper goes on describing our shared vision with regard to this extended functionality as well as discussing the new conceptual basis and its model architecture.

Solenopsis: A Framework for the Development of Ant Algorithms

Network resources management issues in complex and dynamic scenarios require decentralized solutions and adaptive systems to face critical and unattended situations. Bio-inspired techniques such as swarm intelligence algorithms, have proved to be robust and suitable for managing tasks like routing, load-balancing or resource discovery. In this paper we describe Solenopsis, a framework for the development, simulation and deployment of ant-algorithms, which is aimed at supporting network management middlewares. The system provides a modular and scalable environment that can be distributed over a network. Ants are coded using a simple programming language, and are able to migrate across nodes. Two basic load-balancing algorithms are presented and evaluated, as an example of how this tool works and can be used in practice.

MaGate Simulator: A Simulation Environment for a Decentralized Grid Scheduler

This paper presents a simulator for of a decentralized modular grid scheduler named MaGate. MaGates design emphasizes scheduler interoperability by providing intelligent scheduling serving the grid community as a whole. Each MaGate scheduler instance is able to deal with dynamic scheduling conditions, with continuously arriving grid jobs. Received jobs are either allocated on local resources, or delegated to other MaGates for remote execution. The proposed MaGate simulator is based on GridSim toolkit and Alea simulator, and abstracts the features and behaviors of complex fundamental grid elements, such as grid jobs, grid resources, and grid users. Simulation of scheduling tasks is supported by a grid network overlay simulator executing distributed ant-based swarm intelligence algorithms to provide services such as group communication and resource discovery. For evaluation, a comparison of behaviors of different collaborative policies among a community of MaGates is provided. Results support the use of the proposed approach as a functional ready grid scheduler simulator.

Proactive information caching for efficient resource discovery in a self-structured grid

The cornerstone of successful deployment of large scale grid systems depends on efficient resource discovery mechanisms. In this respect, this paper presents a grid information system supported by a self-structured overlay topology and proactive information caching. The proposed approach features an ant-inspired self-organized overlay construction that maintains a bounded diameter overlay, and a selective flooding based discovery algorithm that exploit local caches to reduce the number of visited nodes. The caches are periodically exchanged between neighboring nodes using an epidemic replication mechanism that is based on a gossiping algorithm, thus allowing nodes to have a more general view of the network and its resources. We conducted extensive experimentation that provides evidence that the average number of hops required to efficiently locate resources is limited and that our framework performs well with respect to hit rate and network overhead.

BlåtAnt: Bounding Networks' Diameter with a Collaborative Distributed Algorithm

In this paper we describe BlatAnt, a new algorithm to create overlay networks with small diameters. BlatAnt is a fully distributed and adaptive algorithm inspired by Ant Colony Optimization (ACO), which targets dynamic and evolving networks without requiring a global knowledge. Simulation results show that our approach results in networks with a bounded diameter. This algorithm, implemented and empirically tested, will be the foundation of a fully decentralized resource discovery mechanism optimized for networks with small diameters.

Community-Aware Scheduling Protocol for Grids

Much work has been done to exploit the effectiveness and efficiency of job scheduling upon distributed computational––resources. With regard to existing resource topology and administrative constraints, scheduling approaches are designed for different hierarchic layers, for example, scheduling for job queues of local resource management systems (local scheduling), and scheduling for job queues of high level schedulers (also known as meta-schedulers or grid schedulers). Such scheduling approaches mainly focus on optimizing job queues of the hosting nodes, which are interconnected with computational resources directly or indirectly. In the real world (or in a community-based grid), a grid is comprised of nodes with different computing power and scheduling preferences, which in turn, raise a notable opportunity that is to exploit and optimize the process of job sharing between reachable grid nodes via improving the job allocation and efficiency ratio. In our work, we introduce a novel scheduling protocol which dedicates to disseminate scheduling events happened on each involving node to as many candidate nodes as possible. By means of the proposed protocol, scheduling process of each received job consists of several phases with awareness of grid volatility, and dynamic scheduling and rescheduling is allowed as long as the job execution has not started yet. To this end, a set of concerning algorithms and processing steps are described. A prototype of our scheduling approach is being implemented within the SmartGRID project.