Carmine Spagnuolo

Bringing together efficiency and effectiveness in distributed simulations: The experience with D-Mason

Agent-based simulation models are an increasingly popular tool for research and management in many fields. In executing such simulations “speed” is one of the most general and important issues because of the size and complexity of simulations. But another important issue is the effectiveness of the solution, which consists of how easily usable and portable the solutions are for the users, i.e. the programmers of the distributed simulation. Our study, then, is aimed at efficient and effective distribute simulations by adopting a framework-level approach, with our design and implementation of a framework, D-Mason, which is a parallel version of the Mason library for writing and running simulations of agent-based simulation models. In particular, besides the efficiency due to workload distribution with small overhead, D-Mason at a framework level proves itself effective since it enables the scientists that use the framework (domain expert but with limited knowledge of distributed programming) only minimally aware of the fact that the simulation is running on a distributed environment. Then, we present tests that compare D-Mason against Mason in order to assess the improved scalability and D-Mason capability to exploit heterogeneous distributed hardware. Our tests also show that several massive simulations that are impossible to execute on Mason (e.g. because of CPU and/or memory requirements) can be easily performed using D-Mason.

Exploiting D-Mason on Parallel Platforms: A Novel Communication Strategy

Agent-based simulation models are a powerful experimental tool for research and management in many scientific and technological fields. D-Mason is a parallel version of Mason, a library for writing and running Agent-based simulations. In this paper, we present a novel development of D-Mason, a decentralized communication strategy which realizes a Publish/Subscribe paradigm through a layer based on the MPI standard. We show that our communication mechanism is much more scalable and efficient than the previous centralized one.

From desktop to large-scale model exploration with Swift/T

As high-performance computing resources have become increasingly available, new modes of computational processing and experimentation have become possible. This tutorial presents the Extreme-scale Model Exploration with Swift/T (EMEWS) framework for combining existing capabilities for model exploration approaches (e.g., model calibration, metaheuristics, data assimilation) and simulations (or any “black box” application code) with the Swift/T parallel scripting language to run scientific workflows on a variety of computing resources, from desktop to academic clusters to Top 500 level supercomputers. We will present a number of use-cases, starting with a simple agent-based model parameter sweep, and ending with a complex adaptive parameter space exploration workflow coordinating ensembles of distributed simulations. The use-cases are published on a public repository for interested parties to download and run on their own.

Communication Strategies in Distributed Agent-Based Simulations: The Experience with D-Mason

Agent-Based simulation Models (ABMs) are a very powerful experimental tool of analysis, used in many scientific and technological communities of researchers, to assess and predict the dynamic unfolding of a series of events or processes, according to the imposition of certain conditions, given by the analyst. The computing power usually represents a limit for such simulations and the traditional answer to the need for computing power is to invest in computer resources. D-Mason is a framework for parallelizing simulations developed on top of Mason toolkit. The goal of D-Mason is to exploit wasted computing power in a network of computers, eventually heterogeneous, as a research lab or a cluster of workstation.

On Evaluating Graph Partitioning Algorithms for Distributed Agent Based Models on Networks

Graph Partitioning is a key challenge problem with application in many scientific and technological fields. The problem is very well studied with a rich literature and is known to be NP-hard. Several heuristic solutions, which follow diverse approaches, have been proposed, they are based on different initial assumptions that make them difficult to compare. An analytical comparison was performed based on an Implementation Challenge [3], however being a multi-objective problem (two opposing goals are for instance load balancing and edge-cut size), the results are difficult to compare and it is hard to foresee what can be the impact of one solution, instead of another, in a real scenario. In this paper we analyze the problem in a real context: the development of a distributed agent-based simulation model on a network field (which for instance can model social interactions).

Distributed Agent-Based Simulation and GIS: An Experiment with the Dynamics of Social Norms

In the last decade, the investigation of the social complexity has witnessed the rise of Computational Social Science, a research paradigm that heavily relies upon data and computation to foster our understanding of social phenomena. In this field, a key role is played by the explanatory and predictive power of agent-based social simulations that are showing to take advantage of GIS, higher number of agents and real data. We focus GIS based distibuted ABMs. We observed that the density distribution of agents, over the field, strongly impact on the overall performances. In order to better understand this issue, we analyzes three different scenarios ranging from real positioning, where the citizens are positioned according to a real dataset to a random positioning where the agent are positioned uniformly at random on the field. Results confirm our hypothesis and show that an irregular distribution of the agents over the field increases the communication overhead. We provide also an analytic analysis which, in a 2-dimensional uniform field partitioning, is affected by several parameters (which depend on the model), but is also influenced by the density distribution of agents over the field. According to the presented results, we have that uniform space partitioning strategy does not scale on GIS based ABM characterized by an irregular distribution of agents.

Engaging Citizens with a Social Platform for Open Data

Open Data are valuable initiatives in favour of transparency. Public administrations are increasing the availability of datasets for citizens, associations, innovators and other stakeholders, by releasing their data with open licenses. Open initiatives are achieving less success than expected, mainly due to the lack of engagement. There is a growing demand for approaches to actively engage citizens in exploiting Open Data. This paper introduces SPOD, a Social Platform for Open Data, which aims to engage citizens, local associations and organizations in forming communities of interests, stimulating the interpretation of Open Data and exploiting their use in Data-driven discussions, something not well-supported on traditional social networks. Social collaboration is the key aspect to increase the public value, where citizens participate in the discussions, co-create knowledge and data. The paper describes the engagement of four communities of citizens, which contributed to the public value by discussing topics in the context of Cultural Heritage, generating information from existing and co-created open datasets, by using SPOD.

Datalet-Ecosystem Provider (DEEP): Scalable Architecture for Reusable, Portable and User-Friendly Visualizations of Open Data

This paper presents the DatalEt-Ecosystem Provider (DEEP), an extensible, and scalable Edge-centric architecture to visualize Open Data, retrieved in real time from institutional open data portals. The aim is to engage citizens and stakeholders through reusable, portable and interactive visualizations, named datalets. The DEEP architecture exploits the increasing computing power and capacity of end-users devices, moving the computation to process and visualize data, from the central server, directly to the client-side ensuring data trustiness, privacy, scalability and dynamic data loading. DEEP and its datalets have been fully exploited, in the ROUTE-TO-PA, HORIZON 2020 funded project, by five public administrations across Europe as pilot projects. The project engages and involves citizens in creating, sharing and commenting existing visualizations of Open Data. DEEP is open source, its source code is fully available on GitHub, thus every single component can be reused by other projects.

D-Mason on the Cloud: An Experience with Amazon Web Services

D-Mason framework is a parallel version of the Mason library for writing and running Agent-based simulations – a class of models that, by simulating the behavior of multiple agents, aims to emulate and/or predict complex phenomena. D-Mason has been conceived to harness the amount of unused computing power available in common installations like educational laboratory. Then the focus moved to dedicated installation, such as massively parallel machines or supercomputing centers. In this paper, D-Mason takes another step forward and now it can be used on a cloud environment.

Distributed simulation optimization and parameter exploration framework for the cloud

Abstract Simulation models are becoming an increasingly popular tool for the analysis and optimization of complex real systems in different fields. Finding an optimal system design requires performing a large sweep over the parameter space in an organized way. Hence, the model optimization process is extremely demanding from a computational point of view, as it requires careful, time-consuming, complex orchestration of coordinated executions. In this paper, we present the design of SOF (Simulation Optimization and exploration Framework in the cloud), a framework which exploits the computing power of a cloud computational environment in order to carry out effective and efficient simulation optimization strategies. SOF offers several attractive features. Firstly, SOF requires “zero configuration”, as it does not require any additional software installed on the remote node; only standard Apache Hadoop and SSH access are sufficient. Secondly, SOF is transparent to the user, since the user is totally unaware that the system operates on a distributed environment. Finally, SOF is highly customizable and programmable, since it enables the running of different simulation optimization scenarios using diverse programming languages – provided that the hosting platform supports them – and different simulation toolkits, as developed by the modeler. The tool has been fully developed and is available on a public repository 1 under the terms of the open source Apache License. It has been tested and validated on several private platforms, such as a dedicated cluster of workstations, as well as on public platforms, including the Hortonworks Data Platform and Amazon Web Services Elastic MapReduce solution.