L. Felipe Perrone

Safe: simulation automation framework for experiments

The workflow of a network simulation study requires adherence to best practices in methodology so that results are credible and reproducible by third parties. The opportunities for one to introduce errors start at model description and permeate the process through to the reporting of results. The literature indicates that even publications in respected venues include inadvertent mistakes and poor application of methodology. When experts are liable to fail, it is unreasonable to expect that students would fare any better. This paper presents a system designed to provide guidance for inexperienced users of the popular ns-3 network simulator. SAFE automates the workflow from the initialization of model parameters, to the parallelized execution of experiments, to the processing and persistent storage of output data, and to graphical visualization of results. We discuss the architecture and the implementation of the system in the context of similar contributions in the literature.

On the automation of computer network simulators

Simulation has been an important resource for functional and performance analyses of computer networks. Although the number of widely adopted network simulators is small, new ones continue to be created to address gaps in the functionality of existing tools. It can be argued, however, that the greatest need of the scientific community is to raise the credibility of published simulation studies. In this paper, we show that this need can be addressed by enabling network simulators to provide fool-proof automation of the experimental process. Ideally, the simulators interface would provide users with an environment to minimize set up time for experiments and to guarantee their reproducibility, and to safeguard the statistical rigor of data analysis. We demonstrate that advances toward this goal have been made by three different tools. Our contributions in this paper culminate with the derivation of requirements for automation tools from recent literature and from our own experience in tool construction. Once these requirements are fulfilled, network simulation tools can have a stronger impact in education, in carrying out large simulation studies, and in enhancing the credibility of simulation results.

A Study of On-Off Attack Models for Wireless Ad Hoc Networks

The security of mobile wireless ad hoc networks is a multifaceted topic which in recent years has been the focus of much interest in the research community. While many security issues in these networks can be addressed by protocol design, wireless nodes have inherent physical vulnerabilities that can be exploited by attackers to cause disruptions in network traffic. The nature of these exposures is such that there is little that can be done to eliminate them leaving the network open to denial of service and reduction of quality attacks. It is essential that the impact of such attacks is well-understood before wireless ad hoc networks are used in mission-critical applications. This paper is a step in this risk analysis as our experiments quantify the effects of attacks that exploit physical vulnerabilities. Our contributions in this paper are two fold. First, we introduce a general model that can be used to characterize a physical attack as an on-off process and then we apply this model to two specific attack scenarios. Second, we present the results of a simulation study with these two attack scenarios in the context of a mesh of network nodes using the IEEE 802.11 standard and the AODV routing protocol

The design of an output data collection framework for NS-3

An important design decision in the construction of a simulator is how to enable users to access the data generated in each run of a simulation experiment. As the simulator executes, the samples of performance metrics that are generated beg to be exposed either in their raw state or after having undergone mathematical processing. Also of concern is the particular format this data assumes when externalized to mass storage, since it determines the ease of processing by other applications or interpretation by the user. In this paper, we present a framework for the ns-3 network simulator for capturing data from inside an experiment, subjecting it to mathematical transformations, and ultimately marshaling it into various output formats. The application of this functionality is illustrated and analyzed via a study of common use cases. Although the implementation of our approach is specific to ns-3, this design presents lessons transferrable to other platforms.

Perspectives on languages for specifying simulation experiments

Domain specific languages have been used in modeling and simulation as tools for model description. In recent years, the efforts toward enabling simulation reproducibility have motivated the use of domain specific languages also as the means with which to express experiment specifications. In simulation areas ranging from computational biology to computer networks, the emerging trend is to treat the experimentation process as a first class object. Domain specific languages serve to specify individual sub-tasks in this process, such as configuration, observation, analysis, and evaluation of experimental results. Additionally, they can be used in a broader scope, for instance, to describe formally the experiments goals. The research and development of domain specific languages for experiment specification explores all of these and additional possible applications. In this paper, we discuss various existing approaches for defining this type of domain specific languages and present a critical analysis of our findings.

An experiment automation framework for ns-3

Recent studies have indicated that tools that automate the execution of simulation experiments can serve to enhance both the usability of network simulators and the credibility of the studies developed with them. In this poster, we present the architecture for an experiment automation framework for the ns-3 network simulator. The architecture was designed with two specific goals in mind. First, it raises the level of abstraction of the ns-3 user interface to make the simulator easier to use in large scale experimental studies. Second, it provides functionalities to guide the user along known correct methodologies for modeling and simulation thereby increasing the confidence one can have in the correctness of the experimental results.

FlyLoop: a micro framework for rapid development of physiological computing systems

With the advent of wearable computing, cheap, commercial-grade sensors have broadened the accessibility to real-time physiological sensing. While there is considerable research that explores leveraging this information to drive intelligent interfaces, the construction of such systems have largely been limited to those with significant technical expertise. Even seasoned programmers are forced to tackle serious engineering challenges such as merging data from multiple sensors, applying signal processing algorithms, and modeling user state in real-time. These hurdles limit the accessibility and replicability of physiological computing systems, and more broadly intelligent interfaces. In this paper, we present FlyLoop - a small, lightweight Java framework that enables programmers to rapidly develop, and experiment with intelligent systems. By focusing on simplicity and modularity rather than device compatibility or software dependencies, we believe that FlyLoop can broaden the participation in next-generation user interfaces, and encourage systems that can be communicated and reproduced.

Data visualization for network simulations

As many other kinds of simulation experiments, simulations of computer networks tend to generate high volumes of output data. While the collection and the statistical processing of these data are challenges in and of themselves, creating meaningful visualizations from them is as much an art as it is a science. A sophisticated body of knowledge in information design and data visualization has been developed and continues to evolve. However, many of the visualizations created by the network simulation community tend to be less than optimal at creating compelling, informative narratives from experimental output data. The primary contribution of this paper is to explore some of the design dimensions in visualization and some advances in the field that are applicable to network simulation. We also discuss developments in the creation of the visualization subsystem in the Simulation Automation Framework for Experiments (SAFE) in the context of best practices for data visualization.