Hermes Senger

Scalability limits of Bag-of-Tasks applications running on hierarchical platforms

Bag-of-Tasks applications are parallel applications composed of independent (i.e., embarrassingly parallel) tasks, which do not communicate with each other, may depend upon one or more input files, and can be executed in any order. Each file may be input for more than one task. Examples of Bag-of-Tasks (BoT) applications include Monte Carlo simulations, massive searches (such as key breaking), image manipulation applications and data mining algorithms. A common framework to execute BoT applications is the master-slave topology, in which the user machine is used to control the execution of tasks. In this scenario, a large number of concurrent tasks competing for resources (e.g., CPU and communication links) severely limits application execution scalability. This paper is devoted to study the scalability of BoT applications running on multi-node systems (such as clusters and multi-clusters) organized as hierarchical platforms, considering several communication paradigms. Our study employs a set of experiments that involves the simulation of various large-scale platforms. The results presented provide important guidelines for improving the scalability of practical applications.

Hierarchical scheduling of independent tasks with shared files

Parallel computing platforms such as grids, clusters and multi-clusters constitute promising alternatives for executing applications comprised by a large number of independent tasks. However, some application and architectural characteristics may severely limit performance gains. For instance, tasks with fine granularity, huge data files to be transmitted to or from data repositories, and tasks which share common input files are examples of such characteristics that may cause poor performance. Bottlenecks may also appear due to the existence of a centralized controller in the master-slave architecture, or centralized data repositories within the system. This paper shows how system efficiency decreases under such conditions. To overcome such limitations, a hierarchical strategy for file distribution which aims at improving the system capacity of delivering input files to processing nodes is proposed and assessed. Such a strategy arranges the processors in a tree topology, clusters tasks that share common input files together, and maps such groups of tasks to clusters of processors. By means of such strategy, significant improvements in the application scalability can be achieved.

BSP cost and scalability analysis for MapReduce operations

Data abundance poses the need for powerful and easy‐to‐use tools that support processing large amounts of data. MapReduce has been increasingly adopted for over a decade by many companies, and more recently, it has attracted the attention of an increasing number of researchers in several areas. One main advantage is that the complex details of parallel processing, such as complex network programming, task scheduling, data placement, and fault tolerance, are hidden in a conceptually simple framework. MapReduce is supported by mature software technologies for deployment in data centers such as Hadoop. As MapReduce becomes popular for high‐performance applications, many questions arise concerning its performance and efficiency. In this paper, we demonstrated formally lower bounds on the isoefficiency function for MapReduce applications, when these applications can be modeled as BSP jobs. We also demonstrate how communication and synchronization costs can be dominant for MapReduce computations and discuss the conditions under which such scalability limits are valid. To our knowledge, this is the first study that demonstrates scalability bounds for MapReduce applications. We also discuss how some MapReduce implementations such as Hadoop can mitigate such costs to approach linear, or near‐to‐linear speedups. Copyright

SparkBLAST: scalable BLAST processing using in-memory operations

BackgroundThe demand for processing ever increasing amounts of genomic data has raised new challenges for the implementation of highly scalable and efficient computational systems. In this paper we propose SparkBLAST, a parallelization of a sequence alignment application (BLAST) that employs cloud computing for the provisioning of computational resources and Apache Spark as the coordination framework. As a proof of concept, some radionuclide-resistant bacterial genomes were selected for similarity analysis.ResultsExperiments in Google and Microsoft Azure clouds demonstrated that SparkBLAST outperforms an equivalent system implemented on Hadoop in terms of speedup and execution times.ConclusionsThe superior performance of SparkBLAST is mainly due to the in-memory operations available through the Spark framework, consequently reducing the number of local I/O operations required for distributed BLAST processing.

Survey on Simulation for Mobile Ad-Hoc Communication for Disaster Scenarios

Mobile ad-hoc communication is a demonstrated solution to mitigate the impact of infrastructure failures during large-scale disasters. A very complex issue in this domain is the design validation of software applications that support decision-making and communication during natural disasters. Such disasters are irreproducible, highly unpredictable, and impossible to scale down, and thus extensive assessments cannot be led in situ. In this context, simulation constitutes the best approach towards the testing of software solutions for natural disaster responses. The present survey reviews mobility models, ad-hoc network architectures, routing protocols and network simulators. Our aim is to provide guidelines for software developers with regards to the performance evaluation of their applications by means of simulation.

A model for estimating energy consumption based on resources utilization

Over the recent years, managing the energy efficiency of ICT resources, mainly in data centers, has become a priority. Energy efficiency improvement involves numerous ICT initiatives, from the development of more efficient devices, to virtualization and resource consolidation. However, the management of energy efficiency requires dynamic information on the energy consumption, as well as a thorough understanding on how the utilization of computing resources influences energy consumption. In this paper, we propose and evaluate a computationally inexpensive method that indirectly measures energy consumption with a good accuracy. Our model estimates the dynamic energy consumption (given as the amount of watts consumed at a given time), based on information about the utilization of computing resources coming from the OS (e.g., memory, processors, I/O operations, etc). Experiments demonstrate that our method achieves an accuracy of 85.23% and an average random error of 8.6 watts compared to a power meter. Our studies also show which resources have the most impact on energy consumption. Our model can be easily implemented in software, and deployed in data center machines for energy management.

Personalização em Sistemas de Gerenciamento da Aprendizagem em Conformidade com o Padrão SCORM

In the presence of the actual world context, the importance of knowing the technological resources and making use of them to improve the performance of human activities is realized. Educat ion cannot be laid aside of this reality, on the contrary, it has to be allied and seek for resources to improve even more the process of teaching and learn ing through Information and Communication Technologies (ICTs). Thus, the dof Araucaria/PR, through the Educational Technology Project, has pro vided access to these resources establishing informatics laboratories, ev aluating and reorganizing the used method, as an example, the accomplishment of the Leader-Teacher function related to the laboratory monitor one. Resumo. Diante do contexto do mundo atual, percebe-se a imp ortância de conhecer os recursos tecnologicos e fazer uso deste s para melhorar o desempenho das atividades humanas. A educacao nao pficar a margem desta realidade, ao contrario, deve buscar meios de aprimorar o processo de ensino e aprendizagem por meio das Tecnologias da I nformacao e Comunicacao (TICs). Assim, o municipio de Araucari a/PR, atraves do Projeto de Tecnologia Educacional, tem viabilizado o acesso a estes recursos com a implantacao de laboratorios de informatica, ae reorganizando os metodos empregados, como por exemplo, a implemen tacao da funcao de Professor-orientador em detrimento a funcao do moni tor de laboratorio.

Supporting the collaborative construction of learning objects using the grid

This paper presents the collaborative construction of learning objects (LOs) supported by grid services. The needs of the different users (e.g. teachers, designers, and developers) are taken into account to analyse the cycle of LOs production. The analysis identifies the functionalities that should be made available and the services that enable a grid platform to support the collaboration among these users. The grid infrastructure and respective set of services provide uniform access to distributed and heterogeneous resources for geographically dispersed users in the collaborative construction of LOs. The proposed infrastructure is implemented in a prototype.

Applying Machine Learning to Reduce Overhead in DTN Vehicular Networks

VANETs benefit from Delay Tolerant Networks (DTNs) routing algorithms when connectivity is intermittent because of the fast movement of vehicles. Multi-copy DTN algorithms spread message copies to increase the delivery probability but increasing network overhead. In this work we apply machine learning algorithms to reduce network overhead by discriminating the worst intermediate nodes for the transmission of copies. The scenario is a VANET of public buses that follow specific routes and schedules. This repetitive behavior creates an opportunity for applying trained classifiers to predict the occurrence of performance-related events. As the main contribution, our method decreases overhead without degrading delivery probability.

BOUNDS ON THE SCALABILITY OF BAG-OF-TASKS APPLICATIONS RUNNING ON MASTER-SLAVE PLATFORMS

Bag-of-Tasks applications are parallel applications composed of independent (i.e., embarrassingly parallel) tasks that do not communicate with each other, may depend upon one or more input files, and can be executed in any order. Each file may be input for more than one task. A common framework to execute BoT applications is the master-slave topology, in which the user machine is used to control the execution of tasks. In this scenario, a large number of concurrent tasks competing for resources (e.g., CPU and communication links) severely limits the scalability. In this paper we studied the scalability of BoT applications running on multi-node systems (e.g. clusters and grids) organized as master-slave platforms, considering two communications paradigms: multiplexed connections and efficient broadcast. We prove that the lowest bound possible on the isoefficiency function for master-slave platforms is achievable by those platforms that have an O(1) efficient broadcast primitive available. We also analyze the impact of output file contention in scalability, under different assumptions. Our study employs a set of simulation experiments that confirms and extends the theoretical results (e.g. by simulating TCP links).