Haroon Malik

Automatic detection of performance deviations in the load testing of large scale systems

Load testing is one of the means for evaluating the performance of Large Scale Systems (LSS). At the end of a load test, performance analysts must analyze thousands of performance counters from hundreds of machines under test. These performance counters are measures of run-time system properties such as CPU utilization, Disk I/O, memory consumption, and network traffic. Analysts observe counters to find out if the system is meeting its Service Level Agreements (SLAs). In this paper, we present and evaluate one supervised and three unsupervised approaches to help performance analysts to 1) more effectively compare load tests in order to detect performance deviations which may lead to SLA violations, and 2) to provide them with a smaller and manageable set of important performance counters to assist in root-cause analysis of the detected deviations. Our case study is based on load test data obtained from both a large scale industrial system and an open source benchmark application. The case study shows, that our wrapper-based supervised approach, which uses a search-based technique to find the best subset of performance counters and a logistic regression model for deviation prediction, can provide up to 89% reduction in the set of performance counters while detecting performance deviations with few false positives (i.e., 95% average precision). The study also shows that the supervised approach is more stable and effective than the unsupervised approaches but it has more overhead due to its semi-automated training phase.

Supporting software evolution using adaptive change propagation heuristics

When changing a source code entity (e.g., a function), developers must ensure that the change is propagated to related entities to avoid the introduction of bugs. Accurate change propagation is essential for the successful evolution of complex software systems. Techniques and tools are needed to support developers in propagating changes. Several heuristics have been proposed in the past for change propagation. Research shows that heuristics based on the change history of a project outperform heuristics based on the dependency graph. However, these heuristics being static are not the answer to the dynamic nature of software projects. These heuristics need to adapt to the dynamic nature of software projects and must adjust themselves for the peculiarities of each changed entity. In this paper we propose adaptive change propagation heuristics. These heuristics are metaheuristics that combine various previously researched heuristics to improve the overall performance (precision and recall) of change propagation heuristics. Through an empirical case study, using four large open source systems; GCC (a compiler), FreeBSD (an operating system), PostgreSQL (a database), and GCluster (a clustering framework), we demonstrate that our adaptive change propagation heuristics show a 57% statistically significant improvement over the top-performing static change propagation heuristics.

Automatic Comparison of Load Tests to Support the Performance Analysis of Large Enterprise Systems

Load testing is crucial to uncover functional and performance bugs in large-scale systems. Load tests generate vast amounts of performance data, which needs to be compared and analyzed in limited time across tests. This helps performance analysts to understand the resource usage of an application and to find out if an application is meeting its performance goals. The biggest challenge for performance analysts is to identify the few important performance counters in the highly redundant performance data. In this paper, we employed a statistical technique, Principal Component Analysis (PCA) to reduce the large volume of performance counter data, to a smaller, more meaningful and manageable set. Furthermore, our methodology automates the process of comparing the important counters across load tests to identify performance gains/losses. A case study on load test data of a large enterprise application shows that our methodology can effectively guide performance analysts to identify and compare top performance counters across tests in limited time.

Software agent-based directed diffusion in wireless sensor network

In an environment where node density is massive, placement is heterogeneous and redundant sensory traffic is produced; limited network resources such as bandwidth and energy are hastily consumed by individual sensor nodes. Equipped with only a limited battery power supply, this minimizes the lifetime of these sensor nodes. At the network layer, many researchers have tackled this issue by proposing several energy efficient routing schemes. All these schemes tend to save energy by elevating redundant data traffic via in-network processing and choosing empirically good and shortest routing paths for transfer of sensory data to a central location (sink) for further, application-specific processing. Seldom has an attempt been made to reduce network traffic by moving the application-specific code to the source nodes. We unmitigated our efforts to augment the node lifetime within a sensor network by introducing mobile agents. These mobile agents can be used to greatly reduce communication costs, especially over low bandwidth links, by moving the processing function to the data rather than bringing the data to a central processor. Toward this end, we propose an agent-based directed diffusion approach to increase sensor node efficiency and we present the experimental results.

Multi-agent-based clustering approach to wireless sensor networks

This paper presents a multi-agent system for hierarchical routing in a Wireless Sensor Network (WSN). The agents communicate and collaborate with each other and benefit from learning techniques, more specifically genetic algorithms. The proposed system consists of four types of agents, including regional, interface, cluster and query agents. The regional agent resides on the base-station and performs genetic algorithm intense computing. The interface agent interacts with the users to fulfil their interests. The cluster agents manage all agents within clusters for query dissemination and efficiency in network. The query agents reside in each sensor and acquire, aggregate, process the useful data, and transmit the desired results. A prototype of all agents is simulated. For a given radio model our test results revealed that our multi-agent-based approach for hierarchical routing not only outperforms other routing protocols such as LEACH, but also determines the set of optimum clusters for various topologies.

Pinpointing the Subsystems Responsible for the Performance Deviations in a Load Test

Large scale systems (LSS) contain multiple subsystems that interact across multiple nodes in sometimes unforeseen and complicated ways. As a result, pinpointing the subsystems that are the source of performance degradation for a load test in LSS can be frustrating, and might take several hours or even days. This is due to the large volume of performance counter data collected such as CPU utilization, Disk I/O, memory consumption and network traffic, the limited operational knowledge of analysts about all subsystems of an LSS and the unavailability of up-to-date documentation in a LSS. We have developed a methodology that automatically ranks the subsystems according to the deviation of their performance in a load test. Our methodology uses performance counter data of a load test to craft performance signatures for the LSS subsystems. Pair-wise correlations among the performance signatures of subsystems within a load test are compared with the corresponding correlations in a baseline test to pinpoint the subsystems responsible for the performance violations. Case studies on load test data obtained from a large telecom system and that of an open source benchmark application show that our approach provides an accuracy of 79% and do not require any instrumentation or domain knowledge to operate.

Understanding the rationale for updating a function’s comment

Up-to-date comments are critical for the successful evolution of a software application. When modifying a function, developers may update the comment associated with the function or may not update it. For example, comments associated with a complex function are likely to be updated more often when the function is modified to prevent the code and the comments from drifting apart. Nevertheless, the rationale behind updating a comment has never been studied. In this paper, we present a large empirical study to better understand the rationale for updating comments. We recover the code change history for four large open source projects (GCC: a compiler, FreeBSD: an operation system, PostgreSQL: a database management system, and GCluster: a clustering framework) with an average code history of 10 years. Using the Random Forests algorithm, we investigate the rationale for updating comments along three dimensions: characteristics of the changed function, characteristics of the change itself and time and code ownership characteristics. Our case study shows that we can predict with an accuracy of 80%; the likelihood of updating the comment associated with a modified function. We perform a sensitivity analysis to determine the most important attributes. Our analysis shows that the percentage of changed call dependencies and control statements, the age of the modified function and the number of co-changed functions which depend on it are the most important attributes in determining the likelihood of updating comments.

WSN in cyber physical systems: Enhanced energy management routing approach using software agents

Recently, the cyber physical system has emerged as a promising direction to enrich the interactions between physical and virtual worlds. Meanwhile, a lot of research is dedicated to wireless sensor networks as an integral part of cyber physical systems. A wireless sensor network (WSN) is a wireless network consisting of spatially distributed autonomous devices that use sensors to monitor physical or environmental conditions. These autonomous devices, or nodes, combine with routers and a gateway to create a typical WSN system. Shrinking size and increasing deployment density of wireless sensor nodes implies the smaller equipped battery size. This means emerging wireless sensor nodes must compete for efficient energy utilization to increase the WSN lifetime. The network lifetime is defined as the time duration until the first sensor node in a network fails due to battery depletion. One solution for enhancing the lifetime of WSN is to utilize mobile agents. In this paper, we propose an agent-based approach that performs data processing and data aggregation decisions locally i.e., at nodes rather than bringing data back to a central processor (sink). Our proposed approach increases the network lifetime by generating an optimal routing path for mobile agents to transverse the network. The proposed approach consists of two phases. In the first phase, Dijkstras algorithm is used to generate a complete graph to connect all source nodes in a WSN. In the second phase, a genetic algorithm is used to generate the best-approximated route for mobile agents in a radio harsh environment to route the sensory data to the base-station. To demonstrate the feasibility of our approach, a formal analysis and experimental results are presented.

Towards Identifying the Challenges Associated with Emerging Large Scale Social Networks

Abstract Social networking sites like Facebook, YouTube, Orkut and twitter are among the most popular sites on the internet. Users of these sites form a social network (SN), which provides a powerful mean of sharing, organizing, and finding contents and contacts. However, the rate at which SNs are growing, posses many latent challenges in maintaining the stability of their underlying systems and the members associated with them. We discus some of the current challenges associated with these emerging SNs; In particular the challenge in the analyzing of large scale SN data, challenge in developing the tools and techniques required to extend these large scale SNs, and privacy issue related to SNs.

A methodology to support load test analysis

Performance analysts rely heavily on load testing to measure the performance of their applications under a given load. During the load test, analyst strictly monitor and record thousands of performance counters to measure the run time system properties such as CPU utilization, Disk I/O, memory consumption, network traffic etc. The most frustrating problem faced by analysts is the time spent and complexity involved in analysing these huge counter logs and finding relevant information distributed across thousands of counters. We present our methodology to help analysts by automatically identifying important performance counters for load test and comparing them across tests to find performance gain/loss. Further, our methodology help analysts to understand the root cause of a load test failure by finding previously solved problems in test repositories. A case study on load test data of a large enterprise application shows that our methodology can effectively guide performance analysts to identify and compare top performance counters across tests in limited time thereby archiving 88% counter data reduction.