Christoph Heger

Automated root cause isolation of performance regressions during software development

Performance is crucial for the success of an application. To build responsive and cost efficient applications, software engineers must be able to detect and fix performance problems early in the development process. Existing approaches are either relying on a high level of abstraction such that critical problems cannot be detected or require high manual effort. In this paper, we present a novel approach that integrates performance regression root cause analysis into the existing development infrastructure using performance-aware unit tests and the revision history. Our approach is easy to use and provides software engineers immediate insights with automated root cause analysis. In a realistic case study based on the change history of Apache Commons Math, we demonstrate that our approach can automatically detect and identify the root cause of a major performance regression.

Automatic detection of performance anti-patterns in inter-component communications

Performance problems such as high response times in software applications have a significant effect on the customers satisfaction. In enterprise applications, performance problems are frequently manifested in inefficient or unnecessary communication patterns between software components originating from poor architectural design or implementation. Due to high manual effort, thorough performance analysis is often neglected, in practice. In order to overcome this problem, automated engineering approaches are required for the detection of performance problems. In this paper, we introduce several heuristics for measurement-based detection of well-known performance anti-patterns in inter-component communications. The detection heuristics comprise load and instrumentation descriptions for performance tests as well as corresponding detection rules. We integrate these heuristics with Dynamic Spotter, a framework for automatic detection of performance problems. We evaluate our heuristics on four evaluation scenarios based on an e-commerce benchmark (TPC-W) where the heuristics detect the expected communication performance anti-patterns and pinpoint their root causes.

Deriving Work Plans for Solving Performance and Scalability Problems

The performance of an enterprise application (e.g. response time, throughput, or resource utilization) is an important quality attribute that can have a significant impact on a company’s success. When a performance problem such as a performance bottleneck has been detected, the root cause identified and a solution proposed, developers have to identify the elements of the application often manually that will undergo changes and determine how these elements must be changed in order to implement the solution. Many existing approaches are able to identify the elements that have to be modified but only few are able to determine the necessary types of changes on these elements. Neither of the approaches supports developers with a work plan sketching the implementation steps. In this paper, we propose an approach to point developers the way torwards an implementation of a performance or scalability solution with an ordered set of work activities. Rules are used to derive a work plan sketching the implementation of a solution for the particular application based on an initial set of work activities. The rule-based approach identifies impacted elements and determines how they should be changed. We demonstrate the proposed approach with a solution of a performance bottleneck as an example.

Systematic performance evaluation based on tailored benchmark applications

Performance (i.e., response time, throughput, resource consumption) is a key quality metric of todays applications as it heavily affects customer satisfaction. SAP strives to identify and fix performance problems before customers face them. Therefore, performance engineering methods are applied in all stages of the software lifecycle. However, especially in the development phase continuous performance evaluations can introduce a lot of overhead for developers which hinders their broad application in practice. In order to evaluate the performance of a certain software artefact (e.g. comparing two design alternatives), a developer has to run measurements that are tailored to the software artefact under test. The use of standard benchmarks would create less overhead, but the information gain is often not sufficient to answer the specific questions of developers. In this industrial paper, we present an approach that enables exhaustive, tailored performance testing with minimal effort for developers. The approach allows to define benchmark applications through a domain-specific model and realizes the transformation of those models to benchmark applications via a generic Benchmark Framework. The application of the approach in the context of the SAP Netweaver Cloud development environment demonstrated that we can efficiently identify performance problems that would not have been detected by our existing performance test infrastructure.

Towards Performance Tooling Interoperability: An Open Format for Representing Execution Traces

Execution traces capture information on a software system’s runtime behavior, including data on system-internal software control flows, performance, as well as request parameters and values. In research and industrial practice, execution traces serve as an important basis for model-based and measurement-based performance evaluation, e.g., for application performance monitoring (APM), extraction of descriptive and prescriptive models, as well as problem detection and diagnosis. A number of commercial and open-source APM tools that allow the capturing of execution traces within distributed software systems is available. However, each of the tools uses its own (proprietary) format, which means that each approach building on execution trace data is tool-specific.

Expert-Guided Automatic Diagnosis of Performance Problems in Enterprise Applications

Application performance management (APM) is a necessity to detect and solve performance problems during operation of enterprise applications. While existing tools provide alerting and visualization capabilities when performance requirements are violated during operation, the isolation and diagnosis of the problems real root cause is the responsibility of the rare performance expert, often resulting in a boring and recurring task. Main challenges for APM adoption in practice include that initial setup and maintenance of APM, and particularly the diagnosis of performance problems are error-prone, costly, and require a high manual effort and expertise. In this paper, we present preliminary work on diagnoseIT, an approach that utilizes formalized APM expert knowledge to automate the aforementioned recurring APM activities.

Application Performance Management: State of the Art and Challenges for the Future

The performance of application systems has a direct impact on business metrics. For example, companies lose customers and revenue in case of poor performance such as high response times. Application performance management (APM) aims to provide the required processes and tools to have a continuous and up-to-date picture of relevant performance measures during operations, as well as to support the detection and resolution of performance-related incidents. In this tutorial paper, we provide an overview of the state of the art in APM in industrial practice and academic research, highlight current challenges, and outline future research directions.

Generic Instrumentation and Monitoring Description for Software Performance Evaluation

Instrumentation and monitoring plays an important role in measurement-based performance evaluation of software systems. To this end, a large body of instrumentation and monitoring tools exist which, however, depend on proprietary and programming-language-specific instrumentation languages. Due to the lack of a common instrumentation language, it is difficult and expensive to port per se generic measurement-based performance evaluation approaches among different application contexts. In this work-in-progress paper, we address this issue by introducing a performance-oriented, generic meta-model for application-independent and tool independent description of instrumentation instructions. Decoupling the instrumentation description from its realization in a concrete application context, by a concrete instrumentation tool allows to design measurement based performance evaluation approaches in a generic and portable way.

AIM: adaptable instrumentation and monitoring for automated software performance analysis

Instrumentation and monitoring plays an important role in measurement-based performance analysis of software systems. However, in practice the performance overhead of extensive instrumentation is not negligible. Experiment-based performance analysis overcomes this problem through a series of experiments on selectively instrumented code, but requires additional manual effort to adjust required instrumentation and hence introduces additional costs. Automating the experiments and selective instrumentation can massively reduce the costs of performance analysis. Such automation, however, requires the capability of dynamically adapting instrumentation instructions. In this paper, we address this issue by introducing AIM, a novel instrumentation and monitoring approach for automated software performance analysis. We apply AIM to automate derivation of resource demands for an architectural performance model, showing that adaptable instrumentation leads to more accurate measurements compared to existing monitoring approaches.

Systematic guidance in solving performance and scalability problems

The performance of enterprise software systems affects business critical metrics like conversion rate (proportion of visitors who become customers) and total cost of ownership for the software system. Keeping such systems responsive and scalable with a growing user base is challenging for software engineers. Solving performance problems is an error-prone and time consuming task that requires deep expert knowledge about the system and performance evaluation. Existing approaches to support the resolution of performance problems mainly focus on the architecture level neglecting influences of the implementation. In this proposal paper, we introduce a novel approach in order to support software engineers in solving performance and scalability problems that leverages known solutions to common problems. The known solutions are evaluated in the context of the particular software system. As a result, a detailed plan is derived that helps and guides software engineers in resolving the problem. We plan to conduct an industrial case study at SAP.