Jianmei Guo

Variability-aware performance prediction: a statistical learning approach

Configurable software systems allow stakeholders to derive program variants by selecting features. Understanding the correlation between feature selections and performance is important for stakeholders to be able to derive a program variant that meets their requirements. A major challenge in practice is to accurately predict performance based on a small sample of measured variants, especially when features interact. We propose a variability-aware approach to performance prediction via statistical learning. The approach works progressively with random samples, without additional effort to detect feature interactions. Empirical results on six real-world case studies demonstrate an average of 94% prediction accuracy based on small random samples. Furthermore, we investigate why the approach works by a comparative analysis of performance distributions. Finally, we compare our approach to an existing technique and guide users to choose one or the other in practice.

Comparison of exact and approximate multi-objective optimization for software product lines

Software product lines (SPLs) allow stakeholders to manage product variants in a systematical way and derive variants by selecting features. Finding a desirable variant is often difficult, due to the huge configuration space and usually conflicting objectives (e.g., lower cost and higher performance). This scenario can be characterized as a multi-objective optimization problem applied to SPLs. We address the problem using an exact and an approximate algorithm and compare their accuracy, time consumption, scalability, parameter setting requirements on five case studies with increasing complexity. Our empirical results show that (1) it is feasible to use exact techniques for small SPL multi-objective optimization problems, and (2) approximate methods can be used for large problems but require substantial effort to find the best parameter setting for acceptable approximation which can be ameliorated with known good parameter ranges. Finally, we discuss the tradeoff between accuracy and time consumption when using exact and approximate techniques for SPL multi-objective optimization and guide stakeholders to choose one or the other in practice.

Coevolution of variability models and related artifacts: a case study from the Linux kernel

Variability-aware systems are subject to the coevolution of variability models and related artifacts. Surprisingly, little knowledge exists to understand such coevolution in practice. This shortage is directly reflected in existing approaches and tools for variability management, as they fail to provide effective support for such a coevolution. To understand how variability models and related artifacts coevolve in a large and complex real-world variability-aware system, we inspect over 500 Linux kernel commits spanning almost four years of development. We collect a catalog of evolution patterns, capturing the coevolution of the Linux kernel variability model, Makefiles, and C source code. Further, we extract general findings to guide further research and tool development.

Cost-Efficient Sampling for Performance Prediction of Configurable Systems (T)

A key challenge of the development and maintenanceof configurable systems is to predict the performance ofindividual system variants based on the features selected. It isusually infeasible to measure the performance of all possible variants, due to feature combinatorics. Previous approaches predictperformance based on small samples of measured variants, butit is still open how to dynamically determine an ideal samplethat balances prediction accuracy and measurement effort. Inthis paper, we adapt two widely-used sampling strategies forperformance prediction to the domain of configurable systemsand evaluate them in terms of sampling cost, which considersprediction accuracy and measurement effort simultaneously. Togenerate an initial sample, we introduce a new heuristic based onfeature frequencies and compare it to a traditional method basedon t-way feature coverage. We conduct experiments on six realworldsystems and provide guidelines for stakeholders to predictperformance by sampling.

Feature-oriented software evolution

In this paper, we develop a vision of software evolution based on a feature-oriented perspective. From the fact that features provide a common ground to all stakeholders, we derive a hypothesis that changes can be effectively managed in a feature-oriented manner. Assuming that the hypothesis holds, we argue that feature-oriented software evolution relying on automatic traceability, analyses, and recommendations reduces existing challenges in understanding and managing evolution. We illustrate these ideas using an automotive example and raise research questions for the community.

Scaling exact multi-objective combinatorial optimization by parallelization

Multi-Objective Combinatorial Optimization (MOCO) is fundamental to the development and optimization of software systems. We propose five novel parallel algorithms for solving MOCO problems exactly and efficiently. Our algorithms rely on off-the-shelf solvers to search for exact Pareto-optimal solutions, and they parallelize the search via collaborative communication, divide-and-conquer, or both. We demonstrate the feasibility and performance of our algorithms by experiments on three case studies of software-system designs. A key finding is that one algorithm, which we call FS-GIA, achieves substantial (even super-linear) speedups that scale well up to 64 cores. Furthermore, we analyze the performance bottlenecks and opportunities of our parallel algorithms, which facilitates further research on exact, parallel MOCO.

Coevolution of variability models and related software artifacts

Variant-rich software systems offer a large degree of customization, allowing users to configure the target system according to their preferences and needs. Facing high degrees of variability, these systems often employ variability models to explicitly capture user-configurable features (e.g., systems options) and the constraints they impose. The explicit representation of features allows them to be referenced in different variation points across different artifacts, enabling the latter to vary according to specific feature selections. In such settings, the evolution of variability models interplays with the evolution of related artifacts, requiring the two to evolve together, or coevolve. Interestingly, little is known about how such coevolution occurs in real-world systems, as existing research has focused mostly on variability evolution as it happens in variability models only. Furthermore, existing techniques supporting variability evolution are usually validated with randomly-generated variability models or evolution scenarios that do not stem from practice. As the community lacks a deep understanding of how variability evolution occurs in real-world systems and how it relates to the evolution of different kinds of software artifacts, it is not surprising that industry reports existing tools and solutions ineffective, as they do not handle the complexity found in practice. Attempting to mitigate this overall lack of knowledge and to support tool builders with insights on how variability models coevolve with other artifact types, we study a large and complex real-world variant-rich software system: the Linux kernel. Specifically, we extract variability-coevolution patterns capturing changes in the variability model of the Linux kernel with subsequent changes in Makefiles and C source code. From the analysis of the patterns, we report on findings concerning evolution principles found in the kernel, and we reveal deficiencies in existing tools and theory when handling changes captured by our patterns.

Performance Prediction of Configurable Software Systems by Fourier Learning (T)

Understanding how performance varies across a large number of variants of a configurable software system is important for helping stakeholders to choose a desirable variant. Given a software system with n optional features, measuring all its 2n possible configurations to determine their performances is usually infeasible. Thus, various techniques have been proposed to predict software performances based on a small sample of measured configurations. We propose a novel algorithm based on Fourier transform that is able to make predictions of any configurable software system with theoretical guarantees of accuracy and confidence level specified by the user, while using minimum number of samples up to a constant factor. Empirical results on the case studies constructed from real-world configurable systems demonstrate the effectiveness of our algorithm.

Towards system analysis with variability model metrics

Variability models are central artifacts in highly configurable systems. They aim at planning, developing, and configuring systems by describing configuration knowledge at different levels of formality. The existence of large models using a variety of modeling concepts in heterogeneous languages with intricate semantics calls for a unified measuring approach. In this position paper, we attempt to take a first step towards such a measurement. We discuss perspectives of metrics, define low-level measurement goals, and conceive and implement metrics based on variability modeling concepts found in real-world languages and models. An evaluation of these metrics with real-world models and codebases provides insight into the benefits of such metrics for the defined perspectives.

Empirical comparison of regression methods for variability-aware performance prediction

Product line engineering derives product variants by selecting features. Understanding the correlation between feature selection and performance is important for stakeholders to acquire a desirable product variant. We infer such a correlation using four regression methods based on small samples of measured configurations, without additional effort to detect feature interactions. We conduct experiments on six real-world case studies to evaluate the prediction accuracy of the regression methods. A key finding in our empirical study is that one regression method, called Bagging, is identified as the best to make accurate and robust predictions for the studied systems.