Sergiy S. Kolesnikov

Predicting performance via automated feature-interaction detection

Customizable programs and program families provide user-selectable features to allow users to tailor a program to an application scenario. Knowing in advance which feature selection yields the best performance is difficult because a direct measurement of all possible feature combinations is infeasible. Our work aims at predicting program performance based on selected features. However, when features interact, accurate predictions are challenging. An interaction occurs when a particular feature combination has an unexpected influence on performance. We present a method that automatically detects performance-relevant feature interactions to improve prediction accuracy. To this end, we propose three heuristics to reduce the number of measurements required to detect interactions. Our evaluation consists of six real-world case studies from varying domains (e.g., databases, encoding libraries, and web servers) using different configuration techniques (e.g., configuration files and preprocessor flags). Results show an average prediction accuracy of 95%.

Scalable Prediction of Non-functional Properties in Software Product Lines

A software product line is a family of related software products, typically, generated from a set of common assets. Users can select features to derive a product that fulfills their needs. Often, users expect a product to have specific non-functional properties, such as a small footprint or a minimum response time. Because a product line can contain millions of products, it is usually not feasible to generate and measure non-functional properties for each possible product of a product line. Hence, we propose an approach to predict a products non-functional properties, based on the products feature selection. To this end, we generate and measure a small set of products, and by comparing the measurements, we approximate each features non-functional properties. By aggregating the approximations of selected features, we predict the products properties. Our technique is independent of the implementation approach and language. We show how already little domain knowledge can improve predictions and discuss trade-offs regarding accuracy and the required number of measurements. Although our approach is in general applicable for quantifiable non-functional properties, we evaluate it for the non-functional property footprint. With nine case studies, we demonstrate that our approach usually predicts the footprint with an accuracy of 98% and an accuracy of over 99% if feature interactions are known.

Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption

A software product line is a family of related software products, typically, generated from a set of common assets. Users can select features to derive a product that fulfills their needs. Often, users expect a product to have specific non-functional properties, such as a small footprint or a minimum response time. Because a product line can contain millions of products, it is usually not feasible to generate and measure non-functional properties for each possible product of a product line. Hence, we propose an approach to predict a products non-functional properties, based on the products feature selection. To this end, we generate and measure a small set of products, and by comparing the measurements, we approximate each features non-functional properties. By aggregating the approximations of selected features, we predict the products properties. Our technique is independent of the implementation approach and language. We show how already little domain knowledge can improve predictions and discuss trade-offs regarding accuracy and the required number of measurements. Although our approach is in general applicable for quantifiable non-functional properties, we evaluate it for the non-functional property footprint. With nine case studies, we demonstrate that our approach usually predicts the footprint with an accuracy of 98% and an accuracy of over 99% if feature interactions are known.

Exploring feature interactions in the wild: the new feature-interaction challenge

The feature-interaction problem has been keeping researchers and practitioners in suspense for years. Although there has been substantial progress in developing approaches for modeling, detecting, managing, and resolving feature interactions, we lack sufficient knowledge on the kind of feature interactions that occur in real-world systems. In this position paper, we set out the goal to explore the nature of feature interactions systematically and comprehensively, classified in terms of order and visibility. Understanding this nature will have significant implications on research in this area, for example, on the efficiency of interaction-detection or performance-prediction techniques. A set of preliminary results as well as a discussion of possible experimental setups and corresponding challenges give us confidence that this endeavor is within reach but requires a collaborative effort of the community.

Access control in feature-oriented programming

In feature-oriented programming (FOP) a programmer decomposes a program in terms of features. Ideally, features are implemented modularly so that they can be developed in isolation. Access control mechanisms in the form of access or visibility modifiers are an important ingredient to attain feature modularity as they allow programmers to hide and expose internal details of a modules implementation. But developers of contemporary feature-oriented languages have not considered access control mechanisms so far. The absence of a well-defined access control model for FOP breaks encapsulation of feature code and leads to unexpected program behaviors and inadvertent type errors. We raise awareness of this problem, propose three feature-oriented access modifiers, and present a corresponding access modifier model. We offer an implementation of the model on the basis of a fully-fledged feature-oriented compiler. Finally, by analyzing ten feature-oriented programs, we explore the potential of feature-oriented modifiers in FOP.

A comparison of product-based, feature-based, and family-based type checking

Analyzing software product lines is difficult, due to their inherent variability. In the past, several strategies for product-line analysis have been proposed, in particular, product-based, feature-based, and family-based strategies. Despite recent attempts to conceptually and empirically compare different strategies, there is no work that empirically compares all of the three strategies in a controlled setting. We close this gap by extending a compiler for feature-oriented programming with support for product-based, feature-based, and family-based type checking. We present and discuss the results of a comparative performance evaluation that we conducted on a set of 12 feature-oriented, Java-based product lines. Most notably, we found that the family-based strategy is superior for all subject product lines: it is substantially faster, it detects all kinds of errors, and provides the most detailed information about them.

Quantifying structural attributes of system decompositions in 28 feature-oriented software product lines

A key idea of feature orientation is to decompose a software product line along the features it provides. Feature decomposition is orthogonal to object-oriented decomposition—it crosscuts the underlying package and class structure. It has been argued often that feature decomposition improves system structure by reducing coupling and by increasing cohesion. However, recent empirical findings suggest that this is not necessarily the case. In this exploratory, observational study, we investigate the decompositions of 28 feature-oriented software product lines into classes, features, and feature-specific class fragments. The product lines under investigation are implemented using the feature-oriented programming language Fuji. In particular, we quantify and compare the internal attributes import coupling and cohesion of the different product-line decompositions in a systematic, reproducible manner. For this purpose, we adopt three established software measures (e.g., coupling between units, CBU; internal-ratio unit dependency, IUD) as well as standard concentration statistics (e.g., Gini coefficient). In our study, we found that feature decomposition can be associated with higher levels of structural coupling in a product line than a decomposition into classes. Although coupling can be concentrated in very few features in most feature decompositions, there are not necessarily hot-spot features in all product lines. Interestingly, feature cohesion is not necessarily higher than class cohesion, whereas features are more equal in serving dependencies internally than classes of a product line. Our empirical study raises critical questions about alleged advantages of feature decomposition. At the same time, we demonstrate how our measurement approach of coupling and cohesion has potential to support static and dynamic analyses of software product lines (i.e., type checking and feature-interaction detection) by facilitating product sampling.

On the relation between internal and external feature interactions in feature-oriented product lines: a case study

The feature-interaction problem has been explored for many years. Still, we lack sufficient knowledge about the interplay of different kinds of interactions in software product lines. Exploring the relations between different kinds of feature interactions will allow us to learn more about the nature of interactions and their causes. This knowledge can then be applied for improving existing approaches for detecting, managing, and resolving feature interactions. We present a framework for studying relations between different kinds of interactions. Furthermore, we report and discuss the results of a preliminary study in which we examined correlations between internal feature interactions (quantified by a set of software measures) and external feature interactions (represented by product-line-specific type errors). We performed the evaluation on a set of 15 feature-oriented, Java-based product lines. We observed moderate correlations between the interactions under discussion. This gives us confidence that we can apply our approach to studying other types of external feature interactions (e.g., performance interactions).

Tradeoffs in modeling performance of highly configurable software systems

Modeling the performance of a highly configurable software system requires capturing the influences of its configuration options and their interactions on the system’s performance. Performance-influence models quantify these influences, explaining this way the performance behavior of a configurable system as a whole. To be useful in practice, a performance-influence model should have a low prediction error, small model size, and reasonable computation time. Because of the inherent tradeoffs among these properties, optimizing for one property may negatively influence the others. It is unclear, though, to what extent these tradeoffs manifest themselves in practice, that is, whether a large configuration space can be described accurately only with large models and significant resource investment. By means of 10 real-world highly configurable systems from different domains, we have systematically studied the tradeoffs between the three properties. Surprisingly, we found that the tradeoffs between prediction error and model size and between prediction error and computation time are rather marginal. That is, we can learn accurate and small models in reasonable time, so that one performance-influence model can fit different use cases, such as program comprehension and performance prediction. We further investigated the reasons for why the tradeoffs are marginal. We found that interactions among four or more configuration options have only a minor influence on the prediction error and that ignoring them when learning a performance-influence model can save a substantial amount of computation time, while keeping the model small without considerably increasing the prediction error. This is an important insight for new sampling and learning techniques as they can focus on specific regions of the configuration space and find a sweet spot between accuracy and effort. We further analyzed the causes for the configuration options and their interactions having the observed influences on the systems’ performance. We were able to identify several patterns across subject systems, such as dominant configuration options and data pipelines, that explain the influences of highly influential configuration options and interactions, and give further insights into the domain of highly configurable systems.