Aleksander Fabijan

Customer Feedback and Data Collection Techniques in Software R&D: A Literature Review

In many companies, product management struggles in getting accurate customer feedback. Often, validation and confirmation of functionality with customers takes place only after the product has been deployed, and there are no mechanisms that help product managers to continuously learn from customers. Although there are techniques available for collecting customer feedback, these are typically not applied as part of a continuous feedback loop. As a result, the selection and prioritization of features becomes far from optimal, and product deviates from what the customers need. In this paper, we present a literature review of currently recognized techniques for collecting customer feedback. We develop a model in which we categorize the techniques according to their characteristics. The purpose of this literature review is to provide an overview of current software engineering research in this area and to better understand the different techniques that are used for collecting customer feedback.

The Benefits of Controlled Experimentation at Scale

Online controlled experiments (for example A/B tests) are increasingly being performed to guide product development and accelerate innovation in online software product companies. The benefits of controlled experiments have been shown in many cases with incremental product improvement as the objective. In this paper, we demonstrate that the value of controlled experimentation at scale extends beyond this recognized scenario. Based on an exhaustive and collaborative case study in a large software-intensive company with highly developed experimentation culture, we inductively derive the benefits of controlled experimentation. The contribution of our paper is twofold. First, we present a comprehensive list of benefits and illustrate our findings with five case examples of controlled experiments conducted at Microsoft. Second, we provide guidance on how to achieve each of the benefits. With our work, we aim to provide practitioners in the online domain with knowledge on how to use controlled experimentation to maximize the benefits on the portfolio, product and team level.

Time to Say 'Good Bye': Feature Lifecycle

With continuous deployment of software functionality, a constant flow of new features to products is enabled. Although new functionality has potential to deliver improvements and possibilities that were previously not available, it does not necessary generate business value. On the contrary, with fast and increasing system complexity that is associated with high operational costs, more waste than value risks to be created. Validating how much value a feature actually delivers, project how this value will change over time, and know when to remove the feature from the product are the challenges large software companies increasingly experience today. We propose and study the concept of a software feature lifecycle from a value point of view, i.e. how companies track feature value throughout the feature lifecycle. The contribution of this paper is a model that illustrates how to determine (1) when to add the feature to a product, (2) how to track and (3) project the value of the feature during the lifecycle, and how to (4) identify when a feature is obsolete and should be removed from the product.

The Lack of Sharing of Customer Data in Large Software Organizations: Challenges and Implications

With agile teams becoming increasingly multi-disciplinary and including all functions, the role of customer feedback is gaining momentum. Today, companies collect feedback directly from customers, as well as indirectly from their products. As a result, companies face a situation in which the amount of data from which they can learn about their customers is larger than ever before. In previous studies, the collection of data is often identified as challenging. However, and as illustrated in our research, the challenge is not the collection of data but rather how to share this data among people in order to make effective use of it. In this paper, and based on case study research in three large software-intensive companies, we (1) provide empirical evidence that ‘lack of sharing’ is the primary reason for insufficient use of customer and product data, and (2) develop a model in which we identify what data is collected, by whom data is collected and in what development phases it is used. In particular, the model depicts critical hand-overs where certain types of data get lost, as well as the implications associated with this. We conclude that companies benefit from a very limited part of the data they collect, and that lack of sharing of data drives inaccurate assumptions of what constitutes customer value.

Early Value Argumentation and Prediction: An Iterative Approach to Quantifying Feature Value

Companies are continuously improving their practices and ways of working in order to fulfill always-changing market requirements. As an example of building a better understanding of their customers, organizations are collecting user feedback and trying to direct their R&D efforts by e.g. continuing to develop features that deliver value to the customer. We 1 develop an actionable technique that practitioners in organizations can use to validate feature value early in the development cycle, 2 validate if and when the expected value reflects on the customers, 3 know when to stop developing it, and 4 identity unexpected business value early during development and redirect R&D effort to capture this value. The technique has been validated in three experiments in two cases companies. Our findings show that predicting value for features under development helps product management in large organizations to correctly re-prioritize R&D investments.

Competitive Online Clique Clustering

Clique clustering is the problem of partitioning a graph into cliques so that some objective function is optimized. In online clustering, the input graph is given one vertex at a time, and any vertices that have previously been clustered together are not allowed to be separated. The objective here is to maintain a clustering the never deviates too far in the objective function compared to the optimal solution. We give a constant competitive upper bound for online clique clustering, where the objective function is to maximize the number of edges inside the clusters. We also give almost matching upper and lower bounds on the competitive ratio for online clique clustering, where we want to minimize the number of edges between clusters. In addition, we prove that the greedy method only gives linear competitive ratio for these problems.

Commodity eats innovation for breakfast: A model for differentiating feature realization

Once supporting the electrical and mechanical functionality, software today became the main competitive advantage in products. However, in the companies that we study, the way in which software features are developed still reflects the traditional ‘requirements over the wall’ approach. As a consequence, individual departments prioritize what they believe is the most important and are unable to identify which features are regularly used – ‘flow’, there to be bought – ‘wow’, differentiating and that add value to customers, or which are regarded commodity. In this paper, and based on case study research in three large software-intensive companies, we (1) provide empirical evidence that companies do not distinguish between different types of features, which causes poor allocation of R&D efforts and suppresses innovation, and (2) develop a model in which we depict the activities for differentiating and working with different types of features and stakeholders.

Differentiating feature realization in software product development

Context: Software is no longer only supporting mechanical and electrical products. Today, it is becoming the main competitive advantage and an enabler of innovation. Not all software, however, has an equal impact on customers. Companies still struggle to differentiate between the features that are regularly used, there to be for sale, differentiating and that add value to customers, or which are regarded commodity. Goal: The aim of this paper is to (1) identify the different types of software features that we can find in software products today, and (2) recommend how to prioritize the development activities for each of them. Method: In this paper, we conduct a case study with five large-scale software intensive companies. Results: Our main result is a model in which we differentiate between four fundamentally different types of features (e.g. ‘Checkbox’, ‘Flow’, ‘Duty’ and ‘Wow’). Conclusions: Our model helps companies in (1) differentiating between the feature types, and (2) selecting an optimal methodology for their development (e.g. ‘Output-Driven’ vs. ‘Outcome-Driven’).

Experimentation that Matters: A Multi-case Study on the Challenges with A/B Testing

From having been exclusive for companies in the online domain, feature experiments are becoming increasingly important for software-intensive companies also in other domains. Today, companies run experiments, such as e.g. A/B tests, to optimize product performance and to learn about user behaviors, as well as to guide product development and innovation. However, although experimentation with customers has become an effective mechanism to improve products and increase revenue, companies struggle with how to leverage the results of the experiments they run. In this paper, we study the reasons for this and we identify three key challenges that make feature experimentation a difficult task. Our research reveals the following challenges: (1) the impact of experiments doesn’t scale, (2) business KPIs and team level metrics are not aligned and (3) it is unclear if the available solutions are applicable across domains.