Why Software Testing Fails: Common Pitfalls Observed in a Critical Smart Metering Project

Abstract

Over the last decades a considerable share of software engineering research has been dedicated to the area of software testing. Still, however, testing often fails or causes major problems in practice. In this paper we present insights and experiences from a large project in the energy sector. The obligatory switch from analog energy meters to smart metering technology poses a big challenge for many energy providers. Apart from technical issues concerning meters and transmission technology, the adaption of the internal business processes together with the development of backend software can turn out to be more difficult than expected. The criticality, size and complexity of the analyzed project are reflected in software and system testing, where the underestimated effort, mistakes, and wrong decisions caused serious difficulties. In our work we describe the observed testing problems and the underlying causes. Subsequently, we compare the identified problems with a catalogue of commonly known testing pitfalls and anti-patterns. The results show that the majority of the observed problems are not new or specific to the studied project. Furthermore, additional candidates for extending the list of common pitfalls are identified. Besides recommendations on how to mitigate the problems in the studied project, we conclude with the general insight that there is a great potential to improve software testing practice by developing measures for early recognition, communication, and avoiding of common mistakes.