The multiple facets of test case quality: analyzing effectiveness and going beyond

Abstract

Nowadays, software pervades our life. Being software so deeply rooted into our society, software failures can cause enormous consequences. Unit test cases represent the first line of defense against the introduction of software bugs and a pillar of any software development pipeline. \nHigher is their quality, the better they can fulfill their role. This research aims at supporting developers in measuring and optimizing test suite quality. To fulfill this goal, we fist characterized the test code quality aspects deemed important by practitioners. We learned that test quality does not have an exact definition and includes a variety of different facets. We also discovered that, while developers value test effectiveness, they believe it is not sufficient to achieve test quality since non-functional aspects also play a crucial role in it. \nThese insights motivated us to devise novel approaches to measure and optimize test effectiveness and non-functional quality aspects both in the context of manually written and automatically generated tests. \nWhile mutation testing is widely used to measure effectiveness, its computational cost hinders its practical usage. We tackled the problem by exploiting machine learning (ML) models trained on source code features to estimate test effectiveness. \nWe relied on similar features to tackle the problem of code coverage prediction in the context of test case generation (TCG). \nThe ML models we proposed are able to suggest developers whether TCG is able to produce satisfactory result for their software projects. \nTo optimize non-functional aspects along with code coverage in TCG, we proposed an adaptive search-based algorithm suitable to arbitrary secondary objectives. We instantiated it to focus on test resource demands, obtaining more parsimonious tests at equal levels of code coverage.