How Students Unit Test: Perceptions, Practices, and Pitfalls

Abstract

Unit testing is reported as one of the skills that graduating students lack, yet it is an essential skill for professional software developers. Understanding the challenges students face during testing can help inform practices for software testing education. To that end, we conduct an exploratory study to reveal students perceptions of unit testing and challenges students encounter when practicing unit testing. We surveyed 54 students from two universities and gave them two testing tasks, one involving black-box test design and one involving white-box test implementation. For the tasks, we used two software projects from prior work in studying test-first development among software developers. We quantitatively analyzed the survey responses and test code properties, and qualitatively identified the mistakes and smells in the test code. We further report on our experience running this study with students. Our results regarding student perceptions show that students believe code coverage is the most important outcome for test suites. For testing practices, most students were ineffective in finding known defects. This may be due to the task design and/or challenges with understanding the source code. For testing pitfalls, we identified six test smells from student-written test code; the most common were ignoring setups in the test code and testing happy path only. These results suggest the students needed more introduction to these common testing concepts and practices in advance of the study activity. Through this experience, we have identified testing concepts that require emphasis for more effective future studies on testing behavior among students.