Reproduction for Insight: Towards Better Understanding the Quality of Students Tests

Abstract

Unit testing is an essential part of computer science education. Students leaving the university should be able to test at a high quality. Traditionally, teaching assistants or teachers themselves check projects and assignments manually. Clearly, this does not scale well. Furthermore, such methods are not suitable to easily get an overview of the overall student performance. A more efficient approach is the so-called all-pairs method, applied by the study of Edwards and Shams (ITiCSE, 2014). Each student implements both the unit (e.g. a class) and a corresponding test suite. Additionally, the teacher creates a reference implementation. The measurement is done by running all tests against all implementations. The present study builds on that research. Our fundamental question is how teachers can get insight into the testing ability of their students. In particular, we develop a software framework and a replicable method. Students have to solve an implementation assignment. Assembling the results, the software provides an overview to the teachers about the test quality of each student s work as well as of the overall testing ability of students. In this study, we replicate and improve the method of Edwards and Shams. We perform measurements to get a better understanding of the testing ability of our students ($N=98$). Additionally, we lay the foundations to make the measurement of students test quality replicable; this enables us later to compare different study programs or the same student population before and after a given course.