Use Bots to Improve GitHub Pull-Request Feedback

Abstract

Rising enrollments make it difficult for instructors and teaching assistants to give adequate feedback on each student s work. In our software engineering course, we have 50-120 students each semester. Our course projects require students to submit GitHub pull requests as deliverables for their open-source software (OSS) projects. We have set up a static code analyzer and a continuous integration service on GitHub to help students check code style and functionality. However, these tools cannot enforce system-specific customized guidelines and do not explicitly display detailed information. In this study, we discuss how we bypass the limitations of existing tools by implementing three Internet bots. The Expertiza Bot can help detect violations of more than 35 system-specific guidelines. The Travis CI Bot can explicitly display instant test execution results on the GitHub pull-request page. The Code Climate Bot can insert pull-request comments to remind students to fix issues detected by the static code analyzer. These bots are either open source or free for OSS projects, and can be easily integrated with GitHub repositories. Our survey results show that more than 70% of students think the advice given by the bots is useful. We tallied the amount of feedback given by the bots and the teaching staff for each GitHub pull request. Results show that bots can provide significantly more feedback (six times more on average) than teaching staff. Bots can also offer more timely feedback than teaching staff and help student contributions avoid more than 33% system-specific guideline violations.