Daniel Zingaro

Reviewing CS1 exam question content

Many factors have been cited for poor performance of students in CS1. To investigate how assessment mechanisms may impact student performance, nine experienced CS1 instructors reviewed final examinations from a variety of North American institutions. The majority of the exams reviewed were composed predominantly of high-value, integrative code-writing questions, and the reviewers regularly underestimated the number of CS1 concepts required to answer these questions. An evaluation of the content and cognitive requirements of individual questions suggests that in order to succeed, students must internalize a large amount of CS1 content. This emphasizes the need for focused assessment techniques to provide students with the opportunity to demonstrate their knowledge.

Peer Instruction in computing: The value of instructor intervention

Research has demonstrated that Peer Instruction (PI) is an attractive pedagogical practice in computer science classes. PI has been shown to improve final exam performance over standard lecture, reduce failure rates, contribute to increased retention, and be widely valued by students. In addition, a recent study using isomorphic (same-concept) questions found that students are learning during peer discussion and not merely copying from neighbors. Though this prior work is useful for evaluating peer discussion, it does not capture learning that takes place after peer discussion when the instructor further expands on the concept through a whole-class discussion. In the present work, isomorphic questions were used to determine the value of a PI question from start to finish: solo vote, group discussion, group vote, and instructor-led classwide discussion. The analysis revealed that the value of the instructor-led classwide discussion was evident in increased student performance over peer-discussion alone (raw gains of 22% compared to 14%). Moreover, the instructor-led discussion was highly valuable for all groups of students (weak, average, and strong) and was of particular value for weak students. Importantly, the largest gains were associated with more challenging PI questions, further suggesting that instructor expertise was valuable when students struggled.

Predicting student success using fine grain clicker data

Recent research suggests that the first weeks of a CS1 course have a strong influence on end-of-course student performance. The present work aims to refine the understanding of this phenomenon by using in-class clicker questions as a source of student performance. Clicker questions generate per-lecture and per-question data with which to assess student understanding. This work demonstrates that clicker question performance early in the term predicts student outcomes at the end of the term. The predictive nature of these questions applies to code-writing questions, multiple choice questions, and the final exam as a whole. The most predictive clicker questions are identified and the relationships between these questions and final exam performance are examined.

Comparing outcomes in inverted and traditional CS1

We compare a traditional CS1 offering with an inverted offering delivered the following year to a comparable student population. We measure student attitudes, grades, and final course outcomes and find that, while students in the inverted offering do not report increased enjoyment and are no more likely to pass, learning as measured by final exam performance increases significantly. This increase is not simply a function of a more onerous inverted offering, as students report spending similar time per week in the traditional and inverted offerings. Contrary to our hypotheses, however, we find no evidence that the the inverted offering disproportionally helps beginners or those not fully fluent in English.

Computer science concept inventories: past and future

Concept Inventories (CIs) are assessments designed to measure student learning of core concepts. CIs have become well known for their major impact on pedagogical techniques in other sciences, especially physics. Presently, there are no widely used, validated CIs for computer science. However, considerable groundwork has been performed in the form of identifying core concepts, analyzing student misconceptions, and developing CI assessment questions. Although much of the work has been focused on CS1 and a CI has been developed for digital logic, some preliminary work on CIs is underway for other courses. This literature review examines CI work in other STEM disciplines, discusses the preliminary development of CIs in computer science, and outlines related research in computer science education that contributes to CI development.

Facilitating code-writing in PI classes

We present the Python Classroom Response System, a web-based tool that enables instructors to use code-writing and multiple choice questions in a classroom setting. The system is designed to extend the principles of peer instruction, an active learning technique built around discussion of multiple- choice questions, into the domain of introductory programming education. Code submissions are evaluated by a suite of tests designed to highlight common misconceptions, so the instructor receives real-time feedback as students submit code. The system also allows an instructor to pull specific submissions into an editor and visualizer for use as in-class examples. We motivate the use of this system, describe its support for and extension of peer instruction, and offer use cases and scenarios for classroom implementation.

Peer instruction in computing: the role of reading quizzes

Peer Instruction has recently gained interest in computing as an effective active learning pedagogy. The general focus of PI research has been on the in-class portion of PI: multiple choice questions and group discussion. Here, our focus is the reading quizzes completed by students for purposes of class preparation. These quizzes contain content questions but also ask for difficulties or confusion with course material. Consistent with expectations, we demonstrate that providing correct responses to quiz questions positively correlates with other course assessments. Somewhat counter-intuitively, we find that identifying confusions, noting problematic sections, or asking questions about the reading are also correlated with lab grades.

Importance of early performance in CS1: two conflicting assessment stories

It is generally assumed that early success in CS1 is crucial for success on the exam and course as a whole. Particularities of students, densely-connected CS1 content, and recurring core topics each suggest that it is difficult to rebound from early misunderstandings. In this paper, we use Peer Instruction (PI) data, in addition to exam data, to explore relationships between in-class assessments and performance at the end of term and on the exam. We find that early course performance very quickly and strongly predicts performance on the final exam and that subsequent weeks provide no major increase in that predictive power. In contrast, early performance is similarly predictive of performance in the last weeks of PI questions, but subsequent weeks are increasingly more predictive. We speculate on what this means for the content of these assessments and potential future assessment practices.

Peer instruction: a link to the exam

In computer science, the active learning pedagogical practice of Peer Instruction (PI) has been shown to improve final exam performance, reduce student failure rates, and improve student retention. PI consists of two major parts: group discussion and follow-up instructor intervention. We expect that PI performance as a whole will correlate with final exam performance, but it is unclear whether or how each piece of PI is involved in these relationships. In this work, we use isomorphic questions to isolate the effects of peer discussion and instructor intervention, and examine scores on a final exam and its code-writing and code-tracing questions. We find that both pieces of PI correlate with the final exam as a whole, code-tracing question (similar to PI questions), and code-writing question (not similar to PI questions). This is further evidence that both PI components are important to the success of PI.

Examining Interest and Grades in Computer Science 1: A Study of Pedagogy and Achievement Goals

Computer Science 1 (CS1), the first course taken by college-level computer science (CS) majors, has traditionally suffered from high failure rates. Efforts to understand this phenomenon have considered a wide range of predictors of CS success, such as prior programming experience, math ability, learning style, and gender, with findings that are suggestive but inconclusive. The current quasiexperimental study extends this research by exploring how the pedagogical approach of the course (traditional lecture vs. Peer Instruction (PI) and clickers) in combination with student achievement goals (mastery goals vs. performance goals) relates to exam grades, interest in the subject matter, and course enjoyment. The research revealed that students with performance goals scored significantly lower on final exams in both the lecture and PI conditions. However, students with performance goals reported higher levels of subject matter interest when taught through PI. Students with mastery goals, in both conditions, scored significantly higher on the final exam, had higher levels of interest, and reported higher levels of course enjoyment than their performance-oriented counterparts. The results suggest that PI may improve the level of subject-matter interest for some students, thereby indicating the importance of studying pedagogical approach as we seek to understand student outcomes in CS1.