Toward Predicting Success and Failure in CS2: A Mixed-Method Analysis
TToward Predicting Success and Failure in CS2: A Mixed-MethodAnalysis
Lucas Layman
University of North Carolina,WilmingtonWilmington, North Carolina, [email protected]
Yang Song
University of North Carolina,WilmingtonWilmington, North Carolina, [email protected]
Curry Guinn
University of North Carolina,WilmingtonWilmington, North Carolina, [email protected]
ABSTRACT
Factors driving success and failure in CS1 are the subject of muchstudy but less so for CS2. This paper investigates the transitionfrom CS1 to CS2 in search of leading indicators of success in CS2.Both CS1 and CS2 at the University of North Carolina Wilmington(UNCW) are taught in Python with annual enrollments of 300 and150 respectively. In this paper, we report on the following researchquestions:(1) Are CS1 grades indicators of CS2 grades?(2) Does a quantitative relationship exist between CS2 coursegrade and a modified version of the SCS1 concept inventory?(3) What are the most challenging aspects of CS2, and howwell does CS1 prepare students for CS2 from the student’sperspective?We provide a quantitative analysis of 2300 CS1 and CS2 coursegrades from 2013–2019. In Spring 2019, we administered a modifiedversion of the SCS1 concept inventory to 44 students in the firstweek of CS2. Further, 69 students completed an exit questionnaireat the conclusion of CS2 to gain qualitative student feedback ontheir challenges in CS2 and on how well CS1 prepared them forCS2.We find that 56% of students’ grades were lower in CS2 thanCS1, 18% improved their grades, and 26% earned the same grade.Of the changes, 62% were within one grade point. We find a sta-tistically significant correlation between the modified SCS1 scoreand CS2 grade points. Students identify linked lists and class/objectconcepts among the most challenging. Student feedback on CS2challenges and the adequacy of their CS1 preparations identifypossible avenues for improving the CS1-CS2 transition.
CCS CONCEPTS • Social and professional topics → CS1 ; Student assessment . KEYWORDS
CS1, CS2, student assessment
ACM Reference Format:
Lucas Layman, Yang Song, and Curry Guinn. 2020. Toward Predicting Suc-cess and Failure in CS2: A Mixed-Method Analysis. In
ACMSE 2020, April 2–4, 2020, Tampa, FL, USA © 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM.This is the author’s version of the work. It is posted here for your personal use. Not forredistribution. The definitive Version of Record was published in , https://doi.org/10.1145/3374135.3385277.
Conference (ACMSE 2020), April 2–4, 2020, Tampa, FL, USA.
ACM, New York,NY, USA, 8 pages. https://doi.org/10.1145/3374135.3385277
CS1 and CS2 are important first courses for those interested incomputing. CS1 is often an introduction to computer science andprogramming, and CS2 is often an introduction to data structuresand algorithms, though the subject matter varies across institu-tions [11]. Students’ performance and experience with the introduc-tory programming sequence have a major impact on retention in theComputer Science major and has been often studied (e.g., [17, 28].While students in CS1 may be dipping their toe in the water, stu-dents in CS2 have committed to a curricular path and failure in theCS2 course can be more impactful. We focus our research on the transition from CS1 to CS2 . We want to improve student outcomesin CS2 by both identifying challenges with CS2 topics in particular,and ensuring that CS1 is best preparing students for CS2.Both CS1 and CS2 at the University of North Carolina Wilm-ington (UNCW) are taught in Python with annual enrollments of300 and 150 respectively. Approximately 15% of students enrolledin CS2 at UNCW do not earn the requisite ’C’ grade to proceedto the next course in the sequence. Prior research has identifiedindicators of CS2 final grade, including overall Grade Point Aver-age (GPA), number of absences, and performance in prerequisitecourses including CS1 [5, 9], but these indicators are not necessarilyactionable.Our long term research goal is to investigate successful andunsuccessful transitions from CS1 to CS2 to identify interventionsand pedagogical improvements in those courses. We begin with thefollowing research questions:(1) Are CS1 grades indicators of CS2 grades?(2) Does a quantitative relationship exist between CS2 coursegrade and a modified version of the Second Computer Sci-ence 1 (SCS1) concept inventory [18]? The SCS1 evaluatesstudents’ understanding of sequencing, selection, iteration,and other CS1 concepts.(3) What are the most challenging aspects of CS2, and howwell does CS1 prepare students for CS2 from the student’sperspective?For (1), we perform a quantitative analysis of 2300 course gradesin CS1 and CS2 at UNCW from 2013-2019 to evaluate the rela-tionship between CS1 and CS2 grades. For (2), we quantitativelyevaluate SCS1 assessment [18] scores of CS1 skills against studentgrades for two sections of CS2 offered in Spring 2019. For (3), weanalyze responses to a student questionnaire at the end of CS2 thatobtained feedback on challenges faced in the course and on the a r X i v : . [ c s . OH ] F e b CMSE 2020, April 2–4, 2020, Tampa, FL, USA Lucas Layman, Yang Song, and Curry Guinn adequacy of CS1 in preparing them for CS2. We present the mostcommon responses from students and discuss their implications onavenues for pedagogical and curricular improvements. The rest of the paper is organized as follows: Section 2 discussesprior research on CS2 performance, Section 3 describes the CS1and CS2 courses at UNCW, Sections 4 and 5 present quantitativeanalyses of CS1 vs. CS2 grade and modified SCS1 score vs. CS2grade, Section 6 presents our qualitative questionnaire findings,and we conclude in Section 7.5.
Our CS1 and CS2 courses are taught in Python. Koulouri et al. [13]report statistical findings that students better learned introductoryprogramming concepts in Python (a "syntactically simple" language)than in Java, however Alzharani et al. found that beginning studentsstruggle with assignments in Python as much or more than inC++ [1]. Educators have argued that that Python is a good choicefor a first programming language [14, 22] because data typing,memory management, and object references are implicit, and thatbeginning programmers may find such concepts confusing. Theseconcepts are usually important in a CS2 data structures course,though Enbody et al. [9] found no statistical difference in a C++CS2 data structures course between students who took CS1 inPython vs. C++.We are interested in indicators of unsuccessful transitions fromCS1 to CS2. In this paper, we use failure to pass CS2 as an indicatorof an unsuccessful transition as it can be studied at scale – more fine-grained measures, such as failing early CS2 homeworks or exams,are warranted in future work. A host of literature reports the manypersonal [4, 17] societal [15], institutional [7], and pedagogical [29,32] drivers that lead to student success and failure in CS coursesin general. Overall GPA, CS1 grade, and perceived difficulty ofprerequisite courses [5, 9] have been correlated with CS2 finalgrades, but these indicators are not actionable. Carter et al. [6]used measures of compilation, debugging, and execution in thedevelopment environment coupled with the frequency of messageboard interaction to develop a model that accounted for much ofthe variance in students’ final CS2 grade in a C++ data structurescourse. Falkner and Falkner [10] find that students who turn intheir first assignments on time statistically perform better acrossthe CS curriculum.Prior studies [23, 31] suggest that common CS2 topics are so-called threshold concepts [16] that transform one’s understanding ofcomputing and programming, including data structures, classes andinheritance, abstraction, and pointers. In general, research on CS1concept acquisition greatly outnumbers research on CS2 conceptacquisition, though CS2 is receiving increased focus (e.g., [12, 19]).Zingaro et al. [33] recently presented the most comprehensive studyof data structure misconceptions based on qualitative analysis of 279students’ responses to final exam questions. Their study and Porteret al.’s [20] report on data structures learning goals led to the recentpublication of the Basic Data Structures Inventory (BDSI), a multi-language, validated mechanism for assessing student understandingof linked list, array, binary tree, and binary search tree concepts [21]. This study is approved by UNCW’s Institutional Review Board:
Figure 1: Enrollment History for CS1 and CS2 since Adopt-ing Python
Unfortunately, the BDSI was published after it could be incorporatedinto our current study.Our work expands and adds to this body of literature in severalways. First, our study of the distributions between CS1 and CS2grade is a large sample size (n=614), providing quantitative evidenceon a scale that can overcome biases such as instructor pedagogicaleffects. Second, no studies have been reported that quantitativelyevaluate derivatives of the SCS1 concept inventory [18] (discussedin Section 5) as an indicator of CS2 success. Finally, we ask studentsto qualitatively reflect on CS2 challenges similar to Zingaro etal. [33], but also to consider how well CS1 prepared them for CS2.Educators should try to identify the students who are likely to haveunsuccessful transitions from CS1 to CS2 as early as possible. Byidentifying which factors contribute to failure in CS2, we will beable to focus on how to improve aspects related to both CS1 andCS2 to reduce dropout rates.
UNCW is an R2 Doctoral university per the Carnegie Classifica-tion. The Department of Computer Science offers undergraduatedegree programs in Computer Science ( ~
420 majors), InformationTechnology ( ~
110 majors), and Digital Arts ( ~
60 majors) as well asgraduate programs in Computer Science and Information Systems( ~
35 students) and Data Science ( ~
45 students). Our undergradu-ate major in Computer Science has received ABET accreditationsince 2010-2011. Figure 1 shows enrollment in CS1 and CS2 eachsemester.In this paper, CS1 and CS2 correspond to the terminology ex-pressed in the ACM Computing Curriculum [2] guidelines whereCS1 introduces basic algorithm design and programming conceptswhile CS2 introduces more advanced data abstractions and data oward Predicting Success and Failure in CS2: A Mixed-Method Analysis ACMSE 2020, April 2–4, 2020, Tampa, FL, USA structures. Both CS1 and CS2 are four credits (200 minutes perweek) equally divided between lecture and computer lab activities.No online sections are offered. All lectures and labs are led by afaculty member who is assisted by a graduate student who provideslab assistance, office hours, and grading help.CS1 is CSC 131 – Introduction to Computer Science. Six to eightsections are offered each semester (1-3 sections in the summer)with approximately 20-25 students per section. The prerequisitefor CS1 is a course in college algebra; no programming experienceis assumed. CS1 has been taught using the Python programminglanguage since Fall 2013 (Java was the language before that year).Typically, instructors introduce the IDLE development environmentas well as command-line interpretation. The official learning out-comes for CS1 include a demonstrated ability to: understand andimplement basic programming concepts (data types, conditionals,function definition), apply problem-solving techniques, use pro-gram control structures, practice modular programming, and usefile input/output used to solve a variety of problems. The teachingmethods and course structure are left to each instructor so long asthey address the learning outcomes. All instructors implement a mixof homework and exam, and all courses are taught in a computerlab with programming exercises during most (if not all) sessions.CS2 is CSC 231 – Introduction to Data Structures. Three to foursections are offered each semester (one in the summer) with ap-proximately 20-25 students per section. The prerequisite for CS2 isCS1 (grade of C or higher) and Discrete Mathematics which is listedas a pre- or co-requisite. CS2 has been taught using the Pythonprogramming language since Spring 2014 (Java was the languagebefore that year). Typically, instructors introduce a more advancedIDE such as PyCharm or Visual Studio Code. The official learningoutcomes for CS2 include: learning multiple data structures forordered and unordered data (e.g., lists, trees, hash tables), develop-ing algorithms that implement the main operations of those datastructures, performing Big-O analysis, and implementing projectsthat solve problems using those data structures. Instructors use avariety of in-class programming and written assessments combinedwith homeworks. Some implementations of the class are taughtstrictly in a computer lab, while others have two traditional lecturesections and one longer laboratory meeting each week.Importantly, students must earn a grade of C or higher to enrollin future courses for which CS1 or CS2 is a prerequisite. Thesecourses are required for Computer Science majors and minors aswell as Digital Arts majors. Students who earn less than a C mayrepeat a course to earn a higher grade.
Our first research question is:
Are CS1 grades indicators of CS2grades?
We look to validate previous research that found a cor-relation between CS1 and CS2 grades [5, 9]. Historical grade dis-tributions also provide context and identify potential biases forinterpreting our other analyses.
We collected all registered course grades for CS1 and CS2 fromFall 2013 (when first offered in Python) through Spring 2019. We mapped the letter grades of [F, D-, D, D+, ..., B, B+, A-, A], to a4.0 grade point scale. We collected 1641 CS1 letter grades and 706CS2 letter grades in total. The CS1 sample is larger because CS1is a required course for other non-Computer Science majors andalso satisfies a general University Studies requirement. CS1 andCS2 were taught by 16 and 10 different faculty respectively in thisperiod. Earning a C or higher is required to progress from both CS1and CS2, and students may repeat a course if they earn less than a C.Approximately 7% of students repeat CS1 and CS2. Students’ gradesfrom all course attempts are included in this section’s analysis; weexclude withdrawals and audits.
Figure 2 shows the historical distribution of grades. CS1 has agreater proportion of A and F grades, while CS2 has a greaterproportion of Bs. The difference between frequency distributions ofgrades is statistically significant ( X = . , p < . , d f = The scatterplot in Figure 3 showsno obvious relationship between a students’ CS1 and CS2 grades,though a sizeable portion of students earn As in CS1 and similarlyhigh grades in CS2.A more insightful analysis is to examine the grade point difference of individuals between CS1 and CS2 as shown in Figure 4.
Gradepoint difference is calculated by subtracting CS1 grade points earnedfrom CS2 grade points earned on the standard 4.0 scale. All studentsin this sample must have passed CS1 with at least a C (with manyearning As), thus the distribution is biased to be skew left. The dataindicates that most student grades (56%) dropped slightly. Only 18%improved their letter grades, and 26% earn the same grade bolsteredby the 118 students who earned As in both courses. Figure 4 sup-ports previous findings that CS1 grade is related to CS2 grade [5, 9]in the sense that 62.1% of grade point changes are within one stan-dard deviation ( σ = . K = . , p < . We use the most recent grade, as opposed to an average grade across repeated attempts,to simplify the analysis.
CMSE 2020, April 2–4, 2020, Tampa, FL, USA Lucas Layman, Yang Song, and Curry Guinn
Figure 2: CS1 and CS2 Course Letter Grade Distributions,Fall 2013—Spring 2019, CS1 n = , CS2 n = . Percent-ages Are Shown to Normalize Values between CoursesFigure 3: Scatterplot of Students’ Most Recent CS1 Grades vs.CS2 Grades, n = amounts to only 14 students in our dataset, but they make up adisproportionate percentage of those who do not move on fromCS2 ( X = . , p < . , d f = Figure 4: Grade Point Change from CS1 to CS2, n = Table 1: Proportions of Students Who Earned a C or Betterin CS2 Grouped by CS1 Grade
C or better in CS2?
CS1 - Final Grade False True TotalC 14 (27%) 37 (7%) 51C+ 5 (10%) 24 (4%) 29B- 4 (8%) 41 (7%) 45B 9 (17%) 77 (14%) 86B+ 6 (11.5%) 51 (9%) 57A- 6 (11.5%) 60 (11%) 66A 8 (15%) 272 (48%) 280Total 52 562 614insights into what drives these students’ performances yet, but thatis the subject of future work.
The second research question we investigate is:
Does a quantitativerelationship exist between CS2 course grade and a modified SCS1concept inventory score?
Our second research objective is to evaluate a modified version(explained below) of the
SCS1 concept inventory [18] as a leadingindicator of CS2 performance. The SCS1 is a validated instrumentfor evaluating students’ understanding of select CS1 concepts. Arelationship between our modified SCS1 (mSCS1) score and CS2final grade may indicate that the mSCS1 can be used early in theCS2 semester to identify areas of weakness. The scatterplot shows students who earned grades in CS1 and CS2 at UNCW; studentswho earned CS1 credit at other institutions are excluded which is why the number ofdata points (614) is less than the number of CS2 grades collected (706). oward Predicting Success and Failure in CS2: A Mixed-Method Analysis ACMSE 2020, April 2–4, 2020, Tampa, FL, USA
Note:
We are not at liberty to disclose the SCS1 or mSCS1 perthe license of the SCS1 authors, but a request for free educatoraccess to the SCS1 can be made via its Google Group located athttps://goo.gl/MiYOFk.The Second CS1 Assessment (SCS1) [18] is derived from theFoundational CS1 (FCS1) Assesment [26]. The FCS1 was shownto have a significant, moderate correlation with CS1 Final ExamScores [26, 27] in a large, multi-institutional, multi-programminglanguage study. The SCS1 is an "isomorphic version" of the FCS1available to educators while the FCS1 is copyrighted by its author.Both assessments are written in pseudocode and inventory the CS1concepts of arrays, basics, for-loops, function parameters, functionreturn values, if-else, logical operators, recursion, and while-loops.Each of the 27 questions has five multiple choice answers, and eachtopic has a definition-oriented, code tracing, and code completionquestion. Both the FCS1 and SCS1 were intended for a 60-minuteexam period. We chose the SCS1 as a candidate for leading indi-cators because it is language agnostic, has been validated at scale,and is freely available to educators.We modified the SCS1 for our study in response to communityfeedback on the SCS1 Google group and to fit our institutions CS1learning outcomes and language (Python). Questions on recursionwere dropped as recursion is not part of CS1 learning objectives atour institution. We translated the questions to Python as we foundthe pseudocode non-intuitive in places and did not wish to burdenthe students with interpreting the pseudolanguage. Finally, onequestion for each concept was dropped to encourage completion ofthe assessment in a 50-minute lecture session based on feedbackfrom students and instructors in the FCS1 study [27] and on theSCS1 Google Group states that 60 minutes is not enough time tocomplete the original 27 questions. Questions with very low correctresponse rates (<20%) reported in [27] were dropped along withthose that did not directly translate to Python.Two sections of CS2 were administered the mSCS1 assessmentconsisting of 16 questions translated to Python. The assessmentwas given closed-book in a 50-minute class session. Scrap paperwas permitted. A total of 44 students were administered the test,and 39 of those completed the semester.
Students’ CS2 course grade points are plotted against their mSCS1scores in Figure 5. D’Agostino’s statistical test does not reject thenull hypothesis that the mSCS1 scores are drawn from a normaldistribution ( K = . , p = . K = . , p = . ρ = . , p < . Figure 5: Modified SCS1 Score vs. CS2 Grade Points Earned, n = The final research question we investigate is:
What are the mostchallenging aspects of CS2, and how well does CS1 prepare studentsfor CS2 from the student’s perspective?
In the final weeks of the semester, CS2 students were given aquestionnaire that asked the following questions:(1) Rate your agreement with the following statement: "Overall,I think that CS1 (or equivalent) adequately prepared me forthis course". (1-Strongly Disagree through 5-Strongly Agree)(2) Which concepts or topics from CS1 were most helpful to youin this course? (free text)(3) Which concepts or topics from CS1 would you like to havebeen better at prior to taking this course? (free text)(4) What were the most challenging aspects of this course? (freetext)Of the 79 students enrolled at the end of the semester, 69 (87%)completed the exit questionnaire. The questionnaire was not anony-mous so that student responses could be linked to course perfor-mance; that analysis will be a part of future work, and we acknowl-edge that the lack of anonymity may alter the truthfulness of an-swers.Of the 69 respondents, 59% agreed, 19% were neutral, and 22%disagreed with the statement that CS1 prepared them well for CS2.Ideally, student responses would be in the "agree" categories, butthere are many personal [4, 17] and pedagogical reasons [29, 33]why this would not be the case irrespective of any disconnectbetween exit criteria from CS1 and starting expectations in CS2.Responses to Questions 2–4 were transcribed into an Excel fileand open coded [25] by the first author to identify topics/conceptsin each response. A single response could mention multiple topics.Unique responses or responses that did not readily identify a topicor concern were not coded. Tables 2, 3, and 4 show codes (topics
CMSE 2020, April 2–4, 2020, Tampa, FL, USA Lucas Layman, Yang Song, and Curry Guinn and concepts) appearing in more than 10% of the responses to eachquestion.Question 2 asked, "Which concepts or topics from CS1 were mosthelpful to you in this course?" The student responses (Table 2) re-flect the skills required to interact with data structures implementedas Python classes. Reading data from files, iterating over lists, andlist manipulation are skills used in nearly every assignment. Datastructures are implemented as Python classes with accompanyingmethods, e.g., a Stack class with push() , pop() , top() , etc. Thus, astrong grasp of function definition, call, and return is beneficial forstudents. Students who were exposed to recursion in CS1 found itto be useful when it is introduced in CS2. Question 3 asked, "Whichconcepts or topics from CS1 would you like to have been better atprior to taking this course?" The students’ responses (Table 3) againreflect the focus of CS2 on Python classes, reading files for datacontent, and manipulating lists. File I/O, Python lists, and functiondefinitions are covered extensively in our CS1 curriculum, whereasrecursion and class definition are not part of the CS1 learning out-comes. The responses to Questions 2 and 3 are useful for informingdiscussions of what could be covered in CS1, but more importantly,these responses identify the concepts that should be reviewed, re-inforced, and possibly quantified as potential leading indicators ofperformance at the beginning of CS2.Question 4 asked "What were the most challenging aspects ofthis course? (open-ended)" concerning CS2. The responses to thisquestion were predictably diffuse, but students did agree on someissues. Linked lists and its emphasis on pointers and class/objectdefinition are challenging topics, reinforcing findings in Simon etal. [24] and Yeomans et al. [31]. Responses discussing the Linked Table 2: Questionnaire: Which Concepts or Topics from CS1Were Most Helpful to You in this Course?
Topic ResponsesFoundations (basic expressions, boolean values,syntax, vocabulary) 29 (43%)Iteration, for-loops, or while-loops 25 (37%)File manipulation (reading, writing, loading into adata structure) 15 (22%)Recursion 15 (22%)Lists and list operations 10 (15%)Function call and definition 10 (15%)if-else, conditionals 8 (12%)
Table 3: Questionnaire: Which Concepts or Topics from CS1Would You Like to Have Been Better at Prior to Taking thisCourse?
Topic ResponsesClasses/objects 18 (26%)File manipulation (reading, writing, reading into adata structure) 12 (17%)Lists and list operations 11 (16%)Recursion 11 (16%)Dictionaries 8 (12%)
Table 4: Questionnaire: What Were the Most Challenging As-pects of this Course?
Topic ResponsesLinked list / doubly linked list 11 (16%)Creating code from scratch 10 (15%)Classes/objects 8 (12%)Pace of instruction / work 7 (10%)List and Class/Objects topics often reference challenges in trans-lating the concepts into code. That is, students could understandthe notion of Linked Lists and Objects but had difficulty imple-menting the data structures and applying them to problems. Thislikely relates to students’ mental models of how the computer pro-cesses programs and points to a particular area where pedagogycan be improved. The "Creating Code from Scratch" topic refersto responses where students remarked on how past instructorsrequired them to complete a half-finished Python script, but nowthey were required to create Python scripts from scratch or usinga minimal code template. CS2 instructors should be aware of thisparadigm. More importantly, student grades (particularly in CS2)may be predicated on their ability to decompose problems and ini-tiate a solution with more independence than was required in thepast. Thus, instructors may need to include these skills in coursecontent and explicitly add them to course learning outcomes andassessments.
We discuss this study’s threats to validity according to the categoriesof Cook and Campbell [8].
Conclusion validity refers to issues that affect the ability to asso-ciate treatment and outcome.
Low statistical power is not a concernfor our CS1-to-CS2 grade point comparison as the sample sizesare sufficiently large, however, the correlation between mSCS1score and CS2 Grade Points Earned (Figure 5) had a sample size of39, which is relatively low. Appropriate non-parametric statisticaltests were used in all analyses as none of the data were normallydistributed. The reliability of the mSCS1 assessment has not beenevaluated, and thus we cannot be certain if the outcomes of thatassessment are replicable across contexts, especially outside of thePython language. Further, students’ answers are likely influencedby the specific pedagogical style of the instructor, course content,and assignment content. The coding performed in this section wasonly performed by the first author due to time constraints, and thusinterrater reliability is not available. We will continue administer-ing the questionnaire in future semesters to improve the reliabilityof those responses.
Internal validity concerns surround the causal inference betweentreatment and outcome should a relationship be detected. Some stu-dents took CS2 multiple times, introducing a potential history effect wherein their questionnaire answers and classroom performance oward Predicting Success and Failure in CS2: A Mixed-Method Analysis ACMSE 2020, April 2–4, 2020, Tampa, FL, USA was perhaps different than those of students enrolled only onetime. We do not isolate these students in our analyses and cannotquantify any such effects. Table 5 shows the number of repeats foreach course; approximately 6% and 7% of students repeat CS1 andCS2 respectively. Only students’ most recent grades are included inthe calculation of Grade Change between CS1 and CS2 (Figure 4).Finally, our questionnaire in Section 6 was not anonymous, andthus students may not have been compelled to answer truthfully.
Table 5: Individuals’ Times Taken Excluding Withdrawalsand Audits
Times Taken CS1 Count CS2 Count1 1434 6092 91 443 7 34 1 -Total 1533 656
Construct validity concerns issues around whether the analysisresults are connected to the driving theory, in our case, whether weadequately captured issues related to success in CS2 and the CS1-CS2 transition. One goal of our study (hopefully the first in a series)was to help better operationalize these constructs. Additional meth-ods, such as semi-structured interviews and analyses of individualhomework assignments and exam questions, will be used in thefuture to better capture which topics from CS2 are most challeng-ing for students. Further, CS2 Grade is only one indicator of CS2performance. In the future, we will also use the validated Basic DataStructures Inventory (BDSI) [21] to assess student understandingof CS2 concepts.
External validity concerns the transferability of results outside thestudy context. We describe the content and environment of our CS1and CS2 courses in Section 3, and it is up to the reader to decide ifour Python-based courses resemble their own. One point of noteis that our failure rate for CS1 is lower than that reported in otherlarge scale studies, and thus our average CS1 grades may be inflated.No such data is available for CS2.
Our long term research goal is to investigate successful and un-successful transitions from CS1 to CS2 . As the first phase of thatresearch, we provide a quantitative analysis of CS1 and CS2 gradepoint statistics–an evaluation of a modified version of the SCS1 con-cept inventory as an indicator of CS2 performance–and performedqualitative analysis of a questionnaire that obtained feedback fromCS2 students on helpful CS1 preparation and challenges in CS2.In a sample of 614 students, we find that CS1 grades appear tohave a statistical relationship with CS2 grades. This corroboratesearlier findings [5, 9], but we provide a much larger sample size.
Further, the majority (56%) of students’ CS2 grades were lower than their CS1 grade, though 62% of the grade point changes were withinone standard deviation . A disproportionate number of C studentsfrom CS1 do not pass CS2 owing to the mean grade drop.Our goal as educators is to ensure that students exiting CS1 havethe necessary capabilities to succeed in CS2. To this end, we ad-ministered a modified SCS1 concept inventory [18] and correlatedthe scores with CS2 grade points earned.
We found a statisticallysignificant correlation between mSCS1 score and CS2 grade points , sug-gesting that the concepts on the mSCS1 and/or the way the mSCS1tests those concepts measure useful information about beneficialskills in CS2. We believe, anecdotally, that strong performance onthe mSCS1 requires solid mental models of how programs are in-terpreted by the machine. We believe that these mental models arecentral to both program debugging and abstraction. We will con-tinue to investigate the individual questions and concepts capturedin the mSCS1 for continued predictive power of CS2 grade and toidentify possible improvements in CS1 structure.From our questionnaire, the majority of students believed thatCS1 adequately prepared them for CS2. The only data structureexplicitly mentioned as challenging was linked lists . This may bea consequence of using Python as the programming language ofchoice. Unlike C/C++, Python (and Java) does not make the con-cept of pointers or memory references explicit.
Our findings arein line with studies have identified pointers and classes as thresholdconcepts [23, 31] for students learning to program. Students indi-cated that coding a data structure to solve a problem, rather thanfilling in missing parts of an implementation, was challenging. Thisfinding is congruous with previous papers who found that of datastructure application is a challenging addition to the finer points ofdata structure implementation [20, 24].A tremendous amount of computer science education researchexists on success and failure factors in CS1, and a growing setof literature is being produced for CS2. We will continue to in-vestigate cross-course indicators of success, specifically throughmore thorough investigations of the SCS1 inventory, the new Ba-sic Data Structures Inventory [21], in-process code execution andclass participation metrics (e.g., [6]), and qualitative feedback fromthe students themselves. We hope that we can improve studentsoutcomes in CS2 (and beyond) through interventions driven byquantitative observation, and ensure that those students who com-mit to CS2 as part of a computing path receive the best opportunityto gain the skills and knowledge for career success.
ACKNOWLEDGMENTS
Thanks to the students of CSC231 who contributed feedback tothis study. Thanks also to Elham Ebrahimi, Clayton Ferner, Srid-har Narayan, Toni Pence, Geoffrey Stoker and other CS1 and CS2instructors for their valuable participation and input in this study.
REFERENCES [1] N. Alzahrani, F. Vahid, A. Edgcomb, K. Nguyen, and R. Lysecky. 2018. PythonVersus C++: An Analysis of Student Struggle on Small Coding Exercises inIntroductory Programming Courses. In
SIGCSE 2018 - Proceedings of the 49th ACMTechnical Symposium on Computer Science Education . Association for ComputingMachinery, Inc, Baltimore, MD, USA, 86–91. https://doi.org/10.1145/3159450.3160586[2] R. H. Austing, B. H. Barnes, D. T. Bonnette, G. L. Engel, and G. Stokes. 1979.Curriculum ’78: Recommendations for the Undergraduate Program in Computer
CMSE 2020, April 2–4, 2020, Tampa, FL, USA Lucas Layman, Yang Song, and Curry Guinn
Science – A Report of the ACM Curriculum Committee on Computer Science.
Commun. ACM
22, 3 (3 1979), 147–166. https://doi.org/10.1145/359080.359083[3] J. Bennedsen and M. E. Caspersen. 2007. Failure Rates in Introductory Program-ming.
ACM SIGCSE Bulletin
39, 2 (6 2007), 32. https://doi.org/10.1145/1272848.1272879[4] S. Bergin, R. Reilly, and D. Traynor. 2005. Examining the Role of Self-RegulatedLearning on Introductory Programming Performance. In
Proceedings of the 1stInternational Computing Education Research Workshop, ICER 2005 . ACM Press,Seattle, WA, USA, 81–86. https://doi.org/10.1145/1089786.1089794[5] H. Bisgin, M. Mani, and S. Uludag. 2018. Delineating Factors that Influence StudentPerformance in a Data Structures Course. In . IEEE, San Jose, CA, USA, 1–9. https://doi.org/10.1109/FIE.2018.8659300[6] A. S. Carter, C. D. Hundhausen, and O. Adesope. 2017. Blending Measures ofProgramming and Social Behavior into Predictive Models of Student Achievementin Early Computing Courses.
ACM Transactions on Computing Education
17, 3 (82017), Article 12. https://doi.org/10.1145/3120259[7] J. M. Cohoon. 2001. Toward Improving Female Retention in the Computer ScienceMajor.
Commun. ACM
44, 5 (2001), 108–114.[8] T. D. Cook and D. T. Campbell. 1979.
Quasi-Experimentation: Design & AnalysisIssues for Field Settings . Rand McNally, Chicago.[9] R. J. Enbody, W. F. Punch, and M. McCullen. 2009. Python CS1 as Preparationfor C++ CS2. In
SIGCSE’09 - Proceedings of the 40th ACM Technical Symposiumon Computer Science Education . ACM Press, Chattanooga, TN, 116–120. https://doi.org/10.1145/1508865.1508907[10] N. J. Falkner and K. Falkner. 2012. A Fast Measure for Identifying At-Risk Studentsin Computer Science. In
ICER’12 - Proceedings of the 9th Annual InternationalConference on International Computing Education Research . ACM Press, Auckland,New Zealand, 55–62. https://doi.org/10.1145/2361276.2361288[11] M. Hertz. 2010. What Do "CS1" and "CS2" Mean? Investigating Differences in theEarly Courses. In
SIGCSE’10 - Proceedings of the 41st ACM Technical Symposiumon Computer Science Education . ACM Press, Milwaukee, WI, USA, 199–203. https://doi.org/10.1145/1734263.1734335[12] K. Karpierz and S. A. Wolfman. 2014. Misconceptions and Concept InventoryQuestions for Binary Search Trees and Hash Tables. In
SIGCSE 2014 - Proceedingsof the 45th ACM Technical Symposium on Computer Science Education . ACM Press,Atlanta, GA, USA, 109–114. https://doi.org/10.1145/2538862.2538902[13] T. Koulouri, S. Lauria, and R. D. Macredie. 2014. Teaching Introductory Program-ming: A Quantitative Evaluation of Different Approaches.
ACM Transactions onComputing Education
14, 4 (12 2014), Article 26. https://doi.org/10.1145/2662412[14] R. P. Loui. 2008. In Praise of Scripting: Real Programming Pragmatism.
Computer
41, 7 (7 2008), 22–26. https://doi.org/10.1109/MC.2008.228[15] J. Margolis and A. Fisher. 2002.
Unlocking the Clubhouse: Women in Computing .The MIT Press, Cambridge, Massachusetts.[16] J. H. F. Meyer and R. Land. 2005. Threshold Concepts and Troublesome Knowledge(2): Epistemological Considerations and a Conceptual Framework for Teachingand Learning.
Higher Education
49, 3 (4 2005), 373–388. https://doi.org/10.1007/s10734-004-6779-5[17] I. O. Pappas, M. N. Giannakos, and L. Jaccheri. 2016. Investigating FactorsInfluencing Students Intention to Dropout Computer Science Studies. In
AnnualConference on Innovation and Technology in Computer Science Education, ITiCSE .ACM Press, Arequipa, Peru, 198–203. https://doi.org/10.1145/2899415.2899455[18] M. C. Parker, M. Guzdial, and S. Engleman. 2016. Replication, Validation, and Useof a Language Independent CS1 Knowledge Assessment. In
Proceedings of the2016 ACM Conference on International Computing Education Research (ICER ’16) .ACM, Melbourne, Australia, 93–101. https://doi.org/10.1145/2960310.2960316[19] W. Paul and J. Vahrenhold. 2013. Hunting High and Low. In
Proceeding of the 44thACM Technical Symposium on Computer Science Education - SIGCSE ’13 . ACM Press, Denver, CO, USA, 29–34. https://doi.org/10.1145/2445196.2445212[20] L. Porter, D. Zingaro, C. Lee, C. Taylor, K. C. Webb, and M. Clancy. 2018.Developing Course-Level Learning Goals for Basic Data Structures in CS2.In
SIGCSE 2018 - Proceedings of the 49th ACM Technical Symposium on Com-puter Science Education . ACM Press, Baltimore, MD, USA, 858–863. https://doi.org/10.1145/3159450.3159457[21] L. Porter, D. Zingaro, S. N. Liao, C. Taylor, K. C. Webb, C. Lee, and M. Clancy. 2019.BDSI: A Validated Concept Inventory for Basic Data Structures. In
Proceedings ofthe 2019 ACM Conference on International Computing Education Research - ICER’19 . ACM Press, Toronto, ON, Canada, 111–119. https://doi.org/10.1145/3291279.3339404[22] A. Radenski. 2006. "Python First": A Lab-Based Digital Introduction to ComputerScience. In
Working Group Reports on ITiCSE on Innovation and Technology inComputer Science Education 2006 . ACM Press, Bologna, Italy, 197–201. https://doi.org/10.1145/1140124.1140177[23] K. Sanders, J. Boustedt, A. Eckerdal, R. Mccartney, J. E. Moström, L. Thomas,and C. Zander. 2012. Threshold Concepts and Threshold Skills in Computing. In
ICER’12 - Proceedings of the 9th Annual International Conference on InternationalComputing Education Research . ACM Press, Auckland, New Zealand, 23–30. https://doi.org/10.1145/2361276.2361283[24] B. Simon, M. Clancy, R. McCartney, B. Morrison, B. Richards, and K. Sanders. 2010.Making Sense of Data Structures Exams. In
ICER’10 - Proceedings of the Interna-tional Computing Education Research Workshop . ACM Press, Aarhus, Denmark,97–105. https://doi.org/10.1145/1839594.1839612[25] A. Strauss. 1987.
Qualitative Analysis for Social Scientists . Cambridge UniversityPress, New York. https://doi.org/10.1017/CBO9780511557842[26] A. E. Tew. 2010.
Assessing Fundamental Introductory Computing Concept Knowl-edge in a Language Independent Manner . Ph.D. Dissertation. Georgia Insitute ofTechnology.[27] A. E. Tew and M. Guzdial. 2011. The FCS1: A Language Independent Assessmentof CS1 Knowledge. In
Proceedings of the 42Nd ACM Technical Symposium onComputer Science Education (SIGCSE ’11) . ACM, Dallas, TX, USA, 111–116. https://doi.org/10.1145/1953163.1953200[28] E. H. Turner, E. Albert, R. M. Turner, and L. Latour. 2007. Retaining Majors throughthe Introductory Sequence. In
SIGCSE 2007: 38th SIGCSE Technical Symposiumon Computer Science Education . ACM Press, Covington, KY, USA, 24–28. https://doi.org/10.1145/1227310.1227321[29] A. Vihavainen, J. Airaksinen, and C. Watson. 2014. A Systematic Review ofApproaches for Teaching Introductory Programming and their Influence onSuccess. In
ICER 2014 - Proceedings of the 10th Annual International Conferenceon International Computing Education Research . ACM Press, Glasgow, Scotland,19–26. https://doi.org/10.1145/2632320.2632349[30] C. Watson and F. W. Li. 2014. Failure Rates in Introductory Programming Revisited.In
ITICSE 2014 - Proceedings of the 2014 Innovation and Technology in ComputerScience Education Conference . ACM Press, Uppsala, Sweden, 39–44. https://doi.org/10.1145/2591708.2591749[31] L. Yeomans, S. Zschaler, and K. Coate. 2019. Transformative and Troublesome?Students’ and Professional Programmers’ Perspectives on Difficult Concepts inProgramming.
ACM Transactions on Computing Education
19, 3 (1 2019), Article23. https://doi.org/10.1145/3283071[32] D. Zingaro. 2015. Examining Interest and Grades in Computer Science 1: A Studyof Pedagogy and Achievement Goals.
ACM Transactions on Computing Education
15, 3 (7 2015), Article 14. https://doi.org/10.1145/2802752[33] D. Zingaro, C. Taylor, L. Porter, M. Clancy, C. Lee, S. N. Liao, and K. C. Webb.2018. Identifying Student Difficulties with Basic Data Structures. In