Characterizing active learning environments in physics using network analysis and COPUS observations
CCharacterizing active learning environments in physics using network analysis andCOPUS observations
Kelley Commeford and Eric Brewe
Drexel University3141 Chestnut St, Philadelphia, PA 19104
Adrienne Traxler
Wright State University3640 Colonel Glenn Hwy, Dayton, OH 45435 (Dated: August 14, 2020)This study uses social network analysis and the Classroom Observation Protocol for Undergradu-ate STEM (COPUS) to characterize six research-based introductory physics curricula. Peer Instruc-tion, Modeling Instruction, ISLE, SCALE-UP, Context-Rich Problems, and Tutorials in Introduc-tory Physics were investigated. Students in each curriculum were given a survey at the beginningand end of term, asking them to self-identify peers with whom they had meaningful interactionsin class. Every curriculum showed an increase in the average number of student connections fromthe beginning of term to the end of term, with the largest increase occurring in Modeling Instruc-tion, SCALE-UP, and Context-Rich Problems. Modeling Instruction was the only curriculum witha drastic change in how tightly connected the student network was. Transitivity increased for allcurricula except Peer Instruction. We also spent one week per research site in the middle of the termobserving courses using COPUS. From these observations, the student COPUS profiles look nearlythe same for Tutorials, ISLE recitations, and Context-Rich Problems discussion sections. This islikely due to the large resolution of activities that can be coded as other group activity, suggestingthe need for a more detailed observation instrument.
I. INTRODUCTION
After decades of research in discipline based educationresearch communities, it is well established that activelearning is more effective than standard passive lectureat improving student outcomes [1, 2]. A vast majorityof active learning research still uses passive lecture meth-ods as a baseline measurement of learning techniques.As such, Freeman et. al. recommends a second wave ofdiscipline based education research initiatives that studyactive learning methods independently of passive lecturemethods, so we can understand the mechanisms throughwhich active learning promotes increased student out-comes.However, studies of active learning pedagogies as in-dependent entities have encountered difficulties becauseimplementations vary widely. When comparing learninggains of individual implementations of Tutorials in Intro-ductory Physics, with different instructors and studentpopulations, there have been inconsistent gains based oninstructor background and student buy-in [3, 4]. Addi-tionally, a study of several Peer Instruction classroomsat the same university, but with different instructors,shows that the implementation of the same pedagogi-cal methods vary greatly, resulting in differing classroomnorms [5]. Implementation of a given active learning ped-agogy can also be severely limited based on external fac-tors, such as assigned classroom, class size, and studentpreparation [6]. Before we can even think of directlycomparing pedagogies to each other, we need to under-stand the roles that different mechanisms play when im-plementing a pedagogy, and what impact those mecha- nisms have on students interacting with the curriculum.To understand how active learning mechanisms impactstudent achievement, we first need to develop an appro-priate vocabulary to describe the distinguishing featuresof an active learning environment. Doing so will allow re-searchers to discuss these implementations as individualentities, rather than as umbrella “active learning meth-ods” to be compared to passive lecture.Active learning environments vary based on studentand instructor tasks and behaviors. There have beena handful of observational characterization studies donefor Peer Instruction [7] and small group physics work-shops [8]. This study aims to broaden that lens to in-clude six active learning pedagogies commonly used inphysics, and we propose to characterize these activitiesusing two complimentary methods, the Classroom Obser-vation Protocol for Undergraduate STEM (COPUS) [9]and social network analysis.Active learning, at its core, provides opportunities forstudents to interact with each other. COPUS can beused to record instructor and student interaction cate-gories with the aim of creating a unique profile for eachimplementation. Social network analysis can be usedas a means to quantify student interactions that ariseduring a given active learning implementation. There isevidence that a student’s position in the classroom so-cial network improves course outcomes and persistencein physics [10, 11], so it is natural to surmise that activelearning contributes to social network development.With COPUS profiles and classroom social networkdata, we can begin describing how classroom tasks andbehaviors can correlate to student social network devel- a r X i v : . [ phy s i c s . e d - ph ] A ug opment and mobility. By understanding what studentsare doing and how it impacts their social position withinthe class, future research can aim to better understandhow different kinds of interactions lead to student growth.The goal of this paper is to describe the characteristics ofsix distinct pedagogies in physics, as measured by CO-PUS and network analysis. We do not aim to directlycompare pedagogies.We begin with a brief literature review to introducethe observation protocol, social network analysis con-cepts, and the active learning pedagogies that are stud-ied. Then, we describe how observation sites were chosen,the demographic information of each site, and how datawas collected. Finally, we show data from each observedpedagogy and discuss how the in-class tasks and behav-iors present themselves in the social network data. II. LITERATURE REVIEW
In this section, we introduce the observation protocolsthat were considered for this study, and discuss the cho-sen protocol (COPUS). We then introduce social networkanalysis and define relevant terminology used throughoutthis paper. Finally, we summarize the active learningpedagogies that are studied in this project.
A. Observation Protocol
A research-based pedagogy is generally accepted as ‘ac-tive learning’ if the students are involved in the learningprocess in some meaningful way. Additionally, a peda-gogy is considered ‘active learning’ if it is based on re-search with regards to development and implementation,and students consistently show learning gains when thepedagogy is implemented in the classroom [12]. However,it is difficult to measure how much active learning is oc-curring in a classroom, largely due to the wide array ofactive learning methods that exist. The most straightfor-ward way to measure ‘active learning-ness’ is to conductclassroom observations using some sort of protocol. Sev-eral observation protocols exist that serve various pur-poses, some of which will be discussed briefly here.There are two categories of observation protocols:open-ended and structured [9]. Open-ended protocolstypically provide the observer with prompting questions,to which they provide feedback. Structured protocolsprovide the observer with some sort of framework to in-put observation data. Since we aim to have concrete mea-surements of how much and what kind of active learningis occurring, we discarded all protocols that were 100%open-ended, as they rely on observer judgements and gen-erally focus more on opinion statements rather than mea-surable quantities. A discussion of structured protocolsthat were considered for this study follows.The Reformed Teaching Observation Protocol(RTOP) [13] has observers make holistic judgements about the quality of lesson design and implementation,classroom culture, and content coverage. While itincludes the use of Likert scales to assign numericalvalues to these categories, these categories are largelysubjective. Additionally, the RTOP observation itemsdo not fully capture the time-dependence of studentand instructor activities and interactions. RTOP alsohas a long training time due to the in-depth theoreticalframework that it is built upon. Finally, RTOP givesa numeric score where higher is better, which wewanted to avoid because our purpose is comparison anddescription rather than ranking. The RTOP may bepromising for future studies delving into the quality of agiven pedagogical implementation, assuming inter-raterreliability could be reached.The Real-Time Instructor Observation Tool, or RIOT,is a computer-based tool that records instructor behav-iors in real time [8]. This protocol gives a fine-grainedtemporal observation of instructor interactions with stu-dents. While this tool is valuable for understanding howinstructors are leading conversations in the classroom, itwas ultimately not chosen due to the computational hard-ware requirement and lack of student observation codes.This protocol is best suited for studying instructor be-haviors independently of student behaviors.The Teaching Dimensions Observation Protocol(TDOP) [14] has 46 codes to delineate student and in-structor behaviors in the classroom, as well as a handfulof open-ended responses. The observer codes interac-tions during a two minute time interval; if the behav-ior occurred for longer than five seconds, it gets coded.This protocol requires an extensive three-day training toachieve inter-rater reliability. While this protocol gives abroad overview of what is happening in the classroom, itwas deemed impractical for our needs due to long train-ing and large number of codes to keep track of duringlive observations.The Classroom Observation Protocol for Undergrad-uate STEM (COPUS) [9] is similar in structure to theTDOP, but has 25 codes instead of 46, and does nothave observation codes that require the observer to makejudgements about the quality of instruction. It has beenshown to have strong inter-rater reliability after a mereone and a half hour training period. The non-judgmentalcoding schemes provide a quantitative view of the class-room in two-minute intervals. Due to the small numberof codes, it is well known for its ease of use in a liveclassroom observation [9].We ultimately chose COPUS as the observation pro-tocol for this study because we wanted to get an over-all picture of what was happening in the classroom, asopposed to how effective or well implemented the ped-agogy was. In this study, we intentionally chose devel-opment sites or well-regarded secondary sites to ensurehigh-fidelity implementations, so the ‘instruction qual-ity’ codes were deemed unnecessary for our initial in-vestigation. Additionally, we want our investigations tobe easily reproducible, so the shorter training and highinter-rater reliability were highly desirable. While CO-PUS is limited in its ability to measure code duration,as codes are recorded in two-minute intervals instead ofinstantaneously, it includes student and teacher behaviorcodes that are appropriate for examining active learn-ing pedagogies. Codes included in the COPUS protocolcan be seen in Table I. Additional protocols that addressthe time resolution concern were not appropriate or notpublished at the time the project started.
B. Network Analysis
Social network analysis has been used as an analy-sis tool in a Modeling Instruction classroom previously.Brewe et. al. showed that Modeling Instruction pro-duced classroom networks that were structurally uniquefrom a network formed in a standard passive lecture en-vironment [15], which was a large motivation for thisproject. Meanwhile, Zwolak and Dou showed how socialnetwork analysis can be used to correlate student socialpositioning with other factors, such as persistence[11] andself-efficacy[16]. The survey methods deployed by Breweet. al. were refined by Zwolak and Dou, which subse-quently influenced the survey methods of this study.Social network analysis has also been done in the con-text of upper division physics courses, where homeworkgrades were found to be strongly correlated with studentcentrality in their homework problem-solving network.These centrality measures remained stable across differ-ent courses and types of assignments [17]. Vargas et. al.showed the potential benefits of studying physics class-rooms from a network perspective to understand how stu-dent gains manifest.We ultimately chose network analysis as a tool to studyactive learning environments due to its applicability torelational data. Vargas and Brewe demonstrated howsocial network analysis can be used in a physics context,which we expand upon by collecting social network datafrom six pedagogies to identify the network structuresthat arise when a given pedagogy is used. Before webegin, we need to take a moment to define some networkterms.We have provided two toy networks to help illustratethese terms in figures 1 and 2.
Directed/Undirected – A network can be directed,meaning the edges have arrows attached to them toillustrate which node initiated contact (such as in figure1), or undirected, where the edge exists regardless ofwhich node initiated contact (such as figure 2). We useundirected networks in this study. The following termsare defined in the context of undirected networks.
Node – Sometimes referred to as an actor, thenode is the noun in the network. In our toy network,the nodes are represented by the orange dots.
Edge – Sometimes referred to as a tie, an edge is
FIG. 1: A directed toy network, for illustrationpurposes only. The direction of the arrows indicates thedirection of interaction.
FIG. 2: An undirected toy network, for illustrationpurposes only. If an interaction occurred, the edgeexists regardless of which node initiated the interaction.the verb in the network. Edges are represented by thelines connecting the nodes in the toy network diagrams.
Degree – Degree measures how many edges a givennode is connected with. For example, in figure 2, node1 is connected to nodes 2 and 4, for a total of twoconnections. Node 1 has a degree of 2.
Density – The density is the ratio of actual num-ber of edges in a network to possible number of edges.In our undirected toy network (figure 2), there are 4edges, but 10 possible unique edges, giving us a densityof 2/5.
Diameter – If we calculate the geodesic distance(shortest path) between any two nodes, and then do thisfor every pair of nodes in the network, the longest ofthose paths is the diameter of the network. In our toynetwork (fig. 2), our longest-shortest path is from 3 → → →
1, for a diameter of 2.TABLE I: COPUS Codes for Instructor and Student Activities [9]
Students are DoingL
Listening to instructor/taking notes, etc.
Ind
Individual thinking/problem solving. Only mark when an instructor explicitly asks students to thinkabout a clicker question or another question/problem on their own CG Discuss clicker question in groups of 2 or more students WG Working in groups on worksheet activity OG Other assigned group activity, such as responding to instructor question
AnQ
Student answering a question posed by the instructor with rest of class listening SQ Students asks question WC Engaged in whole class discussion by offering explanations, opinions, judgment, etc. to whole class,often facilitated by instructor
Prd
Making a prediction about the outcome of demo or experiment SP Presentation by student(s) TQ Test or quiz W Waiting (instructor late, working on fixing AV problems, instructor otherwise occupied, etc.) O Other
Instructor is DoingLec
Lecturing (presenting content, deriving mathematical results, presenting a problem solution, etc.)
RtW
Real-time writing on board, document projector, etc. (Often checked off along with Lec)
FUp
Follow-up/feedback on clicker question or activity to entire class PQ Posing non-clicker question to students (non-rhetorical) CQ Asking a clicker question (mark the entire time the instructor is using a clicker question, not justwhen first asked)
AnQ
Listening to and answering student questions with entire class listening MG Moving through class guiding ongoing student work during active learning task
One-on-one extended discussion with one or more individuals, not paying attention to the rest ofclass (can be along with MG or AnQ)
D/V
Showing or conducting a demo, experiment, simulation, video, or animation
Adm
Administration (assign homework, return tests, etc.) W Waiting when there is an opportunity for an instructor to be interacting with or observing/listeningto student or group activities and the instructor is not doing so O Other
Transitivity – In an undirected network, this isthe ratio of closed triangles to triads. A closed trianglein an undirected network looks like the arrangement ofnodes 1 → →
4, while a triad would be one step smallerand missing the closure, like 1 → →
3. High levels oftransitivity are usually indicative of a collaborativeenvironment, so this measure will be of particularimportance when considering network formation inactive learning environments.
Giant component – The giant component is thenumber of nodes that are all connected in the largestcluster of nodes. In our toy model, our largest clusterhas 4 nodes in it, so the giant component would be 4.
Isolate – An isolate is a node that is not connected toany other nodes, such as node 5 in figure 2.
C. Active Learning Pedagogies
This project investigated six active learning pedago-gies commonly used in introductory physics courses atthe university level. All of these pedagogies have beenfeatured in the New Faculty Workshops run by the Amer-ican Association of Physics Teachers [6] and have re-search articles describing their development, implemen-tation, and outcomes (and can thus be considered activelearning according to Meltzer, et al.[12]).
1. Tutorials in Introductory Physics
Tutorials in Introductory Physics have been iterativelyresearched and developed at the University of Washing-ton [18]. This curriculum is typically implemented in atraditional lecture/lab/recitation setup, with the bulk ofthe tutorial material presented in recitation (aka tuto-rial section). Tutorials focus on building a strong con-ceptual understanding of the material before introducingcalculations. The tutorial curriculum materials consistof pre-tests, group worksheets, homework problems, andpost-tests. These materials are scaffolded and promptstudents to confront and resolve common misconceptions.Each tutorial section has one or two teaching assistantswho are trained to guide students through the miscon-ception confrontation process [19].
2. ISLE
The Investigative Science Learning Environment(ISLE) approach was developed at Rutgers Univer-sity [20]. The ISLE approach is intended to be imple-mented in all parts of a course, but in some cases, it ispossible to use the ISLE philosophy only in a lab. TheISLE approach helps students learn physics while treat-ing them as neophyte scientists. Students are encouragedto use an iterative process in their learning, much likethey would be expected to do in a scientific career. Thisprocess typically begins with observing a simple, care-fully chosen “observational experiment”. Students, work-ing in small groups, then try to explain the experimentbased on their observations, and use their explanation tomake predictions about the outcomes of new “testing”experiments that they design. When there is a mismatchbetween the prediction and the outcome of the testingexperiment, the students revise the explanation. Mul-tiple explanations are encouraged for the observationalexperiments. To develop and test explanations, studentsuse multiple representations. Unlike several of the othercurricula studied in this project, ISLE focuses on build-ing up student’s correct intuition rather than debunkingmisconceptions. It is used in large and small enrollmentcollege physics courses and in many high school physicscourses [21].
3. Modeling Instruction
Modeling Instruction for university physics was devel-oped collaboratively [22], modeled after the high schoolModeling Instruction curriculum developed by Wells andHestenes [23]. Modeling Instruction is ideal for large,open classrooms with the ability for large group collabo-ration. It is a curriculum that focuses on having studentsbuild their fundamental understanding of physics fromthe ground up, by observing phenomena and then creat-ing models to describe the phenomena. Students use mul-tiple representations to explain their models, and deploythose models to future situations until they break. A typ-ical day in class begins with a cooperative group activity.After the activity, students then engage in ‘whiteboardmeetings’, where they circle up with several other groupsto share their results. This forces students to consolidatetheir ideas into an understandable, presentable format,and allows discussion between groups [24]. There is verylittle lecture instruction; all material is learned through the activities and model building.
4. Peer Instruction
Peer Instruction was popularized by Eric Mazur atHarvard University [25]. Peer Instruction is typicallyused in large lecture halls as a way to integrate activelearning into traditional lecture-style courses, but canalso be used in smaller courses. A typical cycle of in-struction begins with approximately ten minutes of lec-ture followed by a clicker question. The question is posed,students answer individually, students discuss with theirneighbors, and are sometimes allowed to answer again.This curriculum is commonly facilitated with personal-response systems, such as clickers, color-coded cards, oran online response program. While Peer Instruction ismeant to refer to a specific routine and style of question-ing [26], the implementation of Peer Instruction varieswildly between instructors [5].
5. Context-Rich Problems
Context-Rich Problems refers specifically to the Min-nesota Model for Large Introductory Courses that wasdeveloped as a physics curriculum at the University ofMinnesota [27]. Context-Rich Problems works within astandard course structure consisting of lectures, labs, andrecitation/discussion sections. This pedagogy uses thecognitive apprenticeship model [28] with an emphasis onproblem solving skills as a means to organize content inthe course. For example, during lectures, the instruc-tor solves problems using an expert-like framework, illu-minating the hidden decision-making processes that arenecessary in physics problems. During the labs and dis-cussion sections, students practice solving problems whilegiving and receiving coaching from instructors and otherstudents. This practice takes place in groups of 2 to 4,structured by the principles of cooperative group work[29].The context-rich problems that students encounterwith this pedagogy differ from typical textbook prob-lems in that they provide a realistic reason for calculat-ing something. Students are encouraged to follow thesame expert-like problem solving strategy their instruc-tor demonstrates in lecture [30]. These types of problemstypically contain extraneous information, require the useof estimation, or require students to recall commonlyknown values. All of these activities immerse studentsin a culture of expert practice similar to a traditionalapprenticeship.
6. SCALE-UP
Student Centered Activities for Large Enrollment Un-dergraduate Programs, or SCALE-UP , was developed byRobert Beichner et al [31]. It is an integrated learningenvironment designed for large-enrollment physics classeswith up to 100 students. The goal of SCALE-UP was totake a studio-style environment and make it accessiblefor larger courses. A typical SCALE-UP course does notinclude much lecture time, but instead relegates informa-tion transfer to assigned readings outside of class. Thisleaves class time for cooperative group problem solving,experiments, or answering questions that students mayhave. SCALE-UP refers more to the environment thanthe specific pedagogy; instructors are able to implementany pedagogy they wish, and have it easily translated toa large class via the room layout. A typical SCALE-UPclassroom has large, round tables, capable of holding upto nine students. Within these tables, students are inteams of 3 with whom they solve problems and work onexperiments. Having multiple teams at the same tableallows for group-to-group interaction without wreakinghavoc on classroom management. The room can eitherhave whiteboards along the perimeter wall or individualwhiteboards at the tables. These are used for group prob-lem solving and sharing with the class. SCALE-UP hasone session with the students, there is no separate lab orrecitation section; everything is done in the same roomwhen the lesson calls for it.
III. METHODOLOGY
In this section, we describe how research sites werechosen and provide an overview of the selected site de-mographics. We broadly describe the social network andCOPUS data collection protocols, then further describethe methods for each site. We also include descriptionsof each pedagogical implementation.
A. Research Site Selection
To determine which curricula to study, we began byreviewing past New Faculty Workshops, hosted by theAmerican Astronomical Society, the American PhysicalSociety, and the American Association of Physics Teach-ers. There were three main criteria we looked for whendeciding on pedagogies: Does the pedagogy have devel-oped curriculum materials? Does the pedagogy havean established body of research? Is this pedagogy stillwidely used in actual physics classrooms?After identifying the six pedagogies used in this study,we reached out directly to the developers of the curricu-lum to identify high-fidelity implementations of each ped-agogy. When possible, the institution that developed thepedagogy was used, however, this was not always feasible.Secondary institutions were identified via recommenda-tion of instructors with extensive training or research ex-perience with the pedagogy in question. Research siteswere compensated for participating in this study. Demo-graphic information for the chosen sites can be seen in Table II.
B. Social Network Data Collection
Research sites provided rosters of students enrolled inthe introductory physics course for the term that wouldbe observed. Students were invited via email to completean online survey during the first week of the term andagain at the end of the term. Students were asked toselect their name from a list of people enrolled in theclass. Given the same list, they were presented with thequestion“Please choose from the list of people thatare enrolled in your physics class the namesof any other student with whom you hada meaningful interaction in class during thepast week, even if you were not the main per-son speaking.”Students self-identified what counted as a ‘meaningful’interaction. This approach to network data collectionhas been used successfully in physics education researchpreviously [11, 15, 16]. Instructors were asked to promotethe survey in class to encourage participation.Students were included in the social network if theyfilled out the survey, or were named by someone whofilled out the survey in either the pre or post distribu-tion. If a student did not fill out the survey, or was notnamed by someone else in either the pre or post distri-bution, they were not included in the data. Additionally,students under 18 were not included as respondents dueto IRB restrictions, but they could still be included inthe network by someone else naming them.
C. COPUS Data Collection
Each institution was visited for a full week during theterm to conduct COPUS observations for as many sec-tions as possible. Every two minutes, instructor and stu-dent codes were marked if said code occurred during thatinterval. COPUS profiles were created by taking the frac-tion of time slots observed versus the total length of theobservation, and recorded as a percentage of class timefor each code.
D. Tutorials in Introductory Physics: DataCollection
The tutorials site was an R1 institution in the North-western United States. The course format included a lec-ture section, a laboratory section, and a tutorial section.The network surveys were distributed by lecture section,as all students in a lecture section were distributed intothe same subset of tutorial sections. In the survey, nameswere grouped by tutorial section to facilitate studentsTABLE II: Institution-level demographics of research sites as provided by Integrated Post-Secondary EducationData System (IPEDS) [32]. Class-level demographics were not available for this study. For demographics that donot sum to 100%, the missing entries are ‘unknown’.
Tutorials ISLE Peer Inst. Context-Rich Modeling SCALE-UP selecting peers they worked with in their tutorial class.Three lecture sections were included in this study, whichcorrelated to 24 tutorial sections. The tutorial sectionshad approximately 20 students per section. Nineteen tu-torial sections and one lecture section were observed.All tutorials occurred in the same classroom, and at-tendance was graded. Students were seated at small ta-bles designed for four students, but group sizes rangedfrom two to five students. Students chose their owngroups, and did not necessarily have to work with thesame people each week. The tables had whiteboards inthe middle to facilitate group discussion. Each sectionhad one lead teaching assistant and one grader. Thegrader spent the first few minutes passing back gradedpapers, and then joined the lead TA as a teaching assis-tant. For the purposes of the COPUS observations, thelead TA was observed using instructor codes. Grader in-teractions were also coded, but not included in analysis.Tutorial sessions were 50 minutes long, but observationwas only done for 20 minutes in the middle of the session.
E. ISLE: Data Collection
The ISLE site was an R1 public institution in the Mid-Atlantic region of the United States. There are two vari-ations of ISLE at this institution, both of which wereincluded in this study. The first is a lab-only implemen-tation, where ISLE is used in the lab course and accom-panied by a standard lecture and recitation. The secondis a whole-course implementation, where ISLE principlesare used across lab, recitation, and lecture.
1. Lab-Only Implementation
In the lab-only ISLE implementation, the lab courseis taken separately from a lecture and recitation course.The lab does not have to be taken in the same term asthe lecture and recitation, so only the lab sections wereobserved with COPUS and given network surveys. Eachsection had approximately 28 students enrolled. Twentylab sections were surveyed from this implementation. Ofthose, five sections were observed using COPUS.The physics laboratory is a typical lab-bench set-upwith one computer per station. Students work in groupsof 2-3 to complete the activity, and use shared GoogleDocs to draft and submit their lab report. COPUS ob-servations were taken for the entire 3 hour session.
2. Whole-Course Implementation
The whole-class ISLE implementation consisted of alecture component, and then small lab and recitation sec-tions where group activities were performed. All sectionsof the whole-course implementation must be taken con-currently, so network surveys were distributed for eachof the nine recitation sections as well as each of the ninelab sections. All students were in the same lecture, so alecture survey was not distributed. Both lab and recita-tion sections had approximately 28 students per section.Two lecture sessions were observed with COPUS, as wellas four recitations and one lab.The lecture was held in a stadium-seating auditoriumstyle lecture hall. The lab and recitations were held ina separate lab/recitation room, which was set up withsmall round tables to encourage group cooperation. Eachtable held a group of 4 students, where they worked inpairs and shared with the other pair at the same table.On recitation days, the students worked through ISLEworkbook activities. Recitations were an hour and a halflong. On lab days, students performed experiments fol-lowing the ISLE protocol. Labs were three hours long.COPUS observations were taken for the full length ofeach session.
F. Modeling Instruction: Data Collection
The Modeling Instruction site was an R1 institution inthe Southeastern part of the United States. The courseused a studio format (integrated lab and lecture com-ponents) that met twice a week. Network surveys weredistributed in each individual class section. Three sec-tions were included in this study, with sizes ranging from64 to 92 students. All three sections were observed usingCOPUS.All course activity occurred in the same room, whichwas a large open format room with large tables for groupactivities. There was a large whiteboard and projec-tors at the front of the room. Students were distributedamong the large tables in groups ranging from 2 to 6 peo-ple (assigned group size was 6 but due to absences couldbe as small as 2 on any given day). Student groups wereassigned by the instructor and changed twice during theterm. The tables had whiteboards in the middle to facil-itate group discussion; in addition, the boards were usedduring ‘board meetings’.Within the groups, students ranged from working col-laboratively on the assigned activity to working alonewhile sitting next to other students. However, the groupshad to create a whiteboard summarizing their work topresent to their peers during large whiteboard meetings,held 1-3 times during the class. Even if students workedalone on the activity, they were required to collaboratewith their table-mates to create the whiteboard. Duringthe whiteboard meeting, 3-5 groups made a large circle todiscuss their work. Each group shared their whiteboards,and students discussed whether they obtained the sameanswers/results. Each section had several teaching assis-tants; in the COPUS observations, only the lead instruc-tor was coded. Modeling Instruction sections were twohours long, of which COPUS observations began 10-15minutes into the session to allow students to get settled.Both days of instruction were aggregated into a singleCOPUS profile for each section. We felt it produced amore meaningful “snapshot” of the curriculum to providea week-long observation rather than a single class period.
G. Peer Instruction: Data Collection
The Peer Instruction site was an R1 private institutionin the Mid-Atlantic region of the United States. Thecourse format included a lecture, a laboratory section,and a recitation section. The network surveys were dis-tributed by lecture section, since that is where the PeerInstruction curriculum was implemented. One section ofPeer Instruction was surveyed and observed using CO-PUS, with an enrollment of 113 students.Class was held in a large stadium seating lecture hall.Students sat wherever they wanted. The laboratory andrecitation sections were completely open during registra-tion, so it was possible to have students mixing betweenlecture sections in the smaller lab and recitation sections.The survey rosters only included students in the lecturesection in which they are enrolled in order to constrainthe network. The network survey was distributed to thestudents in class as part of a ”Learning Catalytics” ac-tivity, which resulted in a significantly higher responserate. Lectures were 50 minutes long, of which twentyminutes were observed using COPUS, in the middle ofthe session.
H. Context-Rich Problems: Data Collection
The Context-Rich Problems site was a 2-year commu-nity college in the Mid-Pacific region of the United States.The course format included a lecture, discussion section,and lab section. There were two distinct sections of stu-dents. Each section attended the same lecture and dis-cussion, and then was split into two lab courses. Networksurveys were distributed at the lecture/discussion level,with enrollments of 48 and 45 for each section. COPUSobservations were performed for both discussion sections,all four lab sections, and two sessions of lecture per sec-tion, for a total of four lecture observations.The lab component was held in a traditional style labroom with lab benches that had computers at each sta-tion. Each group had 3-4 students. At the beginning ofclass, a question was projected on screen to help primestudents for the activity. They were instructed to usewhiteboards to design their experiment based on whatthey were trying to learn, and relate the data they weregoing to take to the concepts learned in class. Lab sec-tions were 170 minutes long, of which the entire time wascoded with COPUS.The lecture was held in a small lecture hall withslightly tiered rows of stationary desks. The lecture was80 minutes long, of which the entire time was coded withCOPUS. The discussion section was held in the sameroom as the lecture. Despite the desks being immobile,students would physically turn in their seats to work ingroups of 2-3 on a worksheet. They were provided white-boards to facilitate discussion. An additional teaching as-sistant was present– although not included in the COPUSobservations; they spent the entire time guiding discus-sions (MG and 1o1). Discussion sections were 50 minuteslong, of which the entire time was coded. The studentsworked in the same groups in the lab and discussion ses-sions. These groups were assigned by the instructor andmaintained for 3-4 weeks at a time before being reorga-nized so that students could work with different peers.
I. SCALE-UP: Data Collection
The SCALE-UP site was a public R2 institution in theGreat Plains region. The course used a studio formatthat met three times a week. There was only one sectionof the course, taught by one instructor, with an enroll-ment of 71 students.The observed SCALE-UP curriculum was performedin a large room designed for active learning. There wereseveral large, round tables with microphones at the cen-ter. The perimeter of the room was covered with whiteboards. There were television screens around the roomin place of a projector, to allow for viewing of presenta-tion materials from multiple angles. The instructor hada wireless microphone to allow for mobility without sac-rificing sound quality.Class was an hour and fifteen minutes long, of whichthe entire time was coded with COPUS. Three days ofCOPUS observations were aggregated into a single CO-PUS profile, as we felt it better represented the curricu-lum to give a week-long “snapshot” instead of a singleclass period.
IV. RESULTS
While five of the six curricula had data collected frommore than one section, we chose the section with thehighest survey response rate for network analysis. TableIII depicts the calculated network metrics for the socialnetworks presented in this paper. These values will bediscussed in detail in section V A.
A. Tutorials in Introductory Physics
1. COPUS observations
The tutorial section presented here was chosen at ran-dom from the subset of tutorial sections included in thetop level lecture section. In the tutorial section, studentsworked in groups of 2-5 students. Within the groups, stu-dents ranged from working as a fully collaborative groupon the tutorial worksheet assignment, to working alonewhile sitting next to other students. Figure 3a shows thestudent codes to illustrate the overwhelmingly commoncodes, SQ (student asks a question) and WG (working ingroups). In these observations, the SQ COPUS code wasonly marked when a student explicitly raised their hand to ask the TA or grader a question, not questions duringan already in progress one-on-one session.Figure 3b shows the instructor codes for this tutorialsection to illustrate the three overwhelmingly commoncodes. The TA moved around the room and promptedstudents to work on the activity (MG) and stopped tohave extended discussions with the student groups (1o1).The observations for the tutorial sections looked ex-tremely similar, with small variations depending if theTA walked around the room and interjected themselvesinto student groups in an effort to drive conversation(coded as MG, moving around the room and guiding dis-cussions, typically followed by 1o1 when a longer discus-sion arose), or if the TA waited at the front of the roomuntil a student raised their hand with a question (codedas W-waiting).The tutorial section presented here was taken from thesame lecture section as presented in the networks. How-ever, the observed lecture was not the same as presentedin the network plots, but had the same instructor, so weassumed high levels of similarity between the two sectionsfor COPUS.The observed lecture took place in a traditional, audi-torium stadium-seating lecture hall with a large projectorand chalk boards. The lecture began with a group activ-ity about vectors, then used a response collection systemto give students a short quiz. Students used phones orcomputers to answer the questions, and were allowed towork together as long as they gave their own answer. Af-ter the quiz, an interactive lecture was given using Pow-erPoint slides. The COPUS codes for the lecture sectioncan be seen in figures 4a and 4b.
2. Network data
The network data for Tutorials in Introductory physicswas collected at the lecture section level. The lecturesection with the highest response rate is presented here.Figures 5a and 5b show the pre and post-network di-agrams, respectively. The networks are characterized bygroupings of 2-5 students, indicative of small group struc-ture. There were also larger chain-like patterns, likelydue to the geometry of the lecture hall and the interactiv-ity of the lecture-based activities. Additionally, we sawlinking between smaller clusters, indicating cross-groupinformation transfer.Figures 6a and 6b were colored by tutorial section forthe pre and post network, respectively. We saw in bothcases clustering by tutorial section, indicating that stu-dents largely worked with their tutorial group to learnphysics, even in the lecture portion of the class.Figures 7a and 7b were colored by lab section. Therewas still some clustering by section, but not as distinctas the tutorial sections. Tutorial and lab section overlapranged between 1 and 6 students, which could explainclustering consistent in both sections.0 S − L S − A n Q S − W S − O S − I N D S − C G S − S Q S − W G S − W C S − S P S − O G S − P r d S − T / Q Student Code F r a c t i on o f O b s e r v a t i on T i m e Student Activities−Tutorials: Tutorial section (a) Student COPUS codes for a tutorial section ofTutorials in Introductory Physics curriculum. Studentsworked in small groups on tutorial worksheet activity,and occasionally asked questions to the TA. I − P Q I − A n Q I − A d m I − W I − C Q I − O I − Le c I − R t W I − F up I − M G I − I − D / V Instructor Code F r a c t i on o f O b s e r v a t i on T i m e Instructor Activities−Tutorials: Tutorial section (b) Instructor COPUS codes for the tutorial section ofTutorials in Introductory Physics curriculum. Tutorialsections had a lead TA and a grader/TA. The lead TAwas coded with COPUS and shown here.
FIG. 3: COPUS profiles of one tutorial section in the Tutorials in Introductory Physics curriculum. Observationswere twenty minutes long out of the fifty minute section. S − L S − A n Q S − W S − O S − I N D S − C G S − S Q S − W G S − W C S − S P S − O G S − P r d S − T / Q Student Code F r a c t i on o f O b s e r v a t i on T i m e Student Activities−Tutorials: Lecture (a) Student COPUS codes for the lecture portion of theTutorials in Introductory Physics curriculum. I − P Q I − A n Q I − A d m I − W I − C Q I − O I − Le c I − R t W I − F up I − M G I − I − D / V Instructor Code F r a c t i on o f O b s e r v a t i on T i m e Instructor Activities−Tutorials: Lecture (b) Instructor COPUS codes for the lecture portion ofthe Tutorials in Introductory Physics curriculum.
FIG. 4: COPUS profiles of the lecture portion of the Tutorials in Introductory Physics curriculum. Observationswere twenty minutes long during the fifty minute section.1 l ll lll ll l llll ll ll llll ll ll ll lll l l ll ll lll l l ll ll ll lllll l ll lll l ll llll llll l ll l lll ll l ll l l ll ll l ll lll lll ll lll l lll ll l lll ll lll ll ll l lll lll ll ll lll lll l ll llll ll l
Tutorials: Pre (a) Beginning of term social network for Tutorials inIntroductory Physics. lll lll llll l ll ll l llll l lll lll ll ll l lll l l ll lllll l l lllll ll lll ll ll lll lll ll lll lll lll ll ll lll ll ll l llll l ll ll l lll ll ll lll l l ll lll lll l lll l ll lll lll l ll llll ll ll ll ll
Tutorials: Post (b) End of term social network for Tutorials inIntroductory Physics.
FIG. 5: Pre and post-term social networks for Tutorial in Introductory Physics. The network survey was distributedat the lecture level. Small groupings of students are visible, likely due to tutorial groups. The color shading on theseplots indicates which students filled out the survey. (a) Beginning of term social network for Tutorials inIntroductory Physics, colored by tutorial section. (b) End of term social network for Tutorials inIntroductory Physics, colored by tutorial section.
FIG. 6: Pre and post-term social networks for Tutorial in Introductory Physics, colored by tutorial section. We seeclustering by color, indicating that students form connections with peers in the same tutorial section.2TABLE III: Network metrics for six active learning curricula in physics. The section with the highest response ratewithin each pedagogy is reported.
Curriculum Pre/Post Diameter Avg.Degree Density Transitivity GiantComponent Numberof Nodes
Tutorials Pre 13 1.370 0.00941 0.1340 41 147Tutorials Post 13 1.580 0.01080 0.2640 67 147ISLE-Lab only Pre 6 1.410 0.05410 0.2310 15 27ISLE-Lab only Post 7 1.930 0.07410 0.3160 13 27ISLE-Whole Class Lab Pre 3 1.750 0.07610 0.4140 5 24ISLE-Whole Class Lab Post 4 2.250 0.09780 0.5900 11 24ISLE-Whole Class Rec Pre 3 0.667 0.03330 0.0000 4 21ISLE-Whole Class Rec Post 4 1.430 0.07140 0.5560 9 21Modeling Instruction Pre 10 2.510 0.03390 0.1790 62 75Modeling Instruction Post 5 5.600 0.07570 0.2100 72 75Peer Instruction Pre 13 1.830 0.01740 0.2330 61 106Peer Instruction Post 12 2.230 0.02120 0.2310 80 106Context-Rich Problems Pre 8 1.910 0.04340 0.1980 28 45Context-Rich Problems Post 6 3.640 0.08280 0.2450 41 45SCALE-UP Pre 7 1.710 0.02510 0.3360 27 69SCALE-UP Post 8 4.090 0.06010 0.5170 58 69(a) Beginning of term social network for Tutorials inIntroductory Physics, colored by lab section. (b) End of term social network for Tutorials inIntroductory Physics, colored by lab section.
FIG. 7: Pre and post-term social networks for Tutorial in Introductory Physics, colored by lab section. While thereis still some clustering by color, the tutorial section seems to be the main driver for student connections.
B. ISLE: Lab-only implementation
1. COPUS observations
Figure 8a shows the student COPUS codes for one sec-tion of the ISLE lab-only implementation. The entireclass period was spent working on the group lab activity(OG). Students would occasionally raise their hands toask for help from the teaching assistant (SQ). In some sections, like the one shown here, the TA spent a fewminutes during class going over concepts used in the labactivities, in which case the students listened (L) andanswered questions posed to the entire class (AnQ).Figure 8b shows the instructor COPUS codes from theobserved TA behavior. The TA spent most of their timemoving around the room and guiding the student activ-ities (MG), and frequently stopped at groups to engagethem in conversation about the activity (1on1). In some3sections, like this one shown, the TA spent a few minutesgoing over concepts used in the lab activities, which wascoded as Lec and RtW. During this short lecture time,the TA also posed questions to the entire class (PQ) andanswered questions that the students had (AnQ). Theyalso spent some time handing back papers and discussinggrades with students (Adm), waiting for students to raisetheir hands (W), or talking to other TAs who happenedto stop by (O).4 S − L S − A n Q S − W S − O S − I N D S − C G S − S Q S − W G S − W C S − S P S − O G S − P r d S − T / Q Student Code F r a c t i on o f O b s e r v a t i on T i m e Student Activities−ISLE, lab−only: Lab (a) Student COPUS codes for the laboratory section ofthe lab-only ISLE curriculum. I − P Q I − A n Q I − A d m I − W I − C Q I − O I − Le c I − R t W I − F up I − M G I − I − D / V Instructor Code F r a c t i on o f O b s e r v a t i on T i m e Instructor Activities−ISLE, lab−only: Lab (b) Instructor COPUS codes for a laboratory section ofthe lab-only ISLE curriculum.
FIG. 8: COPUS profiles of a lab section in the lab-only ISLE curriculum. Observations were of the entire three hourperiod.5
2. Network data
Figure 9a shows the early-term network diagram forthe lab-only implementation. While there was a largecluster of students, the structure seemed to indicate thatthree students in particular were responsible for the ma-jority of student interactions (the darkest shaded nodes).Meanwhile, in the post-network of figure 9b, the group-ings of three students were much more pronounced, sug-gesting that lab groups were the dominant driver of stu-dent interactions.
C. ISLE: Whole-course implementation
1. Recitations COPUS observations
Figure 10a shows the COPUS codes for the studentsduring the recitation section of the whole-course ISLEimplementation. The majority of the time was spentworking on the active learning workbook activity (WG).Students occasionally raised their hands to ask the in-structor a question (SQ), and spent some time listeningto the instructor go over midterm instructions and in-structions for registering for the next term (L).Figure 10b shows the instructor COPUS codes for therecitation section. The instructor spent most of the timemoving between groups and guiding the activity (MG),and frequently stopped for extended one on one discus-sions (1o1). Some time was spent describing how to reg-ister for the next term and midterm logistics (Adm). Theremaining time was spent waiting for a student to raisetheir hand with a question (W) or talking to the TA (O).
2. Recitation Network Data
The recitation sections for the COPUS and the net-work plots presented here are not from the same sec-tion. We chose to present the network with the highestresponse rate, which did not have a matching COPUSobservation. Since all sections were taught by the sameinstructor and used the same activities, we assumed themto be similar enough to be considered side by side.The early term network shown in figure 11a shows alargely isolated student population. During recitation,students worked on workbook activities, so it could beexplained as students working alone despite being at thesame table as other students. In the end of term networkin figure 11b, we can see some more prominent grouping,but still a large number of isolates.
3. Lab COPUS Observations
Figure 12a shows the COPUS student codes for thewhole-class implementation during the lab section. The majority of the lab course was spent working on the as-signed activity (OG). Students occasionally raised theirhands to ask the instructor a direct question (SQ). Theinstructor spent some time going over midterm instruc-tions, with students listening (L). At the end of the day,students cleaned up their stations and prepared to gohome (O). Figure 12b shows the instructor codes forthe lab section of the whole-course ISLE implementa-tion. The instructor spent the majority of the time mov-ing around the room and guiding the activity (MG), andfrequently stopped to have extended one-on-one discus-sions (1o1). Some time was spent describing the midterm(Adm), talking with the TA (O), briefly going over a com-mon misconception (Lec), or waiting at the side of theroom for a group to raise their hands (W).
4. Lab Network Data
The network plots for the lab sections show very dis-tinct groupings, indicating that students did not speakto anyone outside of their assigned groups (figure 13a).As the term progressed, the distinct groups remained thedominant structure, but introduced some cross-group in-teraction (Figure 13b).
5. Lecture COPUS Observations
During the lecture, students listened to the instructor(L) and worked together on a short problem solving ac-tivity (OG). The instructor lectured (L) and wrote onthe chalkboard (RtW), and posed an extended questionto the entire class (PQ). Students were not expected torespond with clickers, but instead put their answer on apaper to be handed in. The rest of the time was spent fol-lowing up the extended question (Fup) and talking aboutthe midterm (Adm). Network data was not collected atthe lecture level.
D. Modeling Instruction
1. COPUS observations
We present the section with the highest response rateon the network survey. Two COPUS observations weredone for each section. The two corresponding COPUSobservations were aggregated into one graph to representa week worth of class time. This particular section had11 groups of students, ranging in size from 2 to 6 stu-dents. There were three teaching assistants. The firstday of class was spent working on a worksheet in thesmall groups, and the second day extended that activityto include an investigative lab experiment.Figure 15a shows the student COPUS codes for theModeling Instruction course. While students spent mostof the time working in groups on their worksheets (WG),6 l ll l ll ll ll lll lll l lll lll llll
ISLE, lab only: Pre (a) Beginning of term social network for one lab sectionof the lab-only ISLE curriculum. ll l ll ll ll l ll ll l lll ll l lll ll l
ISLE, lab only: Post (b) End of term social network for one lab section of thelab-only ISLE curriculum.
FIG. 9: Pre and post-term social networks for the lab-only implementation of ISLE. The network survey wasdistributed within individual lab sections. The color shading on these plots indicates which students filled out thesurvey. S − L S − A n Q S − W S − O S − I N D S − C G S − S Q S − W G S − W C S − S P S − O G S − P r d S − T / Q Student Code F r a c t i on o f O b s e r v a t i on T i m e Student Activities−ISLE, whole course: Recitation (a) Student COPUS codes for the recitation section ofthe whole-course ISLE curriculum. I − P Q I − A n Q I − A d m I − W I − C Q I − O I − Le c I − R t W I − F up I − M G I − I − D / V Instructor Code F r a c t i on o f O b s e r v a t i on T i m e Instructor Activities−ISLE, whole course: Recitation (b) Instructor COPUS codes for the recitation section ofthe whole-course ISLE curriculum.
FIG. 10: COPUS profiles of one recitation section of the whole-course ISLE curriculum. Observations were of theentire 75 minute section.7 l lll lll ll l lllll lll ll l
ISLE, whole course: Rec., Pre (a) Beginning of term social network for one recitationsection of the whole-course ISLE curriculum. lllll ll ll l llll lllll ll
ISLE, whole course: Rec., Post (b) End of term social network for one recitation sectionof the whole-course ISLE curriculum.
FIG. 11: Pre and post-term social networks for a recitation section of the whole-course implementation of ISLE. Thenetwork survey was distributed in the individual recitation sections. Small pairings of students are visible, likely dueto partner collaboration on recitation activities. The color shading on these plots indicates which students filled outthe survey. S − L S − A n Q S − W S − O S − I N D S − C G S − S Q S − W G S − W C S − S P S − O G S − P r d S − T / Q Student Code F r a c t i on o f O b s e r v a t i on T i m e Student Activities−ISLE, whole course: Lab (a) Student COPUS codes for the lab section of thewhole-course ISLE curriculum. I − P Q I − A n Q I − A d m I − W I − C Q I − O I − Le c I − R t W I − F up I − M G I − I − D / V Instructor Code F r a c t i on o f O b s e r v a t i on T i m e Instructor Activities−ISLE, whole course: Lab (b) Instructor COPUS codes for a lab section of thewhole-course ISLE curriculum.
FIG. 12: COPUS profiles of a lab section in the whole-course ISLE curriculum. Observations were of the entirethree hour period8 l l ll lll l lll ll ll ll ll ll lll
ISLE, whole course: Lab, Pre (a) Beginning of term social network for one lab sectionof the whole-course ISLE curriculum. lll l lll ll lll l lll lll lll ll
ISLE, whole course: Lab, Post (b) End of term social network for one lab section of thewhole-course ISLE curriculum.
FIG. 13: Pre and post-term social networks for the lab section of the whole-course implementation of ISLE. Thenetwork survey was distributed within the individual lab sections. Small groupings of students are visible, likely dueto collaborative lab groups. S − L S − A n Q S − W S − O S − I N D S − C G S − S Q S − W G S − W C S − S P S − O G S − P r d S − T / Q Student Code F r a c t i on o f O b s e r v a t i on T i m e Student Activities−ISLE, whole course: Lecture (a) Student COPUS codes for the lecture section of thewhole-course ISLE curriculum. I − P Q I − A n Q I − A d m I − W I − C Q I − O I − Le c I − R t W I − F up I − M G I − I − D / V Instructor Code F r a c t i on o f O b s e r v a t i on T i m e Instructor Activities−ISLE, whole course: Lecture (b) Instructor COPUS codes for the lecture section ofthe whole-course ISLE curriculum.
FIG. 14: COPUS profiles of one lecture section in the whole-course ISLE curriculum. Observations were of theentire hour long section.9they also spent time having whole class discussions viawhiteboard meetings (WC). The experiment was codedas OG for other group activity. SQ was only coded whena student explicitly raised their hand to draw a TA tothe group.Figure 15b shows the instructor COPUS codes. Mostof the time was spent moving around the room, guid-ing the activities (MG) and frequently stopping for ex-tended one on one discussions (1o1). During the white-board meetings, the instructor would frequently lead thediscussion by posing questions (PQ) to the students oranswering questions the students still had after the dis-cussion.
2. Network Data
The early-term network plot in figure 16a shows highlevels of inter-connectivity. However, there were still twodistinct “islands”, only connected to the main cluster viaa single person. The late-term network shown in figure16b shows a much higher level of inter-connectivity; thedensity of this network more than doubled throughoutthe ten week period.
E. Peer Instruction
1. COPUS observations
The Peer Instruction COPUS data had the greatestrange of significantly present activities recorded. Figure17a shows the COPUS student codes for the observedPeer Instruction section. Students spent most of the classlistening and taking notes (L), or responding to clickerquestions. Clicker questions were evenly split betweenindividual thinking (IND) and group discussion (CG), in-dicating that the instructor followed the suggested PeerInstruction model for clicker questions, which includedtime for both individual and group discussion during theanswer portion. There were also a few instances of stu-dents asking questions (SQ) and students being calledon to answer questions (AnQ). The rest of the time wasspent waiting for the instructor to enable the clicker re-sponse system (W).The instructor COPUS codes can be seen in figure 17b.The instructor spread their time across multiple activi-ties. Less than half of the class time was spent lecturingand writing on the board (Lec and RTW). More than halfof the time was spent posing individual questions (PQ)or clicker questions (CQ). After a question was posed,the instructor spent time following up by going over theanswer (Fup) and answering questions the students stillhad (AnQ). For this class period, a lot of time was spentwaiting for the students to answer the questions (W) orworking through technical issues and activating the stu-dent response system (O). Administrative tasks were alsoperformed (Adm) in the form of handing back midterm exams while the students were thinking through clickerquestions.
2. Network Data
Figure 18a shows the student network at the beginningof the term. There were string-like structures in this net-work, likely caused by the geographical restriction thatstudents were sitting in rows and thus limited to interac-tions with students in their immediate vicinity. Branchescould be explained by speaking to fellow students in frontof or behind the student. There were a large percentageof isolated students, as students were not required to sitnear or interact with their peers. The post-term networkin figure 18b showed a very similar structure, but withslightly larger levels of connectivity, indicating that stu-dents were more likely to engage in discussion with theirpeers towards the end of the term.
F. Context-Rich Problems
1. COPUS observations
COPUS observations were taken for each part of theContext-Rich Problems curriculum. The discussion sec-tion observations can be seen in Figure 19. Studentsspent the majority of the time working in groups on theworksheet activity (WG). They frequently raised theirhand to ask the TA/instructor a question (SQ). The in-structor occasionally made announcements about the ac-tivity to the whole class, which students listened to (L).Whiteboards were available to facilitate group coopera-tion, which were gathered and put away at the beginningand end of class (O). The instructor spent the majorityof the time walking around the room and guiding theactivity (MG), frequently stopping to have extended dis-cussions with student groups (1o1). The instructor alsohanded back papers (Adm) and went over points of diffi-culty with the activity to the whole class (Lec and Fup).The lab section observations can be seen in Figure 20.Students spent the first portion of the class listening tothe instructions (L). Students then solved a context-richproblem in their groups that was related to the lab activ-ity (OG), and used the problem to predict the outcomeof their experiment (Prd). The lab activity was then per-formed for the rest of the period (OG). When studentshad questions, they raised their hands to attract the in-structor to their group (SQ). There were limited numbersof some apparatus, so groups would have to wait for themto become available (W). At the end of the class, the ac-tivity was cleaned up for the next class (O).The instructor began the class by giving instructionsfor the activity and reviewing key concepts that would beused (Lec, RtW). They then posed a context-rich prob-lem to the students that was related to the activity anddiscussed the experiment as a class (PQ, Fup). Once the0 S − L S − A n Q S − W S − O S − I N D S − C G S − S Q S − W G S − W C S − S P S − O G S − P r d S − T / Q Student Code F r a c t i on o f O b s e r v a t i on T i m e Student Activities− Modeling Instruction (a) Student COPUS codes for one section of theModeling Instruction curriculum. I − P Q I − A n Q I − A d m I − W I − C Q I − O I − Le c I − R t W I − F up I − M G I − I − D / V Instructor Code F r a c t i on o f O b s e r v a t i on T i m e Instructor Activities− Modeling Instruction (b) Instructor COPUS codes for one section of theModeling Instruction curriculum. Only the instructorwas coded, despite the presence of 3-4 teachingassistants.
FIG. 15: COPUS profiles for one section of the Modeling Instruction curriculum. Both days of instruction, each twohours long, were combined to create profiles representing a weeks worth of instruction. lll lll ll l llll lll ll ll l l lll l lll ll ll l ll lll lllll ll lllll ll l ll l ll lll ll lll ll ll ll ll
Modeling Instruction: Pre (a) Beginning of term social network for one section ofthe Modeling Instruction curriculum. lll llll ll l ll l ll ll lll llll l l ll l l l lll llllll l lll llll l llll lll lll ll l l l lll lll lll ll
Modeling Instruction: Post (b) End of term social network for one section of theModeling Instruction curriculum.
FIG. 16: Pre and post-term social networks for Modeling Instruction. Network surveys were distributed by section.The pre-network is tightly connected, but still has two distinct islands at the top and right side of the networkdiagram. The post network shows a highly integrated class structure. The color shading on these plots indicateswhich students filled out the survey.1 S − L S − A n Q S − W S − O S − I N D S − C G S − S Q S − W G S − W C S − S P S − O G S − P r d S − T / Q Student Code F r a c t i on o f O b s e r v a t i on T i m e Student Activities−Peer Instruction (a) Student COPUS codes for one section of the PeerInstruction curriculum. I − P Q I − A n Q I − A d m I − W I − C Q I − O I − Le c I − R t W I − F up I − M G I − I − D / V Instructor Code F r a c t i on o f O b s e r v a t i on T i m e Instructor Activities−Peer Instruction (b) Instructor COPUS codes for one section of the PeerInstruction curriculum.
FIG. 17: COPUS profiles of one section of the Peer Instruction curriculum. Observations were twenty minutes of thefifty minute long section. l ll ll lll ll lllll lll l lll l ll ll l ll l llll ll l lll ll lll llll lll ll l ll lll ll l llll llll lll ll ll ll l l llll l ll l lll l ll ll ll l ll ll
Peer Instruction: Pre (a) Beginning of term social network for one section ofthe Peer Instruction curriculum. lll l lll ll lll l l llllll llll lll l ll ll lll lll ll l ll l ll l l ll ll ll l lll ll lll l llll lll l l ll llll lll l llll lll ll lll ll ll lll ll ll
Peer Instruction: Post (b) End of term social network for one section of thePeer Instruction curriculum.
FIG. 18: Pre and post-term social networks for Peer Instruction. The network survey was distributed at the lecturelevel. Small pairings of students are visible, likely due to think-pair-share types of clicker questions. We also see ahigh number of isolates, likely due to the open-seating arrangement that does not require students to interact withtheir peers. Additionally, we see long chains of students, likely due to the layout of the lecture hall. The colorshading on these plots indicates which students filled out the survey.2 S − L S − A n Q S − W S − O S − I N D S − C G S − S Q S − W G S − W C S − S P S − O G S − P r d S − T / Q Student Code F r a c t i on o f O b s e r v a t i on T i m e Student Activities−Context−Rich Problems: Disc. (a) Student COPUS codes for the discussion section ofthe Context-Rich Problems curriculum. I − P Q I − A n Q I − A d m I − W I − C Q I − O I − Le c I − R t W I − F up I − M G I − I − D / V Instructor Code F r a c t i on o f O b s e r v a t i on T i m e Instructor Activities−Context−Rich Problems: Disc. (b) Instructor COPUS codes for the discussion sectionof the Context-Rich Problems curriculum.
FIG. 19: COPUS profiles of one discussion section in the Context-Rich Problems curriculum. Observations were ofthe entire fifty minute sectionactivity began, the instructor moved around the room(MG) and frequently stopped for extended small groupdiscussions (1o1). Equipment sometimes needed trou-bleshooting (O), and papers were passed back to the stu-dents (Adm).The lecture section observations can be seen in Figure21. The students listened to the instructor (L), answereddirected questions (AnQ), and answered clicker questionsindividually and as pairs (CG, IND). Students also askedthe instructor questions with the whole class listening(SQ). When demos were presented, students were askedto predict the outcome (Prd) and solve a related context-rich problem in pairs (OG).The instructor would lecture and write on the board(Lec, RtW) to present new content, and to follow upquestions (Fup). Numerous clicker questions were posed(CQ), as well as non-clicker questions (PQ). The instruc-tor would use demonstrations to illustrate non-clickerContext-Rich Problems (D/V), and answered individualstudent questions (AnQ). Papers were also handed backduring group thinking time (Adm).
2. Network Data
Figure 22a shows the early-term network for one lec-ture/discussion section of the Context-Rich Problemscurriculum. There were a couple of small groupings,likely from lab or discussion, while the rest of the classwas lightly connected. Figure 22b shows the late-termnetwork, where students were much more heavily con- nected.
G. SCALE-UP
1. COPUS Observations
The week-long observation period allowed for threeclass sessions to be documented. The COPUS datashown in figures 23a and 23b was aggregated over allthree observation periods to represent a week’s worth ofclass time.Similar to other curricula already discussed, studentssolved problems in groups using a whiteboard (OG), asshown in figure 23a. However, this whiteboard was onthe perimeter of the room, so students physically got upand walked to the whiteboards. Students also spent timelistening to short lectures (L), answering clicker questionsboth alone and in their groups (IND and CG), and ask-ing questions to the instructor (SQ). When the studentgroups were called on during a whiteboard or clicker ac-tivity, it was marked as AnQ.The instructor codes shown in figure 23b hit all of thecategories available with COPUS. There was some timespent giving short lectures via powerpoint (Lec), posingindividual (PQ) and clicker questions (CQ), and follow-ing up those questions (Fup) with discussion and some-times whiteboard explanations (RTW). During problemsolving or clicker question time, the instructor movedaround the room (MG) and engaged in discussions withthe individual groups (1o1). During part of the lecture3 S − L S − A n Q S − W S − O S − I N D S − C G S − S Q S − W G S − W C S − S P S − O G S − P r d S − T / Q Student Code F r a c t i on o f O b s e r v a t i on T i m e Student Activities−Context−Rich Problems: Lab (a) Student COPUS codes for the lab section of theContext-Rich Problems curriculum. I − P Q I − A n Q I − A d m I − W I − C Q I − O I − Le c I − R t W I − F up I − M G I − I − D / V Instructor Code F r a c t i on o f O b s e r v a t i on T i m e Instructor Activities−Context−Rich Problems: Lab (b) Instructor COPUS codes for the Lab section of theContext-Rich Problems curriculum.
FIG. 20: COPUS profiles of the lab section in the Contex-Rich Problems curriculum. Observations were of theentire three hour period.period, a short PhET [33] simulation was shown (D/V).
2. Network Data
The structure of the room is evident in the social net-work graphs, seen in figures 24a and 24b. The clusteringof the students is indicative of the large table set-up ofthe classroom.
V. DISCUSSION
The goal of this project was to develop a vocabulary todescribe active learning pedagogies as individual entities.In this section, we will discuss how the network metricsvaried with curricula and noticeable trends. We also dis-cuss the overall trends within the COPUS profiles.
A. Network Analysis
Table III shows the calculated network metrics for thepresented curricula.Diameter is a metric that is limited by class size, sothe absolute value of the diameter should not be com-pared across curricula. However, the change in diam-eter from the beginning to the end of term for eachnetwork varied with curriculum. Most curricula had achange in diameter of ± S − L S − A n Q S − W S − O S − I N D S − C G S − S Q S − W G S − W C S − S P S − O G S − P r d S − T / Q Student Code F r a c t i on o f O b s e r v a t i on T i m e Student Activities−Context−Rich Problems: Lecture (a) Student COPUS codes for the lecture section of theContext-Rich Problems curriculum. I − P Q I − A n Q I − A d m I − W I − C Q I − O I − Le c I − R t W I − F up I − M G I − I − D / V Instructor Code F r a c t i on o f O b s e r v a t i on T i m e Instructor Activities−Context−Rich Problems: Lecture (b) Instructor COPUS codes for the discussion sectionof the Context-Rich Problems curriculum.
FIG. 21: COPUS profiles of the lecture section in the Context-Rich Problems curriculum. Observations were of theentire ninety minute section. lll lll ll l ll ll ll ll llll ll l l lll lll ll lll lll ll lll l
Context−Rich Problems: Pre (a) Beginning of term social network for one section ofthe Context-Rich Problems curriculum. ll ll lll lll l l l lll ll lll ll l ll ll l lll lll lll ll l llll
Context−Rich Problems: Post (b) End of term social network for one section of theContext-Rich Problems curriculum.
FIG. 22: Pre and post-term social networks for Context-Rich Problems. The network survey was distributed at thelecture/discussion level. Small groupings of students are visible in the pre network, likely due to discussion or labgroups. The post network is significantly more connected. The color shading on these plots indicates which studentsfilled out the survey.5 S − L S − A n Q S − W S − O S − I N D S − C G S − S Q S − W G S − W C S − S P S − O G S − P r d S − T / Q Student Code F r a c t i on o f O b s e r v a t i on T i m e Student Activities− SCALE−UP (a) Student COPUS codes for the SCALE-UPcurriculum. I − P Q I − A n Q I − A d m I − W I − C Q I − O I − Le c I − R t W I − F up I − M G I − I − D / V Instructor Code F r a c t i on o f O b s e r v a t i on T i m e Instructor Activities− SCALE−UP (b) Instructor COPUS codes for the SCALE-UPcurriculum.
FIG. 23: COPUS profiles for the SCALE-UP curriculum. All three days of instruction, each 75 minutes long, werecombined to create profiles representing a weeks worth of instruction. lll l lll ll l ll lll l llllll lll ll llll l lll ll ll ll lll lll llll ll l l ll l lll l lll l lll
SCALE−UP: Pre (a) Beginning of term social network for the SCALE-UPcurriculum. lll ll llllll ll l lll llll llllll ll lllll l lll llll l lll ll ll lll lll ll l l lll llll ll
SCALE−UP: Post (b) End of term social network for the SCALE-UPcurriculum.
FIG. 24: Pre and post-term social networks for SCALE-UP. The network survey was distributed at the section level.Larger groupings of students are visible, likely due to the large table dynamic within the SCALE-UP room. Thecolor shading on these plots indicates which students filled out the survey.6creased for all curricula except Peer Instruction, whichremained approximately constant. All curricula have afocus on collaborative group interactions, with the ex-ception of Peer Instruction, which instead promotes part-ner interactions. The spatial limitations of the Peer In-struction classroom may have also inhibited transitivitygrowth, as the ability for students to engage with morepeers than are present in their immediate vicinity washindered. As such, a stable level of transitivity through-out the term may be indicative of a Peer Instruction class-room.The giant component in all but ISLE:lab-only networksincreased at the end of the term, indicative of more con-nectivity throughout the entire class. As with diameter,the absolute value of the giant component depends onclass size, so change is more meaningful when discussingthis metric.While we did notice promising features during this de-scriptive analysis, it is unclear whether certain featuresarise due to the curriculum itself or the classroom layout.If we compare the post-network diagrams from ModelingInstruction (Fig. 16b) and SCALE-UP (Fig. 24b), wesee completely different structures despite similar class-room layout. Both Modeling Instruction and SCALE-UPhad students sitting at large tables and working collab-oratively. However, the Modeling Instruction social net-work developed into a tightly knit learning community,while the SCALE-UP social network retained distinctgroupings based on the students’ physical locations inthe room. This may be attributed to the addition of thewhole-class whiteboard meetings in Modeling, whereasSCALE-UP retains individual group identities duringdiscussions.While this project has showed promise for a method ofdescribing active learning curricula independent of lec-ture methods, there are some serious limitations thatneed to be addressed. For one, the need for studentsto respond to a survey to develop social networks intro-duces opportunity for data loss via poor response rates.In a few cases, the instructor provided time in class tofill out the survey. Response rates for sections that wereallotted class time to fill out the survey were significantlyhigher than those that did not. While network metricsare typically robust to missing data [34, 35] due to thereciprocity of ties between students, poor response ratecan render an incomplete picture.We combated low response rates in a few ways. Reci-procity of ties allowed us to include members of the classwho were underage or declined to participate in the sur-vey. This means that if a student did not fill out the sur-vey, or were later cut for being under 18, they could stillbe presented in the network if someone else named themas a meaningful interaction. Second, we used an undi-rected network, meaning that if person A named personB, the edge between person A and person B existed, evenif person B did not also name person A. Finally, datacuts only included students that filled out either the preor post-survey, or were named in the pre or post-survey.
B. COPUS
At this level of descriptive analysis, we see that the stu-dent COPUS profiles in Tutorials (Fig. 3a), ISLE recita-tion (Fig. 10a), and Context-Rich Problems discussion(Fig. 19a) are very similar. COPUS only has two codesto refer to student group activities; working in groups ona worksheet (WG) and other group activity (OG). Thecode WG was appropriate for Tutorials, ISLE recitation,and Context-Rich Problems discussion, rendering similarprofiles. However, all other group activities were indis-tinguishable within the OG category. ‘Other group activ-ity’ was coded for experiments, lab report creation, whiteboard collaboration, and non-clicker problem group dis-cussion. While two codes for small-group student collab-oration may be appropriate for a Peer Instruction setting,which COPUS was designed for, we lost the capability tomeaningfully distinguish curricula at the student-grouplevel.Similarly, the COPUS code ‘O- Other’ was markedfor numerous student and instructor activities. Forthe instructor, this code included instances of bathroombreaks, conversations with the TA or another instructor,organizing lab materials, eating a snack during an ex-tended teaching block, and troubleshooting equipment.For students, this code was used for clean-up, gatheringmaterials, miming unit-vectors, turning in homework, ortaking short breaks. The wide range of activities thatcan be classified as ‘O’ is troublesome, as instructionalactivities can sometimes fall into this category that islargely dominated by non-instruction behaviors.
VI. CONCLUSION
Despite losing resolution at the student-group level,and the broad brush of activities included as ‘other’, itis still possible that COPUS will be able to differentiatebetween curricula as a whole; further study using latentprofile analysis is underway [36]. However, while a valu-able tool for interactive lecture environments, COPUSfailed to distinguish student group activities. This couldsuggest that these pedagogies are not as distinct as welike to think, or that COPUS is an inadequate tool formaking these measurements.Network analysis illuminated possible distinguishingfeatures, such as a large decrease in network diameterwith Modeling Instruction, and static transitivity withPeer Instruction. Now that we see promise in using thismethod to characterize active learning environments, alarger scale study would be advised. Additionally, weplan to use exponential random graph modeling to deter-mine if these metric trends are coincidental or a featureof the pedagogy [37].Our goal with this project was to develop a vocabu-lary to discuss active learning curricula independently oflecture, which we have begun by using COPUS obser-vations and network analysis. More in-depth analysis of7both network metrics and COPUS profiles will be fea-tured in future papers.
VII. ACKNOWLEDGEMENTS
We would like to thank the institutions that partic-ipated in data collection and observation arrangements.We would also like to express our heart-felt thanks to theinstructors that allowed us to infiltrate their classroomsfor a week, with an additional thank you to those whoalso provided thorough and thoughtful feedback on thispaper. Lastly, we would like to thank the National Sci-ence Foundation (DUE 1711017 and 1712341) for theirgenerous support of this project. [1] S. Freeman, S. L. Eddy, M. McDonough, M. K. Smith,N. Okoroafor, H. Jordt, and M. Pat Wenderoth, Proc.Natl. Acad. Sci. USA , 8410 (2014).[2] R. R. Hake, Am. J. Phys. , 64 (1998).[3] S. J. Pollock and N. D. Finkelstein, Phys. Rev. ST Phys.Educ. Res. , 010110 (2008).[4] N. D. Finkelstein and S. J. Pollock, Phys. Rev. ST Phys.Educ. Res. , 010101 (2005).[5] C. Turpen and N. D. Finkelstein, Phys. Rev. ST Phys.Educ. Res. , 020101 (2009).[6] C. Henderson, Am. J. Phys. , 179 (2008).[7] A. K. Wood, R. K. Galloway, R. Donnelly, and J. Hardy,Phys. Rev. Phys. Educ. Res. , 010140 (2016).[8] E. A. West, C. A. Paul, D. Webb, and W. H. Potter,Phys. Rev. ST Phys. Educ. Res. , 010109 (2013).[9] M. K. Smith, F. H. M. Jones, S. L. Gilbert, and C. E.Wieman, CBE Life Sci. Educ. , 618 (2013).[10] J. Bruun and E. Brewe, Phys. Rev. ST Phys. Educ. Res. , 020109 (2013).[11] J. P. Zwolak, R. Dou, E. A. Williams, and E. Brewe,Phys. Rev. Phys. Educ. Res. , 010113 (2017).[12] D. E. Meltzer and R. K. Thornton, Am. J. Phys. , 478(2012).[13] D. Sawada, M. D. Piburn, E. Judson, J. Turley, K. Fal-coner, R. Benford, and I. Bloom, School Sci. Math. ,245 (2002).[14] M. T. Hora, A. Oleson, and J. J. Ferrare, Teaching Di-mensions Observation Protocol (TDOP) User’s Manual (2008).[15] E. Brewe, L. H. Kramer, G. E. OBrien, C. Singh,M. Sabella, and S. Rebello, AIP Conf. Proc , 85(2010).[16] R. Dou, E. Brewe, J. P. Zwolak, G. Potvin, E. A.Williams, and L. H. Kramer, Phys. Rev. Phys. Educ.Res. , 020124 (2016).[17] D. L. Vargas, A. M. Bridgeman, D. R. Schmidt, P. B.Kohl, B. R. Wilcox, and L. D. Carr, Phys. Rev. Phys.Educ. Res. , 020112 (2018).[18] L. C. McDermott and P. S. Shaffer, Tutorials in Introduc-tory Physics , 1st ed. (Prentice Hall, Upper Saddle River,NJ, 2002).[19] R. E. Scherr and A. Elby, AIP Conf. Proc. , 46 (2007).[20] E. Etkina and A. V. Heuvelen, in
Reviews in PER:Research-Based Reform of University Physics , Vol. 1,edited by E. F. Redish and P. J. Cooney (American As-sociation of Physics Teachers, College Park, MD, 2007).[21] E. Etkina, Am. J. Phys. , 669 (2015). [22] “PhysPort Methods and Materials: University ModelingInstruction,” .[23] M. Wells, D. Hestenes, and G. Swackhamer, Am. J.Phys. , 606 (1995).[24] E. Brewe, Am. J. Phys. , 1155 (2008).[25] E. Mazur, Peer Instruction: A User’s Manual , 1st ed.(Prentice Hall, Upper Saddle River, NJ, 1997).[26] C. Crouch, J. Watkins, A. Fagen, and E. Mazur, in
Reviews in PER: Research-Based Reform of UniversityPhysics , Vol. 1, edited by E. F. Redish and P. J. Cooney(American Association of Physics Teachers, College Park,MD, 2007).[27] “University of Minnesota Physics Education Researchand Development: Minnesota Model for Large Introduc-tory Courses,” .[28] A. Collins, J. S. Brown, and A. Holum, Am. Educ. ,6 (1991).[29] D. W. Johnson and R. Johnson, Cooperation and compe-tition: Theory and research (Interaction Book Company,Edina, MN, 1989).[30] P. Heller and M. Hollabaugh, Am. J. Phys. , 637(1992).[31] R. J. Beichner, J. M. Saul, D. S. Abbott, J. J. Morse,D. Deardorff, R. J. Allain, S. W. Bonham, M. H. Dancy,and J. S. Risley, in Reviews in PER: Research-Based Re-form of University Physics , Vol. 1, edited by E. F. Re-dish and P. J. Cooney (American Association of PhysicsTeachers, College Park, MD, 2007).[32] U.S. Department of Education. Institute of EducationSciences, National Center for Education Statistics, “Inte-grated Postsecondary Education Data System (IPEDS),”(2018).[33] K. Perkins, W. Adams, M. Dubson, N. Finkelstein,S. Reid, C. Wieman, and R. LeMaster, Phys. Teach. , 18 (2006).[34] J. A. Smith and J. Moody, Soc. Netw. , 652 (2013).[35] J. A. Smith, J. Moody, and J. H. Morgan, Soc. Netw. , 78 (2017).[36] M. Stains, J. Harshman, M. K. Barker, S. V. Chasteen,R. Cole, S. E. DeChenne-Peters, M. K. Eagan, J. M. Es-son, J. K. Knight, F. A. Laski, M. Levis-Fitzgerald, C. J.Lee, S. M. Lo, L. M. McDonnell, T. A. McKay, N. Mich-elotti, A. Musgrove, M. S. Palmer, K. M. Plank, T. M.Rodela, E. R. Sanders, N. G. Schimpf, P. M. Schulte,M. K. Smith, M. Stetzer, B. Van Valkenburgh, E. Vin-son, L. K. Weir, P. J. Wendel, L. B. Wheeler, and A. M.Young, Science , 1468 (2018). [37] G. Robins, P. Pattison, Y. Kalish, and D. Lusher, Soc.Netw.29