How Well Do Remote Labs Work? A Case Study at Princeton University
HHow Well do Remote Labs Work?A Case Study at Princeton University
Saumya Shivam and Kasey Wagoner Joseph Henry Laboratories of Physics, Jadwin Hall, Princeton University, Princeton, NJ,USA 08544August 12, 2020
The onset of the COVID-19 pandemic forced many universities to move to virtual instruction during thespring 2020 semester. The transition to remote learning was abrupt and overwhelming for teachers ofall subjects, all across the US. Nowhere was this more true than in science lab courses. The experiencenevertheless provides an opportunity to investigate the optimal design of remote labs, with similar learninggoals as in-person labs. In this study we explore the three most common approaches to remote labs: recordedexperiments, applet-based experiments, and at-home projects. We use surveys and interviews to make twocomparisons: remote labs vs. in-person labs; the different types of remote labs. Examining these twoquestions we find that remote labs perform as well as in-person labs and students learn the most from athome physics experiments while also enjoying those the most.We first describe the structure of the labs, and the different types of remote labs, then describe thesurveys and interviews we use to study the efficacy of such labs, and finally analyze the results from thesurveys.
This study was conducted in Physics 101 at Princeton University. In the spring of 2020 the course had 62students, 40% sophomores, 50% juniors, and 10% seniors. To first order, this course is a typical introductoryalgebra-based, course for non-majors. The pedagogical structure follows the Investigative Science LearningEnvironment (ISLE) model [1] and when on campus, the labs are typical ISLE labs [2]. The primary goal ofour labs is for students to develop an understanding of the process of experimental physics, not to reinforceconcepts.
Princeton University transitioned to remote instruction at the midway point of the spring 2020 semester.After the transition, students completed three labs: one recorded experiment, one project experiment, andone virtual experiment; each of these are described below. The primary motivation for assigning differenttypes of labs was to create a situation where we could assess the efficacy of the different approaches.The first remote lab that the students completed involved a set of recorded experiments. The experimentswere exactly what the students would have done, had they been on campus. The lab covers magneticinduction with “observation” and “testing” experiments which are typical of the ISLE pedagogy. Thelab consisted of three videos of instructors performing experiments. After watching the videos, the students1 a r X i v : . [ phy s i c s . e d - ph ] A ug orked through a worksheet in which they were supposed to use observations to develop a model of magneticinduction; to this point they had not encountered the magnetic induction in any part of the course.The second remote experiment was a three-week long, student-defined project. Students were given threedifferent project options. This was done to ensure that all students would be able to complete the project,regardless of their circumstances or ability to access tools (e.g. internet, equipment, etc.); each of thoseoptions is described below. For each of these options, students were given examples projects to learn fromand detailed rubrics that described exactly how their projects would be graded. The only restriction on thecontent of the project was that it should come from material covered that semester (E&M, optics, circuits,atomic physics, nuclear physics). • Virtual Experiment: For this type of project students were tasked with using a web applet to “exper-imentally” answer a question which they had defined themselves (e.g. PhET [3]; students were givena spreadsheet with links to a large number of applets). They were required to design an experimentalmethod and perform the measurements using the applet. They then used standard methods to analyzethe data and answer the question. • At Home Physics: For this type of project students were tasked with using materials available to themto experimentally answer a question which they had defined themselves. They were required design anexperimental method and perform the measurements. They then used standard methods to analyzethe data and answer the question. To aid them, they were provided with suggestions software thatmake use of the numerous sensors in their smart devices. • The students in the class were divided into three sections of ∼
20 students each. Within each section thestudents were recommended to form a group of three (occasionally two) at the beginning of the semester.During the first half of the semester, with in-person labs, two AIs (Assistant Instructors, who are typicallygraduate students in the Physics department) were present for each lab session. The AIs typically summarizedthe concepts and equipment function at the beginning of each lab, and assisted the students during the labas needed.After the transition to remote instruction, the nature of interactions between students and the AIschanged as well. For the recorded experiment, the AIs were available to answer questions about the labthrough email. For the project lab, each AI was assigned a section, with the idea that the same AI wouldguide a given lab group for the three weeks of the project. During the first week of the project the aim wasto help the students in choosing a well defined question that they would begin to answer. In this processthey would learn about the different types of applets available online, think about the equipment and setupthat could be used for a home experiment, and become familiar working with Algodoo. At the same time,the students were expected to relate their experiments with the content of the course covered up to thatpoint (including electrostatics, DC circuits, magnetic induction and optics). The second week was typicallyabout consolidating the experiment and answering questions about analysing any initial data, and the finalweek was supposed to help the students put everything together and resolve any errors or discrepancies. For2ach of these three weeks, the AIs scheduled a virtual meeting with each lab group, keeping track of theirprogress and providing guidance whenever required. In addition to the AIs, the instructor was also availablefor discussions during the office hours for the course. For the third remote experiment, a virtual experimenton radioactivity, each lab group was provided the option to sign up for a virtual meeting with an AI wherethey could ask questions or seek clarifications.Both before moving to remote instruction and after, each lab group was required to submit a reportsummarizing their experiment, with the students expected to follow the standard practices with aspects likecreating plots and calculating uncertainties in the result. This report was then graded by the AIs, usingISLE rubrics [1]. New rubrics were developed for the project experiment. These rubrics were inspired by theISLE rubrics, but were expanded to include topics such as creativity. The full set of rubrics for the projectlab can be seen in Appendix A.
In order to assess the efficacy of our off-campus labs, we employed a number of different instruments. Wegave students three different surveys and did a series of interviews.
Our students completed two different surveys from which we can draw information about the remote labs.The first was the E-CLASS survey developed by Zwickl et. al [5]. The second was an optional, anonymousone created by the course instructors to get feedback on the transition to remote instruction.
E-CLASS : The E-CLASS survey series assesses how students’ responses to a number of questions compareto the responses of experts. It is a pre/post series, so it can assess how students’ alignment with expertsshifted during the time between the pre and post surveys. Our students completed the series two differenttimes; it was the same pool of students taking both survey series. The series was first taken during the fallsemester, when all labs were in-person (69 students, 58% of students responded); the pre was taken beforeany labs, and the post was taken after all labs were complete. The series was taken the second time duringthe spring, when we transitioned to remote learning (43 students, 69% of students responded); the pre wastaken mid-semester before any remote labs, and the post was taken after all remote labs were complete. Thisapproach allowed us to compare E-CLASS results for in-person labs, to results for remote labs.
Instructor Created Survey : The second survey was and end-of-the-semester evaluation created by theinstructors specifically for getting anonymous feedback on the transition online (completed by 15 students,24% of full enrollment). In this survey there were a few lab-specific questions. Each of these surveys gaveus information to make comparisons: (i) in-person labs vs. remote labs; (ii) different types remote labs.
After the semester was complete, we conducted Zoom interviews with eight students (13% of full enrollment).During the interview, students were asked only questions specific to the labs. Appendix B has a full list ofquestions.The demographics of the interviewees closely followed those of the full course enrollment: 4 completedAt Home Physics (AHP) projects, 4 completed Virtual Experiment (VE) projects (full enrollment: 43.5%AHP, 50% VE projects, 6.5% Algodoo projects); course grades 5 As, 3 Bs (full enrollment 66% As, 32% Bs);3 women, 5 men (full enrollment: women 63%, men= 37%).3 .2 Comparing in-person labs to remote labs
To compare the in-person labs to the remote labs, we draw from all of the instruments. We use the E-CLASSsurvey first to get a general sense of the impact of switching to remote instruction for a part of the semester.The overall E-CLASS score was reasonably close to the typical score in such classes. A comparison of thescores, before and after the semester, is shown in Figure 1. In order to compare with an otherwise usualsemester with all in-person labs, we also show the overall score for the same cohort of students from fall2019. We see that despite the abrupt shift to remote instruction, the overall score for the spring semesterremained nearly the same. Figure 2 shows the change in physics interest after the semester. We see a similarbehaviour as fall, physics interest increased for a large number of students, more so than in a typical course.This increase is higher than typical classes at the same level. However, the proportion of the class for whichinterest increases is slightly smaller than in the fall. This could be indicative of the fact that the remotelabs had to be prepared on a short notice and the students might have taken some time to adjust to the newparadigm, yet not having an adverse impact on the overall scores.Figure 1: Shown is a comparison of how expert-like student responses were to all of the questions on theE-CLASS. The fall semester (completely in-person) is shown on the left, while the scores on the right arefor the spring semester, with remote instruction for part of the semester. Despite the sudden shift to remotelabs, the overall score doesn’t significantly decrease for the spring semester.Figure 2: E-CLASS results indicating the change in physics interest are shown. Students’ interest in physicsshows a significant increase in both the fall (left) and spring semesters (right). The interest increased slightlyless in spring than it did in the fall.To see the differences between in-person labs and remote labs for the spring semester more explicitly, weconsider the results from the instructor-created survey, where students weigh the two paradigms based on4nterest and instructional quality. Students were asked “Compared to in-person labs, I found the off-campuslabs to be as instructive” and “Compared to in-person labs, I found the off-campus labs to be as interesting”.Responses to both questions were given on a Likert scale. The scale of the first question ranged from 1 =“Far less instructive” to 5 = “Far more instructive”. The scale of the second question ranged from 1 = “Farless interesting” to 5 = “Far more interesting”. The results of both questions can be seen in Figure 3.Figure 3: Responses to an optional instructor created survey to assess how instructive and interesting theoff-campus labs were when compared to in-person labs. Student responses to the two questions asked onthe instructor-created survey. For both questions 1 corresponds to “Far less instructive/interesting” and 5corresponds to “Far more instructive/interesting”. The upper panel indicates that students found remotelabs to be roughly as instructive as in-person labs. The lower panel indicates that students found remotelabs to be slightly more interesting than in-person labs.The interviews give us some additional information for this comparison. Each student was asked “Com-pared to in-person labs, how ‘real’ would you say this project was?”. The responses indicate that studentsthink in-person labs feel real because of the professional equipment, but the process of the remote projectlab felt more real. Below are some particularly interesting responses to this question. • “In in-person labs students were very reliant on TAs. If they had a hurdle, they would just approachthe TAs. With the AHP project, they didn’t have the luxury of easily talking to TAs. It put greaterfocus on the student doing work independently, which is a goal of lab work.” • “The project was more scientific because it was so free form and students had to define the question.” • “In the lab things felt more real, because the equipment was more professional. But overall the Projectfelt like more scientific.” 5 “Being on campus, because it was more organized. Because there you were following a set of predefinedsteps. And there you have the right tools for the job, you don’t have to search for them.”Pulling all this information together, it appears that our remote labs were at least as effective as ourin-person labs. Additionally, it seems that students felt the remote project lab was better than in-personlabs at replicating the process of experimentation while also being slightly more interesting. The statistics ofour study are such that no strong conclusion can be drawn, but it does seem promising that our transitionto remote learning left us with labs that were at least as effective as our in-person labs. We feel this is asignificant result given the hasty nature in which we had to migrate to remote learning. Perhaps the most interesting thing to investigate in this study is how different types of remote labs compareto one another. For this comparison we use the instructor-created survey and the interviews, with the latterproviding significant information.The instructor-created survey asked two questions which directly compare the different types of remotelabs: “Of the three off-campus labs, which was the most instructive for you?”; “Of the three off-campus labs,which was the most interesting for you?”. The results of these two questions can be seen in Figure 4. Theresponses to these questions seem to indicate that students found the project lab and virtual experimentequally interesting, while finding the project lab more instructive. It is clear from the responses to bothquestions that few students found the recorded experiment to be the most instructive or interesting. Theseresults are supported by the responses to interview questions, which are described below.Figure 4: The left panel shows the remote lab which students found the most interesting. The right panelshows the remote lab which students found the most instructive.During the interview, students were asked a series of questions that directly compare the three remotelabs they completed. Before being asked the questions they were given the instructions “For all the questions,keep in mind the goals of the labs are to make the student think critically about how to do experiments,and to teach experimental and data analysis skills.” The goal of the questioning was to find out which labsstudents enjoyed the most and in which they thought they learned the most.The first relevant question was “What was your favorite of the three labs? Why? How do these compareto the on-campus labs?”. The responses indicate a roughly equal split between virtual and project labs. Thestudents that enjoyed the virtual experiment labs indicated this was because the were easier and provided aclearer understanding of the content. The students that enjoyed the project labs indicated this was becausethe process felt more like experimentation in a research lab. So while half of the students enjoyed thevirtual experiment, their reasoning didn’t align with the lab goals. Below are a few particularly interestingresponses. • “The virtual experiment was neat, and easy at a time when I needed it, but I didn’t feel like I learnedthat much. The project lab was the most interesting in the sense that I got to choose something I aminterested in, that was fun.” 6 “Definitely the project was the best of the three. It was far and away my favorite, even compared toon-campus. It felt more like labs I am accustomed to in research work in biology labs.” • “Project, virtual, recorded (most to least favorite). Compared to on campus, the virtual was goodbecause you could get a good understanding of the concepts by manipulating parameters. The projectshifted thinking toward what a real physicist would do, even more so than on-campus lab.” • “The virtual experiment was my favorite, because the way the questions flowed and were easy tounderstand. The way it was setup really helped me learn the concepts.”The second question relevant here was “‘In which lab do you think you learned the most? Why?”. Allbut one student indicated that they learned the most in the project lab, and that student’s response was“The virtual experiment. For the project we picked a topic we knew already, so I didn’t learn concepts. Inthe project I learned the most about experimentation.” The students that said they learned the most in theproject lab indicated that this was because it made them understand all details of the lab, and be involvedin every step. Below are a few particularly interesting responses. • “The project lab, it made me think about what I needed, why, and how. It also forced me to come upwith a question to test, and that was a new thing.” • “The project because I had to be involved in every step, and I really had to understand every singlething.” • “The project lab, there I learned mostly process, rather than content. It’s harder to learn content, buteasier to learn process. I think there is potential with recorded experiment, but if you don’t understandwhat’s going on, you totally miss everything.” • “The project because it allowed me to get a hands-on understanding that I couldn’t get from the othertwo.”The final question relevant to this discussion was “Given these goals, what type of labs would you assign?”For this 75% of the students said the project lab and the remaining 25% said a mix of the project and thevirtual experiment lab. Below are two particularly interesting responses. • “Watching videos wasn’t as helpful as physically manipulating stuff. The project lab was my favorite,and the most fun I have ever had doing an experiment.” • “A mix of the virtual experiment and at home physics. There’s a lot to learn from having to comeup with an experiment yourself. It forces you to learn a lot more. I had do a lot of reading up andlearning. Then I had to get my data, see what it says, and see how I would present it. The at homephysics was more rigorous. The virtual experiment was a bit easier, so a mix would be nice.”In addition to responses from these questions, a majority of the students interviewed (75%) indicatedthat that they found the process of the project lab the most interesting, saying things like “The AHP wassuper fun. I had to figure how I was going to do everything, and I did a lot of trouble shooting. It wasway more involved, which forced me to focus, and learn.” and “The project tasks were delegated and it wasinteresting to find a way to work as a team. And then the experimenter had to figure out everything ontheir own. It was time consuming, but the process of how to figure out how to optimize things was great. Itwas even more interesting than on-campus labs. It was very gratifying to see it finally work out.”As previously mentioned, students were given three options for the project lab. We were interested inknowing if students of different achievement levels would choose different types of projects. For instance,do the highly motivated (and presumably highly achieving) students chose the potentially more involved AtHome Physics project? Or do the students who are primarily interested in good grades choose the VirtualExperiment because it was perceived to be easier? To get at this, we investigated the student course gradesas a function of project choices. The results are shown in Figure 5. There doesn’t seem to be a significantcorrelation between project selection and course grade (note that students doing Algodoo projects are notdisplayed, as there were only four such students). 7igure 5: A comparison between project selection and course grade is shown. This indicates no significantcorrelation between high-achieving students and project selection. Note that only 4 students did Algodooprojects and the grades of those students are not displayed; 1 of these students received an A and 3 receiveda B. To summarize, we have used three different instruments to assess the efficacy of remote labs in the intro-ductory physics course at Princeton University (with 62 students), introduced halfway in the spring 2020semester due to the COVID-19 pandemic. The three different types of labs studied, besides the in-personlabs, were, a recorded experiment, a student defined three week project experiment, and an applet-basedvirtual experiment. We find that the overall learning attitude determined by the E-CLASS survey remainsalmost the same after the semester as before, corroborated by an instructor created survey, which revealsthat the remote labs were perceived more interesting than in-person labs on an average. The same surveyalso brings out the distinction between the three remote labs, indicating that the project experiment and thevirtual experiment generated more interest than the recorded experiments, while the project experimentswere considered the most instructive. Interviews with 8 representative students explored the origins of thesedifferences and a consensus emerged about project labs being perceived as an ideal experiment, whereinthe students learned the most, while the virtual experiment was popular due to its ease and a well definedstructure. These results are suggestive of an ideal remote lab combining the exploration of realistic physicsexperiments defined by the students themselves, to provide a creative and instructive approach, along withthe ease and structure of a virtual applet based experiment.
Based on this study, our fall labs will be a series of At Home Physics experiments. They will be scaffoldedto slowly introduce the various aspects of designing and executing an experiment. Additionally, given theirefficacy at teaching concepts, we plan to incorporate virtual experiments into the class portion of our course.A full semester dedicated to AHP labs will provide us a better environment to assess their efficacy. Moreover,we will be able to directly compare them to our in-person labs. Our assessment will be similar to that followedhere. Additionally we will implement the Physics Lab Inventory of Critical thinking [6] in order to get anindependent assessment of how well our labs do at making our students think critically.8 eferences [1] E. Etkina, “Millikan award lecture: Students of physics—listeners, observers, or collaborative participantsin physics scientific practices?,”
American Journal of Physics , vol. 83, p. 669, 2015.[2] K. Visnjic, C. Riihimaki, C. Sealfon, and E. Laffey, “ISLE-inspired design laboratory transformation atPrinceton University: Year two results,”
Physics Education Research Conference Proceedings , pp. 347–350, 2015.[3] C. Wieman, W. Adams, and K. Perkins, “PhET: Simulations that enhance learning,”
Science , vol. 322,pp. 682–683, 2008.[4] K. Wagoner, D. Flanagan, and K. Hynes, “Inquiry based labs for large-enrollment, active-learning intro-ductory physics course,”
The Physics Teacher , vol. In prep, 2015.[5] B. Zwickl, T. Hirokawa, N. Finkelstein, and H. Lewandowski, “Epistemology and expectations surveyabout experimental physics: Development and initial results,”
Physical Review Special Topics - PhysicsEducation Research , vol. 10, no. 1, 2014.[6] C. Walsh, K. N. Quinn, C. Wieman, and N. G. Holmes, “Quantifying critical thinking: Developmentand validation of the physics lab inventory of critical thinking,”
Phys. Rev. Phys. Educ. Res. , vol. 15,p. 010135, May 2019. 9
Rubrics
Interview Questions
Questions comparing three different types of labs. • For which of the three labs [Recorded, Project, Virtual] did you find the content the most interesting? • For which of the three labs [Recorded, Project, Virtual] did you find the process the most interesting? • Which of the labs [Recorded, Project, Virtual] felt the most natural to you? That is, which do youthink most closely matched your concept of a real physics experiment? • Which type of lab [Recorded, Project, Virtual] was the most challenging for you? Why? • What was your favorite of the three labs? Why? How do these compare to the usual labs? • In which lab do you think you learned the most? Why? Given these goals, what type of labs wouldyou assign?Questions specific to project lab. • Why did you choose that type? Did you explore the other two projects before deciding or you werecertain about the type from the start? Did the example projects for each type convey the essence ofthe process? • If you had all the resources needed to do any one - any measurement device or tools for At Home, anapp that could examine any phenomena you were interested in, and a hypothetical familiarity withAlgodoo - which one would you pick? • For At Home Physics – How did you decide on your experiment? – What obstacles did you encounter in doing the experiment? (e.g. defining a question, findingequipment, etc.) – How many members in your group could perform the experiment, depending on equipment avail-ability? – Compared to in-person labs, how “real” would you say this project was? • For Virtual Experiments – How did you decide on your experiment? Did you make the final decision based on the scope ofthe app or your interested topic from beforehand? – What obstacles did you encounter in doing the experiment? ––