Influences on Faculty Uptake from a Faculty Learning Community
aa r X i v : . [ phy s i c s . e d - ph ] O c t Influences on Faculty Uptake from a Faculty Learning Community
Lydia G. Bender
Department of Physics, Kansas State University,Manhattan, Kansas 66506
James T. Laverty
Department of Physics, Kansas State University,Manhattan, Kansas 66506 (Dated: October 21, 2020)Professional development is a tool that faculty members can use to become more knowledgeableabout certain fields of study, or to develop a wide variety of skills. One way that college faculty useprofessional development is to learn how to become better teachers. We investigate what influencesaffect the ways in which faculty take up ideas from professional development programs. By employingthe framework of Pedagogical Reasoning and Action, we investigate how faculty take up ideas froma particular Faculty Learning Community (the STEM Teaching & Learning Fellowship) and thefactors that influence their instructional and material design choices. Influences affecting facultywere examined in three different cases. From these cases, we constructed themes, and examinedthose themes across all cases using a cross-case analysis. In this multiple case study we find thatassessment and instructional alignment and the culture of the department and participation indepartmental practices correspond to the extent in which faculty bring new teaching ideas andpractices into the classroom. These findings can be leveraged to help influence the ways in whichdevelopers should design and improve programs as well as inform researchers on future avenues ofresearch.
I. INTRODUCTION
Recently, there has been a movement to improveSTEM education. Efforts that aim to better this fieldfocus on increasing retention rates, and preparing stu-dents for careers in STEM [1–3]. Teachers at the post-secondary level play a large role in increasing STEM liter-acy among their students. As new practices and pedago-gies are created to increase student outcomes, it becomestheir responsibility to learn and apply these techniques totheir classrooms. One way that faculty can develop newskills, techniques, and knowledge is through professionaldevelopment [4].This paper examines faculty members participating ina STEM Teaching & Learning Fellowship, henceforth re-ferred to as the Fellowship. The Fellowship focuses onbringing Three-Dimensional Learning into undergradu-ate STEM classrooms in order to provide students withmeaningful and transferable knowledge. During theirparticipation, Fellows are asked to design material fora unit of their class using ideas from the Fellowship. Thematerial that they create and the process in which thatoccurs allows us to investigate how faculty take up theirideas from the Fellowship and apply it to their class-rooms.The Fellowship is part of a larger project that in-volves bringing Three-Dimensional Learning to under-graduate physics classes. Other research that has beenconducted by this group has focused on assessment [5–7],instruction [8], course transformation [9], and principlesof Three-Dimensional Learning [10]. The Fellowship pro- vides us with the space to investigate different avenues ofresearch within the same context including research onthe program itself.There has been a lot of research done to help un-derstand professional development as a resource for fac-ulty learning. Research on professional development in-cludes the design of programs [11–13], program effective-ness [14–19] , and effects of programs on student out-comes [20, 21]. Review of the literature also reveals thatthere is a desire to conduct research about faculty dur-ing their participation in programs [22–24]. Findingsfrom such research would allow us to explore and improvedesign of professional development programs on facultylearning. Finding ways to help faculty practice new ped-agogies would also aid in the efforts to improve studentoutcomes. The Fellowship provides us with the opportu-nity to investigate faculty participation within the pro-gram.Our research views faculty as talented educators withgood ideas and the appropriate skills to implement them.Because we want faculty to carry out their ideas, it isessential to investigate the influences that affect their use.Moreover, by identifying influences that affect the waysin which faculty take up ideas from a program we areable to pinpoint features of program design that supportfaculty in implementing new ideas and practices.Through the course of this paper, we review the lit-erature pertaining to professional development, facultylearning communities, and faculty uptake in Section II.In Section III, we explore the structure and design ofthe STEM Teaching & Learning Fellowship before arriv-ing at our research questions in Section IV. In Section V,we describe how we employed Pedagogical Reasoning andAction [25] to investigate influences on the design choicesof faculty. Using case studies, we examine three facultymembers, Ron, Charlie, and Molly (pseudonyms), andlook for themes within each case as well as across thosecases with a cross-case analysis in Sections VI-VIII. Wefound that the alignment between assessment and in-struction and the departmental culture directly influencethe ways faculty design and implement classroom andinstructional materials as found in Sections IX & X.
II. LITERATURE REVIEWA. Professional Development
Professional Development is a popular method usedby instructors to develop new skills and techniques thatcan be applied to increase student learning in their class-rooms [4]. Professional development programs use dif-ferent formats such as workshops [14, 21, 26, 27], semi-nars [28, 29], mentoring programs [26, 27, 30], and fac-ulty learning communities [31] to learn about content andtechniques. The content that programs focus on can beanything from teaching practices to integrating technol-ogy into the classroom (e.g. [32–34]).Although professional development is a tool commonlyused by K-12 teachers, it is a less common practice forfaculty in higher education [28, 35, 36]. This is not to saythat higher-ed faculty do not participate in professionaldevelopment, but their participation is generally centeredaround their research interests [37]. Additionally, thereare other factors affecting participation in professionaldevelopment. Most faculty have not formally studiededucation and have not received support when adopt-ing new curriculum and teaching practices [14, 26, 38].Professional development is not seen as part of their jobdescription [35], and is subsequently seen as an inconve-nience [36], or a distraction from other activities facultynormally focus on, such as conducting research, submit-ting publications, and writing grants [39]. Finally, in aphysics department, dialogue about teaching is uncom-mon [40].
B. Approaches to Professional Development
Professional development programs are designed toevoke change and follow several different approaches.Broadly, professional development can be split into twodifferent categories, top-down and bottom-up. A top-down approach occurs when a facilitator comes to teacha group of faculty members about a set of specific in-formation over a short period of time [41]. A commonexample of this would be a workshop focused on deliver-ing a specific set of ideas to faculty in a way that allowsno time for reflection or little say from the faculty at-tending. Overall this approach has been found to be de- motivating [42] and inefficient because the nature of theapproach is imposed on teachers [41] and does not valueteacher’s contextual and professional knowledge [43, 44].A bottom-up approach is built from the needs of thefaculty [41] by allowing them to be involved in decisionsabout program content [42]. The change that comes frombottom-up strategies has the ability to not only changeparticipating faculty but can also spread throughout theinstitution to which they are involved in [45]. Bottom-up approaches have the ability to create this “second-order change” [46] by providing participants the meansto engage in collaboration [47] and support continuousprofessional development [48]. This approach allows par-ticipants to think about and discuss items such as theculture of their university. This addresses the needs forprofessional development to take the culture of the de-partment and university into account [49].Henderson, Beach, and Finkelstein categorized profes-sional development efforts further by conducting a metaanalysis of 191 journal articles that focused on promotingchange [24]. Review of these articles revealed four changecategories that can be distinguished along two axes. Oneaxis describes what the program intends to change (indi-viduals vs. environments and structures) while the otheraxis describes who has control over the purpose of theprogram (prescribed vs. emergent). Professional devel-opment that falls under “Individual” focuses on instruc-tors’ beliefs and behaviors [24], and professional develop-ment that belongs to “Environment” focuses on chang-ing environments and structures that influence instruc-tional choices [24]. The ”Prescribed” and ”Emergent”axis is similar to the top-down or bottom-up approachesabove. Programs considered “Prescribed” are led by oneor more individuals separate from the participants thatcome into the program to teach a predetermined set ofideas. “Emergent” programs do not begin with predeter-mined set of ideas. Instead the content and delivery aredriven by the participants’ needs.Henderson, Beach, and Finkelstein find that strategiesthat align with the change beliefs of instructors over anextended amount of time tend to be most successful [24].Programs that align with these beliefs take an asset-based approach to faculty learning and take a bottom-upapproach [50]. An asset-based view of faculty sees themas experts of their own local contexts, knowledge, val-ues, and tools [51]. It also celebrates and builds uponthe diverse group of faculty participants [52]. Theseapproaches also seek to leverage the ideas and skills offaculty members and can ultimately develop faculty tobecome change agents [46, 51]. One way that profes-sional development can take an asset-based approach isthrough the use of Faculty Learning Communities.
C. Faculty Learning Communities
Faculty Learning Communities (FLCs) are one type ofprofessional development that incorporates different lev-els of faculty participation and ownership [23, 53] throughthe use of a community of faculty who support eachother [54]. FLCs occur over extended amounts of time,where faculty and facilitators come together to investi-gate and discuss different teaching practices and con-cepts [31, 54, 55]. They also provide faculty with thespace and time to reflect on their teaching [55, 56]. FLCscan range from formal communities to more informalcommunities. For example, meeting a coworker for coffeeto talk about work would be considered an independentlearning community [57, 58].One common feature of FLCs is that participants of-ten have varying levels of teaching experience and comefrom different disciplinary backgrounds [31, 59]. Thesefeatures have the ability to instill confidence in youngercolleagues [55], and the interdisciplinary feature of FLCsprovide faculty with the opportunity to interact with newcolleagues and talk about similar problems and sharetheir different approaches to solving those problems [55].The FLC also gives them the platform to develop a com-munity of colleagues with whom they can discuss profes-sional and personal topics outside of the FLC [55].More formal FLCs are often led by facilitators. FLCsare focused on participants’ ideas and needs, the roleof the facilitator is to create a productive space for thecommunity, keep a focus on the big picture, and practiceorganizational skills [56]. The facilitator can also take onthe responsibility of training faculty to use a certain toolor resource [60].FLCs take many different forms in order to account forthe different types of faculty participating within them.Two common types of FLCs are Faculty Online LearningCommunities (FOLC) [61, 62] and University-AffiliatedFaculty Learning Communities (UFLC) [57]. FOLCs oc-cur online and can extend over multiple institutions, thisallows for a more narrow focus on professional develop-ment because faculty who are interested in a specific topiccan join the community virtually [61]. UFLCs are com-posed of faculty that all come from the same university,and tend to participate in a more structured way thanan independent learning community.
D. Faculty Uptake of Pedagogical Practices
Faculty use of practices and skills after participation inprofessional development programs has been the subjectof research. Some research has looked into how facultyreport on their use of materials [14, 63, 64], others look atinfluences of discontinuation of use after participation inprofessional development [39, 65–67], and others investi-gate faculty’s selection and use of new practices [68, 69].Henderson, Dancy, and Niewiadomska-Bugaj foundthat professional development does a good job of mak-ing faculty aware of current research-based instructionalstrategies (RBIS) [65]. This study also identified factorssuch as attending teaching related professional develop-ment, reading teaching journals, interest in RBIS, and gender having an effect on the continued use of RBIS[65]. Other studies have found that the lack of local sup-port [66, 67], student responses [67], and time [67, 70]have an effect on the continued use of new teaching prac-tices. Research of this nature has suggested professionaldevelopment programs should provide support and feed-back during implementation [65, 71], and spend timeaddressing situational barriers [39].Other studies have also looked at the different ways inwhich faculty decide to use and choose to continue withnew pedagogical practices [68, 69]. Zohrabi finds thatcollecting student feedback and intuition benefit facultyin their continued use of practices while departmentaland classroom practices, student engagement, and theuse of classroom materials are all factors that influencehow new practices are applied to their teaching [68].Overall, this study finds that faculty view the processof implementing pedagogical change as a positive experi-ence that becomes easier over time [68]. Another studyconducted by Turpen, Dancy, and Henderson find thatfactors such as using personal experience to gauge effec-tiveness and encouragement from their community anddepartment as influences on adoption and continued useof a new pedagogical practice [69].
III. STEM TEACHING & LEARNINGFELLOWSHIP
TABLE I. Description of each Cohort in the STEM Teach-ing & Learning Fellowship. Each Cohort consists of Fellowsfrom all four disciplines (Mathematics, Physics, Chemistry,and Biology)Cohort Year Participants Institutions1 2014 - 2016 9 Fellows Single site2 2016 - 2018 14 Fellows Single site3 2018 - 2020 20 Fellows Multiple sites4 2019 - 2021 12 Fellows Multiple sites
Our research focuses on faculty participating in theSTEM Teaching & Learning Fellowship. The Fellow-ship started at Michigan State University (MSU) in 2013,but has since expanded to include three more universi-ties: Florida International University (FIU), Grand Val-ley State University (GVSU), and Kansas State Univer-sity (KSU). Since 2013 the Fellowship has had four Co-horts of faculty, each with approximately 10-20 Fellows.The first two Cohorts included faculty only at MSU, andthe last two Cohorts consisted of faculty from all fourinstitutions. Each Cohort had faculty participate fromphysics, chemistry, biology, and math departments. Adescription of each Cohort and the participating facultycan be seen in Table I.The principles of the Fellowship fit closely with fac-ulty learning communities. Although the Fellowship isled by a group of Three-Dimensional Learning (3DL) ex-perts from all four institutions, the topics are selected byfaculty needs, and breakout discussions are led by the Fel-lows. A large portion of the Fellowship is focused on giv-ing Fellows the time and space to reflect on their teach-ing. Participating Fellows met virtually once a month forninety minutes over the course of two academic years, andthis amounts to approximately 16 meetings and 24 hoursof professional development.The Fellowship is centered around the goal of integrat-ing 3DL instruction into undergraduate STEM classes.3DL is the foundation of the Next Generation ScienceStandards [2] and is largely focused on integrating scien-tific practices, crosscutting concepts, and core ideas intothe classroom. Integration of these dimensions into theclassroom help create scientific literate citizens that pos-sess skills needed to start their careers [2]. The ideas of3DL are explored through readings, presentations, anddesign techniques.The Fellowship promotes Learning Goals Driven De-sign (LGDD) as the approach faculty should use to designtheir courses. The principles of LGDD are modeled afterWiggins & McTighe’s model of Backward Design [72].LGDD is a process of designing materials that focuseson three stages: articulating learning objectives, creatingthe material, and collecting feedback [73]. More specifi-cally, LGDD is used in the Fellowship as a tool to haveFellows integrate 3DL into their lessons by articulatingtheir desired 3DL learning outcomes, the evidence neededto reach those outcomes, and how the outcomes would beassessed.Discussions and activities presented in the Fellow-ship are centered around the idea that assessment driveschanges to instruction [74]. In order to change assessmentand align that to instruction, the LGDD model is pre-sented by facilitators and discussed by Fellows in meet-ings and in the forum. As part of learning about LGDD,a significant amount of time and attention is given tocreating learning objectives and assessment that alignwith the 3DL framework. Several meetings are spentdiscussing how to articulate what it is that instructorswant their students to be able to do, and how those objec-tives would be assessed. After creating 3DL learning out-comes, time is spent creating assessment items that canproduce evidence of achieved learning goals, and instruc-tional materials that align with those assessment items.More specifically time is spent reflecting on their own as-sessment practices and utilizing the Three-DimensionalLearning Assessment Protocol [7] to align exam problemswith 3DL.During the Fellowship, Fellows are asked to partici-pate in discussions, activities, and reflections. One of thelargest and non-trivial tasks that is given to Fellows asksthem to create a 3DL unit for their classrooms using theprinciples of LGDD. The scope of the 3DL unit is leftfor the Fellows to decide, but the unit contained learningobjectives, classroom activities/materials, and an assess-ment item. The application and design of the 3DL unitserves as the focus of our research.
IV. RESEARCH QUESTIONS
In order to investigate the influences on how facultytake up ideas from a FLC, we focus on the ways in whichFellows develop and implement their 3DL unit. In partic-ular, we are interested in the following research questions:1. What influences do Fellows encounter outside of theFellowship that affect material design?2. What influences affect Fellows’ plans for the con-tinued use of 3DL?3. How does the FLC support faculty in their adoptionand plan for the continued use of 3DL?Exploration of these questions allows us to look at com-mon mechanisms that influence faculty who are involvedin different departmental, institutional, and social con-texts. Identification of these mechanisms allow us to lookat design features of FLCs, and the ways in which theycan be used to support faculty.
V. THEORETICAL FRAMEWORK
For this study we used Pedagogical Reasoning and Ac-tion (PR&A) to investigate how faculty approach the de-sign of their material. PR&A is a framework proposedby Shulman to explain a teacher’s development of class-room material and instruction from their content knowl-edge [75].PR&A is particularly useful for faculty discussing theirthoughts behind design decisions. Shulman (1987) states,“The following conception of pedagogical reasoning andaction is taken from the point of view of the teacher,who is presented with the challenge of taking what heor she understands and making it ready for effective in-struction” [25]. Stroupe takes on the following definitionof PR&A, “the purposeful coordination of ideas, infor-mation, and values about subject matter, curriculum,learners, and instructional context to plan for, enact, andreflect on instructional practice” [76]. For our researchwe also take on this definition as it will help us examinewhat influences affect the design choices of our fellow’s3DL unit. Research that has used this framework inves-tigates the choices of beginning teachers [76], the signif-icance of content knowledge [77], and uses/or choices oftechnology [78–82].PR&A places significance on the process of thinkingabout instruction and not solely on the observable actsof teaching [25]. Thus, we can explore material designfrom the point of view of a faculty member, focusing ontheir talents, ideas, and practices. Furthermore, PR&Aallows us to examine their reasoning as an educator andas a subject matter specialist [83].PR&A occurs in five stages and ends with faculty cre-ating new comprehension from their experience creatingand applying their material in the classroom [25]. Thesestages do not occur in any order, and include compre-hension, transformation, instruction, evaluation, and re-flection [25].
Comprehension involves faculty thinking about the setof ideas they want to teach, and how those ideas connectto the educational purposes of the class [25]. Comprehen-sion is attained “when teachers understand what they aregoing to teach [82].” As experts in their fields our facultyare knowledgeable in physics content; therefore, in ourresearch we focused on their comprehension of 3DL, andthe ways that it intersects with the educational purposesof their class. The information that is learned from theirexperience implementing the material forms their
NewComprehension , and then becomes part of their compre-hension base as they move forward creating new materi-als.In this study, the
Transformation stage broadly focuseson how faculty turn their 3D design ideas into instruc-tional material. This occurs in five substages: prepara-tion, representation, instructional selections, adaptation,and tailoring [25]. These five substages broadly focus onselecting what is to be taught, representation of ideas,making sure materials are in an instructional format, andmodifying ideas so that they are suitable for the studentsin their classroom. Ultimately this “result[s] in a plan, orset of strategies, to present a lesson, unit, or course” [25].
Evaluation is centered around ways faculty choose toevaluate student understanding inside of the classroomas well as formally testing their understanding. Inside ofthe classroom, faculty may choose to use clicker questionsor reflections to test student understanding of materialduring class time whereas formal evaluation may look likeend of unit exams.The next stage of PR&A is
Instruction ; this stage fo-cuses on the observable acts of instruction [25]. Exam-ples of these observable acts would be classroom man-agement, presentation of material, and interactions withstudents [82]. In our research, video data of the 3DL unitwas not available; therefore, we were unable to observeinstruction of fellow’s 3DL unit. In lieu of video data, fac-ulty were asked to recall and describe their instruction ofthe unit as part of an interview.The
Reflection stage is focused on but is not limitedto recalling impressions and feelings of their teaching andoverall experience of their material [82].In our study wehave Fellows reflect on how they felt their 3DL unit went,and what they learned from implementation.
VI. METHODOLOGY
In this paper, we use a case study approach, whichallows us to deeply explore a phenomenon within itscontext [84]. This is typically done through the use ofmultiple sources of data in order to produce case-basedthemes [84]. For our study, a case is confined to the ex-perience of a fellow participating in the Fellowship. Byexploring each case, we are able to gain a thorough in- sight into their participation in the Fellowship, designof their 3DL unit, the Fellowship itself, and the waysin which these aspects influence their PR&A and hencetheir design choices.In order to explore the different ways in which fac-ulty take up ideas from the Fellowship we selected threedifferent cases. These cases are then cross analyzed tocomplete a multiple case study. A multiple case studygives us the opportunity to explore reoccurring ideas andpatterns in more than one fellow’s experience designingmaterial. Moreover, comparing multiple cases and theircase-based themes together allows us to explore both thesimilarities and differences in 3DL material design andthe underlying mechanisms that influence those choices.
A. Case Selection
We chose to explore the experiences of three Fellows:Ron, Charlie, and Molly. We chose these cases basedon our informal understanding of their attitudes toward3DL and the Fellowship. Our cases encompass a rangeof attitudes. Our first fellow Ron is excited about 3DL,our second fellow Charlie feels skeptical about it, andour final fellow Molly is familiar with 3DL as she alreadyuses principles of 3DL in her teaching. Our cases arealso composed of faculty with different levels of experi-ence teaching and teach in different classroom formats.Finally, these cases also use a range of different toolsand resources that affect the way that they participatein different categories of PR&A. A more comprehensivebackground of each of the Fellows can be found in Ta-ble II.
B. Data Collection
In order to gain a deep understanding of our Fel-lows, we collected data from interviews, homework as-signments, Fellowship meetings, and forum discussions.The interview was semi-structured, lasted for approxi-mately 45 minutes, and occurred either online or in per-son. Ron and Charlie were interviewed after their firstyear in the Fellowship and Molly was interviewed oneyear after she completed the Fellowship. The interviewwas focused on the design of 3D material and was con-structed to specifically address all five stages of PR&A.The interview provided Fellows with an opportunity tovocalize their understanding and thoughts about 3DL,their experience in the Fellowship, and the choices theymake when designing material. In order to anchor theconversation around their 3DL material, we used stimu-lated recall techniques [85] by introducing the materialthey designed during the Fellowship into the interview.The homework assignments that were collected wereposted to a faculty forum and included an end of theyear survey, reading reflections, their 3DL unit, and otherprompts that Fellows were asked to respond to as part of
TABLE II. A summary of the three selected casesCohort Academic Level Classroom Format Attitude Towards 3DLRon Later cohort Pre-Tenure Lecture Excited about 3DLCharlie Later cohort Post-Tenure Lecture, Lab, Recitation Skeptical about the use of 3DLMolly Earlier cohort Pre-Tenure Studio Already uses principles of 3DL the Fellowship. Earlier cohorts, like the one that Mollyparticipated in, did not participate in forum discussionsor end of year surveys, but they did submit their 3DLmaterial. For later cohorts, the end of the year surveyfocused on each fellow’s personal goals and accomplish-ments. It asked them to talk about how the Fellowshipsupported their use of 3DL, and the supports that theyneeded. As well as collecting assignments, the discussionsthat occurred in the comment section of the forum werealso collected.The meeting recordings that were collected were fromdisciplinary Fellowship meetings from the later cohorts.Fellows that attended these meetings were from differentinstitutions but belonged to the same STEM discipline.These meetings focused on each Fellows 3DL unit andserved as a way for Fellows to share ideas and receivefeedback in a smaller, more intentional way.
C. Data Analysis
Data analysis occurred in three separate stages: sort-ing and coding, single case analysis, and cross case anal-ysis.We pulled every response and comment that Fellowsmade within the data set to be further analyzed. Theneach quote was read and categorized into the stage ofPR&A that they belong. Quotes that did not belong toa stage of PR&A were not thrown out but were insteadused to provide more details about the case (e.g. mo-tivation to join the Fellowship). From there we lookedfor repeated patterns of ideas to construct themes withineach case, and then for themes across all three cases.Themes across a singular case were constructed by fo-cusing on each stage of PR&A and the ways in whichthese stages relate to their 3D unit. During the analy-sis of each category we paid close attention to each Fel-low’s ideas of 3DL, the structure and management oftheir class, features of their context that Fellows noteas helping or hindering them, and how these influencesconnect to the design of the Fellowship. After each stageof PR&A was considered individually, the entire set ofstages were examined to find influences that effected mul-tiple parts of a fellow’s PR&A. Influences that were reoc-curring across several stages of PR&A were used to formour themes for each case.In order to construct themes across cases. We lookedfor similar experiences and instances that the Fellows hadand how this affected their material design, instruction,and continued use of 3DL. This allowed us to identify and explore the underlying mechanisms influencing materialdesign.
VII. CASES
Our research used various sources of data, in order todistinguish one data source from the other the presenteddata will end with [I] to signify an interview, [M] for aFellowship/disciplinary meeting, and [F] for forum dis-cussion and homework assignments.Our data analysis revealed three themes that were in-fluencing all three of our Fellows. These themes includeMotivation/Ability to change assessment practices, ma-terial placement and participation in instruction, andengagement in social interactions. These themes arepresent in multiple places of our Fellow’s PR&A, andtheir location can be found in Table III.
A. Ron
Ron is a pre-tenure Physics professor teaching alecture-only introductory physics class. Ron joinedthe Fellowship to “meet like-minded people” and tolearn how to prepare lectures using “evidence-based ap-proaches.” Throughout his experience Ron is very opento discussing and trying new ideas.Ron created his 3DL unit with two other Fellows, Roseand Angelina. Ron’s idea for the 3DL unit was to createan interactive lecture activity that would focus on thephotoelectric effect. In order to do this, Ron wanted toutilize clicker questions and a PhET simulation [86].Due to his participation in the Fellowship, Ron hasthought about how he can integrate 3DL into his ad-vanced lab class that he co-teaches. His ideas for hisadvanced lab course involve finding places in lecture andlab reports to connect to 3DL.Looking throughout Ron’s stages of PR&A we see sev-eral factors influence Ron’s design choices. Ron felt con-strained by a course management system, and it lim-ited his choice of assessment format and how he wasable to test his students’ understanding. Ron also men-tioned the interactions that he has with colleagues aboutteaching ideas and practices numerous times. All ofthese constraints present themselves in the Transforma-tion, Instruction, Evaluation, and Reflection stages ofRon’s PR&A, and contributed to the themes listed inTable /retab:themes. Throughout his experience Ron
TABLE III. A summary of the three themes present in the cases, and where they present themselves in each Fellow’s PR&A.Here T stands for the Transformation stage, E is for evaluation, I is for the instruction stage, and R is for Reflection.Themes Ron Charlie MollyMotivation/Ability to change assessment practices T, E, R T, E, R T, EMaterial placement and participation in instruction T, I, R T, I, R T, ISocial interactions T, E, R T, E, R T, E, remained optimistic about bringing 3DL into his class-rooms.
1. Comprehension
When it comes to the ideas of 3DL and using it inthe classroom Ron felt positive that 3DL aligns with hiseducational goals for his class. When asked about hislearning goals for his class, Ron responded, “What I’veinternalized as my goal for the class is to help them learnproblem solving skills [I].” More specifically Ron wantedhis “students to do [every scientific practice] in the class-room, [. . . ] it doesn’t matter if they are K through 12 orcollege age or 50 physics majors or not physics majors.These are the practices that I’d like to see my studentsdo [I].” These learning goals also extend to students in hisadvanced lab course. When talking about his advancedlab course Ron states, “I mean, there’s just so much of3D that you should be doing [I].” Overall, Ron believesthe principles of 3DL, specifically how it teaches scien-tific practices, aligns well with what he wants studentsto learn.
2. Transformation
Ron’s process of material design is defined by his expe-rience using already constructed classroom material, theuse of clicker questions, the limited availability of timeand content, and the different types of interactions thathe has with colleagues inside and outside of the Fellow-ship.Ron’s process of material design was influenced by theinteractions with other faculty at the beginning of hiscareer. This influences the way that Ron approached de-sign of his 3DL unit. As a new faculty member Ron wasgiven the following strategy: “as junior faculty, I wasjust given all this course material. And so the advice Iwas given was just to go with what we have [I].” Whenit comes to the 3DL unit, Ron starts by “think[ing] ofquestions that were in the vein of some of the exampleswe had seen earlier in the Fellowship [I].” From thereRon looked at the questions “And then we’re trying tothink of, you know, what sort of category [of 3DL] does[the material] fit into [I]?” In order to help him decidewhat category of 3DL that material fit into, Ron createda “cheat sheet with all the cross-cutting ideas and prac-tices and stuff[I].” These sheets allow Ron to ask himself, “does the you know, does this question match one of thesethings? And if not, how can we restate this question soit actually follows 3DL [I]?” Another way that Ron re-fined his material was by thinking about the “commonmisconceptions about the photoelectric effect [I].”Ron felt stuck designing 3DL material for a lecture, butidentifies clicker questions as one way he felt he couldintegrate 3DL into his classroom. The class that Ronteaches is a large lecture setting, and this type of formatis challenging for Ron. In an end of the year survey Ronwrites, “[Developing material that gets students to en-gage in 3DL] was challenging because of the “pull” of thetraditional lecture style is surprisingly hard to resist [F].”Although the “pull” of the lecture is hard to resist Roncreates a class activity for his 3DL unit. One way Ronfelt like he could accomplish this is through the use ofclicker questions. In the interview Ron says, “there areways that you can design clicker questions better, thatallow them to actually use those scientific practices, evenin answering a clicker question. So that’s a place whereI have more freedom in terms of how I run my class [I].”When it comes to content, Ron notices that in orderto integrate 3DL into his classroom material, time andthe amount of content is going to be an issue. Ron liststhree barriers to implementing crosscutting concepts andscientific practices, “Cutting content/lectures would benecessary. Project based two-way discussions would begreat with adequate time for feedback. Material for thestudents to review before coming to lecture [sic] [F].”The content that he chose for his 3DL unit was basedon timing and other available resources. For his 3DL unitRon picked the subject of the photoelectric effect becausehe was “trying to squeeze something into the semester [I]”although he notes the topic as “rich [M]” and “con-nect[ing] to a lot of other things [M].” He also notes theavailability of a PhET photoelectric effect app [86] as apositive, “there was this applet that existed to work withit, so we could actually do something a little bit more in-teractive in class [I].”A lot of the ideas Ron used to create this materialcame from interaction with other Fellows, these inter-actions are very different from the interactions that Ronhas with faculty who do not participate in the Fellowship.When talking about interactions with other Fellows Rondescribes it as being “very helpful because it’s certainlyopened my eyes to what could be done in the large lec-ture room [I].” This is contrasted by the way Ron talksabout interactions with non-Fellows, Ron describes thesefaculty as tending “to have a more old school perspec-tive on the teaching is just, there’s some lecture slides,just give a good performance, the students are going todo how they do and this is really out of your control.And this is kind of the feedback that I get outside theFellowship, to be honest [I].” These types of interactionshave stopped Ron from talking about his ideas to changefeatures of his advanced lab course that he co-teaches.When asked about talking to his co-instructor about inte-grating his ideas of 3DL into the advanced lab classroomRon responds, “Uh, no, I probably should . . . I shouldgive him, I should give the benefit of the doubt. . . [I]”Overall, Ron feels supported by his colleagues in the Fel-lowship, but having a hard time finding support fromother colleagues outside of the Fellowship.Overall, we see that the design of Ron’s materials reliesheavily on tools that are readily available or commonlyused within the department. Ron’s sees his colleaguesinside the department that also participate in the Fel-lowship as positive additions to the design process whilehe sees colleagues that are not participants in the Fellow-ship as more skeptical about Ron’s new ideas. Ron viewstime and content as limiting factors to designing his 3DLunit.
3. Evaluation
Ron’s approach to formative evaluation involves clickerquestions. When it comes to his formal assessment Ronfeels confined to multiple choice exams, and although hefeels positively about integrating 3DL into his exams,other factors such as a commonly used course manage-ment system and interactions with colleagues create dif-ficulties for Ron.During class Ron utilizes clicker questions as a way toevaluate his students; although his use of the clicker ques-tions has evolved. Normally the clicker questions wereused as “a review of the most recent lecture slide [F]”where “most students get it right and collect their bonuspoints for being in that class [F].” Ron wants to moveaway from using clicker questions for participation pointsand move towards using them as an aid to student learn-ing and understanding. Ron moves on from his previousstatements to say, “I would much rather use it as a wayto test their outside class study as well as to set up thenext set of lecture slides so that they are motivated tofind out the answer [F].”When it comes to the structure of his formal exams,Ron feels limited to using a multiple choice format. Hisformal exams are typically multiple-choice questions and“similar to the homework [I].” Ron wants to move awayfrom the way that exams are typically done, but strugglesto do so. One reason for this is because “we use [a coursemanagement system]. And it’s really, really well-suited,well-designed for multiple choice problems [I]” this makesRon feel restricted to using multiple choice formats. An-other factor is that Ron does not get the impression thathe could make 3D questions to put on his exam. In his interview he brings up, “At least from going throughthe Fellowship so far, I didn’t get the impression thatit was. . . It was easy to have something that you couldjust have like four or five multiple choice questions thatyou could have as a 3D assessment [I].” Ron even notesneeding help creating 3D material in the end of the yearsurvey, “I would like to learn how to write 3DL assess-ments within a [course management system] framework.This will help me design new exam questions [F].”The way Ron feels about 3D exams also changes de-pending on who he is talking to. In particular, Ron is“very comfortable [giving a 3D exam] [I]” if he was talk-ing to other Fellows, but Ron knows that his colleaguesoutside of the Fellowship “would recommend against it.”Ron’s issues with 3D exams has nothing to do with hisstudents’ capabilities but more with its alignment in thecourse. When asked about his students, Ron notes, “Interms of my students I think it would be great for themto do it and I think they would rise to the occasion [I].”Lastly, Ron also thinks about how his exams align withinstruction, and the fact that they don’t currently aligncauses Ron to doubt the effectiveness of bringing 3DLinto his classroom. Ron brings up in the interview, “it’sall fine and good if you have examples in class and if yougive the homework like that, but if it’s not, if its alsonot an exam question, then the students aren’t going toreally invest themselves in learning how to do those kindsof problems [I].”Ron wants to change some aspects of his assessment,but the structure of the class, resources used by the de-partment, and interactions with others factor into theways in which Ron thinks about his evaluation.
4. Instruction
Students were presented with “three activities [M]” fol-lowed by “discussion questions [M]” these were done “inclass mostly on by [sic] themselves and if they had anyquestions they could ask [M].” Before the third activitythere was “a small lecture component [M]” where Ronintroduced the work function. Overall, the three activ-ities focused on the “current and wavelength relation-ship [M]”, “threshold and target material”, and “us[ing]the conservation of energy to make sense of the workfunction [M].”For the instruction of the 3DL unit, Ron deviated fromhis typical instruction. During the instruction of Ron’s3DL unit he was not the only instructor. Ron explainsthe instruction of the unit as the following: “when wedid this in the lecture room, [my colleagues] and I wereall [...] there, plus the TA for the class, plus the threeLAs for the class [I].” The role of the instructors for thisunit focused mostly on classroom organization and groupfacilitation by “walking around the outside [I]” to checkon students. This deviates greatly from Ron’s typicalapproach which is “just straight lecturing [I].” In orderto organize the class, instructors “gave [students] a headsup [M]” and “organize[d] the students that didn’t havepartners [M].” Overall, due to the format of the 3DLactivity Ron’s instruction had to change from lecturingto group facilitation.
5. Reflection
After reflecting upon how the implementation of the3DL unit went, Ron notes how there was a “palpabledifference [F]” in the room. In his interview Ron states,“I know that [feeling] shouldn’t be the gauge [. . . ] butputting that aside, they certainly seem to enjoy it moreand we enjoyed it as well [I].”Ron was also able to reflect on certain aspects of theunit that he would like to change for the future. In par-ticular, Ron felt that he needs to keep participating inthese types of instructional activities. Upon reflectionRon explains his thoughts, “This was just one opportu-nity for the students to engage with material in this wayand then they didn’t see it in this format again on thehomework or the exam and I think it would really driveit home [I].”All in all, Ron enjoyed how the unit went and believesthat students enjoyed themselves also, but he believes heneeds to integrate more material into the classroom forthis to become more useful for students.
B. Charlie
Charlie is a tenured physics faculty member teachingan intro physics class for non-physics science majors. Ev-ery couple of semesters Charlie tries to find new ways hecan improve his teaching and this served as his motiva-tion to join the Fellowship. Throughout the Fellowship,Charlie participates in conversations and brings a lot ofdiscussions to the table.Charlie’s idea for his 3DL unit was to create a unitthat could also be used in a class his colleague teaches.This led him to creating a unit that focuses on the idea ofelectrostatic potentials by using the core idea of energyconservation. Charlie also chose to challenge himself bychoosing a scientific practice that was not “using mathe-matics and computational thinking.”By examining Charlie’s PR&A, we find that Charliethinks a lot about his alignment and instructional materi-als, his current assessment practices, and the time he hasavailable. All of these factors contribute to the themesthat we have found influencing all of our Fellows. In Ta-ble III we can find where these themes present themselvesin Charlie’s case.
1. Comprehension
Charlie interprets 3DL as a way for students to learnscience while simultaneously teaching them how to be a scientist. These principles align closely to the educationalgoals Charlie has for his classroom, but he still holds onto some reservations of 3DL.Charlie likes 3DL because it teaches students how tobe scientists while also teaching the content of the course.Charlie describes this as, “three dimensional learningis this goal that science classes should, you know, oneteach the skills necessary to be a scientist and two, youknow, weave in these concepts to connect the variousfields of science together, while at the same time teach-ing this specific branch of physics that is on the title ofthe class [I].”In his class, Charlie has multiple educational goals thatalign with 3DL principles. First, Charlie wants studentsto “try and you know, make that connection between thephysics that is the, the, you know, the primary goal ofthat class, and the scientific field that most of my stu-dents are interested in [I].” He would also like his studentsto “have a feel for, you know, a little bit of a physical in-tuition about the world. [I]” He also wants his studentsto be able to synthesize information from different ar-eas and bring them together to solve a problem. Charliestates, “it’s a physics class and so we do you know, there’sa lot of problems worked out and so that I think thoseare good for, you know, being able to solve problems bysynthesizing information in different areas. In this caseinformation with different areas would be, you know, tak-ing multiple equations that grab different concepts andputting them together [I]” Overall, we see that Charlie’seducational goals of his class is to get students to be ableto connect physics to their primary field of study as wellas real world scenarios that they encounter.Although Charlie does like some ideas of 3DL he alsohas his reservations. His reservation is about “how muchthese proposals [of cross-cutting concepts and scientificpractices] diverged from the teaching that actually oc-curs in the classroom [F].” This has Charlie thinking thatthere are two different questions we could be answering inthe classroom. Charlie writes, “I believe the answer liesin what we want to accomplish in our curriculum. Arewe trying to create a scientifically literate population,or one that is able to think like scientists [F]?” He alsobelieves the questions could be answered on a “course-by-course [F]” basis. Overall, Charlie’s reservations about3DL revolve around the purpose of the class.In summary, Charlie feels 3DL aligns with the edu-cational goals of connecting physics to other fields ofstudy as well as many real world scenarios, but he strug-gles with deciding whether his introductory physics classshould strive to create a scientifically literate populationor create students that are able to think like scientists.
2. Transformation
When it comes to designing new material Charlie isused to designing material from scratch. When creatinghis 3DL unit, Charlie was focused on making it a collab-0orative experience that his colleague could also adapt forhis class, challenging himself to use scientific practicesthat aren’t commonly used in 3DL, and selecting where3DL material should appear within the structure of hisclass.Charlie has always created his own material from thebeginning of his teaching career, and although this tookeffort he wanted to make sure the material was targetingwhat he wanted to target. Since then Charlie has triedadapting problems to fit into his class. This resulted inhis class content changing slightly every year. Upon re-flection, Charlie states that “other help wasn’t offered.And and I guess, you know, now I would, I would askfor it [I].” He then would adapt the material “one slideper year [I]” until lectures are where he wants them tobe. Some problems that Charlie felt like he had to (orwould have to) write from scratch because other profes-sors put “more emphasis on the problem solving and lesson the concepts [I].” The material that he borrows fromonline learning websites usually do not get students toengage with the content that Charlie wants them to. Inthe interview Charlie says, “I’ve gone into some of theonline homework systems to steal multiple choice ques-tions. And by and large, I don’t like those, [. . . ] thoseones that have been that were borrowed or stolen or mod-ified tend to rank low on the ones that I like [I].” Sincecreating his initial body of work, each one of Charlie’s“lecture[s] [improve] by one slide per year. And after fiveyears that’s a lot that adds up [I].”A lot of the design decisions that went into creatinghis 3DL unit came from working with another fellow.While creating his 3DL unit Charlie picked the contentof the unit so it could be a collective effort with anotherFellow. Charlie describes this process as, “so we figuredthat these are very similar concepts. And so we could puttogether a teachable unit that can be used with minormodifications in both, both contexts [F].” Other dimen-sions of 3DL were picked because they would challengeboth Charlie and his colleague out of the physics “com-fort zone.” He talks about his selection of dimensionsas, “so devising something like this, you know, it sort ofseemed like the challenge of it for us was implicitly makesomething that doesn’t involve as much math [I].” Ulti-mately Charlie and his colleague chose to create a unitthat would use a scientific practice that they typically donot integrate into their classroom material.Due to the structure of the class, Charlie also had todecide where he would like to introduce 3DL into theclassroom. Charlie sees “time is at a, at a real pre-mium [I]”, and because of this “spending, you know, 15minutes, even 15 minutes doing the lecture activity wouldhave been brutal [I].” Presenting a 3D activity in therecitation seemed to make the most sense because of thenature of the class. Charlie goes on to say, “The recita-tion seems like the ideal place to put the sort of thingthat’s the time where the students really have time to,you know, get involved and, and get their hands dirty [I].”We see 3DL appearing within the recitation component of Charlie’s class due to the problem-solving environmentthat the class provides.When it came to thinking about how the materialshould be taught, Charlie feels he did not put muchthought into how the material would be taught becausehe spent more time figuring how to represent the prob-lem to students. This is because creating the materialwas “a challenging enough of a task, that, you know,trying to, you know, just the formulation of the questionis seems more difficult than, you know, even trying toteach it [I].” We see time, that time spent designing ac-tivities has an affect on the amount of time that Charliecan spend thinking about other aspects of the 3DL unit.In summary, several factors influenced the way Charlieapproached transforming his ideas into usable materialthese include: working with his colleague, selection ofscientific principles, the structure of his class, and time.
3. Evaluation
Charlie’s evaluation includes clicker questions and unitexams. Charlie has a difficult time wanting to align hisunit exams to 3DL. More specifically, Charlie takes a lotof pride in the current status of his exams and the waythat he feels that they assess student understanding.In order to check his students’ understanding duringlecture, Charlie employs clicker questions. During lec-ture Charlie uses “clickers a lot [I]” to help “bringingthe scores up of those, those tricky multiple-choice ques-tions [I].” Charlie wants to give his students the chanceto “see more of the types of logic that I want them tobe able to do when they get in front of a test [I].” Wesee that clicker questions provide Charlie with the oppor-tunity to test his students understanding as well as givestudents practice for his formal exams.When it comes to formal exams Charlie has a hardertime integrating 3DL. One reason for this is because he“take[s] a fair amount of pride [I]” in his assessment ques-tions. More specifically, he “like[s] the concepts they tar-get”, and “like[s] that, the way that I think they makethe students think [I].”Charlie also likes the concepts his test questions coverand thinks his instruction and assessment complementeach other nicely. He states, “I have been orienting myclass, I know what questions I like to ask [I].” Charliecomes to the conclusion that “my default probably oughtto be to show them questions and recitations that moreclosely connects to types of test questions [I].” OverallCharlie thinks it is best if all of his instructional materialaligns with his exams, he does not think that his 3DLrecitation achieves this due to the type of questions thatare asked in recitation.Charlie also notes that creating new 3DL classroommaterial takes so much effort that the assessment be-comes an afterthought. When speaking about assessinghis recitation problem he mentions, “we spent so muchtime and effort on [the recitation] that it never turned1into an exam problem [I].” He goes on to say “integrat-ing the three dimensions into the test questions is signif-icantly difficult and challenging, that it almost decouplesthe test from the teaching is that it sort of breaks thatconnection there [I].” Ultimately, he sees creating the as-sessment as an “afterthought [I].”The way that Charlie chooses to evaluate his studentsin his class relies on in-class clicker questions and unitexams. When it comes to his unit exams Charlie remainsunsure about aligning them to the 3DL framework.
4. Instruction
For the 3DL unit, students were exposed to 3DL mate-rial in the lecture and recitation. In the lecture studentswere given a series of clicker questions that were used to“show the conversion between you know, kinetic energyand potential energy using a video [I]. After that wasdone Charlie went through the idea of energy conver-sion: “this time the electrostatic context where I’ve gota charge and I’m moving it against electric field now in-stead of a gravitational field [M].” The recitation was the“very next day [M]” and students were presented witha screenshot of the video they saw in lecture and wereasked to “[fill] in the chart for you know, all these differ-ent stages throughout the video [M].”Charlie relied on other instructors and their choice ofclassroom management and content delivery for the 3DLunit. Charlie was present for the instruction of his 3Dmaterial in the lecture and he “really like how this fitinto the lecture [I]”, but he was not able to be presentfor the instruction of the 3D recitation material. Dur-ing the recitation different instructors are responsible forteaching the 3D material. In recitation, instructors givestudents a “problem to work out [. . . ] usually in teams offour and [. . . ] they’ve got a secondary instructor roamingthe room to and hopefully an LA but not this semesterroaming the room to [. . . ] help them through that [M].”During the 3D unit, instructors decided to encourage “abit of a divide and conquer approach [M]” this was mainlydone to “make things go more efficiently [M].” Charliewas able to get this feedback from another fellow whoteaches his recitation although he typically doesn’t “feellike [he] need[s] to check in [I].” Overall, Charlie had torely on the recitation instructor for the instruction of hismaterial.
5. Reflection
After implementing his 3DL unit Charlie reflected onhow different this type of activity was from what he usu-ally asks his students to do, and how he expected therecitation to go. Taking these thoughts into account hereflects on his feelings about 3DL as a whole.Overall, Charlie’s feelings about how the 3DL unitwent are neutral. The problem seemed to be “a fairly big departure from the types of problems they’re usedto seeing in recitation [M].” But overall the unit didn’tseem much different from other units in the class. Char-lie expresses this as a “lateral step [M]”, and describesstudent understanding as “the total amount they learnedwas no more no less [I]” and their grades “being withinthe noise [I].”Charlie did take some lessons away after implementingthe 3D unit. First, Charlie noticed that the problem didnot achieve what he wanted it to. Charlie talks aboutwhat he wanted his students to notice from the problemin his interview, “the hope was that if we if they grindout the numbers, and so and then say, Oh, wait, thesenumbers that we just round out, give exactly, you know,the numbers that would be expected from this model,[. . . ] Yeah, and I don’t think that I, the feedback I gotwas that that was not accomplished [M].” The problemalso seemed to be “much more of a, of a, grind it out justsort of numerically challenging time consuming [M].”Overall Charlie does not think he is going to continueto use 3DL. Although the clicker questions “ended upfitting into the lecture a very nice way that I liked alot [M].” Charlie is not sure what to do with the recitationproblem, he states, “I’m not sure. Whether this is thisshould be tweaked, or whether it’d be more useful forthem to do some more evolved conservation energy typecalculations to get an idea of the utility of, of, of themethod [M].” When asked about continuing use Charliestates, “I don’t know. I mean, at this point, I think theeasy answer is no [I].”After collecting feedback and thinking about how the3DL problem went Charlie has ultimately decided thathe may not wish to continue 3DL.
C. Molly
Molly is a pre-tenure faculty member who teaches astudio-based (integrated lab and lecture) physics classtaken by biology majors. Her class is focused on givingstudents the space to explore physics phenomena throughthe use of experiments, and because of this Molly wantedto learn how to assess her students on these lab skills.This served as her motivation to join the Fellowship. Hertime in the Fellowship did not coincide with Charlie andRon’s, but she talks about the benefit of interacting withother faculty members in the Fellowship.Molly’s ideas to implement in her class were to createassessment items that really assessed lab skills as well ascreating an honors option for students to explore scien-tific phenomena across multiple disciplines.As Molly navigates the through the stages of PR&Ashe does not seem to recall a lot of elements that hinderher from applying her ideas, but she does note influencesthat have helped her. Mainly, a Fellowship coordina-tor that has helped her with assessments and listeningto other Fellows discuss teaching ideas. Where theseideas present themselves and how they contribute to the2themes will be discussed below and can also be found inTable III.
1. Comprehension
The ideas of 3DL connect to Molly’s educational goalsthat involve both skills and content. These goals areshaped by the department, but tailored to her needs.Overall, Molly believes it is beneficial to have both con-tent and practice goals for her classroom.For her class, Molly would like students to develop theuse of multiple tools. Molly teaches a physics class forbiology students. For her class, she thinks it is importantto leave with “a toolbox of skills[I].” This ‘toolbox’ iscomposed of “physics words and representations and likecommon usages of physics [I]” that they can use whenthey “encounter a new situation[I].” These tools shouldalso “help them look at problems in a way that is not thesame as what they get from biology and chemistry[I].”Molly’s educational goals extend beyond developingskills to include content goals as well. When it comes tocontent, Molly agrees with the set of content that she isgiven from the department, but she takes a longer time tofocus on topics that might be more applicable for her bi-ology students. Molly notes, “I definitely agree with, like,so, in terms of content I really prioritize energy thinkingand I prioritize thermodynamics thinking, because thoseare content areas that I know are going to become usefulto those students later in their biology trajectories[I].”She sees her goals and the department goals as intersect-ing, “there’s definitely like content goals I would havethat are coming from my department but are intersect-ing with me knowing who my students are[I].”She also thinks it is beneficial to have content goalsthat are separate from practice goals. Molly states, “thelanguage of like separating content from practices is thatit’s okay sometimes to be like, oh, right now, I want tomake sure you guys have the vocabulary to understandthat voltage and potential difference and potential like,they are all the same thing[I].”On the whole, Molly believes the use of content andpractice goals are both beneficial and align closely withthe 3DL framework.
2. Transformation
When Molly approaches design of classroom material,she likes to adapt borrowed material to fit the needs of herstudents. Integrating 3DL into the classroom fit nicelydue to the studio structure of her classroom and becauseof the ideas she received from other Fellows.Molly has a set approach for material design. When itcomes to designing new material for the classroom Mollyusually starts by “look[ing] for somebody else who hasalready written already fantastic problem on that topic.If it’s like exactly what I want, then I take it, and most often though, it’s not exactly what I want, right it mostoften it’s something close to what I’m looking for, butnot quite there. And so then I started the process ofmodification[I].”For 3DL material, she uses the same process but looksfor different information. Molly states, “I think that re-ally changes for me is that I find myself looking for data,in like real data, like either collected by my students orin a paper or something like that, because I find it eas-ier to ask real practice questions, like interpretation oranalysis or something when there is some sort of datain front of a student[I].” Molly also uses a tool from theFellowship, the Three-Dimensional Learning AssessmentProtocol (3D-LAP) [7]. She describes her experience withthe 3D-LAP as “some sort of protocol where you couldsort of like checkbox, your, your like, you could look atyour own assessments and say, like, how three dimen-sional were they? Yeah, and I had seen that and thoughtit was a useful tool[I].” For the design of 3DL material,Molly starts to prioritize the use of real data in her class-room material. The 3D-LAP also aids Molly ensuringthat her material aligns with the 3DL framework.The structure of the class helps Molly integrate 3DLinto her classroom. Molly recalls what teaching a purelylecture based class was like, “so I think it was a lot harder.And that in that setting to think about some of thosepractices as fitting like if I felt like I should belong in thelab [I].” With the studio structure of her current classstudents are able to engage in practices during lab.Working with Fellows from other disciplines helpedMolly with her material design. During the FellowshipMolly and other Fellows designed a cross disciplinaryproblem for students. Molly recalls this experience as“a cool thing that wouldn’t have come about if I had notbeen in a group of people with Biology and Chemistryand like all of us together[I].”Molly’s general approach for material design remainedunchanged when she started to create 3DL material ex-cept now she started to pay closer attention to the type ofinformation that she presents to her students. The struc-ture of her class and her colleagues in the Fellowship werealso a benefit for Molly.
3. Evaluation
Molly’s formal evaluation of her students takes place inthe form of individual and group exams. The logistics ofthese exams came from discussions with other Fellows.Once she integrated 3DL into her exams, Molly foundthat she was able to assess the practice of science. Afterlearning how to create 3DL formal exams, Molly was ableto change her informal approach of evaluation to alignwith instruction.In Molly’s class she uses individual and group exams.The individual exams are composed of “a bunch of an-alytic problems” and “some coding work [I]” and Mollydescribes her group exams as a place where she is “testing3their experimental understanding [I].”When it came to generating exam ideas, Molly foundthe Fellowship to be very beneficial. In her interview shestates, “I’ve like really enjoyed hearing from people andthe different ways that they write exam questions andthink about grading them. Um, because that’s like oneof the hard parts of faculty member, like grading is theworst [I].” In fact, Molly joined the Fellowship becauseshe “heard from some folks that designing assessmentquestions that people had creative ways of doing that,that I would get some, like exposure to I guess [I].” Shealso found designing 3D exam questions to be useful be-cause it allowed her to accurately assess what she wantedto assess. Her prior work with Ruben (pseudonym), aFellowship coordinator, “really helped me with my examquestions make them better using 3D, 3D dimensionallearning [I].” This process took place by “[Ruben] wouldlook at my exam questions, and he would help me seethat, while implicitly there was a practice there, I wasn’texplicitly asking students to show me their understand-ing of the practice[I].” The end results were very positive.Molly recalls, “my questions just got much cleaner andlike students would produce what I really wanted themto do [I].” 3DL assessments have allowed Molly to paycloser attention to what she was assessing.Molly’s practice with 3DL exams helped her create as-sessments to informally test her student’s understandingin class in the form of clicker questions. Molly states,“so I think since I’ve gotten better at writing those testquestions that do practices and then I can translate thatto like writing clickers style questions that you could usea lecture class [I].”All in all, Molly’s structure of exams has not changeddue to 3DL or the Fellowship although she did benefitfrom hearing from other fellow’s exam practices. Molly’sexams benefited from the use of the 3DL framework be-cause she was able to assess scientific practices. Ulti-mately, her work designing exams also helped her de-sign clicker questions to test her students informal un-derstanding.
4. Instruction
The 3DL assessment items that Molly’s students com-pleted focused on the biological phenomena of the “straincurve of two different component of bone [F].” This ques-tion required students to write down their response fortwo separate parts. Aside from her assessment itemsMolly’s classroom material are “great examples of Three-Dimensional problems [I].”When it comes to instruction, Molly has developed atechnique to her instruction. Molly has been teaching aclass that is an “integrated lab lecture [I]” with multipleinstructors. Due to the set-up of her class, Molly canchoose to “do lab whenever we want to do it [I].” The wayMolly chooses to present and explore material is througha process she calls “applying and extending [I].” In her interview Molly describes this process, “I present them aproblem, a phenomenon of some sort. And, and I askedthem to think about, like, what kinds of things wouldthey like to know about it? And, and then they, theydesigned some experiment, or to figure out some of thestuff that they would like to know about it [. . . ] Andthen we spend the next day seeing how far those ruleswill take us [I].”
5. Reflection
Molly recalls different 3DL materials going differentlyin her classroom. Due to Molly participating in an ear-lier Cohort Molly’s interview occurred one year after herparticipation in the Fellowship, Molly has more difficultyreflecting on how her 3DL unit went. She is able to re-flect on an exam question and classroom problem. Mollyrecalls that one of the 3DL exam problems that she im-plemented was very successful. Molly states, “Everyonegot that question right. I was so like, I remember be-cause it was like so easy to grade [I].” Another problemthat Molly implemented did not go well. Molly recallsthat it “fell on its face [I].” Molly goes on to elaboratewhy, “I remember it as like a really like a thing that Isimplified to the reduced level because that’s what I doin physics, right? These are two point charges connectedby a distance. And for a bunch of students who hadstrong chemistry backgrounds, they saw partial chargesand much more like possible of like bending [I].” Overall,the 3DL material that Molly created went well, but someof the simplifications that Molly made were difficult forstudents with strong STEM backgrounds not in Physics.
VIII. CROSS-CASE ANALYSIS
In order to look for common factors that influenceFellow’s design and implementation of 3DL material wecross-analyzed each case by comparing and contrastingthemes from each one of the 5 stages of a Fellow’s PR&A.From our cross-case analysis we found three themesacross our cases: motivation/ability to change assessmentpractices, material placement and participation in class-room materials, and engagement in social interactions arecommon subject matters present in multiple stages of ourFellow’s PR&A. The location of these themes accordingto each Fellow’s PR&A can be found in Table III.
1. Motivation/Ability to change assessment practices
All three of our Fellows have specific assessment prac-tices. Ron uses a course management system, Charlie’scurrent assessments test his student’s conceptual under-standing, and Molly wants to integrate scientific practicesinto her exams. Charlie has already built his exams fromhis instructional materials and likes the concepts that4they test, and Molly is currently in the process of align-ing her instruction of scientific practices with their assess-ment. Their assessment practices and instructional ma-terials influence each other, and Molly’s ability to assesand instruct scientific practices has a positive influenceon her continued use of 3DL.On the other hand, Ron’s use of a course managementsystem and Charlie’s inability to create an assessmentitem for his 3DL unit causes discomfort with their contin-ued use of 3DL. The course management system that Ronuses restricts him from using multiple choice questions.Ron does not find multiple choice formatting conduciveto the 3DL framework. Charlie was lacking time to cre-ate the assessment for his 3DL activity. Both Ron andCharlie’s discomfort with the lack of alignment betweentheir assessment and instructional materials contributeto their reservations about 3DL. Unlike Charlie it doesnot stop the use of 3DL, but he does note it as a barrierthat he will have to overcome.
2. Classroom structures/Instructional practices
The structure of our Fellow’s classroom effects theirinstructional practices and their overall buy in of 3DL.Ron, Charlie, and Molly all teach classes with differentclassroom structures. Ron teaches a lecture class, Charlieteaches a class with a lecture, lab, and recitation com-ponents, and Molly teaches a studio class. The studioformat of Molly’s class allows her to focus on scientificpractices by exploring biological phenomena, and the ed-ucational resources that Ron uses allows him to create astudent led activity in his lecture activity. Unlike Mollyand Ron, Charlie had to choose where to place his 3DLactivity. Due to the teaching progression Charlie hasfor his class, he decided to create a 3DL activity for therecitation component of his class.The placement of their material influences their in-structional practices. Ron and Molly were able to bepresent for the instruction of their unit. Their participa-tion in instruction resulted in them developing a personalinsight on the ways in which students responded to thematerial and how they felt while teaching the material.The impressions that they were able to form from instruc-tion served as a main motivating factor to continue withthe use of 3DL. Charlie’s recitation activity was taughtby his recitation instructor. Charlie had to rely on stu-dent grades and the feedback his colleague provided. Thefeedback he got from his colleagues and the grades wereneutral, but he formed a negative impression about con-tinuing with 3DL.
3. Social interactions
We see that all three Fellows are influenced by socialinteractions. The interactions that all three Fellows haveinside the Fellowship influence the ways in which they learn about new teaching ideas and supports them intheir material development. All three decide to workcollaboratively on their material design. Ron and Mollyalso discuss how the Fellowship helps them come up withnew ideas for activities and assessment practices. The so-cial interaction that Fellows have inside of the Fellowshiphelp them take up 3DL and integrate it into their class-rooms.Ron is also influenced by interactions outside of theFellowship. Ron classifies his interactions as ’old school’and does not view these interactions has helpful or sup-portive of his new ideas. In fact, Ron refrains from shar-ing his new ideas with a co-instructor because of the in-teractions that he has had outside of the Fellowship. Thesocial interactions with people outside of the Fellowshiphinder the spread of 3DL throughout Ron’s department.Interactions with students also influences Ron andMolly’s view of their instruction. Interacting with hisstudents sparked Ron’s motivation to join the Fellowship.During the Fellowship, Ron and Molly were able to par-ticipate in the instruction of their entire 3DL unit. Thisleft a positive impression on Ron and Molly as they wereable to reflect upon the conversations they were havingwith students. Their positive experience interacting withtheir students also served as a reason for continued useof 3DL.
IX. DISCUSSIONA. Alignment of Assessment and Instruction
The ability to align assessment and instruction directlyinfluences whether faculty buy into 3DL. Fellows whodid not have the chance to align their instruction withtheir assessment items had mixed reactions on planningto continue the use of 3DL in their classrooms.The Fellowship focuses on the principles of LGDD tohelp them align all of their materials to their learninggoals. All three Fellows created learning goals and in-structional items, but both Ron and Charlie never cre-ated an assessment item. For Ron this is due to thecourse management system that his department uses, andfor Charlie it is due to a lack of time and because his ex-ams already assess what he wants to evaluate. Molly isable to align her exams with the framework after work-ing closely with Ruben. She feels very positive abouther class and has confidence that she is assessing howstudents engage in doing science. Their views on theirassessments ultimately affects their instructional materi-als. Both Charlie and Ron see being unable to align theirinstructional materials with their assessment as a prob-lem. For Charlie this makes him want to stop the use of3DL materials; although, it does not fully stop Ron fromthe use of 3DL, but he does note it as a barrier that hewill need to overcome.All three Fellows mention the importance of aligninginstruction and assessment, but unlike Molly both Ron5and Charlie struggle to align their assessment with in-struction. The Fellowship is designed around the premisethat changing assessment leads to change in instruc-tion [74], and our research suggests that there is a correla-tion between the ability to align instruction to assessmentand continued use of said materials. More specifically wefound that assessment design is influenced by the cur-rent state of assessments tools/technology, and how wellit can be integrated into instruction.There is a lot of work out there that outlines ways forfaculty to align their instruction with assessment [72, 73,87], but there is little work that investigates the chal-lenges faculty encounter when aligning their materials ordeveloping assessment items. Although the Fellowshipspent time talking about assessment design and the pro-cess of aligning assessment with instruction, it did notspend time to understand the logistics of assessment de-sign and the current state of their assessments beyondtalking about what it is that they want to assess. FLCsshould address the topics surrounding current assessmentdevelopment as a way to help faculty adopt new instruc-tional practices.
B. Spread of 3DL
We see that the culture of the department workingthrough classroom structures and course managementsystems stops the use of 3DL. Furthermore, social inter-actions within the department stops the spread of ideassurrounding 3DL.
1. Classroom Structure
We see the classroom structures imposed by depart-ments affect material design and play a part in theplan for the continued use of 3DL. Ron, Charlie, andMolly all teach courses with different classroom struc-tures. The structure of Ron and Molly’s classes allowfor one class component, lecture for Ron and studio forMolly. Whereas Charlie has three components in hisclass: lecture, lab, and recitation. Due to the structuresof their classrooms Charlie has to decide what sectionhe would like to implement his 3DL material unlike Ronand Molly. Throughout all of Charlie’s stages of PR&Ahis decision to implement his material in the recitationaffects the way he participates in instruction, how hegauges the effectiveness of the unit, and eventually histhoughts on continuing with 3DL material.The placement of the 3DL unit puts constraints on theinstruction of the material. Ron and Molly were able tobe present for the entire activity and were able to choosehow management of the classroom and explanation of thematerial would look like. Charlie had to rely on anotherinstructor, George, to implement his unit, and Georgemade decisions concerning instruction and managementof the 3DL unit. By looking across the stages of Instruc- tion and Reflection in PR&A we see that their abilityto participate in the instruction of the 3DL unit directlyaffects their feelings about their 3DL unit. In their re-flections, Ron and Molly are able to speak to how theinstruction went and how they felt about the material.However, Charlie was not able to do this and could onlyreflect on how students performed on the material, andbecause the grades did not differ between the 3DL unitand his old material Charlie does not see the benefit forcontinued use of 3DL.The structure of the classroom, and faculty participa-tion in instruction influence Fellow’s plans for the con-tinued use of 3DL. Structures of classrooms are often outof the control of instructors, and this can impact howfaculty interact with their classrooms. More specifically,classrooms that rely on more than one instructor have anadded barrier as other instructors might not be aware ofthe main instructor’s goals [88]. Other research has foundthat faculty often use their intuition and general feelingsto see if a technique worked [68], and that the types of in-teractions instructors have with students during instruc-tion also affect their use of pedagogical practices [18].This all goes to support Turpen, Dancy, and Hender-son’s claim that faculty are more convinced by their ownexperiences rather than with data [69]. Support providedfrom professional development leaders rarely includes ob-servation and feedback during implementation [65]. Onesolution for this could be the use of peer-to-peer obser-vations [18]. FLCs should encourage faculty to use theirintuition and experience as well as student outcomes tosupport faculty buy-in. This can be done by providingfaculty with opportunities to participate in and reflecton their own experience with new material as well as theability to learn from student outcomes.
2. Course Management Systems
Ron struggles to use 3DL because of the departmentalhistory with a course management system. The use ofthe course management system affects the design of hismaterial. Ron mentions the use of this course manage-ment system when describing his exams, but becomes alarger problem when Ron tries to align his exams to hisinstruction. Ron notes this as one of his main concernswhen reflecting upon his experience.Ron’s trouble with the course management systemechoes the work of Zohrabi that finds faculty’s motiva-tions for adopting RBIs can be negatively impacted dueto the use of certain textbooks or online platforms [68].But, unlike Zohrabi we see that Ron’s use of the coursemanagement system affects his use of assessment ratherthan motivation to change it.6
3. Social Interactions
One way that the Fellowship supports faculty isthrough interactions with other Fellows. Yet the socialinteractions Ron has outside of the Fellowship affects hisplan for the continued use of 3DL.Ron, Charlie, and Molly all had positive experiencesinteracting with Fellows inside of the Fellowship. Allthree of them were able to collaborate with Fellows ontheir materials, this proved to be a positive experience forour Fellows. The conversations that occurred as part ofFellowship meetings gave our Fellows new ideas to bringinto their classrooms. These interactions are products ofthe design of the FLC.Ron is also negatively impacted by the social inter-actions he has outside of the Fellowship. Outside ofthe Fellowship, Ron has less than meaningful conversa-tions about new teaching practices. We see that thisdoes not stop Ron from pursuing and trying new instruc-tional/teaching strategies, but it does stop him from col-laborating with his co-instructor. The experience Ronhas had with other colleagues in his department hasstopped him from sharing his ideas about 3DL and doesnot support the purpose of the bottom-up approach ofFLCs.
4. Departmental Culture
Defining departmental culture as the use of a shared setof beliefs, customs, practices, and artifacts [49]. We seethat departmental practices such as classroom structuresinfluence Charlie’s participation in instruction. Ron’s useof departmental used artifacts affect his ability to createassessment items. Customs and beliefs in the depart-ment also affect his social interactions, and discouragesRon from sharing new ideas. This ultimately hinders thegoals of bottom-up approaches to professional develop-ment by stopping the spread of certain teaching ideas.The Fellowship was created based on the hope that fac-ulty involved in the Fellowship would spread ideas acrosscolleagues and hopefully though departments and insti-tutions. We see the spread of new practices and ideasdepends on the tools and environments used/created bythe department and the people within them. In order toutilize the bottom-up feature of FLCs professional devel-opment needs to focus on the departmental culture thatthey are a part of, and be attentive to the way thesecultures could affect the spread of new ideas.
C. Limitations
The limitations present in our study come from ourfocus on Physics, our case selections, and our use of thePR&A framework.We chose to focus on Physics Fellows because ourPhysics expertise would lend itself to understanding the design of Physics classroom materials as well as the phys-ical and cultural contexts that our Fellows participatein. Since this study was conducted with Physics faculty,the claims that are made are confined to Physics depart-ments and professional development programs. Althoughit would be reasonable to assume that course structuresand alignment of assessment and instructional materialsare all factors that would influence material design forother STEM Fellows outside of the Physics discipline.The Fellows that we selected for our cases are all fac-ulty members at large and predominantly white institu-tions. They also all freely chose to participate in a long-term teaching fellowship. The demographic make-up ofthe institutions that our Fellows participate in effects theculture of their department as well as the considerationsthat Fellows take into account when designing classroommaterial. These considerations could look vastly differ-ent for Fellows that do not work at similar universities;therefore, our claims about the culture of departmentsare limited to Faculty participating in large and predom-inately white universities. Our Fellows also chose to par-ticipate in a two year teaching fellowship because of thepractices that our Fellows use may not extend to the ex-periences that every faculty member faces when designinginstructional and classroom materials.The use of the framework of Pedagogical Reasoningand Action also limits the scope of our research inves-tigation. By focusing on Fellows’ PR&A we were ableto investigate the influences on the design of a Fellows’classroom material, but not influences that affect otherexperiences that our Fellows face inside of the Fellowship.These experiences may include their motivation to jointhe fellowship, new ideas learned that go beyond instruc-tion and material design, and interactions that influenceother areas of their professional life.
X. CONCLUSION
In our study we investigated influences on faculty up-take by utilizing the PR&A framework. We exploredthe PR&A of three cases in order to determine the in-fluences on material design, and plans for continued useof 3DL. Our cross analysis of Ron, Charlie, and Mollyhave pointed out assessment alignment, participation ininstruction, and social practices in and outside of the Fel-lowship influence the ways faculty can take up ideas froma FLC.Our first research question was concerned about influ-ences outside of the Fellowship that affect material de-sign. We find that the culture of the department thatour Fellows exist in effect their material design. Morespecifically, the departmental culture influences coursemanagement systems, classroom structure, and social in-teractions that all influence their material design.The second idea that we wanted to investigate was theinfluences on the plans for continued use of 3DL. Ourstudy reveals participation in instruction and the abil-7ity to align assessment to instruction all positively im-pact our Fellow’s continued use of 3DL. More specificallywe find faculty participation and classroom structure in-fluence how faculty feel about the change that they aremaking. Therefore, it is important for faculty to collectevidence of student learning as well as participating inteaching experiences in order to use their intuition.Finally, we explored the ways in which the FLC sup-ports Fellow’s adoption and plans for continued use. Wediscover that the FLC provided Fellows with a space toshare new ideas, learn from other’s ideas, and work withother Fellows on material design. The FLC also spenttime talking about the importance of aligning instruc-tion and assessment; although this does not help Ronand Charlie to align their instruction and assessment, itdoes influence how they view the utility of their 3DL unit.
A. Recommendations
Our work suggests that FLC designers should spendtime talking about assessment, encouraging participationin instruction, and supplying the necessary tools to helppromote change. We also recommend that the researchcommunity investigates the ways in which faculty use anddesign assessment items.Unlike Molly who had additional help and applicableresources, Ron struggled to make the assessment itemsthat he wanted to do to the nature of the course man-agement system that requires multiple choice formatting.In order to prevent discontinued use of their design ideasdue to assessment practices, FLC facilitators should in-clude discussions on how to design assessment items andaddress common issues such as formatting. Participation in instruction is a valuable experience,and in our cases we see this directly affect Molly, Ron,and Charlie’s buy in to 3DL. More specifically we notethat Charlie would have benefited from observing hisstudents working on the 3DL activity. In order to en-courage instructors to participate in instruction and usetheir intuition as valuable data, FLC designers need tounderstand that data does not always drive change andencourage participation in instruction through peer ob-servations or reflections.Lastly, we see that all three of our Fellows participatein different types of social interactions. These interac-tions have an effect on the spread of ideas throughout adepartment. This is apparent in Ron’s case as he triesto bring 3DL into a class that he coteaches. In order topromote change beyond the FLC, FLCs need to work to-wards supplying participants with the tools and resourcesthat will help them promote change within their depart-ment especially in regard to moving towards different as-sessment and instructional formats.Along with our results from our research we found thatthe process of assessment design from the perspective ofthe practitioner is also underrepresented in the literature.As a research community we need to further investigatehow faculty use and design assessments.
XI. ACKNOWLEDGEMENTS
A special thanks to the Kansas State UniversityPhysics Department, the faculty involved in interviews,and my 3DL4US colleagues especially Paul Bergeron andMarcos D. Caballero. This material is based upon worksupported by the National Science Foundation underGrant No. 1726360. [1] M. Mohr-Schroeder, C. Jackson, D. C. Schroeder, andJ. Wilhelm, The Next Generation of STEM Teachers:An Interdisciplinary Approach to Meet the Needs of theFuture p. 93 (2019).[2] N. R. Council et al.,
A framework for K-12 science ed-ucation: Practices, crosscutting concepts, and core ideas (National Academies Press, 2012).[3] S. Olson and D. G. Riordan, Executive Office of the Pres-ident (2012).[4] T. R. Guskey,
Evaluating professional development (Cor-win press, 2000).[5] S. M. Underwood, L. A. Posey, D. G. Herrington, J. H.Carmel, and M. M. Cooper, Journal of Chemical Educa-tion , 207 (2018).[6] J. T. Laverty and M. D. Caballero, Physical ReviewPhysics Education Research , 010123 (2018).[7] J. T. Laverty, S. M. Underwood, R. L. Matz, L. A.Posey, J. H. Carmel, M. D. Caballero, C. L. Fata-Hartley,D. Ebert-May, S. E. Jardeleza, and M. M. Cooper, PloSone (2016). [8] K. Bain, L. Bender, P. Bergeron, M. D. Caballero, J. H.Carmel, E. M. Duffy, D. Ebert-May, C. L. Fata-Hartley,D. G. Herrington, J. T. Laverty, et al., PloS one (2020).[9] R. L. Matz, C. L. Fata-Hartley, L. A. Posey, J. T. Laverty,S. M. Underwood, J. H. Carmel, D. G. Herrington, R. L.Stowe, M. D. Caballero, D. Ebert-May, et al., Scienceadvances , eaau0554 (2018).[10] M. M. Cooper, Journal of Chemical Education (2020).[11] A. Olmstead and C. Turpen, Physical Review PhysicsEducation Research , 020136 (2016).[12] A. Olmstead and C. Turpen, Physical Review PhysicsEducation Research , 020123 (2017).[13] W. Sandoval, Journal of the learning sciences , 18(2014).[14] D. Ebert-May, T. L. Derting, J. Hodder, J. L. Momsen,T. M. Long, and S. E. Jardeleza, BioScience , 550(2011).[15] C. A. Manduca, E. R. Iverson, M. Luxenberg, R. H. Mac-donald, D. A. McConnell, D. W. Mogk, and B. J. Tewks-bury, Science Advances , e1600193 (2017).[16] S. Polich, To improve the academy , 106 (2008). [17] E. A. Addis, K. M. Quardokus, D. C. Bassham, P. W. Be-craft, N. Boury, C. R. Coffman, J. T. Colbert, and J. A.Powell-Coffman, Journal of College Science Teaching ,22 (2013).[18] T. L. Tinnell, P. A. Ralston, T. R. Tretter, and M. E.Mills, International Journal of STEM Education , 1(2019).[19] C. Henderson, American Journal of Physics , 179(2008).[20] E. R. Elliott, R. D. Reason, C. R. Coffman, E. J. Gan-gloff, J. R. Raker, J. A. Powell-Coffman, and C. A.Ogilvie, CBE—Life Sciences Education , ar22 (2016).[21] K. L. Sirum and D. Madigan, Biochemistry and Molecu-lar Biology Education , 197 (2010).[22] S. A. Southerland, E. M. Granger, R. Hughes, P. En-derle, F. Ke, K. Roseler, Y. Saka, and M. Tekkumru-Kisa, AERA open , 2332858416674200 (2016).[23] C. J. Daly, Currents in Teaching & Learning (2011).[24] C. Henderson, A. Beach, and N. Finkelstein, Journal ofresearch in science teaching , 952 (2011).[25] L. Shulman, Harvard educational review , 1 (1987).[26] D. W. Sunal, J. Hodges, C. S. Sunal, K. W. Whitaker,L. M. Freeman, L. Edwards, R. A. Johnston, andM. Odell, School Science and mathematics , 246(2001).[27] R. S. Caffarella and L. F. Zinn, Innovative Higher Edu-cation , 241 (1999).[28] M. L. Blanton and D. A. Stylianou, Innovative HigherEducation , 79 (2009).[29] J. Fusco, S. Haavind, J. Remold, and P. Schank, Educa-tional Media International , 139 (2011).[30] S. Weller et al., International Journal of Teaching andLearning in Higher Education , 25 (2009).[31] M. D. Cox, New directions for teaching and learning , 5 (2004).[32] M. D. Caballero, N. Chonacky, L. Engelhardt, R. C.Hilborn, M. L. del Puerto, and K. R. Roos, The PhysicsTeacher , 397 (2019).[33] C. Gopalan, G. Bracey, M. Klann, and C. Schmidt, Ad-vances in physiology education , 648 (2018).[34] S. G. Hoskins, A. J. Gottesman, and K. L. Kenyon, Jour-nal of microbiology & biology education (2017).[35] M. Engin and F. Atkinson, International journal of teach-ing and learning in higher education , 164 (2015).[36] J. Kusch, Journal of College Science Teaching , 24(2016).[37] L. S. Nadelson, S. E. Shadle, and J. K. Hettinger, Learn-ing Communities Journal (2013).[38] M.-A. Mundy, L. Kupczynski, J. D. Ellis, and R. L.Salgado, Journal of Academic and Business Ethics , 1(2012).[39] C. Henderson and M. H. Dancy, Physical Review SpecialTopics-Physics Education Research , 020102 (2007).[40] P. Baker, The social works of higher education pp. 95–109(1999).[41] K. Vangrieken, C. Meredith, T. Packer, and E. Kyndt,Teaching and teacher education , 47 (2017).[42] M. Wyatt and E. Onˇcevska Ager, ELT journal , 171(2017).[43] K. Roseler and M. W. Dentzau, Cultural Studies of Sci-ence Education , 619 (2013).[44] P. A. Bechtel and M. O’Sullivan (2006).[45] J. Goldstein, J. Hazy, and B. Lichtenstein, Complexityand the nexus of leadership: Leveraging nonlinear science to create ecologies of innovation (Springer, 2010).[46] A. Kezar,
How colleges change: Understanding, leading,and enacting change (Routledge, 2013).[47] A. Macias, International Journal of Teacher Leadership , 76 (2017).[48] E. A. Kyza and Y. Georgiou, Science Education Interna-tional , 57 (2014).[49] J. C. Corbo, D. L. Reinholz, M. H. Dancy, S. Deetz,and N. Finkelstein, Physical Review Physics EducationResearch , 010113 (2016).[50] I. Eloff and L. Ebersohn, Perspectives in education ,147 (2001).[51] B. D. Missingham, Community Development , 339(2017).[52] L. M. Swan, S. Kramer, A. Gopal, L. Shi, and S. M. Roth,The Journal of Faculty Development , 21 (2017).[53] L. Richlin and M. D. Cox, New directions for teachingand learning , 127 (2004).[54] L. W. Natkin and T. Kolbe, International Journal of Sus-tainability in Higher Education (2016).[55] J. Layne, J. Froyd, J. Morgan, and A. Kenimer, in (IEEE, 2002), vol. 2, pp.F1A–F1A.[56] M. C. Petrone and L. Ortquist-Ahrens, New directionsfor teaching and learning , 63 (2004).[57] M. Glowacki-Dudka and M. P. Brown (MidwestResearch-to-Practice Conference in Adult, Continuing,and Community . . . , 2006).[58] M. Glowacki-Dudka and M. P. Brown, New Horizons inAdult Education and Human Resource Development ,29 (2007).[59] D. A. Wicks, B. B. Craft, G. N. Mason, K. Gritter, andK. Bolding, The Internet and Higher Education , 53(2015).[60] J. S. Nugent, R. M. Reardon, F. G. Smith, J. A. Rhodes,M. J. Zander, and T. J. Carter, International Journal ofteaching and learning in Higher education , 51 (2008).[61] M. Dancy, A. C. Lau, A. Rundquist, and C. Henderson,Physical Review Physics Education Research , 020147(2019).[62] A. Rundquist, J. Corbo, S. Chasteen, M. Martinuk,C. Henderson, and M. Dancy, in Physics Education Re-search Conference (2015), pp. 279–282.[63] M. D’Eon, L. Sadownik, A. Harrison, and J. Nation,American Journal of Evaluation , 92 (2008).[64] R. Kane, S. Sandretto, and C. Heath, Review of educa-tional research , 177 (2002).[65] C. Henderson, M. Dancy, and M. Niewiadomska-Bugaj,Physical Review Special Topics-Physics Education Re-search , 020104 (2012).[66] M. Stains, M. Pilarz, and D. Chakraverty, Journal ofChemical Education , 1466 (2015).[67] M. H. Dancy and C. Henderson, in AIP Conference Pro-ceedings (American Institute of Physics, 2012), vol. 1413,pp. 163–166.[68] D. Zohrabi Alaee, Ph.D. thesis, Kansas State University(2020).[69] C. Turpen, M. Dancy, and C. Henderson, Physical Re-view Physics Education Research , 010116 (2016).[70] M. D. Sharma, I. D. Johnston, H. Johnston, K. Varvell,G. Robertson, A. Hopkins, C. Stewart, I. Cooper, andR. Thornton, Physical Review Special Topics-PhysicsEducation Research , 020119 (2010). [71] B. D. Blume, J. K. Ford, T. T. Baldwin, and J. L. Huang,Journal of management , 1065 (2010).[72] G. Wiggins and J. McTighe, Understanding by design ,7 (1998).[73] J. Krajcik, K. L. McNeill, and B. J. Reiser, Science Ed-ucation , 1 (2008).[74] J. T. Laverty, S. H. Tessmer, M. M. Cooper, and M. D.Caballero, arXiv preprint arXiv:1407.3260 (2014).[75] S. Pella, Teacher Education Quarterly , 81 (2015).[76] D. Stroupe, Cognition and Instruction , 51 (2016).[77] D. Loewenberg Ball, M. H. Thames, and G. Phelps, Jour-nal of teacher education , 389 (2008).[78] M. L. Niess and H. Gillow-Wiles, Australasian Journalof Educational Technology (2017).[79] L. Starkey, Teachers and Teaching: theory and practice , 233 (2010).[80] Y. Feng and K. Hew, in Society for Information Technol-ogy & Teacher Education International Conference (As-sociation for the Advancement of Computing in Educa-tion (AACE), 2005), pp. 3173–3180. [81] V. Smart, C. Sim, and G. Finger, in
Society for Informa-tion Technology & Teacher Education International Con-ference (Association for the Advancement of Computingin Education (AACE), 2015), pp. 3418–3425.[82] V. Smart, C. Sim, and G. Finger, International Societyfor Technology in Education (ISTE), San Antonio, Texas,United States (2013).[83] J. L. Endacott and J. Sturtz, The journal of social studiesresearch , 1 (2015).[84] J. W. Creswell, Qualitative inquiry and research design:Choosing among five approaches , 53 (2007).[85] S. M. Gass and A. Mackey, Stimulated recall methodologyin second language research (Routledge, 2000).[86]
Photoelectric effect (2016), URL https://phet.colorado.edu/en/simulation/legacy/photoelectric .[87] R. J. Mislevy and M. M. Riconscente, in
Handbook of testdevelopment (Routledge, 2011), pp. 75–104.[88] E. A. West, C. A. Paul, D. Webb, and W. H. Potter,Physical Review Special Topics-Physics Education Re-search , 010109 (2013)., 010109 (2013).