Associations between learning assistants, passing introductory physics, and equity: A quantitative critical race theory investigation
Associations between learning assistants, passing introductory physics, and equity:A quantitative critical race theory investigation
Ben Van Dusen and Jayson Nissen Department of Science Education, California State University Chico, Chico, California 95929, USA J. M. Nissen Consulting, Corvallis, Oregon 97333, USA (Received 3 December 2019; accepted 20 March 2020; published 9 April 2020)Many science, technology, engineering, and math degrees require passing an introductory physicscourse. Physics courses often have high failure rates that disproportionately harm students who arehistorically and continually marginalized by racism, sexism, and classism. We examined the associationsbetween learning assistant (LA) supported courses and equity in nonpassing grades [i.e., drop, fail, orwithdrawal (DFW)] in introductory physics courses. The data used in the study came from 2312 students in41 sections of introductory physics courses at a regional Hispanic serving institution. We developedhierarchical generalized linear models of student DFW rates that accounted for gender, race, first-generation status, and LA-supported instruction. We used a quantitative critical race theory (QuantCrit)perspective focused on the role of hegemonic power structures in perpetuating inequitable studentoutcomes. Our QuantCrit perspective informed our research questions, methods, and interpretations offindings. The models associated LAs with overall decreases in DFW rates and larger decreases in DFWrates for Black, Indigenous, and people of color than their White peers. While the inequities in DFW rateswere lower in LA-supported courses, they were still present.
DOI: 10.1103/PhysRevPhysEducRes.16.010117
I. INTRODUCTION
Introductory physics courses are required for manyscience, technology, engineering, and math (STEM) degrees.The high rates of students earning nonpassing grades [i.e.,drop, fail, or withdrawal (DFW)] in these courses creates asignificant barrier to many students earning a STEM degree[1 – – – – Published by the American Physical Society under the terms ofthe Creative Commons Attribution 4.0 International license.Further distribution of this work must maintain attribution tothe author(s) and the published article ’ s title, journal citation,and DOI. PHYSICAL REVIEW PHYSICS EDUCATION RESEARCH = = = II. LITERATURE REVIEW
There are a limited number of quantitative studiespublished on equity in physics education. What does existhas largely not examined issues of race and we know of nostudies that examined outcomes for gender nonconforming,gender nonbinary, trans, or two-spirit physics students. Inour literature review, we tried to highlight contributionsfrom authors from historically marginalized groups.The dynamic nature of the LA model has enabledinstructors using LAs to achieve a wide range of goals.Researchers have investigated the association between LAsand student learning [38,39], course transformation [40],departmental transformation [41,42], teacher recruitment[30], teacher preparation [43,44], equity [45,46], support-ing students beyond learning [47,48], and supporting novellearning environments [49,50]. While the associationbetween LAs and student DFW rates interests manyconstituents [51], we only know of three publications thathave examined it [34,35,52].Alzen et al. [52] examined the association between LAsand DFW rates in introductory physics courses for 4941students at the University of Colorado Boulder over a10-year period. They analyzed the data using logisticregression models. The variables included in their finalmodel were gender, race, first-generation status, instructor,financial aid, high school GPA, credits at entry, admissiontest scores, LA exposure, and a placeholder variable for theinstructor with no interaction effects between the variables[53]. Their model found that students in LA-supportedcourses had lower DFW rates than their peers in non-LA-supported courses. The positive effects were present for allstudent demographic groups, with the white men having thesmallest predicted decrease (4% – – et al. [34] expandedtheir dataset to include 32 071 University of ColoradoBoulder students across multiple STEM disciplines over16 years. This investigation accounted for additionalstudent attributes, such as high school grade point average,credits at entry, and admission test scores. In creating theirlogistic regression models, they only included statisticallysignificant variables. This led to the removal of interactionterms between LAs and all demographic variables otherthan gender. Their final model predicted that, after account-ing for prior preparation, LA support predicted lower DFWrates across all demographic groups. Results showed thatmen had higher DFW rates than women and that DFW rateswere higher for nonwhite students and first-generationstudents. Unlike their prior investigation, they found thatmen had larger predicted decreases in DFW rates in LA-supported courses than women. Both of Alzen et al. ’ s studies[34,52] were situated in the same institutional context,limiting the generalizability of their findings. Specifically,the University of Colorado Boulder is a research intensiveuniversity and while it is a public institution, 41.7% [54] ofthe students come from out of state and only 15.7% ofstudent receive Pell Grants compared to 32% nationally. TheUniversity of Colorado Boulder is also the institution thatdeveloped the LA model. The students and instructors in thestudy had different resources to support their success thanmany of their peers in other institutions with LA programs.Close et al. [35] examined the associations betweenLAs and introductory physics student outcomes at TexasState University San Marcos (a Hispanic serving institu-tion). They found that over the seven years of data in theirstudy, the cohorts of students in LA-supported courses hadsemester-over-semester decreases in DFW rates. The DFWrate fell from its historical rate of 25% –
40% to below 20%in the final year of the study. They attributed the gradualdecrease in DFW rate to faculty gradually becomingproficient at implementing LA-supported pedagogies andfor the programmatic reforms to become institutionalized.These findings build on those of Alzen et al. [34] byexpanding the set of contexts that LAs have been associatedwith improvements in DFW rates.Researchers have identified other physics course featuresassociated with improvements in student DFW rates. Forexample, research has associated decreases in physicscourse DFW rates with course transformations that featuresmall group and sense-making activities [55] and the use of4-point grading scales instead of percentage grading scales[56]. Other researchers have focused on how studentfeatures, such as self-efficacy, can predict DFW rates.BEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16,
40% to below 20%in the final year of the study. They attributed the gradualdecrease in DFW rate to faculty gradually becomingproficient at implementing LA-supported pedagogies andfor the programmatic reforms to become institutionalized.These findings build on those of Alzen et al. [34] byexpanding the set of contexts that LAs have been associatedwith improvements in DFW rates.Researchers have identified other physics course featuresassociated with improvements in student DFW rates. Forexample, research has associated decreases in physicscourse DFW rates with course transformations that featuresmall group and sense-making activities [55] and the use of4-point grading scales instead of percentage grading scales[56]. Other researchers have focused on how studentfeatures, such as self-efficacy, can predict DFW rates.BEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16, et al. [57] found that studentswith higher levels of select self-efficacy subconstructspassed introductory physics courses at higher rates thantheir peers.Beyond the contexts of LAs and physics courses, someresearch investigated the relationship between supplemen-tal instructors (SI) [58] and DFW rates. SI is a near-peermodel where students who previously passed a course holdoptional sessions in which they lead small groups workingon testlike problems. Stanich et al. [59] examined achemistry program that had expanded on the traditionalSI support model to create a two-credit SI course thataccompanied a traditional introductory chemistry course.They found mixed results on the association between the SIsupport course and DFW rates for underrepresented racialand ethnic minorities. After controlling for background andpreparation variables, the students from underrepresentedracial and ethnic groups had roughly the same DFW rates inboth the control and SI groups. When controlling for whovolunteered to be in the SI group, however, they foundthat the students from underrepresented racial and ethnicgroups who were randomly accepted into the SI group mayhave had lower DFW rates than those in the non-SI group.The study lacked the statistical power to make any strongclaims about differences in DFW rates across demographicgroups.In a large-scale investigation of DFW rates across theSTEM disciplines at Northern Arizona University, Benfordand Gess-Newsome [60] found that women had loweraverage DFW rates than men in every course they exam-ined. They also examined race as a predictor for DFW rates.To have sufficient statistical power for their analysis, theyaggregated all of the STEM courses. In their descriptivestatistics, they found meaningful differences in DFW ratesbetween racial groups. White students had the lowest DFWrates (22%) and Native American (43%) and AfricanAmerican (40%) students had the highest DFW rates.While there were differences in the descriptive statisticsacross demographic groups, they excluded both gender andrace from their predictive models. The lack of studentdemographic variables in their models limited the abilityof their findings to identify inequities. They found thatstudent-centered courses, as measured by the research ofteaching observation protocol [61], had lower overall DFWrates. While the findings of Benford and Gess-Newsome[60] were not specific to physics, they indicate that there aredifferences in DFW rates across course types and demo-graphic groups.These studies show that student-centered interventionscan improve overall
DFW rates for physics students. Thesestudies also found consistent differences in DFW ratesfavoring white, female, continuing-generation students incollege science courses and physics courses. They providelittle guidance, however, on what interventions wouldimprove the equity of DFW rates for students in physics courses. In the present study, we build upon this prior workby using a critical perspective to investigate DFW ratesacross demographic groups and across introductory physicscourses that did and did not use LAs.
III. CONCEPTUAL FRAMEWORKA. Critical theory
Critical race theory (CRT) began in the 1970s and 1980sas a movement among U.S. legal scholars of varied racialbackgrounds to address social injustices and racial oppres-sion [62 – –
72] hasbeen taken up by scholars to examine the intersection ofracism and disabilities. To analyze and interpret our find-ings, we used a quantitative CRT (QuantCrit) [25,37,73]perspective.
B. QuantCrit
Critical research has historically used qualitativeapproaches to investigate the lived experiences of margin-alized people and the social processes that create racist,sexist, and classist power structures [25,28,73]. QuantCritemerged as a quantitative perspective [37] aligned with thecore principles of critical research. QuantCrit complementsqualitative studies by using large-scale data to representstudent outcomes in ways that reveal structural inequitiesthat reproduce injustices [37]. A QuantCrit perspective alsopushes researchers to identify where society fails tomeasure the outcomes for marginalized groups, such asour societies failure to look at pregnancy outcomes gen-erally and particularly for women of color [74]. Below, wedescribe three principles of QuantCrit [73] and the ways westrove to embody them in this investigation:(1)
The centrality of oppression . — We assumed thatracism, sexism, and classism are complex anddynamic processes present throughout society thatwe must explicitly examine lest our statisticalmodels legitimize existing inequities. Educationalinequities come from hegemonic power structurescreating educational and societal systems that caterto students from dominant groups. The continualmarginalization of specific student populationsASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16,
The centrality of oppression . — We assumed thatracism, sexism, and classism are complex anddynamic processes present throughout society thatwe must explicitly examine lest our statisticalmodels legitimize existing inequities. Educationalinequities come from hegemonic power structurescreating educational and societal systems that caterto students from dominant groups. The continualmarginalization of specific student populationsASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16,
Categories are neither natural nor given . — All dataare socially constructed and reflect the hegemonicpower structures that created them. Grades, forexample, are social constructs created by instructorsand codified by our educational systems. Howinstructors assign grades is an idiosyncratic processthat reflects the values and beliefs of individualinstructors and the power structures of their disci-pline and university, rather than an abstract truthabout a student.Our models aggregate students by race, gender,and class. These categories do not represent anynatural or scientific truth about students but aresocial constructs that maintain hegemonic powerstructures. The dynamic socially-negotiated naturesof race, gender, and class does not diminish the veryreal effects of racism, sexism, and classism asso-ciated with them. We strive to clarify that our modelsare not measuring innate difference in students basedon their race, gender, or first-generation (FG) status,but the impacts of multidimensional oppressivepower structures on students marginalized by thesesocial constructs. One way that we reflect this in ourwriting is through the explicit naming of racism,sexism, and classism in interpreting our models.(3)
Data are not neutral and cannot speak for them-self . — We reject the idea that data is neutral and canspeak for itself. Racist, sexist, and classist assump-tions can shape every stage of collecting, analyzing,and interpreting data [26]. All of the demographicdata used in our analysis reflected an institution ’ sattempt to categorize and quantify aspects of studentidentities that fell along spectrums. For example,CSU Chico only allowed students to identify as abinary gender in their university paperwork. Thispractice marginalized students who identify as transor gender nonconforming and limited our ability toexamine some dimensions of oppression. The in-stitutional classification of student racial identitiesand FG status were also socially constructed cat-egories with their own sets of assumptions. Moredetails on the demographic data are provided inSec. VA. In analyzing this data, we used methods that we felt produced the most meaningful repre-sentation of the impacts of racism, sexism, andclassism knowing that the data and methods wereimperfect.Some methods we used allowed us to create moreinclusive and nuanced findings. For example, ouruse of Akaike information criterion corrected (AICc)to select our models and not using p values tointerpret them allowed us to model and discussinequities in student outcomes that would have beenlost using more traditional methods. Other methods,however, were necessary to develop our models buthad clear limitations. For example, a small portion( < ) of our students data had no response forgender. We did not know if this happened through anerror in the campus systems or these students chosenot to identify a gender on their university forms.It is possible that some of this missing data arosefrom trans and gender nonconforming students nothaving a nonbinary option and choosing not toanswer the question. To account for this missingdata, we included an additional gender variable(Gen.unknown) in our statistical models. Becausethe number of students without gender data wassmall, the model ’ s predictions for the them are notmeaningful and we do not discuss them in ouranalysis of our findings. This solution had advantagesand drawbacks. The primary advantage was that itallowed us to include their data in the models withoutmaking any assumptions about their gender iden-tities. A drawback of this solution is that the group ’ soutcomes are excluded from our discussion of thefindings. In creating and interpreting our models, wedid our best to speak for the data in ways that identifyinjustices while acknowledging that our findingswere shaped by our own imperfect methods.To give voice to our findings, we operationalizedequity in two competing ways (see Sec. III C) andused them to interpret our findings from multipleperspectives.The underrepresentation of students from marginalizedgroups makes it difficult to collect large enough samples toinvestigate inequities with dependable statistical analyses.These challenges are exacerbated for studies that disag-gregate across intersecting marginalized identities, such asfor women of color. Some investigations incorrectly claimthey found no differences across demographic groupsbecause the analyses were underpowered and did not finddifferences with a p value below 0.05. Lack of a sta-tistically significant p value should not be confused withlack of a meaningful effect [76 – ’ intersectional identities andtheir learning outcomes. While our work ’ s foundations lieBEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16,
Data are not neutral and cannot speak for them-self . — We reject the idea that data is neutral and canspeak for itself. Racist, sexist, and classist assump-tions can shape every stage of collecting, analyzing,and interpreting data [26]. All of the demographicdata used in our analysis reflected an institution ’ sattempt to categorize and quantify aspects of studentidentities that fell along spectrums. For example,CSU Chico only allowed students to identify as abinary gender in their university paperwork. Thispractice marginalized students who identify as transor gender nonconforming and limited our ability toexamine some dimensions of oppression. The in-stitutional classification of student racial identitiesand FG status were also socially constructed cat-egories with their own sets of assumptions. Moredetails on the demographic data are provided inSec. VA. In analyzing this data, we used methods that we felt produced the most meaningful repre-sentation of the impacts of racism, sexism, andclassism knowing that the data and methods wereimperfect.Some methods we used allowed us to create moreinclusive and nuanced findings. For example, ouruse of Akaike information criterion corrected (AICc)to select our models and not using p values tointerpret them allowed us to model and discussinequities in student outcomes that would have beenlost using more traditional methods. Other methods,however, were necessary to develop our models buthad clear limitations. For example, a small portion( < ) of our students data had no response forgender. We did not know if this happened through anerror in the campus systems or these students chosenot to identify a gender on their university forms.It is possible that some of this missing data arosefrom trans and gender nonconforming students nothaving a nonbinary option and choosing not toanswer the question. To account for this missingdata, we included an additional gender variable(Gen.unknown) in our statistical models. Becausethe number of students without gender data wassmall, the model ’ s predictions for the them are notmeaningful and we do not discuss them in ouranalysis of our findings. This solution had advantagesand drawbacks. The primary advantage was that itallowed us to include their data in the models withoutmaking any assumptions about their gender iden-tities. A drawback of this solution is that the group ’ soutcomes are excluded from our discussion of thefindings. In creating and interpreting our models, wedid our best to speak for the data in ways that identifyinjustices while acknowledging that our findingswere shaped by our own imperfect methods.To give voice to our findings, we operationalizedequity in two competing ways (see Sec. III C) andused them to interpret our findings from multipleperspectives.The underrepresentation of students from marginalizedgroups makes it difficult to collect large enough samples toinvestigate inequities with dependable statistical analyses.These challenges are exacerbated for studies that disag-gregate across intersecting marginalized identities, such asfor women of color. Some investigations incorrectly claimthey found no differences across demographic groupsbecause the analyses were underpowered and did not finddifferences with a p value below 0.05. Lack of a sta-tistically significant p value should not be confused withlack of a meaningful effect [76 – ’ intersectional identities andtheir learning outcomes. While our work ’ s foundations lieBEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16, – p valuesin our findings [82], we focus on transparency by reportingthe point estimates and uncertainties from our models. Thismethod prevents our results from focusing solely on groupswell represented in the data but rather on differences thatwarrant attention. C. Operationalizing equity
Because data cannot speak for themself, we follow theadvice of Rodriguez et al. [83] and Stage [37] and opera-tionalized equity to interpret our findings from a social-justice perspective. Specifically, we offer two competingoperationalizations of equity that we used to interpret ourfindings: (i) equality of outcomes and (ii) equity of indi-viduality. We grounded these operationalizations of equityin the literature but we renamed them to ease the readers ’ interpretation and to align with Lee ’ s [84] definition ofequity and equality.Equality of outcomes occurs when students from differ-ent demographic groups have the same average achieve-ment at the end of a course regardless of their backgrounds.This perspective on equity has been called equity of parity[83,85] and equality on average across social groups [86].This perspective takes a strong social-justice stance byplacing the onus on the education system to allocateresources to eliminate inequalities and redress educationaldebts. In these scenarios, students from marginalizedgroups receive support that begins to repay their educa-tional debts by overcoming the impacts of prior injustices.Equity of individuality occurs when an interventionimproves the outcomes of students from marginalizedgroups [83]. This perspective gets away from makingcomparisons with white, middle-class students and whatGuti´errez and Dixon-Román [87] refer to as “ gap gazing. ” Rather, it focuses on research and interventions designed toadvance the needs of individuals who are marginalized as aresult of group identity. Guti´errez [88] argues that the focuson achievement gaps supports a deficit model of studentsfrom marginalized groups. By only focusing on margin-alized students, however, equity of individuality ignores theimpact of interventions on students from dominantgroups. By excluding students from dominant groups,equity of individuality may miss opportunities for interestconvergence that promote equitable practices, thereby exacerbating the existing educational debts. For this reason,our examination of equity of individuality included theassociations between LAs and outcomes for students fromdominant groups.
D. Positionality
Feminist theory has shown that all knowledge is markedby those who create it [89]. To be transparent about theposition of the researchers in this work in relation to thepower structures under investigation, we offer positionalitystatements [90] for each of the authors.The following is the first author ’ s, Ben Van Dusen,positionality statement. I identify as a White, cisgender,heterosexual, continuing-generation (CG) man with a colorvision deficiency. I was raised in a pair of lower-incomehouseholds but I now earn an upper-middle class income.I am an assistant professor and director of the LA programat the Hispanic serving institution where the study wasperformed. My experiences working with marginalizedstudents, particularly those whom I have had the honorto mentor as LAs and as researchers, have motivated myattempts to use my position and privilege to dismantleoppressive power structures. As someone who seeks to bean ally it is easy to overlook my own privileges. I try tobroaden my perspective through feedback from those withmore diverse lived experiences than my own.The following is the second author ’ s, Jayson Nissen,positionality statement. My identity as a White, cisgen-dered, heterosexual, nondisabled man has provided mewith power and opportunities denied to others in Americansociety. I use my experience growing up in a poor home andas a veteran of the all-male submarine service to motivatereflecting on and working to dismantle my privilege. Mywork on this project was shaped by the post-positivistscientific traditions I was educated in and my activist goalto pursue scientific knowledge that can help identify anddismantle policies and systems of oppression. Because I amnot a woman or a person of color and I now live in a higherincome household, I brought a limited perspective to thiswork on racism, sexism, and classism.To address potential homogeneity of author positionalityand perspective, we elicited feedback from a diverse set ofpeers and employed Radical Copy [91] to perform anequity audit of the publication. IV. RESEARCH QUESTION
This study investigated the intersectionality of racism,sexism, and classism in physics courses. Specifically, weexamined the associations between LA support in physicscourses and DFW rates of marginalized student popula-tions. To better understand these associations we addressedtwo research questions:(1) To what extent are Learning Assistants (LAs)associated with decreases in DFW rates overallfor students in introductory physics courses?ASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16,
This study investigated the intersectionality of racism,sexism, and classism in physics courses. Specifically, weexamined the associations between LA support in physicscourses and DFW rates of marginalized student popula-tions. To better understand these associations we addressedtwo research questions:(1) To what extent are Learning Assistants (LAs)associated with decreases in DFW rates overallfor students in introductory physics courses?ASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16,
V. METHODSA. Data collection
It was possible that the instructors who integrated LAsinto their courses had different DFW rates than instructorswho never integrated LAs into their courses. To control forpotential self-selection bias between these groups of instruc-tors, the analysis only included instructors who had taughtthe course with LA support for at least one semester (Fig. 1).We did not collect data from sections taught by instructorsthat never integrated LAs into any of their sections.The data for the study came from 9 instructors whotaught 41 sections of first semester introductory physics courses to 2312 students. The average DFW rate across allthe courses was 26.1% (Table I). Forty-two percent of thedata came from students in LA-supported courses. Five ofthe nine LA-using faculty in this study completed an onlinesurvey with open-ended questions about their use of LAs.The relevant questions on the survey were: Does havingLAs change the way you teach your course(s)? If so, how?The responses were unanimous in stating that in thesemesters they had LAs, the LAs allowed them to reducethe time spent lecturing and increase use of collaborativelearning activities. The details of what and how collabo-rative learning activities were implemented varied byinstructor.The LA-supported physics courses typically had a ratioof 24 students/LA. The demographics of the LAs in oursample was not known, but the overall LA population atChico State was similar to that of the student population.All of the LAs took the same LA pedagogy course duringtheir first semester as LAs. The LA pedagogy course,however, was modified over the course of the investigationto include more activities focused on helping the LAssupport marginalized student populations.When collecting data on student race, CSU Chico onlyallowed students to select a single response from a listof groups that combined race and ethnicity (e.g., White,Black, Hispanic, and Asian). Our sample size limited ourability to disaggregate many of the race categories in ourmodels. We combined all responses other than white into asingle category labeled Black, Indigenous, and people ofcolor (BIPOC). Within the BIPOC category, the responsesincluded Hispanic (52%), Non-Resident Alien (12%),Unknown (12%), and Asian (11%), two or more races(8%), Black/African American (3%), American Indian/Alaskan Native ( < ), and Pacific Islander ( < ). Wehad no way to determine the race of students who selectedNon-Resident Alien, unknown, or two or more races. TheDFW rates for Non-Resident Alien students were moder-ately above the overall averages and for the unknownstudents were slightly below the overall averages.The gender variable included three response categories:male, female, and unknown. The institution did not allowstudents to choose trans, gender nonconforming, nonbi-nary, and two-spirit students. The response categories, male FIG. 1. The three types of instructors in the physics department.To control for instructors self-selection into using LAs, we onlyanalyzed data for courses taught by the occasional and consistentLA-using faculty.TABLE I. Descriptive statistics for the first semester courses in each of the two introductory physics sequences.Courses StudentsGender Race FG statusMath-basis Instructors Sections DFW(%) Total LA supported(%) Men(%) Women(%) Unknown(%) White(%) BIPOC(%) FG(%) CG(%)Algebra 3 9 20.9 862 74.8 57.4 41.9 0.7 46.5 53.5 50.1 49.9Calculus 6 32 29.2 1450 22.4 84.4 14.6 1.0 39.7 60.3 43.1 56.9Total 9 41 26.1 2312 42.0 74.4 24.8 0.9 42.2 57.8 45.7 54.3
BEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16,
BEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16,
B. Data analysis
To analyze the data, we generated hierarchical general-ized linear models that predicted DFW outcomes while accounting for the nested structure of our dataset (studentsin sections) using the hglm [92] and lme4 [93] packages inRStudio v.3.5.1 [94]. Our hglm model parameters were fitusing the extended quasilikelihood method. To determinewhich model was the best fit for our data, we used thedredge function in the MuMin package [95] to calculate thecorrected Akaike information criterion (AICc) [96] for eachcombination of the variables. We used the model with thelowest AICc score as our final model. We used the AICcscore, rather than variance explained, to select our finalmodel for several reasons. AICc scores take into accountthe explanatory power of each variable without overlyweighting model parsimony. While it is a common practiceto use variance explained to select a final model (e.g., ourown prior work [45,97]), using it to select models wheninvestigating marginalized populations risks the modelsfalling prey to a tyranny of the masses. Using predictedvariance to select variables risks excluding marginalizedstudents from groups with small representations in the databecause the variance explained by a variable is proportionalto its the sample size.To examine whether the nested structure of our datanecessitated the use of hierarchical models, we generatedan unconditional model without predictor variables andcalculated the intraclass correlation [98]. We found 8.6% ofthe variance at the course level; therefore, the best practicewas to account for the hierarchical structure of the data inour model [97].To generate our model for equity in DFW rates, webegan by creating a model that included use of LAs as theonly predictor variable so we could model the overall DFWrate in the introductory physics courses with and withoutLAs. We then explored using a level-2 course variable(use of LAs) and level-1 student demographic variables(race, gender, FG status) and the interaction effectsbetween the student demographic variables with each otheras well as with the use of LAs. Our examination of AICcidentified the model that included some two-way inter-actions, but not three-way or four-way interactions as ourbest model of equity in DFW rates. The followingequations describe the models of overall DFW rates andequity in DFW rates.
1. Overall model
Level-1 equations (student level)DFW ij ¼ β j þ r ij Level-2 equations (course level) β j ¼ γ þ γ × LA þ μ j TABLE II. Descriptive statistics disaggregated by demographicsand instruction.Gender Race FG status Instruction type N DFW rate(%)Men BIPOC FG Traditional 316 37.7LA 218 28.9CG Traditional 316 42.1LA 123 17.9White FG Traditional 134 30.6LA 86 22.1CG Traditional 314 21.7LA 212 13.2Women BIPOC FG Traditional 86 26.7LA 135 17.8CG Traditional 73 27.4LA 69 10.1White FG Traditional 33 21.2LA 49 16.3CG Traditional 70 20.0LA 78 10.3Unknown All All Traditional 13 17.9LA 7 0
ASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16,
ASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16, . Equity model
Level-1 equations (student level)DFW ij ¼ β j þ β j ×Woman þ β j ×Gen : unknown þ β j ×BIPOC þ β j ×FG þ β j ×BIPOC×FG þ r ij Level-2 equations (course-level) β j ¼ γ þ γ × LA þ μ j β j ¼ γ β j ¼ γ β j ¼ γ þ γ × LA β j ¼ γ þ γ × LA β j ¼ γ To help interpret the uncertainty around the predictionsof our models, we included standard error values for eachcoefficient. We do not include p values because of theirconsistent misuse in the sciences [76,78] and in research onequity [99]. Using p values to examine equity is, in part,problematic because scientists often interpret them as gono-go tests. Since p values are sample size dependent, theycan show that large and meaningful effects for small groupsof students are not statistically significant. Scientists andscience consumers often think a result that is not sta-tistically significant is not meaningful, but this is incor-rect [78].The statistical packages we used to generate the pre-dicted DFW rates for each demographic group (Table IIIand Fig. 2) in the investigation of equity of outcomes allowed us to easily calculate the uncertainty for eachestimate using the standard errors. However, our inves-tigation of equity of individuality examined the differencebetween predicted DFW rates (Fig. 3). This simple sub-traction complicated the calculation of standard errors. Toaddress this challenge, we used bootstrapping [100] with1000 subsamples resampled at the section level to estimatethese means and standard errors for the differences betweenstudent DFW rates in LA and traditional courses. FIG. 2. Predicted DFW rates for each group of students with and without LAs, after accounting for instructor types. Error barsrepresent þ = − S : E : . FIG. 3. Predicted DFW rate decreases for each demographicgroup in LA-supported courses versus traditional courses, ac-counting for instructor types. Error bars represent 95% confidenceintervals. BEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16,
Level-1 equations (student level)DFW ij ¼ β j þ β j ×Woman þ β j ×Gen : unknown þ β j ×BIPOC þ β j ×FG þ β j ×BIPOC×FG þ r ij Level-2 equations (course-level) β j ¼ γ þ γ × LA þ μ j β j ¼ γ β j ¼ γ β j ¼ γ þ γ × LA β j ¼ γ þ γ × LA β j ¼ γ To help interpret the uncertainty around the predictionsof our models, we included standard error values for eachcoefficient. We do not include p values because of theirconsistent misuse in the sciences [76,78] and in research onequity [99]. Using p values to examine equity is, in part,problematic because scientists often interpret them as gono-go tests. Since p values are sample size dependent, theycan show that large and meaningful effects for small groupsof students are not statistically significant. Scientists andscience consumers often think a result that is not sta-tistically significant is not meaningful, but this is incor-rect [78].The statistical packages we used to generate the pre-dicted DFW rates for each demographic group (Table IIIand Fig. 2) in the investigation of equity of outcomes allowed us to easily calculate the uncertainty for eachestimate using the standard errors. However, our inves-tigation of equity of individuality examined the differencebetween predicted DFW rates (Fig. 3). This simple sub-traction complicated the calculation of standard errors. Toaddress this challenge, we used bootstrapping [100] with1000 subsamples resampled at the section level to estimatethese means and standard errors for the differences betweenstudent DFW rates in LA and traditional courses. FIG. 2. Predicted DFW rates for each group of students with and without LAs, after accounting for instructor types. Error barsrepresent þ = − S : E : . FIG. 3. Predicted DFW rate decreases for each demographicgroup in LA-supported courses versus traditional courses, ac-counting for instructor types. Error bars represent 95% confidenceintervals. BEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16,
I. FINDINGS
The findings section presents the outputs of DFWmodels and our interpretation of those findings for equalityof outcomes and equity of individuality. The section beginswith a table of the coefficients for the two DFW models(Table IV). The coefficients are logits that are not easilyinterpretable, particularly in isolation. To help make senseof the model outputs, we included plots of the model ’ spredicted DFW rates for each group (Fig. 2) and thedifference between a group in LA versus traditional courses(Fig. 3). A. LAs and overall DFW rates
Our first research question was: to what extent arelearning assistants (LAs) associated with decreases inDFW rates overall for students in introductory physicscourses? The overall model shows a strong associationbetween LAs and lowered DFW rates. When taught by thesame instructor, courses in the introductory physicssequence have predicted DFW rates of 31.1% withoutLAs and 17.1% with LAs (Fig. 2). This difference wasmuch larger than the uncertainty in the measurements.
B. LAs and equality of outcomes
Our second research question was: to what extent areshifts in DFW rates associated with LA support in intro-ductory physics courses mitigating the impacts of racism,sexism, and classism? To answer that question we definedequity in two ways. The first definition was equality ofoutcomes, in which we compared the differences in pre-dicted DFW rates for each demographic group in traditionalcourses and again in LA-supported courses. The model forequity in DFW rates showed meaningful differences inpredicted DFW rates across demographic groups in bothcontexts (Fig. 2). In traditional courses, predicted DFW ratesranged from 40% (CG men of color) to 16% (CG Whitewomen). In LA-using courses, predicted DFW rates rangedfrom 25% (FG men of color) to 12% (CG White women).Regardless of course type, being a man, a BIPOC, or a FGstudent were all associated with higher predicted DFW rates.With only the exception of FG White men in LA-supportedcourses, BIPOC students had the highest predicted DFWrates regardless of FG status or course type. As we discussbelow, these results contrasted some of our expectationsbased on the role of hegemonic power structures in educa-tional debts. First, women were less likely than men toreceive DFW grades no matter the type of instruction.Second, the relationship between FG status and DFW gradesvaried between White students and BIPOC students acrossinstructional types. The data did not indicate an additionaleducation debt for FG men and women of color in traditionalcourses. The differences in predicted DFW rates weregreatly decreased in LA-supported courses. For example,CG women of color, White men, and White women all hadpredicted DFW rates that ranged from 12% –
13% in LAsupported courses. While there was meaningful progresstoward equity of outcomes, some inequities persisted in LA-supported courses showing that equity of outcomes did notoccur in either course context.
C. LAs and equity of individuality
To answer our second research question from anotherperspective we used a second definition of equity, which wecalled equity of individuality. To determine if equity ofindividuality occurred in LA-supported courses, we com-pared the differences in predicted DFW rates for each
TABLE III. Predicted DFW probabilities by student demo-graphic, accounting for instructors. DFW One S.E.Gender Race FG Status Inst. Rate (%) Range (%)Men BIPOC FG Trad. 37.8 34.7 – – – – – – – – – – – – – – – – β j Intercept, γ − . − . γ − . − . β j Intercept, γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) − . β j Intercept, γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) − . β j Intercept, γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) − . β j Intercept, γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) β j Intercept, γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) − . ASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16,
TABLE III. Predicted DFW probabilities by student demo-graphic, accounting for instructors. DFW One S.E.Gender Race FG Status Inst. Rate (%) Range (%)Men BIPOC FG Trad. 37.8 34.7 – – – – – – – – – – – – – – – – β j Intercept, γ − . − . γ − . − . β j Intercept, γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) − . β j Intercept, γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) − . β j Intercept, γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) − . β j Intercept, γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) β j Intercept, γ (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) − . ASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16,
VII. DISCUSSION
The models identified educational debts incurred byracist and classist power structures impacting studentoutcomes in both traditional and LA-supported courses.However, these models did not identify inequities in DFWrates against women. The equity of individuality analysis,however, predicted lower DFW rates, up to 22 percentagepoints, for students from marginalized groups in LA-supported courses. This large decrease in DFW rates acrossall groups indicated the integration of LAs into thesephysics courses achieved an equity of individuality. Interms of passing the course, students from marginalizedgroups have better outcomes in LA-supported courses.The decreases in DFW rates associated with LAs werenot equal across all marginalized groups. The largestpredicted decreases in DFW rates in LA-supported coursesoccurred for BIPOC independent of gender (Fig. 3). First-generation (FG) students, however, had smaller predicteddecreases in DFW rates associated with LAs than their CGpeers. These results indicate adding LA support to a coursemay not be sufficient to address the role of classism instudent outcomes.The equity model also found an interaction between FGand BIPOC variables indicating a larger decrease in DFWrates for FG BIPOC students than a model not accountingfor these interactions would have predicted. This highlightsthe importance of taking an intersectional approach thatdoes not assume that the impacts of racism, sexism, andclassism are additive [79]. White and BIPOC studentslikely experience classism differently in their educationalexperience. FG status may also act differently as a proxy ofclass for White and BIPOC students. Understanding therole of classism, and how to investigate it, in physicseducation will take a concerted effort from our community.Class is a taboo subject in the United States. We mustprotect students privacy and their right to informed consent.However, we must also make sure that these values are notco-opted by education systems to hide injustices. While the equity model associated LA support withequity of individuality (i.e., improved outcomes for mar-ginalized students), they did not associate LA support withequality of outcomes (i.e., equal outcomes across demo-graphic groups). For example, the equity model predicted25% of FG men of color received DFW grades in LA-supported courses while only 12% of CG White womenreceived DFW grades. While the model predicted lowerDFW rates in LA-supported courses, the inequities iden-tified educational debts due to racism and classism instudent failure rates in LA-supported courses. The lowerDFW rate for women indicates these courses did notincrease educational debts due to sexism.A major difference between sexism and racism andclassism is that Title IX of the Education Amendments of1972 legally protects women against discrimination. Nosimilar legal protections exist across races and classes.Grades provide an accessible metric of systemic sexismwithin a course, program, or institution. Institutional andsocial pressures around sexism may mean that sexism had asmaller role in these physics courses or it may mean thatgrades no longer reflect the sexism women experienced inthese courses. For example, Seymour [9] found that womenwere highly capable but the culture of many science andengineering courses (e.g., competitive, cutthroat, and hos-tile) pushed women to leave STEM degrees. Research alsoshows many women experience microaggressions andblatant sexism in their physics education [10,11], and thatstereotypical environments ’ lack of representation ofwomen undermine women ’ s sense of belonging and per-formance in STEM learning environments [101]. In theirreview of the literature on gender differences in represen-tation across STEM discipline, Cheryan et al. [102] foundthat masculine cultures, lack of early experiences for girls,and gender differences in self-efficacy explain the largegender differences in participation in physics, engineering,and computer science. The difference in gender represen-tation between the algebra- and calculus-based coursesillustrate these differences in participation across STEMfields within the data used in this study. Better under-standing the way students experience sexism, racism, andclassism in introductory physics courses and the role of LAprograms in mitigating those experiences requires furtherstudy with a diversity of methods. A single metric and asingle study is insufficient for claiming the elimination ofsexism in a course when that course is situated in a field anda society with a long history of sexism.Because of a lack of data on prior preparation, ourmodels did not account for educational debts that studentsbrought into the physics courses. We have no reason tobelieve, however, there were differences in the educationaldebts between students in the LA-supported and traditionalsections at the start of the semester.Without data on prior preparation we cannot account forthe role of sexist power structures beyond the course andBEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16,
The models identified educational debts incurred byracist and classist power structures impacting studentoutcomes in both traditional and LA-supported courses.However, these models did not identify inequities in DFWrates against women. The equity of individuality analysis,however, predicted lower DFW rates, up to 22 percentagepoints, for students from marginalized groups in LA-supported courses. This large decrease in DFW rates acrossall groups indicated the integration of LAs into thesephysics courses achieved an equity of individuality. Interms of passing the course, students from marginalizedgroups have better outcomes in LA-supported courses.The decreases in DFW rates associated with LAs werenot equal across all marginalized groups. The largestpredicted decreases in DFW rates in LA-supported coursesoccurred for BIPOC independent of gender (Fig. 3). First-generation (FG) students, however, had smaller predicteddecreases in DFW rates associated with LAs than their CGpeers. These results indicate adding LA support to a coursemay not be sufficient to address the role of classism instudent outcomes.The equity model also found an interaction between FGand BIPOC variables indicating a larger decrease in DFWrates for FG BIPOC students than a model not accountingfor these interactions would have predicted. This highlightsthe importance of taking an intersectional approach thatdoes not assume that the impacts of racism, sexism, andclassism are additive [79]. White and BIPOC studentslikely experience classism differently in their educationalexperience. FG status may also act differently as a proxy ofclass for White and BIPOC students. Understanding therole of classism, and how to investigate it, in physicseducation will take a concerted effort from our community.Class is a taboo subject in the United States. We mustprotect students privacy and their right to informed consent.However, we must also make sure that these values are notco-opted by education systems to hide injustices. While the equity model associated LA support withequity of individuality (i.e., improved outcomes for mar-ginalized students), they did not associate LA support withequality of outcomes (i.e., equal outcomes across demo-graphic groups). For example, the equity model predicted25% of FG men of color received DFW grades in LA-supported courses while only 12% of CG White womenreceived DFW grades. While the model predicted lowerDFW rates in LA-supported courses, the inequities iden-tified educational debts due to racism and classism instudent failure rates in LA-supported courses. The lowerDFW rate for women indicates these courses did notincrease educational debts due to sexism.A major difference between sexism and racism andclassism is that Title IX of the Education Amendments of1972 legally protects women against discrimination. Nosimilar legal protections exist across races and classes.Grades provide an accessible metric of systemic sexismwithin a course, program, or institution. Institutional andsocial pressures around sexism may mean that sexism had asmaller role in these physics courses or it may mean thatgrades no longer reflect the sexism women experienced inthese courses. For example, Seymour [9] found that womenwere highly capable but the culture of many science andengineering courses (e.g., competitive, cutthroat, and hos-tile) pushed women to leave STEM degrees. Research alsoshows many women experience microaggressions andblatant sexism in their physics education [10,11], and thatstereotypical environments ’ lack of representation ofwomen undermine women ’ s sense of belonging and per-formance in STEM learning environments [101]. In theirreview of the literature on gender differences in represen-tation across STEM discipline, Cheryan et al. [102] foundthat masculine cultures, lack of early experiences for girls,and gender differences in self-efficacy explain the largegender differences in participation in physics, engineering,and computer science. The difference in gender represen-tation between the algebra- and calculus-based coursesillustrate these differences in participation across STEMfields within the data used in this study. Better under-standing the way students experience sexism, racism, andclassism in introductory physics courses and the role of LAprograms in mitigating those experiences requires furtherstudy with a diversity of methods. A single metric and asingle study is insufficient for claiming the elimination ofsexism in a course when that course is situated in a field anda society with a long history of sexism.Because of a lack of data on prior preparation, ourmodels did not account for educational debts that studentsbrought into the physics courses. We have no reason tobelieve, however, there were differences in the educationaldebts between students in the LA-supported and traditionalsections at the start of the semester.Without data on prior preparation we cannot account forthe role of sexist power structures beyond the course andBEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16, VIII. CONCLUSIONS AND IMPLICATIONS
The decreased DFW rate in LA-supported course raisesquestions about how LA support relates to the lower DFWrates. The courses with LAs could have given more passinggrades because they used relaxed grading criteria or usedtests that gave higher average scores. We attempted tomitigate this possibility by excluding instructors who hadnever used LAs in the course. In their responses to aquestion about their grading practices in the online surveyadministered to the instructors, none reported making shiftsin their grading criteria or assessments in the semester theircourses were LA supported. Future work can test thepossibility that DFW rates improved because of morelenient grading by comparing the grade structures and taskdifficulties when instructors do and do not have LAs. Asecond possibility is that the students learned more in thecourses with LAs. This is supported by the instructorsreporting using more student-centered collaborative teach-ing techniques when their courses included LAs. Theseteaching techniques support greater conceptual learningthan teacher-centered lecture [45,103,104]. Van Dusen andNissen [45] found that collaborative instructional practicesincreased conceptual learning across all demographicgroups but that it did not eliminate differences acrossdemographic groups. Greater conceptual learning in theLA-supported courses would have resulted in higher gradesif the grading practices were similar between traditional andLA-supported courses. Future work can test the possibilitythat DFW rates improved because of increased learning bycomparing student concept inventory data when instructorsdo and do not have LAs.The decreases in DFW rates associated with LAs aremeaningful and reduce inequities in physics student out-comes. The models associated LA support with lower DFWrates for all groups of students and the largest decreasesoccurred for BIPOC students. While LA support lead toequity of individuality, it did not eliminate differencesacross demographic groups. LA-supported courses did notachieve equality of outcomes. Achieving equality of out-comes through a single intervention is unlikely because itrequires not only eliminating the impacts of racism, sexism,and classism within the course but also mitigating theimpacts of educational debts incurred before and outside ofthe course. This LA program did not eliminate inequitiesbut was associated with mitigation of some educationaldebts owed marginalized students. Continued investiga-tions within this LA program can aid in further addressingthese injustices.Within each gender and racial group the FG students hadsmaller decreases in DFW rates associated with LAs. Ourdata cannot speak to whether these inequities arise fromdifferences in the needs and resources of FG students, how LAs and FG students interact, or through other mecha-nisms. We are not aware of any large-scale investigations ofthe experiences of FG physics students that could informour conclusions.LA programs represent an appealing tool to improveequity on a large scale because LA programs create aninterest convergence between marginalized students andthose with power. Interest convergence in CRT [66] arguesthat improved outcomes for marginalized students is notsufficient motivation to create change within existing powerstructures. Only when those with power see a program asbenefiting them or their interest groups will the programgarner the resources to scale and sustain it. As both studentsfrom historically marginalized communities and studentsfrom dominant identity groups performed better in LA-supported courses and LA programs offer prestige for aninstitution, LA programs are well positioned to solicitsupport from those with the power to fund them. Theeffects of this interest convergence are illustrated by over300 institutions joining the LA Alliance. The LA model hasthe potential for widespread improvement of equity inSTEM courses.There is a danger, however, that LA programs couldincrease systemic inequities. LA programs require resour-ces including paying LAs and faculty and staff time andexpertise to run the program and teach the pedagogycourse. The institutions with the greatest resources tendto primarily serve White, middle-upper class students. Ifwell resourced institutions disproportionately adopt LAprograms, those programs will perpetuate existing racistand classist power structures by disproportionately ben-efiting White, middle-upper class students. For LA pro-grams to counteract injustices at the national level theyneed support at institutions serving marginalized students,such as Hispanic serving institutions, minority servinginstitutions, historically black colleges and universities,and two-year colleges.
IX. LIMITATIONS AND FUTURE RESEARCH
This study only represents outcomes for students inintroductory physics at a single institution using a singletype of course transformation. Increasing the scale of thiswork to include other near-peer programs (e.g., supple-mental instruction), physics courses, STEM disciplines,and institutions will inform the extent of inequitiesacross institutions and disciplines and the extent to whichdifferent course transformations address those inequities.Accounting for differences in prior preparation acrossinstitutions and course types can reveal oppressive powerstructures may reveal patterns of oppression hidden withindata from a single institution. The study also lacked thestatistical power to disaggregate across racial groups.Most BIPOC in this study self-identified as Hispanic.The impact of racism can vary widely across racial groupsand settings [26,28,45]. Collecting data spanning courseASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16,
This study only represents outcomes for students inintroductory physics at a single institution using a singletype of course transformation. Increasing the scale of thiswork to include other near-peer programs (e.g., supple-mental instruction), physics courses, STEM disciplines,and institutions will inform the extent of inequitiesacross institutions and disciplines and the extent to whichdifferent course transformations address those inequities.Accounting for differences in prior preparation acrossinstitutions and course types can reveal oppressive powerstructures may reveal patterns of oppression hidden withindata from a single institution. The study also lacked thestatistical power to disaggregate across racial groups.Most BIPOC in this study self-identified as Hispanic.The impact of racism can vary widely across racial groupsand settings [26,28,45]. Collecting data spanning courseASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16, et al. [26]. The quantitative data can providethe statistical power for a large-scale QuantCrit analysis and the testimonies can provide the rich descriptions ofstudents ’ lived experiences. ACKNOWLEDGMENTS
This work was funded in part by NSF-IUSE GrantsNo. DUE-1525338 and No. 1928596 and is contributionNo. LAA-065 of the Learning Assistant Alliance. We wouldlike to thank McKensie Mack for leading the equity audit onthis manuscript. We would also like to thank Tray Robinsonand Ian Her Many Horses for their feedback on our work. Weacknowledge and are mindful that the location where ourresearch was performed stands on lands that were originallyoccupied by the first people of this area, the Mechoopda, andwe recognize their distinctive spiritual relationship with thisland and the waters that run through the campus. [1] M. R. Jeffreys, Tracking students through program entry,progression, graduation, and licensure: Assessing under-graduate nursing student retention and success, NurseEduc. Today , 406 (2007).[2] M. Kiss, The California State University bottleneckcourses survey report, J. Collective Bargaining Acad., 2(2014).[3] N. W. Klingbeil, K. S. Rattan, M. L. Raymer, D. B.Reynolds, and R. Mercer, Engineering mathematicseducation at Wright State University: Uncorking the firstyear bottleneck (2007), https://corescholar.libraries.wright.edu/cgi/viewcontent.cgi?article=2411&context=knoesis.[4] C. Moore and N. Shulock, Student Progress TowardDegree Completion: Lessons from the Research Liter-ature et al. , Minority Serving Institutions: America ’ s Underu-tilized Resource for Strengthening the STEM Workforce (National Academies Press, Washington, DC, 2019).[7] X. Chen, First-generation students in postsecondaryeducation: A look at their college transcripts (2005),https://vtechworks.lib.vt.edu/handle/10919/84052.[8] K. Kovacs, When a C isn ’ Talking about Leaving: Why Undergradu-ates Leave the Sciences (Westview Press, Boulder, CO,1997). [10] L. M. Aycock, Z. Hazari, E. Brewe, K. B. H. Clancy, T.Hodapp, and R. M. Goertzen, Sexual harassment reportedby undergraduate female physicists, Phys. Rev. Phys.Educ. Res. , 010121 (2019).[11] R. S. Barthelemy, M. McCormick, and C. Henderson,Gender discrimination in physics and astronomy: Gradu-ate student experiences of sexism and gender micro-aggressions, Phys. Rev. Phys. Educ. Res. , 020119(2016).[12] D. Dortch and C. Patel, Black undergraduate women andtheir sense of belonging in stem at predominantly whiteinstitutions, NASPA J. Women Higher Educ. , 202(2017).[13] A. C. Johnson, Women, race, and science: The academicexperiences of twenty women of color with a passion forscience (2001), https://ui.adsabs.harvard.edu/abs/2001PhDT.......150J/abstract.[14] A. Johnson, Policy implications of supporting women ofcolor in the sciences, J. Women Politics Policy , 135(2006).[15] E. O. McGee and L. Bentley, The troubled success ofblack women in stem, Cognit. Instr. , 265 (2017).[16] E. McPherson, Oh you are smart: Young, gifted AfricanAmerican women in stem majors, J. Women MinoritiesSci. Engin. , 1 (2017).[17] M. Ong, Body projects of young women of color inphysics: Intersections of gender, race, and science, SocialProblems , 593 (2005).[18] S. Hyater-Adams, C. Fracchiolla, N. Finkelstein,and K. Hinko, Critical look at physics identity: Anoperationalized framework for examining race and phys-ics identity, Phys. Rev. Phys. Educ. Res. , 010132(2018).[19] In this article we use the term historically margina-lized to represent both historic and continuing margin-alization. BEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16,
This work was funded in part by NSF-IUSE GrantsNo. DUE-1525338 and No. 1928596 and is contributionNo. LAA-065 of the Learning Assistant Alliance. We wouldlike to thank McKensie Mack for leading the equity audit onthis manuscript. We would also like to thank Tray Robinsonand Ian Her Many Horses for their feedback on our work. Weacknowledge and are mindful that the location where ourresearch was performed stands on lands that were originallyoccupied by the first people of this area, the Mechoopda, andwe recognize their distinctive spiritual relationship with thisland and the waters that run through the campus. [1] M. R. Jeffreys, Tracking students through program entry,progression, graduation, and licensure: Assessing under-graduate nursing student retention and success, NurseEduc. Today , 406 (2007).[2] M. Kiss, The California State University bottleneckcourses survey report, J. Collective Bargaining Acad., 2(2014).[3] N. W. Klingbeil, K. S. Rattan, M. L. Raymer, D. B.Reynolds, and R. Mercer, Engineering mathematicseducation at Wright State University: Uncorking the firstyear bottleneck (2007), https://corescholar.libraries.wright.edu/cgi/viewcontent.cgi?article=2411&context=knoesis.[4] C. Moore and N. Shulock, Student Progress TowardDegree Completion: Lessons from the Research Liter-ature et al. , Minority Serving Institutions: America ’ s Underu-tilized Resource for Strengthening the STEM Workforce (National Academies Press, Washington, DC, 2019).[7] X. Chen, First-generation students in postsecondaryeducation: A look at their college transcripts (2005),https://vtechworks.lib.vt.edu/handle/10919/84052.[8] K. Kovacs, When a C isn ’ Talking about Leaving: Why Undergradu-ates Leave the Sciences (Westview Press, Boulder, CO,1997). [10] L. M. Aycock, Z. Hazari, E. Brewe, K. B. H. Clancy, T.Hodapp, and R. M. Goertzen, Sexual harassment reportedby undergraduate female physicists, Phys. Rev. Phys.Educ. Res. , 010121 (2019).[11] R. S. Barthelemy, M. McCormick, and C. Henderson,Gender discrimination in physics and astronomy: Gradu-ate student experiences of sexism and gender micro-aggressions, Phys. Rev. Phys. Educ. Res. , 020119(2016).[12] D. Dortch and C. Patel, Black undergraduate women andtheir sense of belonging in stem at predominantly whiteinstitutions, NASPA J. Women Higher Educ. , 202(2017).[13] A. C. Johnson, Women, race, and science: The academicexperiences of twenty women of color with a passion forscience (2001), https://ui.adsabs.harvard.edu/abs/2001PhDT.......150J/abstract.[14] A. Johnson, Policy implications of supporting women ofcolor in the sciences, J. Women Politics Policy , 135(2006).[15] E. O. McGee and L. Bentley, The troubled success ofblack women in stem, Cognit. Instr. , 265 (2017).[16] E. McPherson, Oh you are smart: Young, gifted AfricanAmerican women in stem majors, J. Women MinoritiesSci. Engin. , 1 (2017).[17] M. Ong, Body projects of young women of color inphysics: Intersections of gender, race, and science, SocialProblems , 593 (2005).[18] S. Hyater-Adams, C. Fracchiolla, N. Finkelstein,and K. Hinko, Critical look at physics identity: Anoperationalized framework for examining race and phys-ics identity, Phys. Rev. Phys. Educ. Res. , 010132(2018).[19] In this article we use the term historically margina-lized to represent both historic and continuing margin-alization. BEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16,
20] P. Hill Collins and S. Bilge,
Intersectionality (John Wiley& Sons, New York, NY, 2016).[21] K. Crenshaw, Mapping the margins: Intersectionality,identity politics, and violence against women of color,Stan. Law Rev. , 1241 (1990).[22] K. Steinmetz, She coined the term intersectionality over30 years ago. heres what it means to her today, TimeMagazine (2020).[23] In this publication we capitalize all races, includingWhite, to emphasize that there is no default race andthat they are all associated sets of cultural practices.[24] J. F. Dovidio, A. Gluszek, M.-S. John, R. Ditlmann,and P. Lagunes, Understanding bias toward latinos:Discrimination, dimensions of difference, and experienceof exclusion, Journal of Social Issues , 59 (2010).[25] N. López, C. Erwin, M. Binder, and M. J. Chavez,Making the invisible visible: Advancing quantitativemethods in higher education using critical race theoryand intersectionality, Race Ethnicity Educ. , 180(2018).[26] A. Covarrubias, P. E. Nava, A. Lara, R. Burciaga, V. N.V´elez, and D. G. Solorzano, Critical race quantitativeintersections: A testimonio analysis, Race Ethnicity Educ. , 253 (2018).[27] A. Covarrubias, Quantitative intersectionality: A criticalrace analysis of the Chicana/o educational pipeline,J. Latinos Educ. , 86 (2011).[28] S. Tae Jang, The implications of intersectionality onSoutheast Asian female students educational outcomesin the United States: A critical quantitative intersection-ality analysis, Am. Educ. Res. J. , 1268 (2018).[29] D. Joy P´erez, L. Fortuna, and M. Alegria, Prevalence andcorrelates of everyday discrimination among us latinos,J. Commun. Psychol. , 421 (2008).[30] V. Otero, S. Pollock, R. McCray, and N. Finkelstein, Whois responsible for preparing science teachers?, Science , 445 (2006).[31] V. Otero, S. Pollock, and N. Finkelstein, A physicsdepartments role in preparing physics teachers: TheColorado learning assistant model, Am. J. Phys. ,1218 (2010).[32] J. H. Van Driel, N. Verloop, and W. De Vos, Developingscience teachers ’ pedagogical content knowledge, J. Res.Sci. Teach. , 673 (1998).[33] E. Etkina, Pedagogical content knowledge and prepara-tion of high school physics teachers, Phys. Rev. ST Phys.Educ. Res. , 020110 (2010).[34] J. L. Alzen, L. S. Langdon, and V. K. Otero, Alogistic regression investigation of the relationship be-tween the learning assistant model and failure rates inintroductory stem courses, Int. J. STEM Educ. , 56(2018).[35] E. W. Close, J.-M. Mailloux-Huberdeau, H. G. Close, andD. Donnelly, Characterization of time scale for detectingimpacts of reforms in an undergraduate physics program,in Proceedings of the 2017 Physics Education ResearchConference, Cincinnati, OH , edited by L. Ding, A.Traxler, and Y. Cao (AIP, New York, 2018).[36] S. Kanim and X. C. Cid, The demographics of physicseducation research, arXiv:1710.02598. [37] F. K. Stage, Answering critical questions using quantita-tive data, New Dir. Inst. Res. , 5 (2007).[38] X. Herrera, J. Nissen, and B. Van Dusen, Student out-comes across collaborative learning environments, in
Proceedings of the 2017 Physics Education ResearchConference , Washington, DC, edited by A. Traxler, Y.Cao, and S. Wolf (AIP, New York, 2018).[39] N. D. Finkelstein, V. Otero, and S. J. Pollock, Teaching tolearn: The Colorado Learning Assistant programs impacton learning content, Forum Educ. , 11 (2007).[40] S. J. Pollock and N. D. Finkelstein, Sustaining educa-tional reforms in introductory physics, Phys. Rev. STPhys. Educ. Res. , 010110 (2008).[41] R. M. Goertzen, E. Brewe, L. H. Kramer, L. Wells, and D.Jones, Moving toward change: Institutionalizing reformthrough implementation of the learning assistant modeland open source tutorials, Phys. Rev. ST Phys. Educ. Res. , 020105 (2011).[42] N. McHenry, A. Martin, A. Castaldo, and D. Ziegenfuss,Learning assistants program: Faculty development forconceptual change, Int. J. Teach. Learn. Higher Educ. ,258 (2010).[43] K. E. Gray, D. C. Webb, and V. K. Otero, Effects of thelearning assistant model on teacher practice, Phys. Rev.Phys. Educ. Res. , 020126 (2016).[44] K. E. Gray and V. K. Otero, Analysis of former learningassistants views on cooperative learning, AIP Conf. Proc. , 149 (2009).[45] B. Van Dusen and J. Nissen, Equity in college physicsstudent learning: A critical quantitative intersectionalityinvestigation, J. Res. Sci. Teach. , 33 (2019).[46] B. Van Dusen and J. M. Nissen, Systemic inequities inintroductory physics courses: the impacts of learningassistants, in Proceedings of the 2017 Physics EducationResearch Conference, Cincinnati, OH , edited by L. Ding,A. Traxler, and Y. Cao (AIP, New York, 2018).[47] E. W. Close, J. Conn, and H. G. Close, Becoming physicspeople: Development of integrated physics identitythrough the learning assistant experience, Phys. Rev.Phys. Educ. Res. , 010109 (2016).[48] R. M. Talbot, L. M. Hartley, K. Marzetta, and B. S.Wee, Transforming undergraduate science educationwith learning assistants: student satisfaction in large-enrollment courses, J. Coll. Sci. Teach. , 24 (2015).[49] J. J. Chini, C. L. Straub, and K. H. Thomas, Learningfrom avatars: Learning assistants practice physics peda-gogy in a classroom simulator, Phys. Rev. Phys. Educ.Res. , 010117 (2016).[50] G. L. Cochran, A. G. Van Duzor, M. S. Sabella, and B.Geiss, Engaging in self-study to support collaborationbetween two-year colleges and universities, in Proceed-ings of the 2016 Physics Education Research Conference,Sacremento, CA (AIP, New York, 2017), pp. 76 – Proceedings of the 2017 Physics EducationResearch Conference, Cincinnati, OH , edited by L. Ding,A. Traxler, and Y. Cao (AIP, New York, 2018), pp. 36 – ASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16,
Proceedings of the 2017 Physics Education ResearchConference , Washington, DC, edited by A. Traxler, Y.Cao, and S. Wolf (AIP, New York, 2018).[39] N. D. Finkelstein, V. Otero, and S. J. Pollock, Teaching tolearn: The Colorado Learning Assistant programs impacton learning content, Forum Educ. , 11 (2007).[40] S. J. Pollock and N. D. Finkelstein, Sustaining educa-tional reforms in introductory physics, Phys. Rev. STPhys. Educ. Res. , 010110 (2008).[41] R. M. Goertzen, E. Brewe, L. H. Kramer, L. Wells, and D.Jones, Moving toward change: Institutionalizing reformthrough implementation of the learning assistant modeland open source tutorials, Phys. Rev. ST Phys. Educ. Res. , 020105 (2011).[42] N. McHenry, A. Martin, A. Castaldo, and D. Ziegenfuss,Learning assistants program: Faculty development forconceptual change, Int. J. Teach. Learn. Higher Educ. ,258 (2010).[43] K. E. Gray, D. C. Webb, and V. K. Otero, Effects of thelearning assistant model on teacher practice, Phys. Rev.Phys. Educ. Res. , 020126 (2016).[44] K. E. Gray and V. K. Otero, Analysis of former learningassistants views on cooperative learning, AIP Conf. Proc. , 149 (2009).[45] B. Van Dusen and J. Nissen, Equity in college physicsstudent learning: A critical quantitative intersectionalityinvestigation, J. Res. Sci. Teach. , 33 (2019).[46] B. Van Dusen and J. M. Nissen, Systemic inequities inintroductory physics courses: the impacts of learningassistants, in Proceedings of the 2017 Physics EducationResearch Conference, Cincinnati, OH , edited by L. Ding,A. Traxler, and Y. Cao (AIP, New York, 2018).[47] E. W. Close, J. Conn, and H. G. Close, Becoming physicspeople: Development of integrated physics identitythrough the learning assistant experience, Phys. Rev.Phys. Educ. Res. , 010109 (2016).[48] R. M. Talbot, L. M. Hartley, K. Marzetta, and B. S.Wee, Transforming undergraduate science educationwith learning assistants: student satisfaction in large-enrollment courses, J. Coll. Sci. Teach. , 24 (2015).[49] J. J. Chini, C. L. Straub, and K. H. Thomas, Learningfrom avatars: Learning assistants practice physics peda-gogy in a classroom simulator, Phys. Rev. Phys. Educ.Res. , 010117 (2016).[50] G. L. Cochran, A. G. Van Duzor, M. S. Sabella, and B.Geiss, Engaging in self-study to support collaborationbetween two-year colleges and universities, in Proceed-ings of the 2016 Physics Education Research Conference,Sacremento, CA (AIP, New York, 2017), pp. 76 – Proceedings of the 2017 Physics EducationResearch Conference, Cincinnati, OH , edited by L. Ding,A. Traxler, and Y. Cao (AIP, New York, 2018), pp. 36 – ASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16,
53] J. Alzen (personal communication).[54] University of Colorado Boulder Office of Data Analytics,Overall enrollment profile (2019).[55] C. A. Paul, D. J. Webb, M. K. Chessey, and W. H. Potter,Equity of success in clasp courses at UC Davis, in
Proceedings of the 2017 Physics Education ResearchConference, Cincinnati, OH , edited by L. Ding, A.Traxler, and Y. Cao (AIP, New York, 2018), p. 292.[56] D. J. Webb, C. A. Paul, and M. K. Chessey, Relativeimpacts of different grade-scales on student success inintroductory physics, arXiv:1903.06747.[57] V. Sawtelle, E. Brewe, and L. H. Kramer, Exploring therelationship between self-efficacy and retention in intro-ductory physics, J. Res. Sci. Teach. , 1096 (2012).[58] D. R. Arendale, Understanding the supplemental instruc-tion model, New Dir. Teach. Learn. , 11 (1994).[59] C. A. Stanich, M. A. Pelch, E. J. Theobald, and S. Free-man, A new approach to supplementary instructionnarrows achievement and affect gaps for underrepresentedminorities, first-generation students, and women, Chem.Educ. Res. Pract. , 846 (2018).[60] R. Benford and J. Gess-Newsome, Factors affectingstudent academic success in gateway courses at NorthernArizona University (eric document reproduction serviceno. ed495693) (2006), https://eric.ed.gov/?id=ED495693.[61] M. Piburn, D. Sawada, J. Turley, K. Falconer, R. Benford,I. Bloom, and E. Judson, Reformed Teaching ObservationProtocol (rtop) Reference Manual (Arizona Collaborativefor Excellence in the Preparation of Teachers, Tempe,Arizona, 2000).[62] G. Ladson-Billings, Critical race theory what it is not! in,
Handbook of Critical Race Theory in Education (Rout-ledge, London, 2013), pp. 54 – Critical Race Theory: The Key Writings thatFormed the Movement (The New Press, New York, 1995).[64] C. Sleeter and D. Bernal, Critical pedagogy, critical racetheory, and antiracist education: Implications for multi-cultural education,
Handbook of Research on Multicul-tural Education (2nd ed.) , edited by J. A. Banks and C. A.McGee Banks (Macmillan, New York, 2004), pp. 240 – , 388 (2007).[66] D. A. Bell, Jr., Brown v. board of education and theinterest-convergence dilemma, Harvard law review ,518 (1980).[67] R. Delgado, Explaining the rise and fall of AfricanAmerican fortunes-interest convergence and civil rightsgains (2002), https://heinonline.org/HOL/LandingPage?handle=hein.journals/hcrcl37=17&id=&page=.[68] G. Ladson-Billings, Just what is critical race theory andwhat ’ s it doing in a nice field like education?, Int. J.Qualitative Studies Educ. , 7 (1998).[69] G. Ladson-Billings, Critical race theory in education, TheRoutledge International Handbook of Critical Education (Routledge, London, 2009), pp. 110 – , 1(2013).[71] S. A. Annamma, DisCrit: Disability Studies and CriticalRace Theory in Education (Teachers College Press,New York, 2016).[72] A. Traxler and J. Blue, Disability in physics: Lessonsfrom the binary view of gender (2017).[73] D. Gillborn, P. Warmington, and S. Demack, Quantcrit:education, policy, big data and principles for a criti-cal race theory of statistics, Race Ethnicity Educ. , 158(2018).[74] C. DIgnazio and L. Klein, Data feminism (2019),https://play.google.com/store/books/details?id=zZnSDwAAQBAJ&rdid=bookzZnSDwAAQBAJ&rdot=1&source=gbs_vpt_read&pcampaignid=books_booksearch_viewport.[75] G. Ladson-Billings, From the achievement gap to theeducation debt: Understanding achievement in us schools,Educ. Res. , 3 (2006).[76] R. L. Wasserstein, A. L. Schirm, and N. A. Lazar, Movingto a world beyond p < . , Am. Stat. , 1 (2019).[77] J. Ho, T. Tumkaya, S. Aryal, H. Choi, and A. Claridge-Chang, Moving beyond p values: data analysis withestimation graphics, Nat. Methods , 1 (2019).[78] American Statistical Association et al. , Statement onstatistical significance and p-values, Am. Stat. , 129(2016).[79] L. Schudde, Heterogeneous effects in education: Thepromise and challenge of incorporating intersectionalityinto quantitative methodological approaches, Rev. Res.Educ. ’ t expect replication, Am. Stat. , 262(2019).[83] I. Rodriguez, E. Brewe, V. Sawtelle, and L. H. Kramer,Impact of equity models and statistical measures oninterpretations of educational reform, Phys. Rev. STPhys. Educ. Res. , 020103 (2012).[84] O. Lee, Equity implications based on the conceptions ofscience achievement in major reform documents, Rev.Educ. Res. , 83 (1999).[85] W. G. Secada, Equity in Education (Falmer Press,London, 1989).[86] O. Espinoza, Solving the equity-equality conceptualdilemma: A new model for analysis of the educationalprocess, Educ. Res. , 343 (2007).[87] R. Guti´errez and E. Dixon-Román, Beyond gap gazing:How can thinking about education comprehensively helpus (re) envision mathematics education?, in MappingEquity and Quality in Mathematics Education (Springer,New York, 2010), pp. 21 – BEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16,
Handbook of Research on Multicul-tural Education (2nd ed.) , edited by J. A. Banks and C. A.McGee Banks (Macmillan, New York, 2004), pp. 240 – , 388 (2007).[66] D. A. Bell, Jr., Brown v. board of education and theinterest-convergence dilemma, Harvard law review ,518 (1980).[67] R. Delgado, Explaining the rise and fall of AfricanAmerican fortunes-interest convergence and civil rightsgains (2002), https://heinonline.org/HOL/LandingPage?handle=hein.journals/hcrcl37=17&id=&page=.[68] G. Ladson-Billings, Just what is critical race theory andwhat ’ s it doing in a nice field like education?, Int. J.Qualitative Studies Educ. , 7 (1998).[69] G. Ladson-Billings, Critical race theory in education, TheRoutledge International Handbook of Critical Education (Routledge, London, 2009), pp. 110 – , 1(2013).[71] S. A. Annamma, DisCrit: Disability Studies and CriticalRace Theory in Education (Teachers College Press,New York, 2016).[72] A. Traxler and J. Blue, Disability in physics: Lessonsfrom the binary view of gender (2017).[73] D. Gillborn, P. Warmington, and S. Demack, Quantcrit:education, policy, big data and principles for a criti-cal race theory of statistics, Race Ethnicity Educ. , 158(2018).[74] C. DIgnazio and L. Klein, Data feminism (2019),https://play.google.com/store/books/details?id=zZnSDwAAQBAJ&rdid=bookzZnSDwAAQBAJ&rdot=1&source=gbs_vpt_read&pcampaignid=books_booksearch_viewport.[75] G. Ladson-Billings, From the achievement gap to theeducation debt: Understanding achievement in us schools,Educ. Res. , 3 (2006).[76] R. L. Wasserstein, A. L. Schirm, and N. A. Lazar, Movingto a world beyond p < . , Am. Stat. , 1 (2019).[77] J. Ho, T. Tumkaya, S. Aryal, H. Choi, and A. Claridge-Chang, Moving beyond p values: data analysis withestimation graphics, Nat. Methods , 1 (2019).[78] American Statistical Association et al. , Statement onstatistical significance and p-values, Am. Stat. , 129(2016).[79] L. Schudde, Heterogeneous effects in education: Thepromise and challenge of incorporating intersectionalityinto quantitative methodological approaches, Rev. Res.Educ. ’ t expect replication, Am. Stat. , 262(2019).[83] I. Rodriguez, E. Brewe, V. Sawtelle, and L. H. Kramer,Impact of equity models and statistical measures oninterpretations of educational reform, Phys. Rev. STPhys. Educ. Res. , 020103 (2012).[84] O. Lee, Equity implications based on the conceptions ofscience achievement in major reform documents, Rev.Educ. Res. , 83 (1999).[85] W. G. Secada, Equity in Education (Falmer Press,London, 1989).[86] O. Espinoza, Solving the equity-equality conceptualdilemma: A new model for analysis of the educationalprocess, Educ. Res. , 343 (2007).[87] R. Guti´errez and E. Dixon-Román, Beyond gap gazing:How can thinking about education comprehensively helpus (re) envision mathematics education?, in MappingEquity and Quality in Mathematics Education (Springer,New York, 2010), pp. 21 – BEN VAN DUSEN and JAYSON NISSEN PHYS. REV. PHYS. EDUC. RES.16,
88] R. Guti´errez, A gap-gazing fetish in mathematics educa-tion? Problematizing research on the achievement gap,J. Res. Math. Educ. , 357 (2008).[89] G. Rose, Situating knowledges: positionality, reflexivitiesand other tactics, Prog. Human Geography , 305(1997).[90] C. DIgnazio and L. Klein, Data Feminism ,20 (2010).[93] D. Bates, M. Maechler, B. Bolker, S. Walker et al. , lme4:Linear mixed-effects models using eigen and s4, Rpackage version , 1 (2014).[94] J. Allaire, Rstudio: Integrated development environmentfor R, Boston, MA 770 (2012), http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.651.1157&rep=rep1&type=pdf ,18 (2013).[96] J. E. Cavanaugh, Unifying the derivations for the akaikeand corrected akaike information criteria, Stat. Probab.Lett. , 201 (1997).[97] B. Van Dusen and J. Nissen, Modernizing use ofregression models in physics education research: A review of hierarchical linear modeling, Phys. Rev. Phys.Educ. Res. , 020108 (2019).[98] S. W. Raudenbush and A. S. Bryk, Hierarchical LinearModels: Applications and Data Analysis Methods (Sage,Thousand Oaks, CA, 2002), Vol. 1.[99] S. Ziliak and D. N. McCloskey,
The Cult of StatisticalSignificance: How the Standard Error Costs us Jobs,Justice, and Lives (University of Michigan Press, AnnArbor, MI, 2008).[100] B. Efron, Bootstrap methods: another look at the jack-knife, in
Breakthroughs in Statistics (Springer, New York,1992), pp. 569 – , 1045 (2009).[102] S. Cheryan, S. A. Ziegler, A. K. Montoya, and L. Jiang,Why are some stem fields more gender balanced thanothers?, Psychol. Bull. , 1 (2017).[103] R. R. Hake, Interactive-engagement versus traditionalmethods: A six-thousand-student survey of mechanicstest data for introductory physics courses, Am. J. Phys. , 64 (1998).[104] S. Freeman, S. L. Eddy, M. McDonough, M. K. Smith, N.Okoroafor, H. Jordt, and M. P. Wenderoth, Active learn-ing increases student performance in science, engineer-ing, and mathematics, Proc. Natl. Acad. Sci. U.S.A. ,8410 (2014). ASSOCIATIONS BETWEEN LEARNING … PHYS. REV. PHYS. EDUC. RES.16,