Extending Modified Module Analysis to Include Correct Responses: An Analysis of the Force Concept Inventory
Jie Yang, James Wells, Rachel Henderson, Elaine Christman, Gay Stewart, John Stewart
aa r X i v : . [ phy s i c s . e d - ph ] J a n Extending Modified Module Analysis to Include Correct Responses: An Analysis ofthe Force Concept Inventory
Jie Yang, James Wells, Rachel Henderson, Elaine Christman, Gay Stewart, and John Stewart ∗ West Virginia University, Department of Physics and Astronomy, Morgantown WV, 26506 College of the Sequoias, Science Division, Visalia CA, 93277 Michigan State University, Department of Physics and Astronomy, East Lansing MI, 48824 (Dated: January 24, 2020)Brewe, Bruun, and Bearden first applied network analysis to understand patterns of incorrectconceptual physics reasoning in multiple-choice instruments introducing the Module Analysis forMultiple-Choice Responses (MAMCR) algorithm. Wells et al. proposed an extension to the algo-rithm which allowed the analysis of large datasets called Modified Module Analysis (MMA). Thismethod analyzed the network structure of the correlation matrix of the responses to a multiple-choiceinstrument. Both MAMCR and MMA could only be applied to networks of incorrect responses. Inthis study, an extension of MMA is explored which allows the analysis of networks involving bothcorrect and incorrect responses. The extension analyzes the network structure of the partial cor-relation matrix instead of the correlation matrix. The new algorithm, called MMA-P, was appliedto the FCI and recovered much of the structure identified by MMA. The algorithm also identifiedsets of correct answers requiring similar physical reasoning reported in previous studies. Beyondgroups of all correct and all incorrect responses, some groups of responses which mixed correct andincorrect responses were also identified. Some of these mixed response groups were produced when acorrect response was selected for incorrect reasons; some of the groups helped to explain the genderunfairness previously reported for the FCI.
I. INTRODUCTION
The structure of students’ conceptual understanding ofphysics and the evolution of that understanding has beenan important research strand within Physics EducationResearch (PER) since its inception. Research into stu-dent understanding has often relied on multiple-choiceconceptual instruments such as the Force Concept In-ventory (FCI) [1] and the Force and Motion ConceptualEvaluation [2]. Quantitative research examining these in-struments began shortly after their introduction [3]. Re-cently, a new class of quantitative methods using networkanalysis has been applied to further understand commonstudent difficulties with mechanics [4, 5].Network analysis is a broad and powerful set of an-alytic techniques applicable to situations as diverse asidentifying drought resistant genes in plants [6] and ex-ploring the status hierarchies of teenagers [7]. A net-work is formed of nodes and edges where edges connectpairs of nodes. Edges can have numerical weights thatrepresent properties which influence the strength of therelationship. Network analysis seeks to characterize thestructure of the network. One important type of struc-ture is the identification of groups of nodes that are moreconnected to each other than they are to other nodes inthe network. These subgroups are called “communities”or “modules” interchangeably; we will call them com-munities to conform with the naming conventions of the“igraph” package [8] in the “R” software system [9] whichwas used for this analysis. ∗ [email protected] A substantial body of research has investigated differ-ences in the conceptual performance of groups underrep-resented in physics [10, 11]; the majority of this work hasexamined differences between men and women [12]. Thenew network analytic techniques have been used to ex-plore differences in the structure of incorrect reasoningof men and women [5]. We continue this practice andreport results disaggregating men and women.
A. Research Questions
The network analytic method used in this work ex-tends Modified Module Analysis (MMA) introduced byWells et al. [5] as an extension to the original algorithmpioneered by Brewe, Bruun, and Bearden [4]. Both meth-ods were only productive when applied to the incorrectanswers in the FCI; the correct answers formed a singletightly connected community that prevented identifica-tion of additional structure. Modified Module Analysis isdescribed in detail in Sec. I C. The purpose of this studyis to explore an extension of MMA to include correct re-sponses; this extension is called MMA-P. This work wasperformed using the same dataset as the previous MMAstudy; this dataset was also used by Traxler et al. todemonstrate that a subset of the items in the FCI aresubstantially unfair to women [13]. This study will usethe results of the MMA-P algorithm to seek an explana-tion for this unfairness.This study explored the following research questions:
RQ1:
How can Modified Module Analysis be extendedto include correct responses? Are the communi-ties detected by the extended algorithm productivein furthering the understanding of the FCI? Howare the communities identified by the extended al-gorithm related to structures identified in previousstudies?
RQ2:
How do the incorrect communities change betweenthe pretest and the post-test?
RQ3:
How do the communities detected differ betweenmen and women?
B. Prior Studies
This work draws heavily on two prior studies of theFCI performed on the same dataset as used in this study.These studies will be referenced as Study 1 and Study 2in this work.
C. Study 1 : Modified Module Analysis
Brewe, Bruun, and Bearden [4] first introduced net-work analysis to PER as Module Analysis for Multiple-Choice Responses (MAMCR). Their initial work was ap-plied to a sample containing only 143 FCI student re-sponses. Wells et al. [5] sought to replicate this studywith a large sample of FCI pretest and post-test re-sponses ( N pre = N post = etal. [5] study will be referenced as Study 1 in this work.In MMA, the network is formed of nodes representingresponses to the FCI. For example, the network containsa node representing the selection of response D to FCIitem 2, node 2D. The edges in the network connect nodesthat are correlated above some threshold; in Study 1, r > . r is the corre-lation coefficient. The correlation coefficient is used asthe weight of the edge. For example, if responses 2D and5E have a correlation coefficient of r = .
25, then the net-work will contain an edge between nodes 2D and 5E withweight 0 .
25. For productive analysis using MMA, nodesrepresenting correct responses must be removed. If thesenodes were included, they formed a densely connectedcommunity which obscured other structures.Study 1 identified 9 groups of incorrect answers onthe pretest and 11 groups on the post-test. In mostcases, these groups could be explained as common mis-conceptions using the taxonomy of misconceptions intro-duced with the publication of the FCI [1] and refinedby Hestenes and Jackson [14]. A subset of the incorrectcommunities was the result of the use of item blocks inthe FCI. An item block is a group of items that refers tothe same physical system or are written with a commonstem. Most of the pretest and post-test groups identifiedwere the same for men and women. The groups identified also had little relation to the gender unfair items identi-fied by Traxler et al. [13]. Study 1 concluded that thedifferences in the structure of incorrect physical reason-ing between men and women did not explain the genderdifferences reported for the FCI or the unfairness of theitems in the instrument.Study 1 was recently replicated by Wells et al. for theFMCE [15]. Incorrect responses representing the samemisconception were identified in the same communitydemonstrating the algorithm is productive for examin-ing instruments beyond the FCI. The incorrect communi-ties identified, however, were often different for men andwomen both pre- and post-instruction as was the changein the communities from the pretest to the post-test.
D. Study 2: Multidimensional Item ResponseTheory and FCI
In Study 2, Stewart et al. [16] used constrained Mul-tidimensional Item Response Theory (MIRT) to developa detailed model of the physical reasoning needed to cor-rectly solve the FCI. Study 2 identified a number of sub-groups of items in the FCI requiring similar physical rea-soning for their solution. These subgroups were used ex-tensively in Study 1 because the incorrect communitiesfound by MMA were often well aligned with the groupsof items requiring similar correct reasoning identified inStudy 2. These subgroups were {
4, 15, 16, 28 } , {
5, 18 } , {
6, 7 } , and {
17, 25 } . Study 2 showed that the practice ofblocking items was generating correlations between itemsnot related to the physical principles tested and, as such,retained only the first item in an item block. The itemblocks in the FCI are {
5, 6 } , {
8, 9, 10, 11 } , {
15, 16 } ,and {
21, 22, 23, 24 } . While items {
25, 26, 27 } are notexplicitly blocked, both items 26 and 27 refer to item 25and, therefore, these three items should be treated as anitem block.Wells et al. application of MMA to the FMCE [15] alsorelied on a constrained MIRT analysis of the instrumentto identify items requiring similar physical reasoning [17].As in Study 1, incorrect answer communities were oftenformed of items requiring the same correct reasoning. E. Results of Prior Research
Prior research into the structure of the FCI was thor-oughly summarized in Study 2, research into the mis-conceptions measured by the FCI in Study 1, and issuesof the gender fairness and how they relate to the FCI inTraxler et al. [13]. These fairly extensive research strandsare summarized below; readers interested in more detailare directed to these previous works.
1. Exploratory Analyses of the FCI
Hestenes, Wells, and Swackhammer decomposed theconcept of force into six conceptual dimensions and pro-vided descriptions of the concepts each FCI item was in-tended to measure [1]. However, attempts to extract thisfactor structure using exploratory factor analysis (EFA),which uses correlations between all items to select groupsthat measure the same idea, proved unsuccessful. Huff-man and Heller identified only two of these six factors,Newton’s 3rd law and “Kinds of Forces,” in their princi-pal component analysis of a sample consisting of 145 highschool students. The only factor identified in a sample of750 university students was kinds of forces [3].Scott, Schumayer, and Gray found that while only onefactor explained much of the variance in an EFA studyof the FCI post-tests of 2150 college students, 5 fac-tors were needed for an optimal model [18]. Scott andSchumayer identified a similar, but not identical, 5-factormodel when using MIRT on a related dataset, confirm-ing their model and suggesting that traditional EFA andMIRT are complementary techniques [19]. Semak et al. applied factor analysis to the FCI pretest and post-testresponses of 427 college students and found that an op-timal model included five factors for the pretest and sixfor the post-test [20]. Study 2 conducted an exploratoryMIRT analysis of college post-test data and reported thata 9-factor model was optimal [16].
2. Gender and the FCI
Differences between the performance of men andwomen on the FCI and Force and Motion ConceptualEvaluation (FMCE) have been broadly reported withmen outperforming women by 13% on pretests and 12%on post-tests. Many explanations have been advanced toexplain the these differences including differences in highschool physics class election [21–23], cognitive differences[24–27], and psychocultural factors including mathemat-ics anxiety [28, 29], science anxiety [30–32], and stereo-type threat [33].All the above explanations locate the source of the per-formance differences with the students, a separate bodyof research has shown multiple items in the FCI are un-fair to women, and some to men [34–36]. An item isunfair if men and women with the same general abilitywith the material score differently on the item. Gen-der differences in some FCI items have been reported formany years. Recently Traxler et al. [13] provided con-vincing evidence that 5 FCI items, items 14, 21, 22, 23,and 27, were substantially unfair to women using samplesfrom three institutions. In their largest sample, Differ-ential Item Function (DIF) theory [37] identified a totalof eight items with large DIF; six were unfair to women(12, 14, 21, 22, 23, 27) and two to men (items 9 and15). The problematic items had generally, but not con-sistently, been identified in earlier work [34–36].
3. The Structure of Knowledge
MAMCR, MMA, and MMA-P all attempt to identifystructure within student responses to a multiple-choiceinstrument. MAMCR and MMA only investigate incor-rect responses and, thus, attempt to find coherent pat-terns of incorrect reasoning. Understanding common dif-ficulties shared by many students in learning Newtonianmechanics has been an important area of research in PERsince its inception [38–44]. This research led to a system-atic exploration of students’ understanding of Newton’slaws and its epistemological development [2, 45–48].From these investigations and research outside of PER,theoretical frameworks of the structure of students’ con-ceptual knowledge of physics emerged. Within PER, pat-terns of incorrect answering have often been character-ized as “misconceptions,” consistently applied incorrectreasoning specific to the physical concept tested. Othertheories have been advanced to explain incorrect reason-ing; two of the most prominent are the knowledge-in-pieces framework [49, 50] and the ontological categoriesframework [51–53].The knowledge-in-pieces framework models studentknowledge as composed of small pieces of reasoning thatare applied either independently or collectively to solvea problem. Models involveing small fragments of reason-ing have been advanced by many authors and have beencalled phenomenological primitives (p-prims) [49, 50], re-sources [54–56], and facets of knowledge [57]. In theknowledge-in-pieces framework, what PER has calledmisconceptions are the result of problems activating thesame collections of inappropriately applied knowledgepieces. In this framework, misconceptions are not “re-solved,” rather students’ frameworks must be refined toapply to the situation in the problems. The ontologicalcategories model suggests that incorrect student answersare a result of misclassification of a physical concept. Forexample, force may be misclassified as a substance.Scherr [58] provides a concise definition of both themisconception view and the knowledge-in-pieces viewwhich we will adopt for this work. The misconceptionview is “a model of student thinking in which studentideas are imagined to be determinant, coherent, context-independent, stable, and rigid” [58]. The knowledge-in-pieces framework models student conceptual ideas “asbeing at least potentially truth-indeterminate, indepen-dent of one another, context-dependent, fluctuating, andpliable” [58].The research in the current work most closely alignswith the misconception and knowledge-in-pieces views.Study 1 and Wells et al. [15] both observed that commu-nities of incorrect answers are generally formed of itemsrequiring the same physical reasoning for their solutionand shared a common misconception from the taxonomyof Hestenes and Jackson [14]. Because the incorrect rea-soning is applied across multiple similar problems, themisconception view may be more appropriate for the in-correct answer communities identified by MMA or MMA-P. The FCI was developed partially to probe commonmisconceptions [1], so the finding that this frameworkis more appropriate is not a surprise. Network analysiscould potentially identify p-prims as consistently appliedreasoning divorced from the physical context, but thusfar this type of community has not been observed.
II. METHODSA. Instrument
The FCI is a 30-item instrument designed to assess astudent’s conceptual understanding of topics in Newto-nian mechanics including one- and two-dimensional kine-matics and Newton’s laws. The FCI’s coverage of New-tonian mechanics is not exhaustive; the instrument con-tains no questions on common topics from introductorymechanics courses such as conservation of momentumand energy [1]. Each item includes four incorrect re-sponses with distractors intended to elicit common stu-dent misconceptions as well as one correct response. Thisstudy uses the revised FCI released in 1995 [59], which isavailable at PhysPort [60].
B. Sample
The data for this study were collected at a large south-ern land-grant university serving approximately 25,000students. Overall university undergraduate demograph-ics were 79% white, 5% African American, 6% Hispanic,with other groups each 3% or less for the period studied[61].The sample consists of 4509 complete pretest records(3482 men and 1027 women) and 4716 complete post-test records (3628 men and 1088 women) from stu-dents enrolled in an introductory calculus-based mechan-ics course. The class served primarily engineering andphysical sciences majors and was presented by the samelead instructor using a consistent pedagogy throughoutthe study. This instructor implemented interactive en-gagement activities in the lecture and multiple research-based instructional methods were used in the requiredlaboratory sections.
C. Partial Correlation
To allow the investigation of correct and incorrect an-swers together in the same network, the correlation ma-trix used by MMA was replaced by the partial correlationmatrix. Partial correlation measures the association be-tween two variables while eliminating the effect of oneor more other variables. The correlation, r XY , between variable X and variable Y is defined in Eqn. 1 r XY = E [( X − µ X )( Y − µ Y )] σ X σ Y (1)where µ i is the mean of variable i , σ i is the standarddeviation, and E [ X ] is the expectation value.Two variables may be correlated because they are bothrelated to a third variable; the partial correlation controlsfor the relation with a third variable. The partial correla-tion of variable X and variable Y controlling for variable Z , r XY ∣ Z , is defined in Eqn. 2. r XY ∣ Z = r XY − r XZ r Y Z √ − r XZ √ − r Y Z (2)Conceptually, partial correlation can best be under-stood in terms of linear regression. If X , Y , and Z arecontinuous random variables and ǫ XZ are the residualswhen Z is regressed on X (controlling X for Z ) and ǫ Y Z are the residuals when Z is regressed on Y , then the par-tial correlation r XY ∣ Z is the correlation of the residuals,corr( ǫ XZ , ǫ Y Z ). The residuals are the part of the vari-ance of X or Y not explained by the variation of Z .The package “ppcor” [62], part of the R software sys-tem [9], was used to perform the partial correlation anal-ysis. D. Extending Modified Module Analysis
Modified Module Analysis uses community detectionalgorithms on the network defined by the correlationmatrix of all incorrect responses with a threshold ap-plied; Study 1 used r > .
2, as will this study. Cor-relation matrix entries below this threshold are set tozero. The non-zero entries represent edges in the net-work; the edge weight is the strength of the correlation.Study 1 applied a number of additional filters to elimi-nate insignificant edges and rarely selected nodes; thesewere repeated. Nodes selected by fewer than 30 studentswere removed. Edges connecting different responses tothe same item (and thus strongly negatively correlated)were removed; the r > . r > . C , wasdefined as the fraction of the bootstrap replications inwhich node i and j were in the same community. Onlynodes found in C >
80% of the communities were ana-lyzed.
III. RESULTSA. Extended Modified Module Analysis
Modified Module Analysis - Partial was applied to un-derstand the structure of the FCI. The communities de-tected with a partial correlation threshold of r > . C > ⊗ , some were only identified in this study andare marked × , some were only identified in Study 1 andare marked ◯ ; some communities identified in Study 1 merged with correct answers to formed mixed correct-incorrect communities and are marked ⊙ .Figure 1 is a refinement of the network visualizationused in Study 1 and in Wells et al. [15]. In these previ-ous studies, a line was drawn between the two nodes ifthe nodes were identified in the same community 80% ofthe time ( C > { } were detected in the same commu-nity in at least 80% of the replications; however, 21E*and 26E* are weakly correlated. Both are strongly cor-related with 22B*. To provide a better indication of thedegree of connection of two nodes in the individual boot-strap replications of the CDA, the average correlation ofthe two nodes across the 1000 replicates is used as theedge weight in Fig. 1.The degree of connection of a community is charac-terized by the intra-community density, γ , the ratio ofthe number of edges observed to the maximum possiblenumber of edges [64]. For example, a community whichcontains four nodes can be connected with a maximum ofsix edges. If only four of those edges are observed, then γ = /
6. For communities that are not fully connected( γ < γ is presented in parenthesis in Table I. B. The Structure of the Response Communities
1. Completely Incorrect Communities
The original MMA algorithm could only explore incor-rect responses, and therefore, all communities reportedin Study 1 were composed of incorrect responses. Theincorrect communities identified by MMA and MMA-Pwere very similar but not identical.Study 1 showed that there were two classes of incorrectcommunities. One was related to the practice of blockingitems; for other communities, the students seemed to beapplying consistent incorrect reasoning, a misconception.Communities { } , { } , and { } are answers within blocked problems where the secondanswer in the pair would be correct if the first answer wascorrect. These were fairly consistently detected by bothMMA and MMA-P; however, the { } commu-nity merged with some correct responses to form a mixedcommunity post-instruction for both men and womenin MMA-P. This occurred because, post-instruction, thecommunity joined with response 8B* for men and re-sponses 8B*, 9E*, and 15A for women. These new com-munities are discussed with the mixed correct and incor-rect communities.Three incorrect communities were identified by MMA-P, but not MMA. In community { } , both itemsare incorrect responses to the Newton’s 3rd law groupidentified by Study 2; however, they are also part of anitem block. It is, therefore, impossible to determine if thecommunity results from a shared misconception or from Table I. Communities identified in the pretest and post-test at r > . C > γ , for communities where the intra-community density is not one. Communities labeled ⊗ were identifiedin both Study 1 and the current study. Communities labeled ◯ were identified in Study 1 but not in the current study.Communities labeled × were identified in the current study, but not in Study 1. Communities labeled ⊙ were identified inStudy 1 as completely incorrect communities, but combined with correct responses to form mixed correct-incorrect communitiesin the current study.Community Pretest Post-test Misconception/Principle/ExplanationMen Women Men WomenCompletely Incorrect Communities1A, 2C ⊙ Heavier objects fall faster.1D, 2D ⊗ Lighter objects fall faster.4A, 15C, 28D ⊗ ⊗ ⊙ ⊙
Newton’s 3rd law misconceptions.5D, 18D ⊗ ×
Motion implies active forces.5D, 11C, 13C, 18D, 30E ◯ Motion implies active forces.5E, 18E ⊗ ⊗ ⊗ ◯
Motion implies active forces/Centrifugal force.6A, 7A ◯ ⊗ ⊗ ⊗
Circular impetus.8A, 9B ⊗ ⊗ ⊗ ⊗
Blocked item.11B, 29A ⊗ ◯
Motion implies active forces.11C, 13C × ×
Motion implies active forces.11C, 13C, 30E ◯ Motion implies active forces.15D, 16D × Newton’s 3rd law misconceptions.17A, 25D ⊗ ◯
Largest force determines motion.21B, 23C ⊗ ⊗
Blocked item.21C, 22A ⊗ ⊙ ⊙
Blocked item.23D, 24C ⊗ ⊗ ⊗ ⊗
Impetus dissipation.Mixed Correct and Incorrect Communities1A, 2C, 17B* × Unknown4A, 14D*, 15C, 28D × (67%) Newton’s 3rd law and 2D kinematics.4A, 15C, 21E*, 22B*, 28D × Newton’s 3rd law and 2D kinematics.8B*, 9E*, 15A*, 21C, 22A ×
8B and 21C share a similar trajectory; 8B and 9E are blocked.8B, 21C* ×
8B and 21C share a similar trajectory.8B*, 21C, 22A ×
8B and 21C share a similar trajectory.23A, 24A* ×
24A is the correct answer if 23A were correct.Completely Correct Communities4E*,28E* × Newton’s 3rd law.4E*, 15A*, 28E* × (67%) × Newton’s 3rd law.5B*, 18B* × ×
Centripetal acceleration in a curved trajectory.6B*, 7B* × ×
Instantaneous velocity is tangent to the trajectory.11D*, 13D* × Motion under gravity; a force in the direction of motion is not necessary.15A*, 28E* × Newton’s 3rd law.17B*, 25C* × ×
Newton’s 1st law; Addition of forces.17B*, 25C*, 26E* × Newton’s 1st and 2nd law; Addition of forces;(26E) 1D acceleration.21E*, 22B*, 26E* × Newton’s 2nd law; 1D and 2D kinematics. the effect of blocking. Item 16 is inconsistently identifiedwith other Newton’s 3rd law items in exploratory factoranalysis studies [3, 16, 18–20, 65, 66]. The two other in-correct communities identified by MMA-P but not MMAresulted from splitting the { } community into the communities { } and { } . (a) Women - Pretest (b) Men - Pretest(c) Women - Post-test (d) Men - Post-test
1A 1D2C 2D4A 5E 8A9B11B15A* 15C15D 16D17B* 18E21C 22A23D24C 28D28E* 29A4A4E* 5B* 5D6A 6B*7A 7B*8A8B* 9B9E* 11C11D* 13C13D* 14D*15A* 15C 17B*18B* 18D21B21C 21E*22A 22B* 23C 23D 24C 25C*26E* 28D28E*4A 4E* 5E 6A7A8A8B* 9B15A*15C 17B*18E 21C23A23D 24A*24C 25C*28D28E* 4A 4E* 5B*5D 5E6A 6B*7A 7B*8A8B* 9B11C13C 15A*15C 17A 17B* 18B*18D 18E21B21C21E* 22A22B* 23C23D24C 25C*25D 26E*28D 28E*
Figure 1. Communities detected in the FCI partial correlation matrix with r > .
2. The strength of the correlation is representedby the line thickness.
2. Completely Correct Communities
Study 2 identified four groups of items requiring similarsolution structure: {
4, 15, 16, 28 } , {
5, 18 } , {
6, 7 } , and {
17, 25 } . Three of the completely correct communitiescombine items in the first group, { } , { } , and { } ; all of which requireNewton’s 3rd law for their solution. All items in thethree-item communities are strongly correlated in Fig.1. Correct communities { } , { } , and { } were all identified as requiring similar rea-soning for their solution in Study 2. We note the preva-lence of the B response is a result of the extremely un-balanced use of distractors in the FCI [67]. The correctcommunity { } extends the community { } by adding item 26. Both items 17 and 25require the addition of forces and Newton’s 1st law fortheir solution. Item 26 requires the addition of forces,but the forces are unbalanced, requiring Newton’s 2ndlaw and one-dimensional kinematics (a net force in the di-rection of motion causes an object to speed up). Becausethe correct solution structure is substantially different, itseems likely this community is detected because items 25and 26 are in the same item block. This is supported byFig. 1(d) where there is a weak correlation between re-sponses 17B* and 26E* in the community { } .Both items in the correct community { } ask the students to identify the forces on an object inmotion; gravity and a normal force for item 11 and grav-ity alone for item 13. Both also explicitly test the motionimplies active forces misconception. Item 11 was elimi-nated from the constrained MIRT analysis in Study 2because of blocking and, therefore, no item level mea-sure of the discrimination of the item on the use of thenormal force is available. With the identification of thecorrect community, { } , it seems likely thatboth items largely test knowledge of the existence of adownward force of gravity while in motion.The correct community { } containedblocked items 21 and 22 along with item 26. Items 21 and22, while blocked, test fairly different physical principles;however, items 22 and 26 both require the principle thatif a force is applied in the direction of motion, an objectwill speed up. There are strong correlations between re-sponses 21E* and 22B* and between responses 22B* and26E* in Fig. 1(c), but a weak correlation between 21E*and 26E*. It seems likely blocking caused a relation ofitems 21 and 22 and the shared principle produced therelation between items 22 and 26.
3. Mixed Correct and Incorrect Communities
In addition to examining the grouping of incorrect orcorrect items, MMA-P can also detect communities com-bining correct and incorrect items. These communitiescan provide further insight into the functioning of theFCI and the complexities of student thinking.Many of the mixed communities seem to form becausea pair of questions mixes a correct and an incorrect com-munity. For example, responses { } both rep-resent the same trajectory (a parabolic curve); this tra-jectory is the correct answer to item 8 and the incorrectanswer to item 21. This similarity serves to explain themixed communities { } and { } .Items 21 and 22 are blocked items where response 22Ais the correct response if response 21C had been the cor-rect response. Responses 8B* and 22A are weakly relatedin Fig. 1(d) making it more likely that the shared tra-jectory is the cause of the community. If a student isselecting a specific trajectory using both the correct andincorrect reasoning, it may indicate that response 8B* isbeing selected correctly without a solid understanding.The relation of 8B* and 21C partially explains thecommunity { } ; items 8 and 9are blocked and to answer item 9 correctly requires a cor- rect response to item 8. It is unclear why Newton’s 3rdlaw item 15 is associated with this group. Some items inthis group are weakly correlated as shown in Fig. 1(c),but the strong relation of 15A* to 8B*, 9B*, and 22Adoes not have a grounding in either the correct solutionstructure of the items or a shared misconception. It isalso difficult to identify a p-prim that might have beenactivated in each case. Items 21 and 22 were identifiedas substantially unfair to women in Traxler et al. [13].The relation of the correct response 15A* to 22A andcorrect response 8B* to 21C correcting for overall testscore provides evidence that these items are being in-correctly answered for reasons not completely related toNewtonian reasoning ability.Responses 21E* and 22B* are part of an item block.The identification of these items in a community of in-correct Newton’s 3rd law responses, { } for men, may indicate that men are selecting the itemscorrectly for the wrong reasons which would partiallyexplain the unfairness of items 21 and 22 reported byTraxler et al. [13].It is possible a similar effect explains the community { } . If students are selecting correctresponse 14D* for incorrect reasons, it could be more cor-related with the incorrect Newton’s 3rd law items thanwould be predicted by the overall FCI score. This com-munity was only identified for women post-instruction.Item 14 was one of the items identified as substantiallyunfair to women in Traxler et al. [13]; if it is also be-ing answered correctly by women for incorrect reasons,it may be more unfair than previously reported.Responses 23A and 24A* are also part of an item block.Item 23A tests the misconception of impetus dissipationwhere after a force is removed the object returns to itsoriginal trajectory before the force was applied. Item24A* (constant speed) is the correct answer for the cor-rect reasoning for item 23, but it is also the correct answerif the misconception of impetus dissipation actually ap-plied. This suggests that some students are getting item24 correct for the wrong reasons, and that the item blockmay need to be restructured.The community { } all involve items withan object moving under the force of gravity. There islittle additional relation between the items. It is diffi-cult to make a theoretical case for this community eitherbecause of a shared misconception or the correct answerstructure. It may be that these items, identified as a com-munity only for men on the pretest, use less consistentreasoning better modeled using the knowledge-in-piecesframework as a p-prim. Response 1A might be answeredby applying the “heavier is faster” principle, response 2Cthe “heavier travels farther” principle, and response 17B*the “constant implies equal” principles; all examples of afragment of reasoning that “like implies like.” IV. DISCUSSIONA. Research Questions
This study sought to answer three research questions;they will be addressed in the order proposed.
RQ1: How can Modified Module Analysis be extendedto include correct responses? Are the communities de-tected by the extended algorithm productive in furtheringthe understanding of the FCI? How are the communitiesidentified by the extended algorithm related to structuresidentified in previous studies?
Modified Module Analysis was extended to include cor-rect answers by replacing the correlation matrix with thepartial correlation matrix correcting for overall FCI testscore. The modified algorithm was called Modified Mod-ule Analysis - Partial (MMA-P). This modification al-lowed the identification of a number of relatively smallcommunities of correct and incorrect answers producinga network quantitatively similar to the MMA algorithm,but including correct answers.Study 1 could only report completely incorrect com-munities. Table I provides a comparison between the twostudies. Of the 38 communities identified either on thepretest or post-test for men or women, 24 were identifiedin both studies, 11 were identified only in Study 1, and 3only in the present study. Four of the communities identi-fied in only Study 1 merged with correct answers to formmixed communities in this study. As such, MMA andMMA-P produce fairly similar incorrect answer commu-nities. The remaining incorrect communities identified inStudy 1, but not in the present study, may have resultedfrom correlations with overall test score where only veryweak or very strong students selected the items.Until this study, module analysis was not productivefor finding communities of correct answers in the FCI.Study 2 presented both the FCI correlation matrix andthe partial correlation matrix for the correct answers tothe FCI for the dataset used in this study. The corre-lation matrix was sparsely and randomly interconnectedand showed little community structure. Clear communitystructure was evident in the partial correlation matrix.As such, it was not surprising that modifying MMA touse the partial correlation matrix allowed the extractionof compact correct answer communities.Study 2 did not use network analysis to form commu-nities of correct answers, but it did present a taxonomy ofisomorphic and blocked problems that can be comparedwith the community structure. Two items are isomor-phic if they required the same reasoning process for theirsolution. Many of the components of the correct commu-nities in Table I were identified in Study 2 as having sim-ilar solution structure { } , { } , { } , and { } . The students do notconsistently integrate the Newton’s 3rd law items, { } , with different combinations observed formen and women either on the pretest or post-test. Theitems in community { } differ only by the ad- dition of the normal force; both items specifically test themotion implies active forces misconception by providinga distractor which indicates there is a force in the direc-tion of motion. The incorrect community, { } ,of students who apply this misconception was also iden-tified. The addition of the normal force does not seeman important discriminating factor in whether the itemsare answered correctly.The attachment of 26E* to the { } commu-nity of isomorphic items for men likely resulted from theblocking of items 25 and 26; the solution structure of item26 does not suggest it should be directly associated withthe other two items. The correlations in Fig. 1 supportthis interpretation. Item 26E* is also associated with theblocked items 21B*, and 22B* for women; however, theassociation is much stronger with item 22B*; both item22 and 26 require application of the principle that forcesin the direction of motion cause objects to speed up.As such, the correct community structure is largelywhat was suggested by Study 2 with some additional setsof items that are being answered consistently which werenot modeled as having similar structure in Study 2. Someof the connections identified continue to support the neg-ative impacts of blocking items on the interpretability ofthe results of the instrument.No prior studies have examined community structurecombining incorrect and correct responses in the samecommunity. The mixed communities identified in thisstudy seem to originate largely from combinations ofitems that may not be working as intended in the in-strument; items that are being answered correctly forthe wrong reason or items that are possibly being misin-terpreted contributing to instrumental unfairness. RQ2: How do the incorrect communities change be-tween the pretest and the post-test? RQ3: How do thecommunities detected differ between men and women?
The differences from pretest to post-test differ for menand women, and therefore, these two questions will betaken together. Table I shows the communities identifiedby MMA-P. The completely correct, completely incor-rect, and mixed correct-incorrect communities are dis-cussed independently.For completely incorrect communities, incorrect com-munities identified in Study 1 which merged with correctanswers to form mixed communities in this study (labeled ⊙ in Table I) will be included in the incorrect communi-ties; the incorrect items in the community were still de-tected in the same community. For men, 9 communitieswere detected on the pretest and 10 on the post-test; only5 were consistent from pretest to post-test. For women,5 communities were identified on the pretest and 8 onthe post-test; 4 were consistent between the pretest andpost-test. As such, women have more consistent incor-rect reasoning patterns between pretest to post-test, butmen have more communities of incorrect reasoning bothpreinstruction and post-instruction.Comparing men and women, only 4 of the 10 pretestcommunities were consistent for men and women, while08 of the 10 post-test communities were consistent be-tween men and women. As such, there is little consis-tency in incorrect reasoning between men and womenon the pretest, but substantial consistency on the post-test. Men and women also had a similar number of itemstransition from completely incorrect in Study 1 to mixedcorrect-incorrect in this study.Six incorrect communities identified by MMA in Study1 were not identified by MMA-P in this study. Thesemissing communities were often related to new communi-ties identified by MMA-P but not MMA. The community { } identified by MMA became the commu-nity { } using MMA-P for men on the post-test.Response 11C and 13C test the force in the direction ofmotion misconception while response 30E asks about the“force of a hit;” as such, the dissolution of the communityis understandable. The community { } identified by MMA became the communities { } and { } in MMA-P; both can be theoreti-cally justified using the framework of Study 2 and bothtest similar misconceptions in the framework of Hestenesand Jackson [14]. It may be the the identification of the { } was a result of correlationsgenerated by overall test score.Four of the communities identified by MMA but notMMA-P were more difficult to explain. Communities { } , { } , and { } contain incorrectresponses to items identified in Study 2 as having thesame correct solution structure. It is unclear why theincorrect reasoning for these items is not applied consis-tently. The correct answer communities for these itemswere consistently identified post-instruction. It may bethat students are transitioning to correct reasoning andthat the misconception is no longer consistently applied.Pretest community { } also was no longer iden-tified by women; students giving responses in this com-munity fail to account for the normal force. It is unclearwhy women do not do so consistently. Item 29 was iden-tified by Traxler et al. [13] as having poor psychometricproperties, so it may be that the item is simply not func-tioning well.Few mixed communities were detected either prein-struction or post-instruction for men or women; therewere no consistent mixed communities either from pretestto post-test or for men and women.Only three correct communities were detected on thepretest for either men or women; only one was also identi-fied post-instruction. Most correct communities were de-tected post-instruction; four for men and six for women;only two of these communities were consistent betweenmen and women. One of the inconsistent correct com-munities { } likely resulted from block-ing items 25 and 26; if response 26E* were removed thecommunity { } would also be consistent be-tween men and women. Three of the other inconsistentcommunities result from the inconsistent application ofNewton’s 3rd law. Responses 4E*, 15A*, and 28E* wereidentified in the same community and were highly cor- related (Fig. 1(d)) for men on the post-test. Only re-sponses 4E* and 28E* were in the same community forwomen; response 15A* was identified as part of a mixedcommunity for women. If the minor differences in the { } and { } communities areignored, then all correct communities identified for menpost-instruction were also identified for women; two ad-ditional correct communities were identified for women.The items in one of the additional communities, { } , differ by the inclusion of a normal force; this dif-ference affects men but not women. The last inconsis-tent community { } involves theblocked items 21 and 22, but also item 26 which requiressomewhat different physical reasoning. It is unclear whythese items are identified in the same community; how-ever, items 21 and 22 were two items identified as highlyunfair to women in Traxler et al. [13]. B. Other Observations
The MMA-P algorithm identified much of the incor-rect structure identified by MMA while also productivelyidentifying correct structure. The structures identifiedwere consistent with the theoretical framework providedby Study 2 and with the taxonomy of misconceptions ofHestenes and Jackson [14]. The mixed correct-incorrectcommunities allowed the identification of combinationsof responses where symmetric correct and incorrect rea-soning was applied suggesting the correct response wasbeing selected for the incorrect reason. These communi-ties also provide some hint as to items within the FCIwhich were not functioning correctly where correct an-swers were related to a community of consistently ap-plied misconceptions. In general, the MMA-P extensionof MMA provided a richer picture of the instrument.
V. FUTURE
The MMA-P algorithm will be applied to other popu-lar conceptual instruments such as the the FMCE, theConceptual Survey of Electricity and Magnetism [68],and the Brief Electricity and Magnetism Assessment [69]to further understand their structure. The algorithm willalso be applied to multiple samples taken from studentsat different institutions to understand how the incorrectreasoning structures identified change with institutionalsetting.
VI. CONCLUSION
This work extended the Modified Module Analysis al-gorithm of Wells et al. [5] to allow the analysis of correctand incorrect responses simultaneously creating ModifiedModule Analysis - Partial (MMA-P). MMA-P applied1the same methodology as MMA, but to the partial cor-relation matrix rather than the correlation matrix. Thischange allowed MMA-P to identify incorrect communi-ties as did MMA, but also fully correct communities andcommunities that mixed correct and incorrect responses.The incorrect communities identified were generally con-sistent with those identified by MMA. The correct com-munities generally followed those identified by Stewart etal. [16]; therefore, MMA-P productively identified bothcorrect solution structure and incorrect structure at thesame time. MMA-P also identified communities whichmixed correct and incorrect answers. These communi- ties were productive in understanding answering patternswhere the correct answer was possibly selected for the in-correct reason and in understanding some of the genderunfairness identified by Traxler et al. . The post-test rea-soning of men and women was generally consistent.
ACKNOWLEDGMENTS
This work was supported in part by the National Sci-ence Foundation as part of the evaluation of improvedlearning for the Physics Teacher Education Coalition,PHY-0108787. [1] D. Hestenes, M. Wells, and G. Swackhamer, “Force Con-cept Inventory,” Phys. Teach. , 141–158 (1992).[2] R.K. Thornton and D.R. Sokoloff, “Assessing stu-dent learning of Newton’s laws: The Force andMotion Conceptual Evaluation and the evaluationof active learning laboratory and lecture curricula,”Am. J. Phys. , 338–352 (1998).[3] D. Huffman and P. Heller, “What does theForce Concept Inventory actually measure?”Phys. Teach. , 138 (1995).[4] E. Brewe, J. Bruun, and I.G. Bearden, “Using mod-ule analysis for multiple choice responses: A newmethod applied to Force Concept Inventory data,”Phys. Rev. Phys. Educ. Res. , 020131 (2016).[5] J. Wells, R. Henderson, J. Stewart, G. Stew-art, J. Yang, and A. Traxler, “Exploring thestructure of misconceptions in the Force Con-cept Inventory with modified module analysis,”Phys. Rev. Phys. Educ. Res. , 020122 (2019).[6] S. Sircar and N. Parekh, “Functional characterization ofdrought-responsive modules and genes in Oryza sativa:a network-based approach,” Front. Gen. (2015).[7] J.A. Smith and R. Faris, “Movement without mobility:Adolescent status hierarchies and the contextual limitsof cumulative advantage,” Soc. Networks , 139–153(2015).[8] G. Csardi and T. Nepusz, “The igraph soft-ware package for complex network research,”InterJournal, Complex Systems , 1–9 (2006).[9] R Core Team, R: A Language and Environment for Statistical Computing ,R Foundation for Statistical Computing, Vienna, Austria(2017).[10] S. Salehi, E. Burkholder, G.P. Lepage, S. Pollock,and C. Wieman, “Demographic gaps or preparationgaps?: The large impact of incoming preparationon performance of students in introductory physics,”Phys. Rev. Phys. Educ. Res. , 020114 (2019).[11] R. Henderson and J. Stewart, “Racial and ethnic biasin the Force Concept Inventory,” in Physics EducationResearch Conference 2017 , PER Conference (Cincinnati,OH, 2017) pp. 172–175.[12] A. Madsen, S.B. McKagan, and E. Sayre, “Gender gapon concept inventories in physics: What is consistent,what is inconsistent, and what factors influence the gap?”Phys. Rev. Phys. Educ. Res. , 020121 (2013). [13] A. Traxler, R. Henderson, J. Stewart, G. Stewart, A. Pa-pak, and R. Lindell, “Gender fairness within the ForceConcept Inventory,” Phys. Rev. Phys. Educ. Res. ,010103 (2018).[14] “Table II for the Force Concept In-ventory (revised from 081695r),” http://modeling.asu.edu/R&E/FCI-RevisedTable-II_2010.pdf .Accessed 3/17/2019.[15] J. Wells, R. Henderson, A. Traxler, and J. Stewart, “Ex-ploring the structure of misconceptions in the force andmotion conceptual evaluation with modified module anal-ysis,” Submitted to Phys. Rev. Phys. Educ. Res. January,2020.[16] J. Stewart, C. Zabriskie, S. DeVore, andG. Stewart, “Multidimensional item responsetheory and the Force Concept Inventory,”Phys. Rev. Phys. Educ. Res. , 010137 (2018).[17] J. Yang, C. Zabriskie, and J. Stewart, “Multidimensionalitem response theory and the Force and Motion Concep-tual Evaluation,” Phys. Rev. Phys. Educ. Res. , 020141(2019).[18] T.F. Scott, D. Schumayer, and A.R. Gray, “Exploratoryfactor analysis of a Force Concept Inventory data set,”Phys. Rev. Phys. Educ. Res. , 020105 (2012).[19] T.F. Scott and D. Schumayer, “Students’ proficiencyscores within multitrait item response theory,” Phys.Rev. Phys. Educ. Res. , 020134 (2015).[20] M.R. Semak, R.D. Dietz, R.H. Pearson, andC.W. Willis, “Examining evolving performance onthe Force Concept Inventory using factor analysis,”Phys. Rev. Phys. Educ. Res. , 010103 (2017).[21] C. Nord, S. Roey, S. Perkins, M. Lyons,N. Lemanski, J. Schuknecht, and J. Brown,“American High School Graduates: Results of the 2009 NAEP High School Transcript Study,” USDepartment of Education, National Center for EducationStatistics, Washington, DC (2011).[22] B.C. Cunningham, K.M. Hoyer, and D. Sparks, Gender Differences in Science, Technology, Engineering, and Mathematics (STEM) Interest, Credits Earned, and NAEP Performance in the 12th Grade (National Center for Education Statistics, Washington,DC, 2015).[23] B.C. Cunningham, K.M. Hoyer, and D. Sparks,“The Condition of STEM 2016,” ACT Inc., Iowa City,IA (2016).[24] Y. Maeda and S.Y. Yoon, “A meta-analysis on gender dif-ferences in mental rotation ability measured by the Pur- due Spatial Visualization Tests: Visualization of Rota-tions (PSVT: R),” Educ. Psychol. Rev. , 69–94 (2013).[25] D.F. Halpern, Sex Differences in Cognitive Abilities, 4thed. (Psychology Press, Francis & Tayler Group, NewYork, NY, 2012).[26] J.S. Hyde and M.C. Linn, “Gender differences in verbalability: A meta-analysis.” Psychol. Bull. , 53 (1988).[27] J.S. Hyde, E. Fennema, and S.J. Lamon, “Gender dif-ferences in mathematics performance: A meta-analysis.”Psychol. Bull. , 139 (1990).[28] N.M. Else-Quest, J.S. Hyde, and M.C. Linn, “Cross-national patterns of gender differences in mathematics:A meta-analysis.” Psychol. Bull. , 103 (2010).[29] X. Ma, “A meta-analysis of the relationship between anx-iety toward mathematics and achievement in mathemat-ics,” Jour. Res. Math. Educ. , 520–540 (1999).[30] J.V. Mallow and S.L. Greenburg, “Science anxiety:Causes and remedies,” J. Coll. Sci. Teach. , 356–358(1982).[31] M.K. Udo, G.P. Ramsey, and J.V. Mallow, “Science anx-iety and gender in students taking general education sci-ence courses,” J. Sci. Educ. Technol. , 435–446 (2004).[32] J. Mallow, H. Kastrup, F.B. Bryant, N. Hislop,R. Shefner, and M. Udo, “Science anxiety, science atti-tudes, and gender: Interviews from a binational study,”J. Sci. Educ. Technol. , 356–369 (2010).[33] J.R. Shapiro and A.M. Williams, “The role ofstereotype threats in undermining girls’ andwomen’s performance and interest in STEM fields,”Sex Roles , 175–183 (2012).[34] R.D. Dietz, R.H. Pearson, M.R. Semak, and C.W.Willis, “Gender bias in the Force Concept Inventory?” in ,Vol. 1413, edited by N.S. Rebello, P.V. Engelhardt, andC. Singh (AIP Publishing, New York, 2012) pp. 171–174.[35] S. Osborn Popp, D. Meltzer, and M.C. Megowan-Romanowicz, “Is the Force Concept Inventory bi-ased? Investigating differential item functioningon a test of conceptual learning in physics,” in (American Education Research Association, Washington,DC, 2011).[36] L. McCullough and D.E. Meltzer, “Differ-ences in male/female response patterns onalternative-format versions of FCI items,” in ,edited by K. Cummings, S. Franklin, and J. Marx (AIPPublishing, New York, 2001) pp. 103–106.[37] B.E. Clauser and K.M. Mazor, “Using statistical proce-dures to identify differentially functioning test items,”Educ. Meas.–Issues Pra. , 31–44 (1998).[38] L. Viennot, “Spontaneous reasoning in elementary dy-namics,” Eur. J. Sci. Educ. , 205–221 (1979).[39] D.E. Trowbridge and L.C. McDermott, “Investigation ofstudent understanding of the concept of acceleration inone dimension,” Am. J. Phys. , 242–253 (1981).[40] A. Caramazza, M. McCloskey, and B. Green, “Naivebeliefs in “sophisticated” subjects: Misconceptions abouttrajectories of objects,” Cogn. , 117–123 (1981).[41] P.C. Peters, “Even honors students have conceptual dif-ficulties with physics,” Am. J. Phys. , 501–508 (1982).[42] M. McCloskey, “Intuitive physics,” Sci. Am. , 122–131 (1983). [43] R.F. Gunstone, “Student understanding in mechanics:A large population survey,” Am. J. Phys. , 691–696(1987).[44] C.W. Camp and J.J. Clement, Preconceptions in Me-chanics: Lessons Dealing with Students’ Conceptual Dif-ficulties (Kendall/Hunt, Dubuque, IA, 1994).[45] L.C. McDermott, “Students’ conceptions and prob-lem solving in mechanics,” in
Connecting Research inPhysics Education with Teacher Education , edited byA. Tiberghien, E.L. Jossem, and J. Barojas (Interna-tional Commission on Physics Education, 1997) pp. 42–47.[46] R. Rosenblatt and A.F. Heckler, “Systematic study ofstudent understanding of the relationships between thedirections of force, velocity, and acceleration in one di-mension,” Phys. Rev. Phys. Educ. Res. , 020112 (2011).[47] N. Erceg and I. Aviani, “Students’ understanding ofvelocity-time graphs and the sources of conceptual dif-ficulties,” Croat. J. Educ. , 43–80 (2014).[48] B. Waldrip, “Impact of a representational approach onstudents’ reasoning and conceptual understanding inlearning mechanics,” Int. J. Sci. Math. Educ. , 741–765 (2014).[49] A.A. diSessa, “Toward an epistemology of physics,”Cogn. Instr. , 105–225 (1993).[50] A.A. diSessa and B.L. Sherin, “What changes in concep-tual change?” Int. J. Sci. Educ. , 1155–1191 (1998).[51] M.T.H. Chi and J.D. Slotta, “The ontological coherenceof intuitive physics,” Cogn. Instr. , 249–260 (1993).[52] M.T.H. Chi, J.D Slotta, and N. De Leeuw, “From thingsto processes: A theory of conceptual change for learningscience concepts,” Learn. Instr. , 27–43 (1994).[53] J.D. Slotta, M.T.H. Chi, and E. Joram, “Assessing stu-dents’ misclassifications of physics concepts: An ontologi-cal basis for conceptual change,” Cogn. Instr. , 373–400(1995).[54] D. Hammer, “Misconceptions or p-prims: How may al-ternative perspectives of cognitive structure influence in-structional perceptions and intentions,” J. Learn. Sci. ,97–127 (1996).[55] D. Hammer, “More than misconceptions: Multi-ple perspectives on student knowledge and reason-ing, and an appropriate role for education research,”Am. J. Phys. , 1316–1325 (1996).[56] D. Hammer, “Student resources for learning introductoryphysics,” Am. J. Phys. , S52–S59 (2000).[57] J. Minstrell, “Facets of students’ knowledge and relevantinstruction,” in Research in Physics Learning: Theoret-ical Issues and Empirical Studies , edited by R. Duit,F. Goldberg, and H. Niedderer (IPN, Kiel, Germany,1992) pp. 110–128.[58] R.E. Scherr, “Modeling student thinking: An examplefrom special relativity,” Am. J. Phys. , 272–280 (2007).[59] I. Halloun, R.R. Hake, E.P. Mosca, and D. Hestenes,“Force Concept Inventory (revised 1995),” (1995), http://modeling.asu.edu/R&E/Research.html Ac-cessed 7/19/2019.[60] “Physport,” . Accessed8/8/2017.[61] “US News & World Report: Education,” https://premium.usnews.com/best-colleges . Ac-cessed 4/30/2017.[62] S. Kim, “ppcor: An R package for a fast calculationto semi-partial correlation coefficients,” Commun. Stat. Appl. M. , 665 (2015).[63] A. Canty and B.D. Ripley, boot: Bootstrap R (S-Plus)Functions (2017), R package version 1.3-20.[64] K.A. Zweig, Network Analysis Literacy: A Practical Ap-proach to the Analysis of Networks (Springer-Verlag,Wien, Austria, 2016).[65] D. Hestenes and I. Halloun, “Interpreting the Force Con-cept Inventory: A response to March 1995 critique byHuffman and Heller,” Phys. Teach. , 502–502 (1995).[66] P. Heller and D. Huffman, “Interpreting the Force Con-cept Inventory: A reply to Hestenes and Halloun,” Phys.Teach. , 503–503 (1995). [67] S. DeVore, J. Stewart, and G. Stewart, “Examining theeffects of testwiseness in conceptual physics evaluations,”Phys. Rev. Phys. Educ. Res. , 020138 (2016).[68] D.P. Maloney, T.L. O’Kuma, C. Hieggelke, andA. Van Huevelen, “Surveying students’ concep-tual knowledge of electricity and magnetism,”Am. J. Phys. , S12–S23 (2001).[69] L. Ding, R. Chabay, B. Sherwood, and R. Beich-ner, “Evaluating an electricity and magnetism assess-ment tool: Brief Electricity and Magnetism Assessment,”Phys. Rev. ST Phys. Educ. Res.2