Broader terms curriculum mapping: Using natural language processing and visual-supported communication to create representative program planning experiences
Rogério Duarte, ?ngela Lacerda Nobre, Fernando Pimentel, Marc Jacquinet
BB ROADER T ERMS C URRICULUM M APPING : U
SING NATURALLANGUAGE PROCESSING AND VISUAL - SUPPORTEDCOMMUNICATION TO CREATE REPRESENTATIVE PROGRAMPLANNING EXPERIENCES
A P
REPRINT
Rogério Duarte ∗ CINEA, DEM, Escola Superior de Tecnologia de SetúbalInstituto Politécnico de SetúbalCampus do IPS, Estefanilha, 2914-761 Setúbal, Portugal [email protected]
Ângela Lacerda-Nobre
DEG, Escola Superior de Ciências EmpresariaisInstituto Politécnico de SetúbalCampus do IPS, Estefanilha, 2914-503 Setúbal, Portugal [email protected]
Fernando Pimentel
DEM, Escola Superior de Tecnologia de SetúbalInstituto Politécnico de SetúbalCampus do IPS, Estefanilha, 2914-761 Setúbal, Portugal [email protected]
Marc Jacquinet
DCSG, Universidade AbertaRua da Escola Politécnica, 141-147, 1269-001 Lisboa, Portugal [email protected]. A BSTRACT
Accreditation bodies call for curriculum development processes open to all stakeholders, reflectingviewpoints of students, industry, university faculty and society. However, communication difficultiesbetween faculty and non-faculty groups leave unexplored an immense collaboration potential. Usingclassification of learning objectives, natural language processing, and data visualization, this paperpresents a method to deliver program plan representations that are universal, self-explanatory, andempowering. A simple example shows how the method contributes to representative program planningexperiences and a case study is used to confirm the method’s accuracy and utility.
Keywords
Curriculum mapping · Natural Language Processing (NLP) · Network graphs · Learningobjectives classification · Curriculum analytics · Curriculum design · Higher education ∗ Corresponding author. a r X i v : . [ c s . C L ] F e b roader Terms Curriculum Mapping A P
REPRINT
Changing times impose new and radical challenges for societies, urging Higher Education Institutions (HEI) to rethinktheir educational offer. Curriculum development is the process where changes to educational offer are conceivedand, to be successful, this process needs to create opportunities for active and consequent reflection. To create theseopportunities, stakeholders’ participation is essential. This is the unanimous opinion of researchers and accreditationbodies [ABET, 2020, Crawley et al., 2007, Sutherland, 2018] who defend curriculum development open to all stakeholdergroups, expressing the viewpoints of students, industry, university faculty and society.Focusing on program planning, a core process that lays at the heart of curriculum development, open principles aretypically associated with interviews, focus group sessions, where non-faculty stakeholders are asked to express theirviews. Faculty, who hold the central managing role [Wilson and Slade, 2020], is responsible for processing the datacollected in these informational touchpoints, and it is faculty who participate in program planning discussions.Faculty central role in program planning is not only a traditional functional attribute. Program planning requiresscientific and pedagogic skills essential to—and therefore mastered by—faculty, that are not essential to non-facultystakeholders. This opens an important communication gap, and the absence of dialogue between faculty and non-facultygroups [Ornstein and Hunkins, 2018] leaves unexplored an immense collaboration potential.Yet, the challenges imposed by a rapidly changing society; recognizing that responsive and effective program plans aremore likely with participatory (not just informational) involvement of all those concerned, forces HEI to reach out andfind ways to integrate external contributions, valuing non-faculty stakeholders as experts of their own experience.To achieve this objective, to bridge communication gaps, better representations of the program plan are essential. Thispaper proposes the use of a broader terms curriculum mapping method to deliver program plan representations that areuniversal, self-explanatory, empowering.The core objective of this paper is the presentation of the broader terms CM (Curriculum Mapping) method. Becausethis method uses a combination of information and data science techniques, a significant part of the paper is dedicatedto the step by step description—using a simple example—of these techniques, and how they contribute to representativeprogram planning experiences. To verify the method’s accuracy and utility, a case study is used.But, before proceeding to the broader terms CM method presentation, it is important to explain what is meant bycurriculum mapping. How it has been used previously, and what changes are required to make it a vehicle forrepresentative program planning. This is the topic of the next section.
According to Burns [2001], curriculum mapping is a process for recording what content and skills are taught in a studyprogram. The recording relies on a visual medium, typically a chart, table, or map, depicting the building blocks of thestudy program and how these blocks relate to one another. Because different types of building blocks could be used,there are different types of curriculum mapping.When individual courses are the building blocks, curriculum mapping provides a snapshot of existing learning pathwaysconsidering the available courses, helping students navigate the study program. These course mappings use the calendaryear as an organizer to depict vertical (from year to year) and horizontal (within a year) relations between courses[Burns, 2001], and are usually represented as flowcharts. Meij and Merx [2018] provide an example of a coursemapping published online, with course-specific scientific and pedagogic details available as hyperlinked content.For accreditation bodies, the grouping of contents and skills per courses is not as relevant as ensuring these contentsand skills result in expected learning outcomes [Felder and Brent, 2003, Crawley et al., 2007]. For this reason, foraccreditation purposes, learning outcomes are the building blocks, and learning outcomes mappings are used to showthe study program yields the expected learning outcomes. These mappings are typically represented as tables aligningprogram learning outcomes and accreditation standards. Examples of learning outcomes mappings are given in Dyjurand Lock [2016].
Learning outcomes mappings’ purpose goes beyond reporting the alignment with accreditation standards. This type ofcurriculum mapping is used to communicate accreditation bodies vision of transparency, accountability and scientificcurriculum development [Ornstein and Hunkins, 2018]. A vision that becomes reality with HEI adoption of outcomes-based education [Spady, 1988, Harden, 2001] and constructive alignment principles [Biggs and Tang, 2011, p.99].Curriculum mapping is used, therefore, as a tool to shape HEI processes, particularly, program planning.2roader Terms Curriculum Mapping
A P
REPRINT
Willcox and Huang describe another type of curriculum mapping: the concept mapping. Concepts are in this caseused for building blocks, with a concept denoting “the main idea underlying a (typically small) unit of content coveredin a course” [Willcox and Huang, 2017, p.9]. These units of content are linked to
Knowledge Concepts defined by[Koedinger et al., 2012], and Willcox and Huang use concept mappings to provide insight into the relations betweenlearning outcomes and between courses, helping faculty with the precise program plan navigation. Examples of thistype of mapping are given in Seering et al. [2015], Willcox and Huang [2017] or Varagnolo et al. [2020]. These authorsuse circular ideograms [Krzywinski et al., 2009] and/ or network graphs [Rosen, 2009] to detail concepts’ precedencerelations. The visual outputs presented by these researchers are very successful and efficient in conveying visualmeaning to the complex relations found study programs.The analysis of the three types of curriculum mapping reveals important characteristics. Curriculum mapping isused to shape HEI processes, and this ability is valuable for opening program planning discussions to non-facultygroups. Curriculum mapping uses visual-supported communication to represent and discuss study programs, and thedevelopments taking place in the field of information visualization can be used to bridge communication gaps, helpingstakeholders to articulate their expert (non-verbal) knowledge. However, as regards the choice of building blocks, if theobjective is to increase non-faculty groups participatory involvement, broader (not detailed) concepts, requiring lessscientific and pedagogic skills should be preferred.A curriculum mapping method that builds on the practices already available but tailored for non-faculty groupsparticipation in program planning discussions is described in the next section.
This section presents a curriculum mapping method designed for representative program planning. A method thatempowers all stakeholders.A flowchart representing the method steps, respective inputs and outputs, is presented in Figure 1.
Figure 1: Flowchart representing the steps, respective inputs and outputs of the broader terms curriculum mapping method.
The method considers four steps—detailed in the following subsections—: (1) classification of course learning objectivestatements into broader terms; (2) use of Natural Language Processing (NLP) to convert broader terms into quantitativefrequencies of key program concepts; (3) visualization of key program concept frequencies and mappings with linksbetween key concepts and/ or courses; (4) discussion, considering the participation of all stakeholder groups, of themethod’s visual outputs and decision to reclassify or review course learning objectives.To illustrate how these steps apply, a simple example is considered. Table 1 presents data for this example.3roader Terms Curriculum Mapping
A P
REPRINT
Table 1: Learning objectives and respective broader terms classification for 5 courses. Notes: (1) A non-truncated version of thistable is provided with the supplementary material [Duarte, 2020]. (2) Data in this table models style and scope variability frequentlyfound in learning objective statements and considers different levels of detail in broader terms selection.
Course Learning objectives Broader terms ( a ) Mathematics(C1:MATH) Recognize a real-valued function of a real variable; (. . . )Recall the concept of derivative of a real function andexplain its geometric interpretation; (. . . ) Function of a real variable; Differential calculus; Integ-ral calculus; Linear algebra; System of linear equationsApplied Physics(C2:PHY) List fundamental concepts in mechanics and understandtheir importance to engineering; Use the internationalunits (. . . ) Physics; Mathematics; Calculus; Mechanics; Thermody-namics; Fluid flowLogistics & Oper.Manag.(C3:LOGOP) Identify logistic activities in a generic organization; Ex-plain the role of contemporary logistics; (. . . ) Distin-guish components of supply chain management (. . . ) Logistics; Supply chain management; Business; Produc-tion Economics; Operations management; Linear pro-gramming; Lean manufacturing; Process Control (. . . )Energy Manag.(C4:ENER) Discuss the efficient use of energy in industry, buildingsand transports; Recognize applicable legislation and de-fend energy efficiency as (. . . ) Energy efficiency; Organization; Buildings; Facilitymanagement; Logistics; Production planning and con-trol; Solar water heating (. . . ).Financial Manag.(C5:FIN) List fundamental financial management concepts andfunctions; Recognize and explain financial statements;Contrast the economic and the financial analysis (. . . ) Financial management; Accounting; Economics; Fin-ance; Organization; Business governance; Corporatelaw; Trade; Return on invested capital (. . . )(a) Obtained with the Wikipedia index [Wikipedia contributors, 2020].
Table 1 includes the learning objectives for 5 courses—Mathematics (C1:MATH), Applied Physics (C2:PHY), Logist-ics & Operations Management (C3:LOGOP), Energy Management (C4:ENER) and Financial Management (C5:FIN)—of a bachelor degree in Technology and Industrial Management (also used in the case study section). For example, thefirst learning objective statement in the Mathematics (C1:MATH) course is: “Recognize a real-valued function of a realvariable”.The third column of Table 1 presents broader terms derived from the courses learning objectives. The methodologyused to obtain these broader terms is described in the following subsection.
To characterize courses and the program-degree, the broader terms CM method uses course Learning Objectives (LO).According to Felder and Brent [2003, p.19], course LO are defined as “statements of observable actions that serve asevidence of the knowledge, skills and attitudes acquired in a course.” These statements define key program conceptsand through these key concepts the intricate web of course relations is revealed. Course LO provide, therefore, accessto the “mechanics” behind a program plan.The problem of using course LO is that they presume tacit understanding of concepts specific to disciplinary andscientific sub-areas, and this renders LO-statements seldom clear and unequivocal [Ballantyne et al., 2019, Watts andHodgson, 2015]. Even when LO are written according to specific rules (e.g., considering Bloom’s taxonomy, Bloom,1956, Adam, 2004, Felder and Brent, 2003), the variability in style and scope results in a heterogeneous set, includingstatements that are often too abstract or too detailed [Lam and Tsui, 2016, Hussey and Smith, 2003].To disclose their latent information and for effective communication, LO-statements would benefit from techniquesused by library and information science professionals in resource classification. Resource classification indicates what aresource is about, and to achieve this goal a control vocabulary, a set of broader terms (concepts or subject headings,Lancaster, 2003) supporting classification has to be agreed upon. Control vocabularies are usually chosen amongbibliographic classification schemes (such as the Dewey Decimal Classification), lists of subject headings [Library ofCongress, S/D] and thesauri [EUROVOC, S/D, UNESCO, S/D, IEEE, 2019]. More recently, for its comprehensivenessand up-to-dateness, the Wikipedia index [Wikipedia contributors, 2020] is also used (see Joorabchi and Mahdi, 2013and Bergman, 2015 for a discussion of the advantages of using Wikipedia’s index as control vocabulary).This paper considers principles of resource classification to classify course LO. Concepts from Wikipedia indexmatching course LO-statements are used to define broader terms. To illustrate how this is done, consider the excerpt ofLO statements for Mathematics (C1:MATH) in Table 1: “Recognize a real-valued function of a real variable; Recallthe concept of derivative of a real function and explain its geometric interpretation.” Using Wikipedia index, the firststatement could be classified according to the Wikipedia concept, “Function of a real variable”; a matching conceptalmost identical to the original LO. However, broader concepts could be chosen. For example, the second LO statement4roader Terms Curriculum Mapping
A P
REPRINT could be classified (with exaggeration) by the broader Wikipedia concept “Differential calculus”. The third column inTable 1 includes results of course LO-statements classification for the 5 courses.The classification example provided in the previous paragraph, and the comparison of columns two and three in Table 1,shows a significant reduction in the vocabulary used to characterize the courses. Because the reduction in vocabularycould entail an important loss of information, it follows the importance of the conceptual analysis of LO-statements[Lancaster, 2003]. The importance of determining what the LO statement is about—the “aboutness”—and of thetranslation into (the selection of) specific broader terms.Because program planning relies, typically, on the declared curriculum, with LO-statements written by universityfaculty, the classification of LO is frequently done by faculty in collaboration with curriculum planners [Willcox andHuang, 2017, Seering et al., 2015, Varagnolo et al., 2020]. An alternative procedure consists of an initial classificationby an information science professional, subsequently revised by faculty [Ballantyne et al., 2019]. For large databaseclassification, automated machine learning techniques are also used [Golub, S/D, West, 2016]. In this paper, aninitial draft classification of course LO is made by a small multidisciplinary team of university faculty. Once visualoutputs derived with the broader terms CM method are available, a reclassification is made with contributions from allstakeholders (see Step 4 in Figure 1).For effective communication of the program plan, having courses associated with a small subset of broader termsselected from a control vocabulary is an important advantage. Key concepts found in courses can be identified, pavingthe way to their quantification and to the analysis of the relations between courses, i.e., to the analysis of informationflows, such as topics covered, which assessments relate to which topic, and so on.The next section describes in detail the method used in the quantitative processing of broader terms.
This paper uses Natural Language Processing (NLP, Manning and Schütze, 1999, West, 2018) to convert broader termsassigned to courses into quantitative data; i.e., into frequencies of words. It will be assumed that these words—thesetokens as they are called in the NPL literature [Manning and Schütze, 1999]—extracted from broader terms, still carryconceptual meaning and can still be used to characterize courses and the program-degree. For this reason, in this paper, token and key (program or course) concept , K, are used as synonyms.NLP applies a sequence of processing functions to an original set of broader terms. Tokenization, the first of thesefunctions, identifies words in broader terms that are included in a corpus; in a dictionary of tokens. Recalling Section3.1 example of obtaining broader terms for Mathematics (C1:MATH)—“Function of a real variable” and “differentialcalculus” were the resulting broader terms—, and, considering the corpus of English words; after tokenization thefollowing set of tokens { Function, of, a, real, variable, differential, calculus } characterizes the Mathematics course.But the above set includes tokens (i.e., “of” and “a”) that add no value to the course characterization; therefore, thesetokens—known as stop-words—, as well as any punctuation signs and numerals, should be removed. Moreover, wordswritten with capital letters and different conjugations of the same word should be replaced by an adequate “stem-word”(in a process known as stemming, Manning and Schütze, 1999).Denoting the stemming and the purging of meaningless tokens as normalization, if a study program has N courses,after tokenization and normalization of course C i broader terms ( i ∈ { , , . . . , N } ), a multiset K i (allowing multipleinstances of the same token) of m i tokens K i,k is obtained (with k = 1 , , . . . , m i ). For the program-degree as a whole,a set K (no repetitions) with a total of M = | (cid:83) Ni =1 K i | tokens is obtained.With K j the j th token in set K , the frequency of this token in course C i is found from, b i,j = m i (cid:88) k =1 δ i,k ( K j ) , (1)with δ i,k ( K j ) = (cid:26) , K i,k (cid:54) = K j , K i,k = K j . (2)Eq. (1) represents the elements of an N × M matrix B of token frequencies per course.5roader Terms Curriculum Mapping A P
REPRINT
Considering the broader terms for the 5 courses in Table 1 (column three), after tokenization and normalization , theresulting course-token matrix is, B = K1:manag K2:calculus K3:control K4:energi K5:linear K6:logistic (. . . ) . . . ) C1:MATH . . . ) C2:PHY . . . ) C3:LOGOP . . . ) C4:ENER . . . ) C5:FIN , (3) where, given the large number of identified tokens (70), only the columns for the six most frequent are shown.Observe how this matrix attaches quantitative information to courses based on token frequency. Observe, for instance,the link that emerges between courses C1:MATH and C2:PHY via token K2:calculus. Matrix B shows this token isfound twice among the tokens associated with course C1:MATH, and once among those associated with course C2:PHY(see also the underlined words in Table 1).This ability to describe a study program quantitatively is an important breakthrough; a way to bridge the gap created bytacit understanding, unclear LO-statements. But, at the same time, notice how unpractical is the analysis of the data inmatrix format.To achieve a clearer understanding of the quantitative data emerging from NLP, an alternative to matrix or tabularrepresentations of data is essential. An important result from NLP are token frequencies: the column-wise sum of the elements in the course-token matrix.A convenient visual representation of these frequencies is obtained with wordclouds. Figure 2 presents a wordcloudfrom data in matrix B (Eq. 3). Figure 2: Wordcloud of token frequencies for the 5 courses example. Graph obtained using the
Wordcloud package [Fellows, 2018]for the R programming language [R Core Team, 2019]—see supplementary material [Duarte, 2020]. Figure 2 identifies the most frequent key program concepts—manag[ement], calculus, control, energi, linear, logistic—,represented with larger font size in a central position.Figure 2 is adequate to identify the relative importance of different key concepts, but provides no information concerningthe relations between these or between courses. To represent these relations researchers can choose among severalalternatives. One which captures all data in course-token matrix and makes patterns and descriptive statistics visibleis presented in Figure 3 a). It is the circular ideogram representation [Krzywinski et al., 2009] of the data in matrix B = B [1 : 5; 1 : 6] , a submatrix including the first 6 columns of matrix B (Eq. 3). These functions are available in most mathematical and numerical computing tools. R programming code [R Core Team, 2019]used in this section is included in supplementary material, [Duarte, 2020]. A P
REPRINT
Figure 3: Visual representation of matrix B as: (a) a circular ideogram; (b) a multigraph. The circular ideogram [Krzywinskiet al., 2009] was obtained with the Circlize package [Gu et al., 2014] and the multigraph was obtained with the iGraph package[Csardi and Nepusz, 2006]. Both packages for the R programming language [R Core Team, 2019]—see supplementary material[Duarte, 2020]. The outer circumference in Figure 3 a) displays the 5 courses C i on the right side and the 6 tokens K j on the left side.This circumference specifies the number of links (see scale) between courses and tokens. For example, courses C3 andC4 have the largest number of links (8 and 7, respectively) to tokens. Token K1 has the largest number of links (6) tocourses.But the advantage of the circular ideogram comes, especially, from the inner circle in Figure 3 a), and from the stripesthat link courses and tokens. The inner circle in Figure 3 a) emphasizes the (previously mentioned) link between coursesC1:MATH and C2:PHY via token K2:calculus (see purple stripe). But much more is revealed: for example, whilecourse C2:PHY has no further associations, course C1:MATH is also related to course C3:LOGOP through K5:linear(green stripe). The width of the stripes—the strength of the links—connecting C1:MATH to K2:calculus and K5:linearis also larger than the width for the stripes connecting these tokens to courses C2:PHY and C3:LOGOP. Given that“calculus” and “linear” are mathematics-related tokens, these results were expected, and the expert analysis of the 5courses LO-statements (in Table 1) should result in identical conclusions. But in Figure 3, the combination of stripes’curvature, color and width renders the analysis universal, self-explanatory, empowering, uncovering latent informationand helping the verbal articulation of expert (non-verbal) knowledge.Another equally useful visual representation of data in matrix B is presented in Figure 3 b). Consider the C i andK j in the outer circumference of Figure 3 a) as vertices V = { v C1 , v C2 , . . . , v K1 , v K2 , . . . } , and the inner circle stripesas edges E = { e C1-K2 , e
C1-K5 , . . . } of an undirected multigraph G = (cid:104) V, E (cid:105) . Figure 3 b) represents this multigraph withcourse and token vertices laid out in a way that communicates vertex centrality, i.e., where the number (the cardinality)of vertex links determine the vertex position [Fruchterman and Reingold, 1991]. Notice how vertices C3 and C4—withlargest number of links—shape a central cluster, while vertices C1 and C2 protrude to the periphery. Moreover, vertexcentrality is emphasized through course vertices diameter; with larger diameters representing courses with a largernumber of incident links.In matrix format, the multigraph in Figure 3 b) for the 5 courses and 6 most frequent tokens is,7roader Terms Curriculum Mapping
A P
REPRINT A = (cid:32) B T (cid:33) = C1 C2 C3 C4 C5 K1 K2 K3 K4 K5 K6 C1 C2 C3 C4 C5 K1 K2 K3 K4 K5 K6 , (4) a square biadjacency matrix obtained from B (superscript T denotes matrix transpose).Figures 3 a) and 3 b) provide important insights on how key concepts and courses interrelate. However, a simpler andyet very useful representation would consist of the direct links between courses and between tokens.Observing Figure 3 b) and matrix A we conclude that elements of the biadjacency matrix represent the cardinalityof 1-walks between consecutive vertices—with a k -walk defined as the sequence of k edges ( e , e , . . . , e k ) joining k + 1 vertices ( v , v , . . . , v k +1 ) [Rosen, 2009]. For example, matrix A shows that between vertices v C1 and v K2 there are two 1-walks, v C1 · e C1-K2 −−−−→ v K2 . Between vertices v K2 and v C2 there is one 1-walk, v K2 · e K2-C2 −−−−→ v C2 . This isconfirmed in Figure 3 b).A direct link between vertices v C1 and v C2 could be conceived as two 2-walks joining these vertices, represented as v C1 · e C1-K2-C2 −−−−−−→ v C2 , with · e C1-K2-C2 denoting the two available options to go from C1 to C2.For the 5 courses example, using matrix algebra, the number of 2-walks between course vertices and between tokenvertices is found from the 2 nd power of the biadjacency matrix, with the diagonal elements of the resulting matrix madeequal to zero [Rosen, 2009]. With L denoting the 2-walk matrix, it follows L = A − diag (cid:0) A (cid:1) , andreplacing A gives, L = C1 C2 C3 C4 C5 K1 K2 K3 K4 K5 K6 C1 C2 C3 C4 C5 K1 K2 K3 K4 K5 K6 = (cid:32) L
00 L (cid:33) . (5) Submatrices L = L [1 : 5; 1 : 5] and L = L [6 : 11; 6 : 11] in Eq. (5) represent the number of possible2-walks between consecutive courses and consecutive tokens, respectively.To confirm the results discussed previously for the direct link between vertices v C1 and v C2 , notice the value 2 found inmatrix element L [1; 2] (or L [2; 1] , because the graph is undirected).Using submatrices L and L , representations of the direct links between courses and between key concepts arepresented in Figures 4 a) and 4 b), respectively. The strength of the links—the cardinality of possible 2-walks—isgiven both by numbers and by edge widths. Moreover, as for Figure 3 b), vertices layout and vertex diameter provide asuggestive visual depiction of core and peripheral courses/ key concepts.8roader Terms Curriculum Mapping A P
REPRINT
Figure 4: Graphs showing direct links between: a) courses (from L ); b) key concepts (from L ). Numbers and edge widthsrepresent the strength of the link. Graphs produced with the iGraph package [Csardi and Nepusz, 2006] for the R programminglanguage [R Core Team, 2019]—see supplementary material [Duarte, 2020]. Figure 4 a) shows the largest number of possible 2-walks (12) occurs between courses C3:LOGOP and C4:ENER.This value can be verified in Figure 3 b). The way key concepts influence links between courses is clearly reflected incourses C2:PHY and C5:FIN locations. Although C2:PHY and C5:FIN have both a single key concept among the 6most frequent—K2:calculus and K1:manag, respectively (see Eq. 3)—, the fact that “manag[ment]” is more commonthan the mathematics-related concept pulls C5:FIN closer to where core program courses lay, whereas C2:PHY ispushed to a peripheral location.Figure 4 b) confirms the peripheral role played by mathematics-related concept, K2:calculus; and it is interesting tocontrast this graph’s discriminating potential with that of the wordcloud in Figure 2. Indeed, no evidence is found in thewordcloud as to differences between tokens K2 to K6 (because the number of edges incident on vertices v K2 to v K6 isthe same for these tokens, 3).Figures 4 a) and Figure 3 b) provide visual evidence of course C2:PHY detachment from the remaining courses.Obviously, reasons for this should be discussed; in particular, the absence of a (expected) link between C2:PHY andC4:ENER.Results from this section show visual outputs from the broader terms CM method provide evidence-based details onweaknesses (and strengths) in program plans; namely, related to key program concepts and to the interrelations betweenthese and/ or courses. With the adoption of the broader terms CM method the focus of program planning discussions is shifted from thediscussion of written statements of course LO—seldom clear and unequivocal—; from atomized discourses about thelinks between courses, to the interpretation of quantitative data communicated visually in a way understandable to all.Because of the universal, self-explanatory quality of its visual outputs —of the mappings—, the broader terms CMmethod empowers all stakeholders, allowing participatory involvement of non-faculty groups in program planningdiscussions. Because of its quantitative nature, the broader terms CM method nurtures constructive critique, effectivelyaddressing disciplinary and scientific boundaries, hierarchical and functional differences, and atomized discourses.With the objective identification of program plan weaknesses, it is possible to unfreeze [Schein, 1999] long establishedbeliefs, preparing the agreement for change with contributions from all stakeholders (see the review feedback loop inFigure 1).Some of the weaknesses identified in the mappings may derive from course LO classification. Section 3.1 stated aninitial draft classification was made by a small multidisciplinary team of university faculty. During the discussion step,9roader Terms Curriculum Mapping
A P
REPRINT with the help of mappings, classification problems are easily identified, justifying the reclassification feedback loop inFigure 1.As mentioned in Section 3.1, this reclassification carries some subjectivity. Different broader terms could be chosen toclassify a LO-statement; and there could be LO-statements for which an adequate broader term is not included in thecontrol vocabulary. However, the technical nature of control vocabularies and of the classification task makes selectionof broader terms distinctly less subjective than the head-on discussion of LO-statements.Using the mappings for the 5 courses example we elaborate on relevant discussion topics that would benefit from theparticipatory involvement of all stakeholders.Considering frequencies and links between key program concepts, in Figures 3 a) and 4 b), stakeholders (namely,industry and society groups) could contribute with their experience to identify important key concepts, essential links,considering not only scientific and pedagogic arguments, but also the mission of HEI in the context of rapidly changingtechnological, economical, societal and political environments. With respect to the links between courses, and the linksbetween courses and key concepts, in Figures 4 a) and 3 b), student and graduate groups could contribute with theirexperience to contrast the differences between the declared and the enacted curriculum [Arafeh, 2016, Varagnolo et al.,2020].Concerning the lack of an expected link between C2:PHY and C4:ENER, mentioned at the end of last subsection—recallFigure 4 a)—; given that Applied Physics and Energy Management syllabuses are typically linked by thermodynamics,heat transfer and fluid flow topics, to express the importance of this link, stakeholders could use handwritten notes tocommunicate a desirable change, as depicted in Figure 5.
Figure 5: Handwritten notes communicating a desirable change to the mapping in Figure 4 a). Example of how broader termscurriculum mapping can be used by stakeholders during the program planning discussions.
Figure 5 demonstrates the ease with which stakeholders take possession of the mappings. The dashed lines show thepreferred location of C2:PHY (and C1:MATH), closer to core courses. Text in square brackets points to broader termsjustifying the link between C2:PHY and C4:ENER. Figure 5 could be the starting point for the revision of these coursesLO; perhaps considering another forum and using detailed concept mappings.To conclude this section, notice that having used a simple example with only 5 courses, it is not possible to verify if themethod most frequent key program concepts and if the links between these and/ or courses are accurate. To assess theaccuracy of the broader terms CM method, the next section presents a case study.
To evaluate the accuracy of the broader terms CM method, results obtained with this method should be confirmed byactual observations. For this purpose, this section uses a bachelor degree—Technology and Industrial Management(T&IM)—assessed by the Portuguese accreditation agency (A3ES) in 2013. The section starts with the genericpresentation of the T&IM study program and with the presentation of the recommendations issued by A3ES (theresults of the assessment). Afterwards, the broader terms CM method is used to generate mappings from courses10roader Terms Curriculum Mapping
A P
REPRINT
LO-statements. To evaluate the accuracy of the method, the mappings are compared to the recommendations—whichare deemed accurate. The section ends with a discussion of this comparison.
The bachelor degree (180 ECTS credits) in Technology and Industrial Management (T&IM, Lourenço et al., 2013,Duarte et al., 2014, 2018) was conceived in 2006 at the College of Engineering of Instituto Politécnico de Setúbal, aPortuguese public HEI. The degree targeted mature students working in the industry sector in the region of Setúbal.Considering the characteristics of the students—mature blue color workers with formal and informal skills in theirarea of professional expertise—and the advanced technological settings provided by the employing organizations(which include automotive, aeronautic and ship repair industries), the 2007-2012 program plan emphasized managerialcontents at the expense of engineering and mathematics. This emphasis on management topics is made clear in Figure6, a circular dendrogram representing T&IM courses and respective departments. Out of the 38 program courses, 18belonged to the Business Sciences Department.
Figure 6: Circular dendrogram representing T&IM courses (2007-2012 program plan) and respective departments. Courses belongingto the departments of Business Sciences (BScDep), Electrical Engineering (ElecEngDep), Informatics (InfDep), Mathematics(MathDep), Mechanical Engineering (MechEngD) and Process Control (ProcCtrlDep) are represented counterclockwise. Theresponsibility for Internship I&II is shared among departments and the asterisk symbol ( ∗ ) is used to identify elective courses. Another important characteristic represented in Figure 6 is the dispersal of T&IM core and elective courses among 6departments.Six years after it began, the Portuguese accreditation agency assessed the T&IM bachelor degree [A3ES, 2013]. Studyprogram data reporting to the 2007-2012 period was gathered, a self-assessment report was delivered by the HEI, anindependent panel of experts (representing A3ES) visited and met with HEI stakeholders.Regarding the program plan, the A3ES produced the following recommendations:i. Increase program-degree mathematical content.ii. Steer programming skills towards high-level languages with practical use.iii. Strengthen the program plan with important applied industrial management content, namely, in operations manage-ment, supply chain management and operational research.iv. Excessive number of courses, some with little additional content.v. Poor integration of topics taught in the different courses.11roader Terms Curriculum Mapping
A P
REPRINT
Concerning these recommendations, note that: (1) these are considered an accurate expression of weaknesses in the2007-2012 T&IM program plan; (2) the non-prescriptive (and somewhat vague) style of the recommendations resultsfrom accreditation criteria allowing program-degrees to adjust to different HEI missions, to student demographics andavailable resources.
Using LO-statements from the T&IM courses (2007-2012 program) and the methodology described in Figure 1(excluding the feedback loop), after courses LO classification and broader terms NLP, a total of 256 program tokens (norepetitions) was obtained. Figure 7 a) presents a wordcloud with the 200 most frequent key program concepts. Usingthe program biadjacency matrix A T&IM , graphs with direct links between the most frequent key program concepts andwith direct links between courses were obtained—Figures 7 b) and 8, respectively.Note that out of the total 38 courses in Figure 6, three (ETH, NET and CAD) were not taught and were excluded;Internships I&II were also excluded, justifying the analysis of only 33 courses (for the meaning of the course acronymsplease refer to Figure 6).
Figure 7: Mappings for the T&IM degree (2007-2012 program plan). a) Wordcloud with the 200 most frequent key programconcepts. b) Links between the 28 most frequent key program concepts. Wordcloud obtained using the wordcloud package [Fellows,2018]. Undirected network graph obtained from matrix L using the iGraph package [Csardi and Nepusz, 2006]. Both packagesdeveloped for the R programming language [R Core Team, 2019]. With larger font size in Figure 7 a) and at the center of Figures 7 a) and 7 b) lay tokens “manag[ement]” and “busi[ness]”,the most frequent key concepts found in the program broader terms. Besides “manag[ement]” and “busi[ness]”, otherkey concepts lay in the vicinity of the graphs central region, namely, “econom[y]”, “resourc[es]”, “account” (related tomanagement); and, “engi[neering]”, “design” (related to engineering). Because only 28 (out of 256) most frequenttokens are represented in Figure 7 b), all tokens exhibit a fair number of links. The way key concepts are linked inFigure 7 b) defines two distinct groups—or clusters—of key concepts: the management cluster, found towards the thetop of the figure, and the engineering cluster at the bottom. Abstract key concepts such as “process”, “perform[ance]”,“analysi[s]” (“system” or “indic[es]”) are also found (mostly) in the interface between the management and engineeringclusters. 12roader Terms Curriculum Mapping
A P
REPRINT
Figure 8: Links between T&IM program courses. The forward (white) plane presents an enlarged detail with 30 out of the 33 coursespresent in the backward (gray) plane. Network graph obtained from matrix L using the iGraph package [Csardi and Nepusz,2006] for the R programming language [R Core Team, 2019]. Figure 8 presents the links between program courses in two planes. The background (gray) plane is used to show threeultra-peripheral courses with no links: MECHT, MULT and INNOV. The forward (white) plane provides a detail of thecourses laying closer to the graph core region. In this detail all courses are linked. Distant from the graph center laycourses PHY and STAT; at an intermediate distance lay MATH, MAIN, CTRLP, PROG, DRAW, ENVECON, ECON,GLOB and ENG; the remaining 19 courses lay at the central region. A divide similar to the one identified previouslybetween managerial and engineering concepts is also present in Figure 8, with managerial courses clustered to the(upper) left and engineering courses clustered towards the (lower) right of Figure 8 detail.
Comparing Figures 7 and 8 with A3ES recommendations (in Section 4.1) it is possible to evaluate, for each recom-mendation, if the meaning conveyed in writing has a visual equivalent. The comparison of written and visual meaningis used to verify the accuracy of the broader terms CM method.Consider item (i) of the A3ES recommendations—increase program-degree mathematical content. The visual equivalentof this recommendation is the (relative) absence of mathematics-related tokens in the mappings. Indeed, Figure 7 a)includes very few mathematics-related tokens (e.g., mathemat, algebra, theorem), with the small font size of thesetokens confirming the detachment of mathematics from core concepts taught in the T&IM degree. The position of the“mathemat” token in Figure 7 b), distant from central key program concepts, is also consistent with this analysis. Figure8 provides further evidence that some action should be taken concerning mathematics contents. Courses MATH andSTAT relative position and the small number of links to other program courses translates into insufficient integration ofmathematics contents.As regards item (ii) of the A3ES recommendations—steer programming skills towards high level languages withpractical use—, the visual equivalent should be the absence of links between programming and applied key concepts.13roader Terms Curriculum Mapping
A P
REPRINT
An analysis similar to the previous one shows few programming-related tokens in Figure 7 a), with none among the 28most frequent in Figure 7 b). As regards the PROG course, its location in Figure 8 confirms it is among those with lesslinks to central and applied courses.As for item (iii)—strengthen the program plan with important applied industrial management content, namely, inoperations management, supply chain management and operational research—, from Figure 7 b), tokens “oper[ational]”,“logist[ics]” are found among the 28 most frequent. Figure 7 a) includes additional concepts related to the mentionedcourses, such as “suppli”, “chain”, “optim”. Comparing font sizes in Figure 7 a), these latter key concepts are lessfrequent than generic managerial key concepts “resourc”, “financ”, “account”, which could be subjectively deemedless important in a Technology & Industrial Management program plan. A detailed quantitative analysis of tokenfrequencies and of token connections could be made, contributing with relevant insights to the constructive discussionof this recommendation.Using a similar line of inquiry, item (iv) in A3ES recommendations—excessive number of courses, some with littleadditional content—would benefit from the detailed analysis of token frequencies per course and from the equivalent toFigure 3 b) with data from the T&IM study program. This detailed analysis and the graph are obtained with ease frommatrix A T&IM , using the methods and tool considered in supplementary material [Duarte, 2020]. However, from Figure8 it is possible to sort courses based on their connectivity (close to core or peripheral location). This figure depictsultra-peripheral courses (MECHT, MULT, INNOV) in the background plane with no links. These courses are obviouscandidates to detailed scrutiny. A scrutiny that should be extended to courses closer to the graph central region but,nevertheless, showing a small number of links (e.g., GLOB, ENVECON, ECON).Finally, concerning item (v)—poor integration of topics—, as stated previously in Section 4.2, Figures 7 b) and 8denounce the clustering of managerial and of engineering concepts. In addition, courses more detached from the graphcentral region and with less links in Figure 8 (already identified in the previous A3ES recommendation, item iv) areonce more obvious candidates to detailed scrutiny.In light of the above, and considering A3ES recommendations, Table 2 (second column) summarizes the evidence-basedvisual meaning obtained from T&IM mappings.
Table 2: Comparing A3ES recommendations with evidence-based visual meaning conveyed from T&IM mappings.
A3ES recommendation Evidence from mappings ( a ) i. Increase program-degree mathematical content Small number of mathematics-related key concepts and poor in-tegration of mathematics-related coursesii. Steer programming skills towards high-level languages with prac-tical use Extremely small number of programming-related key conceptsand detached location of the programming courseiii. Strengthen the program plan with important applied industrialmanagement content, namely, in operations management, supplychain management and operational research Comparison of frequencies of applied industrial management keyconcepts with frequencies of generic managerial key conceptssheds light on the relative weight of each group in the programplaniv. Excessive number of courses, some with little additional content Identifies and sorts courses with few (and with no) links to coreprogram courses.v. Poor integration of topics taught in the different courses Divide between engineering and management, visible both in keyprogram concept and in course mappings(a) Note these results are obtained exclusively from courses LO-statements, whereas A3ES recommendations consider a visit by an independent panel ofexperts, interviews, focus group sessions, among other inputs. From Table 2, for recommendations (i), (ii) and (v), mappings provide detailed visual evidence supporting theserecommendations. For recommendations (iii) and (iv), the style (the vagueness) of A3ES statements prevents anobjective comparison of visual and written meanings. Yet, these latter recommendations are useful to highlight thestriking difference between an evidence-based analysis—possible with the mappings—and the subjective interpretation—relying on tacit understanding—of A3ES written statements.Because the mappings provide evidence supporting the majority of the A3ES recommendations, it is concluded thatthe broader terms CM method provides an accurate depiction of T&IM program plan weaknesses. Because all T&IMmappings rely on key program concepts, it is also concluded that these key concepts—and the broader terms CMmethod—are useful in program planning. 14roader Terms Curriculum Mapping
A P
REPRINT
Three additional notes are worth mentioning. Firstly, despite the large number of program courses (33), classification,NLP and visualization steps were concluded quickly and with ease, posing no particular difficulty. Secondly, Figure8 shows that a holistic experience of the T&IM program plan, considering interrelations between the 33 courses, ispossible. Lastly, unlike the course mapping of Meij and Merx [2018], the detailed concept mappings of Seering et al.[2015], Willcox and Huang [2017] or Varagnolo et al. [2020], visual outputs from the broader terms CM method donot aim at the tracing of the available learning pathways or at the tracing of detailed precedence relations betweenprogram concepts. Instead, the broader terms CM method maps clusters of key concepts, or courses; maps multipleundirected links between courses and/ or concepts. These maps’ aim is to provide a representation of the program planthat is understandable to all stakeholders, allowing participatory involvement or non-faculty groups without imposingpredefined models or fixed routes. In this sense, broader terms curriculum mapping does not replace, rather precedesand complements other curriculum mapping methods, Addressing the curriculum development process is of paramount importance. This process has profound consequencesbeing responsible for the preparation of future professionals and for laying the foundations for dynamic knowledgetransfer systems affecting local and global realities. At the heart of curriculum development lays program planning.Program planning is of immense strategic value. The effort put into program planning propagates through all levels andsubprocesses of teaching and learning, imprinting the values, intentions and expectations that will guide stakeholders;shaping HEI educational outcomes.To improve program planning more participatory touchpoints to non-faculty groups (i.e., students, industry, society) areneeded. Creating these touchpoints, contributing to representative program planning was the motivation behind thispaper.An important impediment to representative program planning lays in the communication gap between faculty andnon-faculty groups. Curriculum mapping has been used to promote better communication between faculty and shapeprogram planning. This paper collected practices available from different types of curriculum mapping and, usinginformation and data science techniques, tailored a curriculum mapping method for non-faculty groups participation inprogram planning discussions. The resulting method—the broader terms CM (Curriculum Mapping) method—wasillustrated with the help of a simple example—5 courses example. The following conclusions were found:• (Section 3.1) Classification replaces head-on discussion of subjective course LO-statements with the muchmore objective task of selecting broader terms from a control vocabulary.• (Section 3.2) Natural language processing allows the quantitative analysis of the program plan, providing away to cut across disciplinary and scientific boundaries, hierarchical and functional differences, and atomizeddiscourses.• (Section 3.3) Mappings render quantitative results’ interpretation universal, self-explanatory, empoweringstakeholders with evidence-based details on weaknesses (and strengths) in the program plan.• (Section 3.4) Discussion of visual outputs with non-faculty groups allows representative program planning,with these groups’ voice being heard on reclassification and review of course LO-statements.Despite the relevance of the above conclusions—related to the participatory involvement of non-faculty stakeholders—,the simple 5 courses example was unable to answer the question of the broader terms CM method’ accuracy and,therefore, of the method’ utility.To evaluate the method’s accuracy, a case study—the T&IM bachelor degree—was used. Mappings for the case studywere obtained and compared with observations from an independent panel of experts. From this comparison thefollowing was concluded (Section 4.4):• Mappings provide evidence supporting the observations, and the broader terms CM method provides anaccurate depiction of T&IM program plan weaknesses. Wang [2015], based on views derived from Gilles Deleuze and Félix Guatarri, discusses the distinction between mappingand tracing in the context of curriculum mapping. This researcher supports that current practice of curriculum mapping in highereducation is, actually, tracing. Current curriculum mappings represent fixed routes with a linear tree-like structure and an objectivemodel of the curriculum, and this is an example of tracing. Maps have different topological characteristics. Like rhizomes, maps donot aim at guiding to a main road or familiar destination, but to represent the mesh of nodes and the patterns that emerge through themultitude of connections between nodes.
A P
REPRINT • Key concepts obtained from course LO-statements—and the broader terms CM method—are useful in programplanning.Considering the benefit of non-faculty groups’ participation in curriculum development processes, and considering theprogresses made in information systems and relational databases [Chen, 1976, Bagui and Earp, 2003, Leff and Rayfield,2001], the merger of techniques used in the broader terms CM method and HEI information systems would help bringthe method’s benefits into HEI everyday reality; for example, with the inclusion of mappings in information systems’summary dashboards. This merger is just one potential topic for further explorations in this rich and challengingresearch area that joins education, information and data sciences.16roader Terms Curriculum Mapping
A P
REPRINT
References
A3ES. Regulations on the assessment and accreditation procedures—[Portuguese] Agency for the Assessmentand Accreditation of Higher Education, 2013. URL . [Online, accessed27-November-2020].ABET. ABET Homepage, 2020. URL .[Online, accessed 23-July-2020].S. Adam. Using learning outcomes: A consideration of the nature, role, application and implications for europeaneducation of emplying learning outcomes at the local, national and international levels. Technical report, UnitedKingdom Bologna Seminar, Heriot-Watt University, Edinburgh, Scotland, 2004. URL .S. Arafeh. Curriculum mapping in higher education: A case study and proposed content scope and sequence mappingtool.
Journal of Further and Higher Education , 40(5):585–611, 2016. doi:10.1080/0309877X.2014.1000278.S. Bagui and R. Earp.
Database Design Using Entity-Relationship Diagrams . 2003. ISBN 0849315484.N. Ballantyne, K. Hay, L. Beddoe, J. Maidment, and S. Walker. Mapping and Visualizing the Social Work Curriculum.
Journal of Technology in Human Services , 37(2-3):184–202, 2019. doi:10.1080/15228835.2019.1609386.M. Bergman. Shaping Wikipedia into a computable knowledge base, 2015. URL . AI3, Adaptive Information blog [Online,accessed 23-July-2020].J. Biggs and C. Tang.
Teaching for Quality Learning at University . McGraw-Hill & Open University Press, 2011.ISBN 978-0-33-524276-4.B. Bloom.
Taxonomy of Educational Objectives, Book I: The Cognitive Domain.
Addison-Wesley Longman Ltd, 1956.ISBN 978-0582280106.R. Burns. Curriculum mapping, 2001. URL . Association for Supervision and Curriculum Development [Online, accessed17-October-2020].P. Chen. The Entity Relationship Model: Toward a Unified View of Data.
ACM Transactions on Database Systems , 1(1):312–339, 1976.E. Crawley, J. Malmqvist, S. Östlund, and D. Brodeur.
Rethinking engineering education: The CDIO approach .Springer Science, 2007.G. Csardi and T. Nepusz. The igraph software package for complex network research.
InterJournal , Complex Systems:1695, 2006. URL http://igraph.org .R. Duarte. Supplementary material for the paper “Broader terms curriculum mapping: Using natural language processingand visual-supported communication to create representative program planning experiences”. Zenodo, 2020. URL https://doi.org/10.5281/zenodo.4313363 . [Online, accessed 19-December-2020].R. Duarte, A. Ramos Pires, and H. Gonçalves. Identifying at-risk students in higher education.
Total QualityManagement and Business Excellence , 25(7-8):944–995, 2014.R. Duarte, A. L. Oliveira Pires, and A. Lacerda Nobre.
Mature learners’ participation in higher education and flexiblelearning pathways: Lessons learned from an exploratory experimental research , chapter 2, pages 33–53. SpringerNature, 2018. doi:10.1007/978-981-10-8917-6_2.P. Dyjur and J. Lock. Three strategies for moving curriculum mapping online.
Educational Developments , 17(2):15–19,2016. ISSN 1469-3267.EUROVOC. Eurovoc thesaurus, S/D. URL https://op.europa.eu/en/web/eu-vocabularies/th-top-concept-scheme/-/resource/eurovoc/100141?target=Browse . [Online, accessed 23-July-2020].R. Felder and R. Brent. Designing and teaching courses to satisfy the ABET engineering criteria.
Journal of EngineeringEducation , 1(92):7–25, 2003.I. Fellows. wordcloud: Word Clouds , 2018. URL https://CRAN.R-project.org/package=wordcloud . R packageversion 2.6.T. Fruchterman and E. Reingold. Graph drawing by force-directed placement.
Software–Practice & Experience , 21(11):1129–1164, 1991.K. Golub. Automatic subject indexing of text, S/D. URL . Encyclopediaof Knowledge Organization, Hjorland, B. and Gnoli, C. (Eds) [Online, accessed 23-July-2020].17roader Terms Curriculum Mapping
A P
REPRINT
Z. Gu, L. Gu, R. Eils, M. Schlesner, and B. Brors. circlize implements and enhances circular visualization in r.
Bioinformatics , 30(19):2811–2812, 2014.R Harden. AMEE Guide No. 21: Curriculum mapping: a tool for transparent and authentic teaching and learning.
Medical Teacher , 23(2):123–137, 2001. doi:10.1080/01421590120036547.T. Hussey and P. Smith. The uses of learning Outcomes.
Teaching in Higher Education , 8(3):357–368, 2003.doi:10.1080/13562510309399.IEEE. Ieee thesaurus: Version 1.0, 2019. URL . [Online, accessed 23-July-2020].A. Joorabchi and A. Mahdi. Towards linking libraries and Wikipedia: Automatic subject indexing of library recordswith Wikipedia concepts.
Journal of Information Science , 40(2):211–221, 2013. doi:10.1177/0165551510000000.K. Koedinger, A. Corbett, and C. Perfetti. The knowledge-learning-instruction framework: Bridging the science-practicechasm to enhance robust student learning.
Cognitive Science , 36:757–798, 2012.M. Krzywinski, J. Schein, I. Birol, J. Connors, R. Gascoyne, D. Horsman, S. Jones, and M. Marra. Circos: Aninformation aesthetic for comparative genomics.
Genome Research , 2009. doi:10.1101/gr.092759.109. URL http://genome.cshlp.org/content/early/2009/06/15/gr.092759.109.abstract .B. Lam and K. Tsui. Curriculum mapping as deliberation: Examining the alignment of subject learning outcomes andcourse curricula.
Studies in Higher Education , 41(8):1371–1388, 2016. doi:10.1080/03075079.2014.968539.F. Lancaster.
Indexing and abstracting in theory and practice . Facet, 3rd edition, 2003. ISBN 1-85604-482-3.A. Leff and J. Rayfield. Web-application development using the model/ view/ controller design pattern. In
Proceedingsof the Enterprise Distributed Object Computing Conference, EDOC’01. Fifth IEEE International , pages 118–127.IEEE. National Committee of Inquiry into Higher Education, 2001.Library of Congress. Library of Congress Subject Headings, S/D. URL https://id.loc.gov/authorities/subjects.html . [Online, accessed 23-July-2020].R. Lourenço, E. Ferreira, R. Duarte, H. Gonçalves, and H. Duarte. IPS’ technology and industrial management graduatecourse: A curriculum follow-up analysis. In
Proceedings of the 1st European Conference on curriculum studies,future directions: uncertainty and possibility , pages 263–269, 2013.C. Manning and H Schütze.
Foundations of statistical natural language processing . The MIT Press, 2nd edition, 1999.L. Meij and S. Merx. Improving curriculum alignment and achieving learning goals by making the curriculum visible.
International Journal for Academic Development , 23(3):219–231, 2018.A. Ornstein and F. Hunkins.
Curriculum: Foundations, principles and issues . 2018. ISBN 978-1-292-16207-2.R Core Team.
R: A Language and Environment for Statistical Computing . R Foundation for Statistical Computing,Vienna, Austria, 2019. URL .K. Rosen.
Discrete Mathematics and its Applications . McGraw-Hill, 2009.E. Schein. Kurt Lewin’s change theory in the field and in the classroom: Notes toward a model of managed learning.
Reflections , 1(1):59–74, 1999. URL .J. Seering, L. Huang, and K Willcox. Mapping outcomes in an undergraduate aerospace engineering program. In
Proceedings of the 2015 ASEE Annual Conference & Exposition , 2015.W. Spady. Organising the results: The basis of authentic restructuring and reform.
Educational Leadership , October:4–8, 1988.K. Sutherland. Holistic academic development: Is it time to think more broadly about the academic development project?
International Journal for Academic Development , 23(4):261–273, 2018. doi:10.1080/1360144X.2018.1524571.UNESCO. Unesco thesaurus, S/D. URL http://vocabularies.unesco.org/browser/thesaurus/en/ . [Online,accessed 23-July-2020].D. Varagnolo, S. Knorn, K. Staffas, E. Fjällström, and T. Wrigstad. Graph-theoretic approaches andtools for quantitatively assessing curricula coherence.
European Journal of Engineering Education , 2020.doi:10.1080/03043797.2019.1710465.C.-L. Wang. Mapping or tracing? Rethinking curriculum mapping in higher education.
Studies in Higher Education ,40(9):1550–1559, 2015. doi:10.1080/03075079.2014.899343.L. Watts and D. Hodgson. Whole Curriculum Mapping of Assessment: Cartographies of Assessment and Learning.
Social Work Education , 34(6):682–699, 2015. doi:10.1080/02615479.2015.1048217.18roader Terms Curriculum Mapping
A P
REPRINT
J. West. Validating curriculum development using text mining.
The Curriculum Journal , 2016.doi:10.1080/09585176.2016.1261719.J. West. Teaching data science: An objective approach to curriculum validation.
Computer Science Education , 28(2):136–157, 2018. doi:10.1080/08993408.2018.1486120.Wikipedia contributors. Wikipedia indices — Wikipedia, The Free Encyclopedia, 2020. URL https://en.wikipedia.org/w/index.php?search=wikipedia&title=Special%3ASearch&profile=advanced&fulltext=1&advancedSearch-current=%7B%7D&ns0=1 . [Online; accessed 30-October-2020].K. Willcox and L. Huang. Network models for mapping educational data.
Design Science , 3(e18), 2017.doi:10.1017/dsj.2017.18.C. Wilson and C. Slade. From consultation and collaboration to consensus: introducing an alternativemodel of curriculum development.