Journal ranking should depend on the level of aggregation
JJournal ranking should dependon the level of aggregation
L´aszl´o Csat´o * Institute for Computer Science and Control, Hungarian Academy of Sciences (MTA SZTAKI)Laboratory on Engineering and Management Intelligence, Research Group of OperationsResearch and Decision SystemsCorvinus University of Budapest (BCE)Department of Operations Research and Actuarial SciencesBudapest, Hungary
Ich behaupte aber, daß in jeder besonderen Naturlehre nur so viel eigentliche Wissenschaftangetroffen werden k¨onne, als darin Mathematik anzutreffen ist. (Immanuel Kant: Metaphysische Anfangsgr¨unde der Naturwissenschaft ) Abstract
Journal ranking is becoming more important in assessing the quality of academicresearch. Several indices have been suggested for this purpose, typically on the basisof a citation graph between the journals. We follow an axiomatic approach andfind an impossibility theorem: any self-consistent ranking method, which satisfiesa natural monotonicity property, should depend on the level of aggregation. Ourresult presents a trade-off between two axiomatic properties and reveals a dilemmaof aggregation.
Keywords : Journal ranking; citations; axiomatic approach; impossibility
MSC class : 91A80, 91B14
JEL classification number : C44, D71
The measurement of the quality and quantity of academic research plays an increasingrole in the evaluation of researchers and research proposals. This paper will focus on a * E-mail: [email protected] “ I maintain that in each particular natural science there is only as much true science as thereis mathematics. ” (Source: Smith, J. T.: David Hilbert’s 1930 Radio Address – German and English. ) a r X i v : . [ c s . D L ] S e p articular field of scientometrics, that is, journal ranking. Furthermore, since a numberof bibliometric indices have been suggested to assess intellectual influence, and now aplethora of ranking methods are available to measure the performance of journals andscholars (Palacios-Huerta and Volij, 2014), we follow an axiomatic approach becausethe introduction of some reasonable axioms or conditions is able to narrow the set ofappropriate methods, to reveal their crucial properties, and to allow for their comparison.An important contribution of similar analyses can be an axiomatic characterisation,meaning that a set of properties uniquely determine a preference vector. For example,Palacios-Huerta and Volij (2004) give a characterisation of the invariant method, whileDemange (2014) provides a characterisation of the handicap method, both of them usedto rank journals. Results for citation indices are probably even more abundant, includingcharacterisations of the ℎ -index (Kongo, 2014; Marchant, 2009; Miroiu, 2013; Quesada,2010, 2011a,b; Woeginger, 2008b), the 𝑔 -index (Woeginger, 2008a; Quesada, 2011a; Adachiand Kongo, 2015), the Euclidean index (Perry and Reny, 2016), or a class of step-basedindices (Chambers and Miller, 2014), among others. de la Vega and Volij (2018) characterisescholar rankings admitting a measure representation. There are also axiomatic comparisonsof bibliometric indices (Bouyssou and Marchant, 2014, 2016).However, the above works seldom uncover the inevitable trade-offs between differentnatural requirements, an aim which can be achieved mainly by impossibility theorems.Similar results are well-established in social choice theory since Arrow’s impossibility the-orem (Arrow, 1951) and the Gibbard-Satterthwaite theorem (Gibbard, 1973; Satterthwaite,1975; Duggan and Schwartz, 2000) but not so widely used in scientometrics.We provide an impossibility result in journal ranking. In particular, it will be provedthat two axioms, invariance to aggregation and self-consistency, cannot be satisfiedsimultaneously even on a substantially restricted domain of citation graphs. Invarianceto aggregation means that the ranking of two journals is not influenced by the level ofaggregation among the remaining journals, while self-consistency, introduced by Chebotarevand Shamis (1997), is a kind of monotonicity property, responsible for some impossibilitytheorems in ranking from paired comparisons (Csat´o, 2019a,b).The paper is organised as follows. Our setting and notations are introduced in Section 2.Section 3 motivates and defines the two axioms, which turn out to be incompatible inSection 4. Section 5 summarises the main findings and concludes. A journal ranking problem consists of a group of journals and their respective citationrecords (Palacios-Huerta and Volij, 2014). Let 𝑁 = { 𝐽 , 𝐽 , . . . , 𝐽 𝑛 } , 𝑛 ∈ N be a non-emptyfinite set of journals and 𝐶 = [ 𝑐 𝑖𝑗 ] ∈ R 𝑛 × 𝑛 be a | 𝑁 | × | 𝑁 | nonnegative citation matrix for 𝑁 . The entry 𝑐 𝑖𝑗 can be directly the number of citations that journal 𝐽 𝑖 receivedfrom journal 𝐽 𝑗 , or any reasonable transformation of this value, for example, by usingexponentially decreasing weights for older citations.The pair ( 𝑁, 𝐶 ) is called a journal ranking problem . The set of journal ranking problemswith 𝑛 journals ( | 𝑁 | = 𝑛 ) is denoted by 𝒥 𝑛 .The aim is to aggregate the opinions given in the citation matrix into a single judgement.Formally, a scoring procedure 𝑓 is a 𝒥 𝑛 → R 𝑛 function that takes a journal ranking problem( 𝑁, 𝐶 ) and returns a rating 𝑓 𝑖 ( 𝑁, 𝐶 ) for each journal 𝐽 𝑖 ∈ 𝑁 , representing this judgement.A scoring method immediately induces a ranking ⪰ for the journals of 𝑁 (a transitiveand complete weak order on the set of 𝑁 ): 𝑓 𝑖 ( 𝑁, 𝐶 ) ≥ 𝑓 𝑗 ( 𝑁, 𝐶 ) means that journal 𝐽 𝑖 is2anked weakly above 𝐽 𝑗 , denoted by 𝐽 𝑖 ⪰ 𝐽 𝑗 . The symmetric and asymmetric parts of ⪰ are denoted by ∼ and ≻ , respectively: 𝐽 𝑖 ∼ 𝐽 𝑗 if both 𝐽 𝑖 ⪰ 𝐽 𝑗 and 𝐽 𝑖 ⪯ 𝐽 𝑗 hold, while 𝐽 𝑖 ≻ 𝐽 𝑗 if 𝐽 𝑖 ⪰ 𝐽 𝑗 holds but 𝐽 𝑖 ⪯ 𝐽 𝑗 does not hold.A journal ranking problem ( 𝑁, 𝐶 ) has the symmetric matches matrix 𝑀 = 𝐶 + 𝐶 ⊤ =[ 𝑚 𝑖𝑗 ] ∈ R 𝑛 × 𝑛 such that 𝑚 𝑖𝑗 is the number of the citations between the journals 𝐽 𝑖 and 𝐽 𝑗 in both directions, which can be called the number of matches between them in theterminology of sports (K´oczy and Strobel, 2010; Csat´o, 2015).It is sometimes convenient to consider not a general problem, arising from complicatednetworks of citations, but only a simpler one.A journal ranking problem ( 𝑁, 𝐶 ) ∈ 𝒥 𝑛 is called balanced if ∑︀ 𝑋 𝑘 ∈ 𝑁 𝑚 𝑖𝑘 = ∑︀ 𝑋 𝑘 ∈ 𝑁 𝑚 𝑗𝑘 for all 𝐽 𝑖 , 𝐽 𝑗 ∈ 𝑁 . The set of balanced journal ranking problems is denoted by 𝒥 𝐵 . In abalanced journal ranking problem, all journals have the same number of matches.A journal ranking problem ( 𝑁, 𝐶 ) ∈ 𝒥 𝑛 is called unweighted if 𝑚 𝑖𝑗 ∈ {
0; 1 } for all 𝐽 𝑖 , 𝐽 𝑗 ∈ 𝑁 . The set of unweighted journal ranking problems is denoted by 𝒥 𝑈 . In anunweighted journal ranking problem, either there is no citations, or there exists only onecitation between any pair of journals.A journal ranking problem ( 𝑁, 𝐶 ) ∈ 𝒥 𝑛 is called loopless if 𝑐 𝑖𝑖 = 0 for all 𝐽 𝑖 ∈ 𝑁 .The set of unweighted journal ranking problems is denoted by 𝒥 𝐿 . In a loopless problem,self-citations are disregarded.The subsets of balanced, unweighted, and loopless journal ranking problems restrictthe matches matrix 𝑀 .A journal ranking problem ( 𝑁, 𝐶 ) ∈ 𝒥 𝑛 is called extremal if | 𝑐 𝑖𝑗 | ∈ { 𝑚 𝑖𝑗 / 𝑚 𝑖𝑗 } for all 𝐽 𝑖 , 𝐽 𝑗 ∈ 𝑁 . The set of extremal journal ranking problems is denoted by 𝒥 𝐸 . Inan extremal journal ranking problem, only three cases are allowed in the comparison ofjournals 𝐽 𝑖 and 𝐽 𝑗 : there are citations only for 𝐽 𝑖 or 𝐽 𝑗 , or they are tied with respect tomutual citations.Any intersection of these special classes can be considered, too.While a given citation matrix 𝐶 will seldom lead to a balanced, unweighted, loopless,or extremal journal ranking problem in practice, they can still be relevant for applicationsdue to the possible transformation of citations. For example, it may make sense to removeself-citations from matrix 𝐶 , and consider only three types of paired comparisons in thederived matrix ^ 𝐶 : ∙ ^ 𝑐 𝑖𝑗 = 0 if 𝑐 𝑖𝑗 = 0 and 𝑐 𝑗𝑖 = 0; ∙ ^ 𝑐 𝑖𝑗 = 0 if 𝑐 𝑗𝑖 > 𝑐 𝑖𝑗 < 𝑐 𝑗𝑖 / ∙ ^ 𝑐 𝑖𝑗 = 0 . 𝑐 𝑗𝑖 > 𝑐 𝑗𝑖 / ≤ 𝑐 𝑖𝑗 ≤ 𝑐 𝑗𝑖 ; ∙ ^ 𝑐 𝑖𝑗 = 1 if 2 𝑐 𝑗𝑖 < 𝑐 𝑖𝑗 .In other words, two journals are not compared (^ 𝑐 𝑖𝑗 = ^ 𝑐 𝑗𝑖 = 0) if they do not cite each other,their paired comparison is tied (^ 𝑐 𝑖𝑗 = ^ 𝑐 𝑗𝑖 = 0 .
5) if their mutual citations are approximatelybalanced – that is, 𝐽 𝑖 does not refer to 𝐽 𝑗 more than two times than 𝐽 𝑗 refers to 𝐽 𝑖 , andvice versa –, and 𝐽 𝑖 is maximally better than 𝐽 𝑗 (^ 𝑐 𝑖𝑗 = 1 and ^ 𝑐 𝑗𝑖 = 0) if 𝐽 𝑗 cites 𝐽 𝑖 morethan two times than 𝐽 𝑖 cites 𝐽 𝑗 . Then the resulting journal ranking problem (︁ 𝑁, ^ 𝐶 )︁ ∈ 𝒥 𝑛 is unweighted, loopless, and extremal. 3 Axioms of journal ranking
In this section two properties, a natural axiom of aggregation and a variant of monotonicity,are introduced.
The first condition aims to regulate the ranking if two journals are aggregated into one.
Axiom 1.
Invariance to aggregation ( 𝐼𝐴 ): Let ( 𝑁, 𝐶 ) ∈ 𝒥 𝑛 be a journal ranking problemand 𝐽 𝑖 , 𝐽 𝑗 ∈ 𝑁 be two different journals. Journal ranking problem ( 𝑁 𝑖 ∪ 𝑗 , 𝐶 𝑖 ∪ 𝑗 ) ∈ 𝒥 𝑛 − isgiven by 𝑁 𝑖 ∪ 𝑗 = ( 𝑁 ∖ { 𝐽 𝑖 , 𝐽 𝑗 } ) ∪ 𝐽 𝑖 ∪ 𝑗 and 𝐶 𝑖 ∪ 𝑗 = [︁ 𝑐 𝑖 ∪ 𝑗𝑘ℓ ]︁ ∈ R ( 𝑛 − × ( 𝑛 − such that ∙ 𝑐 𝑖 ∪ 𝑗𝑘ℓ = 𝑐 𝑘ℓ if { 𝐽 𝑘 , 𝐽 ℓ } ∩ { 𝐽 𝑖 , 𝐽 𝑗 } = ∅ ; ∙ 𝑐 𝑖 ∪ 𝑗𝑘 ( 𝑖 ∪ 𝑗 ) = 𝑐 𝑘𝑖 + 𝑐 𝑘𝑗 for all 𝐽 𝑘 ∈ 𝑁 ∖ { 𝐽 𝑖 , 𝐽 𝑗 } ; ∙ 𝑐 𝑖 ∪ 𝑗 ( 𝑖 ∪ 𝑗 ) ℓ = 𝑐 𝑖ℓ + 𝑐 𝑗ℓ for all 𝐽 ℓ ∈ 𝑁 ∖ { 𝐽 𝑖 , 𝐽 𝑗 } .Scoring procedure 𝑓 : 𝒥 𝑛 → R 𝑛 is called invariant to aggregation if 𝑓 𝑘 ( 𝑁, 𝐶 ) ≥ 𝑓 ℓ ( 𝑁, 𝐶 )implies 𝑓 𝑘 ( 𝑁 𝑖 ∪ 𝑗 , 𝐶 𝑖 ∪ 𝑗 ) ≥ 𝑓 ℓ ( 𝑁 𝑖 ∪ 𝑗 , 𝐶 𝑖 ∪ 𝑗 ) for all 𝐽 𝑘 , 𝐽 ℓ ∈ 𝑁 ∖ { 𝐽 𝑖 , 𝐽 𝑗 } .The idea behind invariance to aggregation is that any journal ranking problem can betransformed into a reduced problem by defining the union 𝐽 𝑖 ∪ 𝑗 of journals 𝐽 𝑖 and 𝐽 𝑗 asfollows: all citations between them are deleted, while any citations by/to these journals aresummed up for the “aggregated” journal 𝐽 𝑖 ∪ 𝑗 . This transformation is required to preservethe order of the journals not affected by the aggregation.Such an aggregation makes sense, for example, if one is interested only in the rankingof journals from a given field (e.g. economics journals) when journals from other disciplinescan be considered as one entity.Invariance to aggregation is somewhat related to the consistency axiom of Palacios-Huerta and Volij (2004), which is also based on the notion of the reduced problem.However, our property probably takes the information from the missing journal in a morestraightforward way into consideration.Invariance to aggregation has some connections to the famous independence of irrelevantalternatives ( 𝐼𝐼𝐴 ) condition, too, which is used, for example, in Arrow’s impossibilitytheorem (Arrow, 1951). Both axioms require an important aspect of the problem, thecitations between two journals and the individual preferences between two alternatives,respectively, to remain fixed. However, there is a crucial difference: the set of alternatives(corresponding to journals) is allowed to change in the case of 𝐼𝐴 , while the preferences(corresponding to citations) are allowed to change in the case of 𝐼𝐼𝐴 . This axiom, originally introduced in Chebotarev and Shamis (1997) to operators used foraggregating preferences, may require a longer explanation.First, some reasonable conditions are formulated for the ranking derived from anyjournal ranking problem. In particular, journal 𝐽 𝑖 is judged better than journal 𝐽 𝑗 if oneof the following holds: (cid:68) 𝐽 𝑖 has more favourable citation records against the same journals;4 𝐽 𝑖 has more favourable citation records against journals with the same quality; (cid:68) 𝐽 𝑖 has the same citation records against higher quality journals; (cid:68) 𝐽 𝑖 has more favourable citation records against higher quality journals.In addition, journals 𝐽 𝑖 and 𝐽 𝑗 should get the same rank if one of the following holds: (cid:68)
5) they have the same citation records against the same journals; (cid:68)
6) they have the same citation records against journals with the same quality.Principles (cid:68) (cid:68) (cid:68) self-consistency , refers to the fact that this is provided by thescoring procedure itself.The meaning of the requirements above is illustrated by an example.Figure 1: The journal ranking problem of Example 3.1 𝐽 𝐽 𝐽 𝐽 Example 3.1.
Consider the journal ranking problem (
𝑁, 𝐶 ) ∈ 𝒥 𝐵 ∩ 𝒥 𝑈 ∩ 𝒥 𝐿 ∩ 𝒥 𝐸 withthe following citation matrix: 𝐶 = ⎡⎢⎢⎢⎣ ⎤⎥⎥⎥⎦ . This is shown in Figure 1 where a directed edge from node 𝐽 𝑖 to 𝐽 𝑗 indicates that journal 𝐽 𝑖 has received a citation from journal 𝐽 𝑗 .Self-consistency has the following implications for the journal ranking problem presentedin Example 3.1: ∙ 𝐽 ∼ 𝐽 due to rule (cid:68) ∙ 𝐽 ≻ 𝐽 because of rule (cid:68) 𝑐 > 𝑐 and 𝑐 > 𝑐 . ∙ Assume for contradiction that 𝐽 ⪯ 𝐽 . Then 𝑐 > 𝑐 and 𝐽 ⪰ 𝐽 , as well as 𝑐 = 𝑐 and 𝐽 ∼ 𝐽 ⪰ 𝐽 ≻ 𝐽 , so rule (cid:68) 𝐽 ≻ 𝐽 , which is impossible.Consequently, 𝐽 ≻ ( 𝐽 ∼ 𝐽 ). ∙ Assume for contradiction that 𝐽 ⪯ 𝐽 . Then 𝑐 > 𝑐 and 𝐽 ≻ 𝐽 , as well as 𝑐 > 𝑐 and 𝐽 ⪰ 𝐽 ∼ 𝐽 , so rule (cid:68) 𝐽 ≻ 𝐽 , which is impossible.Consequently, ( 𝐽 ∼ 𝐽 ) ≻ 𝐽 . 5o conclude, self-consistency demands the ranking to be 𝐽 ≻ ( 𝐽 ∼ 𝐽 ) ≻ 𝐽 in Ex-ample 3.1.It is clear that self-consistency does not guarantee the uniqueness of the ranking ingeneral (Csat´o, 2019a).Now we turn to the mathematical formulation of this axiom. Definition 3.1.
Competitor set : Let (
𝑁, 𝐶 ) ∈ 𝒥 𝑛𝑈 be an unweighted journal rankingproblem. The competitor set of journal 𝐽 𝑖 is 𝑆 𝑖 = { 𝐽 𝑗 : 𝑚 𝑖𝑗 = 1 } .Journals in the competitor set 𝑆 𝑖 are called the competitors of 𝐽 𝑖 . Note that | 𝑆 𝑖 | = | 𝑆 𝑗 | for all 𝐽 𝑖 , 𝐽 𝑗 ∈ 𝑁 if and only if the ranking problem is balanced.The competitor set is defined only for unweighted journal ranking problem but self-consistency may have implications for any pair of journals which have the same number ofmatches. The generalisation is based on a decomposition of journal ranking problems. Definition 3.2.
Sum of journal ranking problems : Let (
𝑁, 𝐶 ) , ( 𝑁, 𝐶 ′ ) ∈ 𝒥 𝑛 be twojournal ranking problems with the same set of journals 𝑁 . The sum of these journalranking problems is the journal ranking problem ( 𝑁, 𝐶 + 𝐶 ′ ) ∈ 𝒥 𝑛 .The sum of journal ranking problems has a number of reasonable interpretations. Forinstance, they can reflect the citations from different years, or by authors from differentcountries.According to Definition 3.2, any journal ranking problem can be derived as the sumof unweighted journal ranking problems. However, it might have a number of possibledecompositions. Notation . Let (
𝑁, 𝐶 ( 𝑝 ) ) ∈ 𝒥 𝑛𝑈 be an unweighted journal ranking problem. Thecompetitor set of journal 𝐽 𝑖 is 𝑆 ( 𝑝 ) 𝑖 . Let 𝐽 𝑖 , 𝐽 𝑗 ∈ 𝑁 be two different journals and 𝑔 ( 𝑝 ) : 𝑆 ( 𝑝 ) 𝑖 ↔ 𝑆 ( 𝑝 ) 𝑗 be a one-to-one correspondence between the competitors of 𝐽 𝑖 and 𝐽 𝑗 . Then g ( 𝑝 ) : { 𝑘 : 𝐽 𝑘 ∈ 𝑆 ( 𝑝 ) 𝑖 } ↔ { ℓ : 𝐽 ℓ ∈ 𝑆 ( 𝑝 ) 𝑗 } is given by 𝐽 g ( 𝑝 ) ( 𝑘 ) = 𝑔 ( 𝑝 ) ( 𝐽 𝑘 ).Finally, we are able to introduce conditions (cid:68) (cid:68) Axiom 2.
Self-consistency ( 𝑆𝐶 ) (Chebotarev and Shamis, 1997): Scoring procedure 𝑓 : 𝒥 𝑛 → R 𝑛 is called self-consistent if the following implication holds for any journal rankingproblem ( 𝑁, 𝐶 ) ∈ 𝒥 𝑛 and for any journals 𝐽 𝑖 , 𝐽 𝑗 ∈ 𝑁 : if there exists a decomposition ofthe journal ranking problem ( 𝑁, 𝐶 ) into 𝑚 unweighted journal ranking problems – thatis, 𝐶 = ∑︀ 𝑚𝑝 =1 𝐶 ( 𝑝 ) and ( 𝑁, 𝐶 ( 𝑝 ) ) ∈ 𝒥 𝑛𝑈 is an unweighted journal ranking problem for all 𝑝 = 1 , , . . . , 𝑚 – together with the existence of a one-to-one mapping 𝑔 ( 𝑝 ) from 𝑆 ( 𝑝 ) 𝑖 onto 𝑆 ( 𝑝 ) 𝑗 such that 𝑐 ( 𝑝 ) 𝑖𝑘 ≥ 𝑐 ( 𝑝 ) 𝑗 g ( 𝑝 ) ( 𝑘 ) and 𝑓 𝑘 ( 𝑁, 𝐶 ) ≥ 𝑓 g ( 𝑝 ) ( 𝑘 ) ( 𝑁, 𝐶 ) for all 𝑝 = 1 , , . . . , 𝑚 and 𝐽 𝑘 ∈ 𝑆 ( 𝑝 ) 𝑖 , then 𝑓 𝑖 ( 𝑁, 𝐶 ) ≥ 𝑓 𝑗 ( 𝑁, 𝐶 ). Furthermore, 𝑓 𝑖 ( 𝑁, 𝐶 ) > 𝑓 𝑗 ( 𝑁, 𝐶 ) if 𝑐 ( 𝑝 ) 𝑖𝑘 > 𝑐 ( 𝑝 ) 𝑗 g ( 𝑝 ) ( 𝑘 ) or 𝑓 𝑘 ( 𝑁, 𝐶 ) > 𝑓 g ( 𝑝 ) ( 𝑘 ) ( 𝑁, 𝐶 ) for at least one 1 ≤ 𝑝 ≤ 𝑚 and 𝐽 𝑘 ∈ 𝑆 ( 𝑝 ) 𝑖 .In a nutshell, self-consistency implies that if journal 𝐽 𝑖 does not show worse performancethan journal 𝐽 𝑗 on the basis of the citation matrix, then it is not ranked lower, in addition,it is ranked strictly higher when it becomes clearly better.Chebotarev and Shamis (1997) consider only loopless journal ranking problems butthe extension of self-consistency is trivial as presented above.Chebotarev and Shamis (1998, Theorem 5) gives a necessary and sufficient condition forself-consistent scoring procedures, while Chebotarev and Shamis (1998, Table 2) presentssome scoring procedures that satisfy this requirement. See also Csat´o (2019a) for anextensive discussion of self-consistency. 6 The incompatibility of the two axioms
In the following, it will be proved that no scoring procedure can meet axioms 𝐼𝐴 and 𝑆𝐶 .Figure 2: The journal ranking problems of Example 4.1 (a) Journal ranking problem ( 𝑁, 𝐶 ) 𝐽 𝐽 𝐽 𝐽 (b) Journal ranking problem ( 𝑁 ∪ , 𝐶 ∪ ) 𝐽 𝐽 𝐽 ∪ Example 4.1.
Let (
𝑁, 𝐶 ) ∈ 𝒥 𝐵 ∩ 𝒥 𝑈 ∩ 𝒥 𝐿 ∩ 𝒥 𝐸 and ( 𝑁 ∪ , 𝐶 ∪ ) ∈ 𝒥 𝐵 ∪ 𝒥 𝑈 ∩ 𝒥 𝐿 ∩ 𝒥 𝐸 be the journal ranking problems with the citation matrices 𝐶 = ⎡⎢⎢⎢⎣ . . . . . . ⎤⎥⎥⎥⎦ and 𝐶 ∪ = ⎡⎢⎣ . . . . . . ⎤⎥⎦ , respectively . Journal ranking problem ( 𝑁 ∪ , 𝐶 ∪ ) is obtained by uniting journals 3 and 4.This is shown in Figure 2 where a directed edge from node 𝐽 𝑖 to 𝐽 𝑗 indicates thatjournal 𝐽 𝑖 has received a citation from journal 𝐽 𝑗 , and an undirected edge between thenodes means that the two journals are tied by mutual citations. Theorem 4.1.
There exists no scoring procedure that is invariant to aggregation andself-consistent.Proof.
The contradiction of the two properties can be proved by Example 4.1. Take firstthe journal ranking problem (
𝑁, 𝐶 ), which has the competitor sets 𝑆 = 𝑆 = { 𝐽 , 𝐽 } and 𝑆 = 𝑆 = { 𝐽 , 𝐽 } . Assume for contradiction the existence of a scoring procedure 𝑓 : 𝒥 𝑛 → R 𝑛 satisfying invariance to aggregation and self-consistency.Self-consistency has several implications for the scoring procedure 𝑓 as follows: a ) Consider the (identity) one-to-one correspondence 𝑔 : 𝑆 ↔ 𝑆 such that 𝑔 ( 𝐽 ) = 𝐽 and 𝑔 ( 𝐽 ) = 𝐽 . Then 𝑔 satisfies condition (cid:68) 𝑆𝐶 due to 𝑐 = 𝑐 = 0 . . 𝑐 > 𝑐 = 0, thus 𝑓 ( 𝑁, 𝐶 ) > 𝑓 ( 𝑁, 𝐶 ). b ) Consider the (identity) one-to-one correspondence 𝑔 : 𝑆 ↔ 𝑆 such that 𝑔 ( 𝐽 ) = 𝐽 and 𝑔 ( 𝐽 ) = 𝐽 . Then 𝑔 satisfies condition (cid:68) 𝑆𝐶 due to 𝑐 = 𝑐 = 0 . 𝑐 > 𝑐 = 0 .
5, thus 𝑓 ( 𝑁, 𝐶 ) > 𝑓 ( 𝑁, 𝐶 ). c ) Suppose that 𝑓 ( 𝑁, 𝐶 ) ≥ 𝑓 ( 𝑁, 𝐶 ), which implies 𝑓 ( 𝑁, 𝐶 ) > 𝑓 ( 𝑁, 𝐶 ) accordingto the inequalities derived in a ) and b ). Consider the one-to-one correspondence 𝑔 : 𝑆 ↔ 𝑆 such that 𝑔 ( 𝐽 ) = 𝐽 and 𝑔 ( 𝐽 ) = 𝐽 . Then 𝑔 satisfiescondition (cid:68) 𝑆𝐶 due to 𝑐 = 𝑐 = 0 . 𝑐 = 𝑐 = 0 .
5, thus 𝑓 ( 𝑁, 𝐶 ) >𝑓 ( 𝑁, 𝐶 ), a contradiction. 7herefore 𝑓 ( 𝑁, 𝐶 ) > 𝑓 ( 𝑁, 𝐶 ) should hold, when invariance to aggregation resultsin 𝑓 ( 𝑁, 𝐶 ′ ) > 𝑓 ( 𝑁, 𝐶 ′ ). However, self-consistency leads to 𝑓 ( 𝑁, 𝐶 ′ ) = 𝑓 ( 𝑁, 𝐶 ′ ) in thejournal ranking problem ( 𝑁, 𝐶 ′ ) because of the one-to-one mapping 𝑔 : 𝑆 ′ ↔ 𝑆 ′ suchthat 𝑔 ( 𝐽 ) = 𝐽 and 𝑔 ( 𝐽 ) = 𝐽 : the assumption of 𝑓 ( 𝑁, 𝐶 ′ ) > 𝑓 ( 𝑁, 𝐶 ′ ) implies 𝑓 ( 𝑁, 𝐶 ′ ) < 𝑓 ( 𝑁, 𝐶 ′ ) due to condition (cid:68) 𝐽 are more prestigious),while 𝑓 ( 𝑁, 𝐶 ′ ) < 𝑓 ( 𝑁, 𝐶 ′ ) would result in 𝑓 ( 𝑁, 𝐶 ′ ) < 𝑓 ( 𝑁, 𝐶 ′ ) due to condition (cid:68) 𝐽 are more prestigious) again.Hence a scoring procedure cannot meet 𝐼𝐴 and 𝑆𝐶 at the same time.Since Example 4.1 contains balanced, unweighted, loopless, and extremal journalranking problems, there is few hope to avoid the impossibility of Theorem 4.1 by plausibledomain restrictions. Remark . 𝐼𝐴 and 𝑆𝐶 are logically independent axioms as there exist scoring proced-ures that satisfy one of the two properties: the least squares method is self-consistent(Chebotarev and Shamis, 1998, Theorem 5), and the flat scoring procedure, which gives 𝑓 𝑖 ( 𝑁, 𝐶 ) = 0 for all 𝑖 ∈ 𝑁 and ( 𝑁, 𝐶 ) ∈ 𝒥 𝑛 , is invariant to aggregation. We have presented an impossibility theorem in journal ranking: a reasonable method cannotbe invariant to the aggregation of journals, even in the case of a substantially restricteddomain of citation graphs. An intuitive explanation is that invariance to aggregation is alocal property (it modifies only the citations directly affecting the journals to be united),while self-consistency considers the global structure of the citations as it depends on thequality of the journals. The clash between local and global axioms can also be observed inother fields, such as game theory. In addition, the impossibility result clearly shows thatinvariance to aggregation is a rather strong requirement, similarly to its peer independenceof irrelevant alternatives (Malawski and Zhou, 1994). Nevertheless, according to ourfinding, the choice of the set of journals to be compared is an important aspect of everyempirical study which aims to measure intellectual influence.It is clear that the axiomatic analysis discussed here has a number of limitations asit is able to consider indices from only one point of view (Gl¨anzel and Moed, 2013). Forexample, the citation graph is assumed to be known, that is, the issue of choosing anadequate time window is neglected. In addition, this paper has not addressed severalimportant problems of scientometrics such as the comparability of distant research areas,or the proper treatment of different types of publications.To summarise, the derivation of similar impossibility results may contribute to a betterunderstanding of the inevitable trade-offs between various properties, and it means anatural subject of further studies besides axiomatic characterisations.
Acknowledgements
We are grateful to
Gy¨orgy Moln´ar and
D´ora Gr´eta Petr´oczy for inspiration.Two anonymous reviewers provided valuable comments and suggestions on an earlier draft.The research was supported by OTKA grant K 111797 and by the MTA PremiumPostdoctoral Research Program. 8 eferences
Adachi, T. and Kongo, T. (2015). Further axiomatizations of Egghe’s 𝑔 -index. Journal ofInformetrics , 9(4):839–844.Arrow, K. J. (1951).
Social choice and individual values . Wiley, New York.Bouyssou, D. and Marchant, T. (2014). An axiomatic approach to bibliometric rankingsand indices.
Journal of Informetrics , 8(3):449–477.Bouyssou, D. and Marchant, T. (2016). Ranking authors using fractional counting ofcitations: An axiomatic approach.
Journal of Informetrics , 10(1):183–199.Chambers, C. P. and Miller, A. D. (2014). Scholarly influence.
Journal of EconomicTheory , 151:571–583.Chebotarev, P. and Shamis, E. (1997). Constructing an objective function for aggregatingincomplete preferences. In Tangian, A. and Gruber, J., editors,
Constructing Scalar-Valued Objective Functions , volume 453 of
Lecture Notes in Economics and MathematicalSystems , pages 100–124. Springer, Berlin-Heidelberg.Chebotarev, P. Yu. and Shamis, E. (1998). Characterizations of scoring methods forpreference aggregation.
Annals of Operations Research , 80:299–332.Csat´o, L. (2015). A graph interpretation of the least squares ranking method.
SocialChoice and Welfare , 44(1):51–69.Csat´o, L. (2019a). An impossibility theorem for paired comparisons.
Central EuropeanJournal of Operations Research , 27(2):497–514.Csat´o, L. (2019b). Some impossibilities of ranking in generalized tournaments.
InternationalGame Theory Review , 21(01):1940002.de la Vega, C. L. and Volij, O. (2018). Ranking scholars: A measure representation.
Journal of Informetrics , 12(2):510–517.Demange, G. (2014). A ranking method based on handicaps.
Theoretical Economics ,9(3):915–942.Duggan, J. and Schwartz, T. (2000). Strategic manipulability without resoluteness or sharedbeliefs: Gibbard-Satterthwaite generalized.
Social Choice and Welfare , 17(1):85–93.Gibbard, A. (1973). Manipulation of voting schemes: A general result.
Econometrica ,41(4):587–601.Gl¨anzel, W. and Moed, H. F. (2013). Opinion paper: thoughts and facts on bibliometricindicators.
Scientometrics , 96(1):381–394.K´oczy, L. ´A. and Strobel, M. (2010). The world cup of economics journals: A ranking by atournament method. IEHAS Discussion Papers 1018, Institute of Economics, HungarianAcademy of Sciences.Kongo, T. (2014). An alternative axiomatization of the Hirsch index.
Journal of Informet-rics , 8(1):252–258. 9alawski, M. and Zhou, L. (1994). A note on social choice theory without the Paretoprinciple.
Social Choice and Welfare , 11(2):103–107.Marchant, T. (2009). An axiomatic characterization of the ranking based on the h-indexand some other bibliometric rankings of authors.
Scientometrics , 80(2):325–342.Miroiu, A. (2013). Axiomatizing the Hirsch index: Quantity and quality disjoined.
Journalof Informetrics , 7(1):10–15.Palacios-Huerta, I. and Volij, O. (2004). The measurement of intellectual influence.
Econometrica , 72(3):963–977.Palacios-Huerta, I. and Volij, O. (2014). Axiomatic measures of intellectual influence.
International Journal of Industrial Organization , 34:85–90.Perry, M. and Reny, P. J. (2016). How to count citations if you must.
American EconomicReview , 106(9):2722–2741.Quesada, A. (2010). More axiomatics for the Hirsch index.
Scientometrics , 82(2):413–418.Quesada, A. (2011a). Axiomatics for the Hirsch index and the Egghe index.
Journal ofInformetrics , 5(3):476–480.Quesada, A. (2011b). Further characterizations of the Hirsch index.
Scientometrics ,87(1):107–114.Satterthwaite, M. A. (1975). Strategy-proofness and Arrow’s conditions: Existence andcorrespondence theorems for voting procedures and social welfare functions.
Journal ofEconomic Theory , 10(2):187–217.Woeginger, G. J. (2008a). An axiomatic analysis of Egghe’s 𝑔 -index. Journal of Informetrics ,2(4):364–368.Woeginger, G. J. (2008b). An axiomatic characterization of the Hirsch-index.