Inconsistency indices for incomplete pairwise comparisons matrices
aa r X i v : . [ c s . D M ] N ov Inconsistency indices for incomplete pairwisecomparisons matrices
Konrad Kułakowski
AGH University of Science and Technology, Faculty of Electrical Engineering, Automatics,Computer Science and Biomedical Engineering, Kraków, Poland
Dawid Talaga
The Higher Theological Seminary of the Missionaries, Kraków, Poland
Abstract
Comparing alternatives in pairs is a very well known technique of ranking cre-ation. The answer to how reliable and trustworthy ranking is depends on theinconsistency of the data from which it was created. There are many indicesused for determining the level of inconsistency among compared alternatives.Unfortunately, most of them assume that the set of comparisons is complete,i.e. every single alternative is compared to each other. This is not true and theranking must sometimes be made based on incomplete data.In order to fill this gap, this work aims to adapt the selected twelve existinginconsistency indices for the purpose of analyzing incomplete data sets. Themodified indices are subjected to Monte Carlo experiments. Those of them thatachieved the best results in the experiments carried out are recommended foruse in practice.
Keywords: pairwise comparisons; inconsistency; incomplete matrices; AHP
1. Introduction
People have been making decisions since time began. Some of them are verysimple and come easily but other, more complicated, ones require deeper anal-ysis. Often, when many complex objects are compared, it is difficult to choosethe best one. The pairwise comparisons (PC) method may help to solve thisproblem. Probably the first well-documented case of using the PC method is thevoting procedure proposed by
Ramon Llull [13] - a thirteenth century alchemistand mathematician. In
Llull’s algorithm, the candidates were compared in pairs
Email addresses: [email protected] (Konrad Kułakowski), [email protected] (Dawid Talaga)
Preprint submitted to Elsevier November 14, 2019 .2 Motivation
2- one against the other, and the winner was the one who won in the largest num-ber of direct comparisons. Later on, in the eighteenth century,
Llull’s votingsystem was reinvented by
Condorcet [38]. The next step came from
Fechner [20] and
Thurstone [58] who enabled the method to be used quantitatively forassessing intangible social quantities. In the twentieth century, the PC methodwas a significant component of the social choice and welfare theory [2, 54]. Cur-rently, the PC method is very often associated with
The Analytic HierarchyProcess (AHP) . In his seminal work on
AHP, Saaty [50] combined a hierarchytogether with pairwise comparisons, which allowed the comparison of signifi-cantly more complex objects than was possible before. In this work, we dealwith the quantitative and multiplicative PC method, that is, the basis of AHP.The PC method stems from the observation that it is much easier for aman to compare objects pair by pair than to assess all the objects at once .However, comparing in pairs presents us with various challenges. One of themis the selection of the priority deriving method, including the case when the setof comparisons is incomplete. Another, equally important, one is the situationin which different comparisons may lead to different or, even worse, opposingconclusions. All these questions are extensively debated in the literature [29,31, 37, 52]. However, one of them does not seem to have been sufficientlyexplored - the co-existence of inconsistency and incompleteness. Namely, oneof the assumptions of AHP says that the higher the inconsistency of the setof paired comparisons, the lower the reliability of the ranking computed. Thisassumption has its supporters [50] and opponents [23], however, in general, mostresearchers agree that high inconsistency may be the basis for challenging theresults of the ranking. The concept of inconsistency in the PC method hasbeen thoroughly studied and resulted in a number of works [7, 12, 10, 33]. Theoriginal PC method assumes that each alternative has to be compared witheach other. However, researchers and practitioners quickly realized that makingso many comparisons can be difficult and sometimes even impossible. For thisreason, they proposed methods for calculating the ranking based on incompletesets of pairwise comparisons [44, 48, 56, 22, 21, 27].
Inconsistency of complete pairwise comparisons is well understood and thor-oughly studied in the literature [9]. One can easily find at least a dozen well-known indices allowing to determine the level of inconsistency. In the case ofincomplete PC matrices, however, the phenomenon of inconsistency remains arelatively little explored area. There are only a few proposals of inconsistencyindices for incomplete PC. One of them has been proposed by
Harker [27], lateron, developed by
Wedley [59]. The more recent index comes from
Oliva et al .[47].
Bozóki et al. proposed using the value of the logarithmic least squarecriterion as the inconsistency measure [4].The purpose of this work is to provide the readers with other inconsistencyindices for incomplete PC. However, the authors decided not to create newindices but to adapt existing ones so that they could be used in the context ofincomplete PC. As a result, new versions for eight inconsistency indices have .3 Article organization
Koczkodaj’s index [32], triad based indices [42],
Saloand Hämäläinen index [53], geometric consistency index [15, 1],
Golden-Wang index [25],
Barzilai’s relative error [3].One can expect that a useful inconsistency index should be resistant torandom deletion of comparisons (a random increase of incompleteness). Thus,during the Montecarlo experiment, all the newly redefined indices, includingHarker’s index and Bozóki’s criterion, were compared for their robustness in asituation of random data deletion. The proposed approach allows assessing thecredibility of the considered indices concerning incompleteness.In AHP, the assessment of ranking veracity is inseparably linked to the con-cept of inconsistency indices. The purpose of our paper is to propose a varietyof inconsistency indexes for incomplete PC and identify those that may be par-ticularly useful. We also realize that the relationship between inconsistency andincompleteness must be subject to further study [43]. Despite the preliminarynature of Montecarlo results, we believe that this work will contribute to theincrease in the popularity of incomplete pairwise comparisons as the rankingmethod and make it more reliable and trustworthy.
The fundamentals of the pairwise comparisons method, including priorityderiving algorithms for complete and incomplete paired comparisons and theconcept of inconsistency, are introduced in Section 2. Due to the relativelylarge number of indices considered in this work, they are briefly reviewed inSection 3. In (Section 4), we briefly describe the main assumptions of our pro-posal for the extensions of selected indexes. In particular, we propose dividingthe indices into two groups: the matrix based indices and the ranking basedindices. The extensions of the indices from the first group are described in (Sec-tion 5), while modifications of indices from the second group can be found in(Section 6). Proposals for extensions considered in (Sections 4 - 6) are followedby a numerical experiment carried out in order to assess the impact of incom-pleteness on the disturbances of the considered indices (Section 7). Discussionand summary (Section 8) close the article.
2. Preliminaries
The PC method is used to create a ranking of alternatives. Let us denotethem by A = { a , . . . , a n } . Creating a ranking in this case means assigning toeach alternative a certain positive real number w ( a i ) , called weight or priority.To achieve this, the pairwise comparisons method compares each alternativewith all the others, then, based on all these comparisons, computes the prioritiesfor all alternatives. As alternatives are compared by experts in pairs, it isconvenient to represent the set of paired comparisons in the form of a pairwisecomparisons (PC) matrix. .1 The Pairwise Comparisons Method Definition 1.
The matrix C is said to be a PC matrix C = c · · · c n ... · · · ...... · · · . . . ... c n · · · c n,n − , if c ij ∈ R + corresponds to the direct comparisons of the i-th and j-th alterna-tives.For example: if, in an expert’s opinion, the i-th alternative is two times morepreferred than the j-th alternative, then c ij receives the value . Of course, insuch a situation it is natural to expect that the j-th alternative is two times lesspreferred than the i-th alternative, which, in turns, leads to c ji = 1 / . Definition 2.
The PC matrix in which c ij = 1 /c ji is said to be reciprocal , andthis property is called reciprocity .In further considerations in this paper, we will deal only with reciprocalmatrices. If the expert is indifferent when comparing a i and a j then the corre-sponding pairwise comparisons result in c ij = 1 , which means a tie between thecompared alternatives.In practice, it is very often assumed that the results of pairwise comparisonsfall into a certain real and positive interval /s ≤ c ij ≤ s . The value s determinesthe range of the scale. For example, Saaty [51] recommends the use of a discretescale where c ij ∈ { / , / , . . . / , , , . . . , , } . Other researchers, however,suggest other scales [24, 61, 18]. For the purpose of this article, we assume that /s ≤ c ij ≤ s where s = 9 .Based on the PC matrix, the priorities of individual alternatives are calcu-lated (Fig. 1). PC matrix c · · · c n c · · · c n ... ... . . . ... c n c n · · · a a a a a n a n . . . ... Priority calculation w ( a ) w ( a )... w ( a n ) Ranking vector
Figure 1: The PC method
It is convenient to present them in the form of a weight vector (1) so thatthe i-th position in the vector denotes the weight of the i-th alternative. w = [ w ( a ) , w ( a ) , . . . , w ( a n )] T . (1)There are many procedures enabling the construction of a priority vector.The first, and probably still the most popular, is one using the eigenvector .2 Inconsistency C . According to this approach, referred to in the literature as EVM (TheEigenvalue Method), the principal eigenvector of the PC matrix is adopted asthe priority vector w . For convenience, the principal eigenvector is rescaled sothat all its entries add up to . Formally, let Cw max = λ max w max (2)be the matrix equation so that λ max is a principal eigenvalue (spectral radius)of C . Then w max is a principal eigenvector of C (due to the Perron-Frobenustheorem, such a real and positive one exists [46]). Thus, the ranking vector w ev is given as (1) where w ev ( a i ) = w max ( a i ) P ni =1 w max ( a i ) . (3)Another popular priority deriving procedure is called GMM (geometric meanmethod) [16]. In this approach, the priority of an individual alternative isdefined as an appropriately rescaled geometric mean of the i-th row of C . Thus,the priority of the i-th alternative is formally given as: w gm ( a i ) = n qQ nj =1 c ij P ni =1 w gm ( a i ) . (4)There are many other priority deriving methods [31, 36, 44]. In general, all ofthem lead to the same ranking vector unless the set of paired comparisons isinconsistent. Let us look at inconsistency a little bit closer. If we compare two pairs of alternatives ( a i , a k ) and ( a k , a j ) , then the resultsof these two comparisons also provide us with information about the mutualrelationship between a i and a j . Indeed, the result of the comparisons a i vs. a k is a positive and real number c ik being an approximation of the ratio betweenthe priorities of the i-th and k-th alternatives i.e. w ( a i ) w ( a k ) ≈ c ik . (5)Similarly, w ( a k ) w ( a j ) ≈ c kj . (6)This implies, of course, that c ik c kj ≈ c ij . (7)If a PC matrix is consistent, then the above formula turns into equality, i.e. c ik c kj = c ij for every i, k , and j ∈ { , . . . , n } where i = k, k = j and i = j . Letus define these three values c ik c kj and c ij formally. Definition 3.
A group of three entries ( c ik , c kj , c ij ) of the PC matrix C is calleda triad if i, j, k ∈ { , . . . , n } and i = j, j = k and i = k . .3 Incompleteness Definition 4.
A PC matrix C = [ c ij ] is said to be inconsistent if there is atriad c ik , c kj and c ij for i, j, k ∈ { , . . . , n } such that c ik c kj = c ij . Otherwise C is consistent.For the purpose of the article, any three values in the form c ik , c kj and c ij will be called a triad . If there is a triad such that c ik c kj = c ij then the triadand, as follows, the matrix C are said to be inconsistent .It is easy to observe that when the PC matrix is consistent then (5) and (6)are also equalities. Hence, the consistent PC matrix takes the form: C = w ( a ) w ( a ) · · · w ( a ) w ( a n ) ... · · · ...... · · · . . . ... w ( a n ) w ( a ) · · · w ( a n ) w ( a n − ) . In practice, the PC matrix arises during tedious and error-prone work ofthe experts who compare alternatives pair by pair. Therefore, due to variousreasons, inconsistency occurs. Since the data that we use to calculate rankingsare inconsistent, the question arises concerning the extent to which the obtainedranking is credible. A highly inconsistent PC matrix can mean that the expertpreparing the matrix was inattentive, distracted or just lacking sufficient knowl-edge and skill to carry out the assessment. Therefore, most researchers agreethat highly inconsistent PC matrices result in unreliable rankings and shouldnot be considered. On the other hand, if the PC matrix is not too inconsistent,the ranking can be successfully calculated. To determine what the inconsistencylevel of the given PC matrix is, inconsistency indices are used. Because thereare over a dozen of them (sixteen indices are subjected to the Montecarlo ex-periment described in this work), their exact description has been included inSection 3
As stated above, the PC matrix contains mutual comparisons of all alterna-tives taken into account. However, from a practical point of view, completing allnecessary comparisons can be difficult. The first reason is the square increase inthe number of comparisons in relation to the number of alternatives considered(providing reciprocity n alternatives implies at least n ( n − / comparisons).As it is easy to see, for alternatives we need comparisons but for we needas many as comparisons and so on. Therefore, in the case of a large number ofalternatives, gathering all comparisons is just labor-intensive. This is especiallytrue as these comparisons are usually made by experts who, as always, sufferfrom a lack of time. For that reason, Wind and Saaty [60] indicated the optimalnumber of alternatives as ± . Another reason for the lack of comparison .3 Incompleteness ? . Let us define incomplete PC matrices formally. Definition 5.
A PC matrix C = [ c ij ] is said to be an incomplete PC matrixif c ij ∈ R + ∪ { ? } where c ij =? means that the comparison of the i-th and j-thalternatives is missing.In the case of missing values, the reciprocity condition would mean that c ij =? implies c ji =? .Bearing in mind all the above problems with obtaining a complete set ofcomparisons, Harker [27, 28] proposed the extension of EVM for an incompletePC matrix. HM (The Harker’s method) requires the creation of an auxiliarymatrix B = [ b ij ] , in which b ij = c ij if c ij is a real number greater than otherwise m i is the number of unanswered questions in the i-th row of C . (8)Finding and scaling a principal eigenvector of B leads directly to the desirednumerical ranking.The well-known GMM (4) also has its own extension for the incompletePC matrices [4]. According to the ILLS (incomplete logarithmic least square)method, one needs to solve the linear equation: R b w = g (9) b w ( a ) = 0 (10)where R = [ r ij ] is the Laplacian matrix [45] such that r = α if i = j where α is the number of ? in the i-th row − if c ij =?0 if c ij =? , (11) g is the constant term vector g = [ g , . . . , g n ] T where g i = log Y c ij =? j =1 ,...,n c ij , and b w is the logarithmized priority vector b w = [ b w ( a i ) , . . . , b w ( a n )] , i.e. b w ( a i ) =log w ( a i ) for i = 1 , . . . , n where w is the appropriate priority vector . The ILLS In practice, w should also be rescaled so that all its entries sum up to .4 Graph representation It is often convenient to consider a set of pairwise comparisons, a PC matrix,as a graph. For this reason, let us introduce the definition of a graph of thegiven PC matrix.
Definition 6.
A directed graph T C = ( V, E, L ) is said to be a graph of C if V = { a , . . . , a n } is a set of vertices, E ⊂ V \ S ni =1 ( a i , a i ) is a set of orderedpairs called directed edges, L : V → R + such that L ( a i, , a j ) = c ij is the labelingfunction, and C = [ c ij ] is the n × n PC matrix.For example, let us consider the following incomplete PC matrix in which c and c are undefined: C =
23 43 1232 ? 1 The graph T C is shown in Fig. 2a. Providing that the matrix C is reciprocal ,the upper triangle of C contains all the information necessary to create a rank-ing. Therefore, instead of the graph T C one can analyze a graph of the uppertriangle of C . Thanks to this, we obtain a simplified drawing of the graph,without losing essential information. Let UT ( C ) denote the upper triangle of C . The graph of the upper triangle of C is shown in Fig. 2b. / / / / / a a a a (a) T C - graph of C / / / / a a a a (b) T UP ( C ) - thegraph of the uppertriangle of C Figure 2: Graph representations of the matrix C For incomplete PC matrices, one property of the graph is particularly im-portant. This is strong connectivity [49]. In the literature, non-reciprocal PC matrices are also considered [41, 30].
Definition 7.
A directed graph T C is strongly connected if for any pair of dis-tinct vertices a i and a j there is an oriented path ( a i , a r ) , ( a r , a r ) , . . . , ( a r k , a j ) in E from a i to a j .The matrix C for which T C is strongly connected is called irreducible [49]. Itis easy to notice that two distinct alternatives a i and a j (through the orientedpath leading from a i to a j ) can be compared together only if T C is stronglyconnected. This immediately leads to the observation that only when the PCmatrix is irreducible (i.e. the appropriate graph is strongly connected), are weable to compute the ranking [55, 27]. For this reason, in the article, we onlydeal with irreducible PC matrices.For the purposes of this article, let us also define the concept of the cycle ina graph. Definition 8.
An ordered sequence of distinct vertices p = a i , a i , . . . , a i m such that { a i , a i , . . . , a i m } ∈ V is said to be a path between a i and a i m withthe length m − in T C = ( V, E, L ) if ( a i , a i ) , ( a i , a i ) , . . . , ( a i m − , a i m ) ∈ E .and similarly Definition 9.
A path s between a i and a i m with the length m − is said tobe a simple cycle with the length m if also ( a i m , a i ) ∈ E .In the case of a cycle, it is not important which element in the sequence ofvertices is first. Thus, if p = a i , a i , . . . , a i m is a cycle then q = a i , . . . , a i m , a i means the same cycle, i.e. p = q . Definition 10.
Let T C = ( V, E, L ) be a graph of C . Then the set of all pathsbetween a i and a j in T C is defined as P C,i,j df = { p = a i , a i , . . . , a i m is a pathbetween a i and a i m in T C } . Similarly, the set of all cycles longer than q in T C is defined as S C,q df = { s = a i , a i , . . . , a i m is a cycle of C for m > q } .The number of adjacent edges to the given vertex a usually is called thedegree of a and written as deg( a ) . Based on the degree of vertex we can definea degree matrix. Definition 11.
Let T UP ( C ) = ( V, E, L ) be a graph of C . The degree matrix D = [ d ij ] of T UP ( C ) is a diagonal matrix such that d ii = deg( a i ) for and d ij = 0 for i, j = 1 , . . . , n and i = j .
3. Inconsistency indices
In his seminal work,
Saaty [50] proposed a measurement of inconsistencyas a way of determining credibility of the ranking. Since then, many inconsis-tency indices have been created allowing the degree of inconsistency in the setof paired comparisons to be determined . Below, we briefly present several As in the paper we deal with cardinal (quantitative) pairwise comparisons, we do notconsider ordinal inconsistency of the ordinal pairwise comparisons. A good example of theordinal inconsistency index is the generalized consistency coefficient [39]. .1 Inconsistency indices for complete PC matrices
The list of indices is opened by the geometric consistency index (GCI). GCIgiven as: I G = 2( n − n − n X i =1 n X j = i +1 log e ij . (12)where e ij = c ij w ( a j ) w ( a i ) , i, j = 1 , ..., n, (13)was proposed by Crawford and
Williams [17], and then called as the geometricconsistency index by Aguaròn and Moreno-Jimènez [1].In contrast to the previous indices, a measure defined by
Koczkodaj doesnot examine the average inconsistency of the set of paired comparisons [32, 19].Instead, it spots the highest local inconsistency and adopts it as an inconsistencyof the examined matrix. A local inconsistency is determined by means of thetriad index K i,k,j defined as follows: K i,j,k = min (cid:26)(cid:12)(cid:12)(cid:12)(cid:12) − c ik c kj c ij (cid:12)(cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:12) − c ij c ik c kj (cid:12)(cid:12)(cid:12)(cid:12)(cid:27) . (14)The inconsistency index for C obtains the form: K = max { K i,j,k | ≤ i < j < k ≤ n } (15) Kułakowski and
Szybowski proposed two other inconsistency indices [42],which are also based on triads . They both use the Koczkodaj triad index K ijk (14). The indices are designed as the average of all possible K ijk given asfollows : I = 6 P { i,j,k }∈ T K ijk n ( n − n − , (16) I = 6 qP { i,j,k }∈ T K ijk n ( n − n − , (17)where T = {{ i, j, k } : i = j, i = k, j = k and ≤ i, j, k ≤ n } . Both indices canbe combined together to create new coefficients. Based on this observation, theauthors proposed two parametrized families of indices: I α = αK + (1 − α ) I , (18) The first of them was later proposed by Grzybowski [26] Note that (cid:0) n (cid:1) = n ( n − n − . .1 Inconsistency indices for complete PC matrices ≤ α ≤ , and I α,β = αK + βI + (1 − α − β ) I , (19)where ≤ α + β ≤ . Golden and
Wang proposed another inconsistency index [25]. Accordingto this approach the priority vector was calculated using the geometric meanmethod, then scaled to add up to 1. In this way, the vector g ∗ = [ g ∗ , , ..., g ∗ n ] was obtained, where C = [ c ij ] is an n by n PC matrix. Then, every column isscaled so that the sum of its elements is . Let us denote the matrix with therescaled columns by C ∗ = [ c ∗ ij ] . The inconsistency index is defined as follows: GW = 1 n n X i =1 n X j =1 | c ∗ ij − g ∗ i | . (20)The index proposed by Salo and
Hämäläinen [53] requires an auxiliary in-terval matrix R to be prepared. In this matrix, every element is a pair corre-sponding to the highest and the lowest approximation of c ij . The n × n matrix R is given as R = ( r , r ) . . . ( r n , r n ) ... . . . ... ( r n , r n ) . . . ( r nn , r nn ) , (21)where r ij = min { c ik c kj | k = 1 , . . . , n } , and r ij = max { c ik c kj | k = 1 , . . . , n } As every c ik c ji is an approximation of c ij then r ij is the lowest and r ij is thehighest approximation of c ij . Finally, the inconsistency index is: I SH = 2 n ( n − n − X i =1 n X j = i +1 r ij − r ij (1 + r ij ) (cid:0) r ij (cid:1) . (22)For further reference, see [6].The last of the extended indexes was proposed by Barzilai [3]. It requirescalculation of the weight vector using the arithmetic mean method for each rowand the preparation of two auxiliary matrices. Let us denote ∆ i = n P nj =1 b c ij , where b C = [ b c ij ] is an n by n additive PC matrix, i.e. such that c ij ∈ R and c ij = − c ji . The two auxiliary matrices are given as follows: X = [ x ij ] = [∆ i − ∆ j ] , E = [ e ij ] = [ b c ij − x ij ] . Ultimately, the formula for the relative error (consideredas the inconsistency index) is as follows: RE ( b C ) = P ij e ij P ij b c ij . (23) .2 Inconsistency indices for incomplete PC matrices RE was defined for additive PC matrices. Thus, for the purposeof multiplicative PC matrices, Barzilai proposes to transform it using a log function with any base. Thus, for the PC matrix C = [ c ij ] and b C = [log c ij ] weobtain: RE ( C ) df = RE ( b C ) . Inconsistency indexes for incomplete PC matrices are definitely less than forcomplete matrices. The first of them, probably the earliest defined is the Saaty’sindex for incomplete PC matrices defined by Harker [27] as: f CI = e λ max − nn − where e λ max is the principal eigenvalue of the auxiliary matrix B (8). Following[27, p. 356], the consistency index f CI can also be written as: f CI = − n ( n − n X i =1 m i + X ≤ i < j ≤ nc ij =? (cid:18) c ij e w ( a j ) e w ( a i ) + c ji e w ( a i ) e w ( a j ) (cid:19) . where e w = [ e w ( a ) , . . . , e w ( a n )] T is the principal eigenvector of B . Properties of f CI were also tested in [57].Another method of measuring the inconsistencies of incomplete matriceswas proposed by Bozóki et al. [4]. According to this approach all the possiblecompletion of an incomplete PC matrix are considered, then one that minimizesa certain inconsistency criterion is chosen. The value of this criterion for theselected completion can be considered as the inconsistency value of the givenincomplete PC matrix. Adopting the function: LLS ( C, w ) = n X i,j =1 i = j (cid:18) log c ij − log w ( a i )log w ( a j ) (cid:19) as such criterion [16, 4] leads to ILLS method (Sec. 2.3). However, LLS ( C, w ) can also be treated as a ranking based inconsistency index. Thus, following [4],we can adopt e L ( C ) df = LLS ( Q C , w ILLS ) as the inconsistency index for incomplete matrix C , where Q C = [ q ij ] is anoptimal completion of C defined as: q ij = ( c ij if c ij =? w ILLS ( a i ) w ILLS ( a j ) if c ij =? . O ( R ) = ρ ( D − S ) − . In the above equation R is an incomplete multiplicative PC matrix in whichevery missing element is represented by , matrix D is the degree matrix of thegraph T UP ( R ) (Def. 11), ρ stands for spectral radius of a matrix, and S = R− Id ,where Id denotes n × n identity matrix.
4. Extensions of inconsistency indexes for incomplete matrices
Among the indices listed above, two distinct groups can be distinguished.The first group consists of indices based on the concept of a triad (Def. 3) andthe idea of the triad’s inconsistency (Section 2.2). According to this idea, theanalysis of three different entries of a PC matrix is able to reveal the inconsis-tency. Of course, this analysis is local as it is limited to three specific compar-isons. However, if we take into account all possible triads in C , our judgmentas to the inconsistency will become global and may act as an inconsistency in-dex. In this approach, the ranking method is not important. Inconsistency isestimated directly using elements of the PC matrix and the definition of incon-sistency (Def. 4). We will call all the indices for which the above observationholds the matrix based indices . This group includes: • Koczkodaj’s inconsistency index, • Triad based average inconsistency indices, • Salo and Hamalainen index.The second group of indices are those for which calculation of the ranking isindispensable. The general idea behind all of the indices in this group is thatthe ratio w ( a i ) /w ( a j ) needs to be similar or even identical to the value c ij forall i, j ∈ { , . . . , n } (see 5 - 7). Of course, to verify the difference between w ( a i ) /w ( a j ) and c ij we first have to compute the ranking vector (1). For thisreason, each of the indices in this group is closely related to some priorityderiving method. For the purpose of this article, we will call them the rankingbased indices. This group includes : • Geometric consistency index, • Golden-Wang index, • Relative Error index For the purpose of the Montecarlo experiment we also consider Harker’s extension ofSaaty’s consistency index [27], Logarithmic least square criterion [4] and Oliva et al. incon-sistency index [47]. .1 Matrix based indices
Matrix based indices use triads to determine inconsistency. However, inan incomplete PC matrix some triads might be missing. For example, if c ik is undefined the triad c ik , c kj and c ij is also undefined. Of course, one maythink that analysis of the remaining triads allows us to assess the degree ofmatrix inconsistency. Unfortunately, it is not true and in some circumstancesthis strategy fails. This happens when the PC matrix does not have any triadsand yet it is irreducible. Let us consider the following PC matrix: C = / /
72 1 ? 6 4 2 ?? ? 1 4 3 3 / / / /
2? 1 / / /
4? 1 / / /
37 ? ? 2 4 3 1 (24)The graph T UP ( C ) of the upper triangle of the above matrix is shown in Fig.3. As we can see, there is no cycle (Def. 9) with the length , thus there areno triads in C . It is easy to observe that T C is strongly connected, thus C isirreducible, and therefore it is a valid incomplete PC matrix. / / / a a a a a a a Figure 3: T UP ( C ) - the graph of the upper triangle of the PC matrix C which does not containtriads Since we can not use triads in this case, the question arises as to whether weshould not use quadruples ( c ik , c kr , c rj , c ij ) i.e. cycles with the length . As inthe previous case, the answer is negative. We are able to construct a directedgraph without cycles with the length and, as follows, a PC matrix which does Remember that in the full graph T C each edge ( a i , a j ) has its counterpart ( a j , a i ) . .1 Matrix based indices n vertices and only n − edges. In such graphs,every vertex a i is connected with only one other vertex a j by two edges ( a i , a j ) and ( a j , a i ) . If we remove any pair of the existing edges { ( a i , a j ) , ( a j , a i ) } , thegraph would cease to be strongly connected. Conversely, if we add a new pair ofedges { ( a p , a q ) , ( a q , a p ) } to E it would be a cycle in the graph. For this reason,whenever there are cycles in T C , we will try to use them all to inconsistencywithout limiting their length or quantity. However, if there are no cycles in thegraph, the concept of inconsistency loses meaning. Therefore, the only optionis to accept that the considered PC matrix is consistent (an alternative wouldbe to assume that the inconsistency is indeterminate).In order to use the cycle to determine the matrix inconsistency, let us extendthe Definition 4. Definition 12.
A PC matrix C = [ c ij ] is said to be inconsistent if there existsa cycle a i , . . . , a i m in T C such that c i i c i i · . . . · c i m − i m = c i i m . Otherwise C is consistent.For complete PC matrices, both definitions 4 and 12 are equivalent. To provethat, it is enough to show that wherever C is inconsistent in the sense (Def. 4)then it is also inconsistent in the sense (Def. 12), and reversely inconsistency inthe sense (Def. 12) entails inconsistency in the sense (Def. 4). Theorem 13.
Every complete PC matrix C is inconsistent in the sense of (Def.4) if and only if it is inconsistent in the sense of (Def. 12)Proof. “ ⇒ ” Let C be inconsistent in the sense of (Def. 4) i.e. there is a triadsuch that c ik c kj = c ij . Since this triad is also a cycle with the length , C isalso inconsistent in the sense of (Def. 12).“ ⇐ ” Let C be inconsistent in the sense of (Def. 12) i.e. there is a cycle s = a i , . . . , a i m such that c i i c i i · . . . · c i m − i m = c i i m ,and let us suppose for a moment that C is consistent in the sense of (Def. 4).The latter assumption means that every triad is consistent, thus, in particular italso holds that c i i c i i = c i i . Therefore, the first assumption can be writtenin the form: c i i · . . . · c i m − i m = c i i m . Applying the same reasoning many times,we subsequently get that c i i · . . . · c i m − i m = c i i m , c i i · . . . · c i m − i m = c i i m and finally c i i m − · c i m − i m = c i i m . However, as we assume that every triadis consistent, therefore also a i , a i m − , a m is consistent, thus it must hold that c i i m − · c i m − i m = c i i m . Contradiction.Of course, when C is incomplete the two above definitions of inconsistencyare not equivalent. In particular, there may be PC matrices which do nothave any triads, hence they have to be considered as consistent, but the samematrices may have graphs with cycles longer than that might be inconsistent.In general, however, the cycle based definition of inconsistency is more generalthan the Def. 4. We may observe the following property. .2 Ranking based indices Remark . Every PC matrix (complete and incomplete) inconsistent in thesense (Def. 4) is also inconsistent in the sense of (Def. 12), but not reversely.Definition 12 also allows us to quantify the inconsistency. As we will seelater on, the ratio: R s df = c i i · . . . · c i m − i m c i i m (25)defined for a cycle s = a i , . . . , a i m is a useful way for measuring inconsis-tency within the set of m alternatives a i , . . . , a i m . We use this fact to defineseveral matrix based indices for an incomplete PC matrix. The idea of usingcycles for inconsistency measurement can be found in [5, 33]. The ranking based indices need the results of ranking in order to calculateinconsistency. The considered indices use two different priority deriving meth-ods: EVM (2) and GMM (3). Although both methods have been defined for acomplete PC matrix, they have their counterparts for incomplete PC matrices(Section 2.3). These are the HM [28] and ILLS approaches [4].The starting point of both extensions is the assumption that every missingvalue c ij in C should eventually take the value w ( a i ) /w ( a j ) . Therefore, theauthors of extensions replaced the unknown values by the appropriate ratios, andthen tried to solve such modified problems. Let b C be a PC matrix obtained from C by replacing every c ij =? by w ( a i ) /w ( a j ) . In HM, the eigenvalue equation(2) takes the form: b Cw = λ max w, and after the appropriate transformations, we finally get Bw = λ max w where B is an auxiliary matrix (8). Similarly, in the ILLS method [4], theauthors define the priority of the i-th alternative as the geometric mean of thei-th row of b C . The adoption of this assumption leads to the matrix equation(9), whose solution determines the desirable vector of priorities.In general, the ranking based indices define the inconsistency as the dif-ferences between c ij and w ( a i ) /w ( a j ) for i, j = 1 , . . . , n . Since both HM andILLS replace every c ij =? by the corresponding ratio w ( a i ) /w ( a j ) , then themissing judgments do not contribute to the inconsistency, but are consideredas perfectly consistent. From a practical point of view, during construction ofthe ranking based indices for incomplete PC matrices, we can either ignore themissing values or just assume that c ij =? equals w ( a i ) /w ( a j ) . Note that when C is reciprocal then R s does not depend on the choice of m . Indeed: R s = c i i c i i · . . . · c i m − i m c i i m = c i i c i i · . . . · c i m − i m c imi = c i i · . . . · c i m − i m c i m i c i i = . . .
5. Matrix based indices for incomplete PC matrices
The Koczkodaj index is directly based on the concept of a triad and itsinconsistency. Thus, as explained above (Sec. 4.1), a triad’s inconsistencyhas to be replaced by the cycle’s inconsistency. Let C be an irreducible andincomplete PC matrix (Def. 5) and T C be a graph of C (Def. 6). Then let usdefine the inconsistency of a single cycle longer than i.e. s ∈ S C, as K s df = min (cid:8) | − R s | , (cid:12)(cid:12) − R − s (cid:12)(cid:12)(cid:9) (26)Then the Koczkodaj index for the incomplete PC matrix C can be defined as: e K df = ( max { K s : s ∈ S C, } |S C, | > |S C, | = 0 The case in which |S C, | = 0 refers to the situation when the n × n matrix C isirreducible, but it contains exactly n − comparisons i.e. T UP ( C ) is a tree [14]. The method of replacing triads with cycles can be successfully used in thecase of triad based average inconsistency indices. Thus, providing that C is anirreducible and incomplete PC matrix, we have: e I df = ( P s ∈S C, K s |S C, | |S C, | > |S C, | = 0 , and correspondingly, e I df = qP s ∈S C, K s |S C, | |S C, | > |S C, | = 0 . The I α and I α,β indices (18, 19) also need to be changed accordingly. e I α df = α e K + (1 − α ) e I , e I α,β df = α e K + β e I + (1 − α − β ) e I . It is worth noting that if s = a i a k a j then K s = K i,k,j (14, 26). Cycles with the length are always consistent as c ij c ji /c ii = 1 , thus they are not relevantfrom the point of inconsistency of C . .3 Salo and Hamalainen index The
SHI index is based on the observation that every product c ik c kj for any k = 1 , . . . , n is an approximation of c ij [53]. When C is irreducible, due to thestrong connectivity of T C between every two vertices a q and a j there is a path p = a q , a i , a i , . . . , a i m − , a j (Def. 9). This means that c q,i , c i i , . . . , c i m − ,j are defined. Let us denote the product induced by p as π p = c q,i c i i , · . . . · c i m − ,j .Due to (5 - 6), π p is also a good approximation of c ij . Thus, let us define r e ij df = min { π p | p ∈ P C,i,j } and e r ij df = max { π p | p ∈ P C,i,j } The modified I SH index can be defined as e I SH df = 2 n ( n − n − X i =1 n X j = i +1 e r ij − r e ij (1 + e r ij ) (cid:18) r e ij (cid:19) .
6. Ranking based indices for incomplete PC matrices
The geometric consistency index (12) is directly based on observation (5) i.e. w ( a i ) /w ( a k ) ≈ c ik , where w is the priority vector obtained from GMM [15, 1].In an incomplete PC matrix, a priority vector e w has to be computed using theILLS method (Sec. 2.3). Following this method wherever c ij =? it is replacedby e w ( a i ) / e w ( a j ) . This leads to the following formula: e I G df = 2( n − n − X e ∈ E log e, (27)where C = [ c ij ] , e w = [ e w ( a ) , . . . , e w ( a n )] T is the ranking vector calculated usingthe ILLS method, and E df = n e ij = c ij e w ( a j ) e w ( a i ) : c ij =? and i < j o . The secondway to extend the GCI index is to calculate the average of all non-zero log e expressions also possible. In this approach we assume that c ij =? does notcontribute to our knowledge about inconsistency. Hence, the second version ofGCI for incomplete PC matrices is as follows: e I G df = 1 | E | X e ∈ E log e. .2 Golden-Wang index The Golden-Wang index is based on the observation that every column ofa consistent PC matrix equals the ranking vector multiplied by some constantscaling factor [25]. Thus, after scaling every column so that it sums up to ,it holds that c ∗ ij = w ( a i ) , where C ∗ = [ c ∗ ij ] is the consistent PC matrix withthe rescaled columns. The difference between c ∗ ij and w ( a i ) is higher when theinconsistency is greater.Despite the fact that the observations remain true for any priority derivingmethod, the authors recommend using GMM (4). The relationship between c ∗ ij and w ( a i ) also remains valid in the case of an incomplete PC matrix, however,due to the missing values the scaling procedure needs to be modified. Let usconsider the k-th column of the irreducible incomplete PC matrix C and theranking vector w . Let every element of C be either or ? . So it is easy to seethat the ranking vector is composed of the same /n values. For example, for a × matrix we may have: c k c k c k c k c k = , w ( a ) w ( a ) w ( a ) w ( a ) w ( a ) = / / / / / , where k = 1 , or . Then, after scaling (so that the sum of elements is one) thek-th column is: c k c k c k c k c k = / / / as the undefined elements cannot be scaled. It is evident that c ∗ ik = w ( a i ) for i = 1 , , as / = 1 / . The solution is to construct the priority vector w k which has the missing values at the same positions as the k-th column. Let usconsider: c k c k c k c k c k = , w k ( a ) w k ( a ) w k ( a ) w k ( a ) w k ( a ) = / / / In such a case, after scaling, indeed c ∗ ik = w ∗ k ( a i ) = 1 / for i = 1 , , .The above observation shows the way in which the Golden-Wang index canbe extended to incomplete pairwise comparisons. Let us define this extensionmore formally. For this purpose, let us assume that C is an irreducible, incom-plete PC matrix, and w is the ranking vector calculated using the ILLS method.Then let Ω = [ ω ij ] be an n × n matrix such that ω ij df = ( w ( a i ) if c ij =?? if c ij =? . .3 Relative Error Ω and in C so that all the elements inthe column sum up to one (in both cases, undefined values are omitted, i.e.only defined elements are subject to scaling). As a result, we get two matrices C ∗ = [ c ∗ ij ] and Ω ∗ = [ ω ∗ ij ] with the appropriately rescaled columns. The absolutedifferences between the entries of these two matrices form the Golden-Wangindex for incomplete PC matrices. Thus, following (20), we may define: g GW df = 1 n n X i =1 n X j =1 | c ∗ ij − ω ∗ ij | In general, Barzilai’s Relative Error RE has been defined for additive PCmatrices [3]. For the purpose of this paper we use its logarithmized versionsuitable for multiplicative matrices [3]. For an incomplete PC matrix, similarlyto the case of the GCI index, we may assume that w is calculated using the ILLSmethod (for multiplicative matrices Barzilai’s original approach uses GMM).The use of the ILLS method implies the assumption that every c ij =? canbe substituted by w ILLS ( a i ) /w ILLS ( a j ) . In particular, in such a case e ij = 0 ,hence in the formula ( ?? ) they can be omitted. Thus, the relative error for theincomplete multiplicative PC matrix C takes the form: f RE df = P c ij =? h log c ij w ILLS ( a j ) w ILLS ( a i ) i P c ij =? log c ij + P c ij =? log w ILLS ( a i ) w ILLS ( a j ) , where i, j = 1 , . . . , n . Another way of extending Relative Error to incomplete,multiplicative PC matrices is to skip all the expressions requiring missing com-parisons. Similarly to the case of the geometric consistency index, this leads toa shorter formula: f RE df = P c ij =? h log c ij w ILLS ( a j ) w ILLS ( a i ) i P c ij =? log c ij .
7. Numerical experiment
Does increasing incompleteness affect inconsistency? Intuition suggests thatit should not. If decision makers responsible for creating PC matrices are in-consistent in their judgments, then we may assume that their inconsistency willnot depend on whether they answer all or only part of the questions. The onlydifference is that in the case of a complete matrix the experts will more oftendo both: make mistakes and respond correctly. Of course, we implicitly assumethat the experts are able to consider each question with similar attention, i.e.questions are not too many, experts have enough time to think about them, andthey are professionals in the field. So if indeed inconsistency does not dependon incompleteness, then the incomplete matrix can be treated just as a sample1of some complete PC matrix. Of course, there may always be some differencesbetween the sample and the entire population, but it is natural to expect thatthey are reasonably small.In light of these observations, it seems interesting to see how much the in-consistency for the complete matrix will differ from its incomplete sample withreference to the given inconsistency index, i.e. how robust the given inconsis-tency index is for the PC matrix deterioration. To this end, we created consistent × PC matrices. Then, the entries of each matrix were disturbed bymultiplying them by the randomly chosen coefficient γ ∈ [1 /d, d ] . We repeatedthe disturbance procedure times for γ = 1 , . . . , . In this way, we receivedthe set C composed of PC matrices with varying degrees of inconsistency.Every complete × PC matrix contains comparisons (entries above thediagonal). On the other hand, the smallest irreducible × PC matrix has comparisons (as at least six edges are needed to connect seven different verticesof a graph of a matrix). Hence, preserving irreducibility, at most comparisonscan be safely removed from the complete × PC matrix. Therefore, foreach of the complete PC matrices, we prepared 15 randomly incompleteirreducible PC matrices, so that every complete matrix had its “sample” matrixwith , up to missing comparisons. Finally, we received completeand incomplete PC matrices for which we calculated all inconsistency indicesdefined in Sections 5 and 6.In order to check the robustness of different inconsistency indices, we cal-culate the directed distance between the inconsistency of the complete matrixand their incomplete counterparts. Let I ( C ) − I ( C k ) be the ordered distancebetween where I ( C ) means the value of the inconsistency index I calculatedfor a complete matrix C ∈ C , and I ( C k ) denotes inconsistency of the matrixthat was obtained from C by removing k comparisons determined by using I .Of course, different indices may take values from various ranges. Therefore,to allow those indices to be compared with each other, the ordered distance isdivided by the higher component of each difference. Hence, the rescaled ordereddistance ∆ I ( C, C k ) between the inconsistency of two matrices C and C k is givenas: ∆ I ( C, C k ) df = ( I ( C ) − I ( C k )max { I ( C ) ,I ( C k ) } max { I ( C ) , I ( C k ) } > I ( C ) = I ( C k ) = 0 . The above formula also takes into account the situation where I ( C ) = I ( C k ) =0 . In such a case, it is assumed that ∆( C, C k ) = 0 . The final result is theaverage ordered distance: D ( I, k ) df = 1 |C| X C ∈C ∆ I ( C, C k ) . (28)The subsequent values D ( I, , D ( I, , . . . , D ( I, allow the difference betweenthe inconsistency of the complete and incomplete matrix to be assessed withrespect to the given index I and the number of missing comparisons k .In an ideal case, D ( I, k ) should be for all inconsistency indices and everypossible k . In practice, of course, it is impossible as not all comparisons in the2PC matrix are inconsistent to the same extent. Therefore, it is possible that anincomplete matrix will be less (or more) consistent than its complete counter-part. If the incomplete matrix C k is less inconsistent than the complete matrix C , i.e. I ( C ) > I ( C k ) , then ∆ I ( C, C k ) > . Reversely, if the incomplete matrixis more inconsistent than its consistent predecessor, the distance is negative i.e. ∆ I ( C, C k ) < . In other words, the sign (direction) of a distance D I informs usif there is a greater complete or incomplete PC matrix. The closer ∆ I ( C, C k ) isto , i.e. the smaller | ∆ I ( C, C k ) | is, the more resistant to incompleteness is theindex I . For the purpose of this study, a directed distance for all inconsistencyindices has been computed using matrices, then the results have beenaveraged as D ( I, k ) . Any particular value of D ( I, k ) for some fixed I and k canbe interpreted as an average value of directed distance for a × randomly dis-turbed matrix where I is an inconsistency index, and k is the number of missingelements. Of course: the index is better (more robust) when it is closer to theabscissa . Of course it is possible that for some particular Q, R, k the distance | D ( Q, k ) | is greater than | D ( R, k ) | . However, for k + 1 it may turn out that | D ( Q, k + 1) | < | D ( R, k + 1) | . In such a case, it is difficult to indicate the win-ner, as one time Q is better, the other time R is better. Therefore, as the finalmeasure of index robustness, we suggest taking the area between the abscissaand the plot of its rescaled ordered distance (Fig. 4). The discrete counterpartof the size of this area is the sum of the absolute value of subsequent D ( I, k ) i.e. D ( I ) df = X k =0 | D ( I, k ) | . Of course, the smaller D ( I ) the better.In Figure 4 there are fourteen plots corresponding to the average ordereddistance (28) for all the inconsistency indices introduced in 5 and 6 and threeadditional indices found in the literature. It is easy to see that, in general, thematrix based indices perform better than the ranking based indices. The ex-ception here is the index e I , which very quickly reveals high differences betweencomplete and incomplete PC matrices. It is interesting that the incompletematrices are considered by this index as more inconsistent than the completeones (most of the plot is below the abscissa). The behavior of e I is inheritedby the e I α,β index. Here, one can also notice that incomplete matrices are con-sidered as more inconsistent than their complete counterparts. Fortunately, theother matrix based indices perform very well. The best of them is e I , where D ( e I ) = 1 . . Then D ( e I α ) = 1 . , and next D ( e I α,β ) = 1 . . Among theranking based indices, the modification of Salo-Hamalainen index e I SH stands When assessing the robustness of I it is not important whether ∆ I ( C, C k ) takes positiveor negative values. How far ∆ I ( C, C k ) is from the abscissa is more important, i.e. the size of | ∆ I ( C, C k ) | The exact numerical data are presented in the Appendix in Table ?? . ! " ! %’& ! &’(&’(%’& D k D ( ! I , k ) D ( ! K, k ) D ( ! I G , k ) D ( ! I G , k ) D ( ! I , k ) D ( ! I α , k ) , α = 0 . D ( ! I α , β , k ) , α = 0 . , β = 0 . D ( ! GW , k ) D ( ! I SH , k ) D ( ! RE , k ) D ( ! RE , k ) D ( ! CI , k ) D ( ! L , k ) D ( O, k ) Figure 4: Rescaled ordered distance for different inconsistency indices for incomplete PCmatrices with k missing comparisons. out positively as it gets D ( e I SH ) = 4 . . The other ranking based indices, likemodified Barzilai’s relative error index version 1, get higher areas under the plot.Thus, incompleteness influences the assessment of the degree of inconsistencyto a greater extent than for previous indices. All the values of D are shown inTable 1.
8. Discussion and summary
The four best indices (Table 1) achieve very similar results. They are allvery good, which means that the differences in inconsistency measured bythese indices between complete and incomplete matrices is small. For instance, D ( e I ,
4) = 0 . and D ( e I ,
11) = 0 . , which means that for of missingcomparisons we may expect a difference in value of the index smaller than ,and for of missing comparisons this difference should not be greater than4Pos. Notation Name D ( I )1 . e I Cycle based index v. I . . e I α α -index, for α = 0 . . . e K Koczkodaj index . . e I α,β α, β -index, for α = β = 0 . . . e I SH Salo-Hamalainen index . . f RE Barzilai’s relative error index v. II . . e I G Geometric consistency index v. II . . O Oliva-Setola-Scala’s index . . g GW Golden-Wang index . . e L Logarithmic least square condition . . f RE Barzilai’s relative error index v. I . . f CI Saaty consistency index . . e I G Geometric consistency index v. I . . e I Cycle based index v. II . Table 1: The total distance D from the abscissae of the D ( I, k ) plots for all considered indices.The smaller the value, the more robust the given inconsistency index. . − . Such results clearly show that inconsistency measurement for incom-plete PC matrices can indeed be a valuable indication of the quality of decisiondata. Thus, methods for calculating the ranking for incomplete PC matricesmentioned in Section 2.3 also gain methods for estimating data inconsistency.An obvious disadvantage of the matrix based indices defined in Section 5 isthe need to find all cycles in the matrix graph. This can be particularly difficultand time-consuming for larger matrices. A way to deal with a large numberof cycles may be to limit their number. This might be achieved by limitingthe analysis of inconsistency to fundamental cycles only, or just to a randomset of cycles. A similar problem does not occur in the case of the rankingbased indices. The best of them, the modification of Salo-Hamalainen index forincomplete PC matrices, gets the total distance D ( e I SH ) =4 . . It is also agood result, which proves that this index can be effectively used to assess theinconsistency of incomplete PC matrices.The article presents extensions for twelve inconsistency indices that allowthem to also be used for incomplete PC matrices. Thanks to this, users of thepairwise comparison method (including AHP) receive a way to determine thequality of incomplete decision data . The presented research does not determinewhich of the defined indices is the best in practice. Robust indices can bedifficult to implement and calculate. On the other hand, indices that are easierto calculate can be more vulnerable for decision data deterioration. Finding asolution that combines robustness with the simplicity of implementation and EFERENCES
Acknowledgment
The authors would like to show their gratitude to José María Moreno-Jiménez (Universidad de Zaragoza, Spain), Sándor Bozóki (Hungarian Academyof Sciences and Corvinus University of Budapest, Hungary) for their commentson the early version of the paper. Special thanks are due to Ian Corkill for hiseditorial help.
Disclosure statement
No potential conflict of interest was reported by the authors.
Funding
The research is supported by The National Science Centre (Narodowe Cen-trum Nauki), Poland, project no. 2017/25/B/HS4/01617.
References [1] J. Aguarón and J. M. Moreno-Jiménez. The geometric consistency in-dex: Approximated thresholds.
European Journal of Operational Research ,147(1):137 – 145, 2003.[2] K. J. Arrow. A difficulty in the concept of social welfare.
The Journal ofPolitical Economy , 1950.[3] J. Barzilai. Consistency measures for pairwise comparison matrices.
Jour-nal of Multi-Criteria Decision Analysis , 7(3):123–132, 1998.[4] S. Bozóki, J. Fülöp, and L. Rónyai. On optimal completion of incom-plete pairwise comparison matrices.
Mathematical and Computer Mod-elling , 52(1–2):318 – 333, 2010.[5] S. Bozóki and V. Tsyganok. The (logarithmic) least squares optimalityof the arithmetic (geometric) mean of weight vectors calculated from allspanning trees for incomplete additive (multiplicative) pairwise comparisonmatrices.
International Journal of General Systems , 48(4):362–381, 2019.[6] M. Brunelli.
Introduction to the Analytic Hierarchy Process . SpringerBriefsin Operations Research. Springer International Publishing, Cham, 2015.[7] M. Brunelli. A technical note on two inconsistency indices for preferencerelations: A case of functional relation.
Information Sciences , 357:1–5,August 2016.
EFERENCES
Fundam. Inform. , 144(3-4):321–332, 2016.[9] M. Brunelli. A survey of inconsistency indices for pairwise comparisons.
International Journal of General Systems , 47(8):751–771, September 2018.[10] M. Brunelli, L. Canal, and M. Fedrizzi. Inconsistency indices for pairwisecomparison matrices: a numerical study.
Annals of Operations Research ,211:493–509, February 2013.[11] M. Brunelli, A. Critch, and M. Fedrizzi. A note on the proportionalitybetween some consistency indices in the AHP.
Applied Mathematics andComputation , 219(14):7901–7906, March 2013.[12] M. Brunelli and M. Fedrizzi. Axiomatic properties of inconsistency indicesfor pairwise comparisons.
Journal of the Operational Research Society ,66(1):1–15, Jan 2015.[13] J. M. Colomer. Ramon Llull: from ‘Ars electionis’ to social choice theory.
Social Choice and Welfare , 40(2):317–328, October 2011.[14] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein.
Introduction toAlgorithms . MIT Press, 3rd edition, 2009.[15] G. Crawford and C. Williams. The Analysis of Subjective Judgment Ma-trices. Technical report, The Rand Corporation, 1985.[16] G. B. Crawford. The geometric mean procedure for estimating the scale ofa judgement matrix.
Mathematical Modelling , 9(3–5):327 – 334, 1987.[17] R. Crawford and C. Williams. A note on the analysis of subjective judge-ment matrices.
Journal of Mathematical Psychology , 29:387 – 405, 1985.[18] Y. Dong, Y. Xu, H. Li, and M. Dai. A comparative study of the numer-ical scales and the prioritization methods in AHP.
European Journal ofOperational Research , 186(1):229–242, March 2008.[19] Z. Duszak and W. W. Koczkodaj. Generalization of a new definitionof consistency for pairwise comparisons.
Information Processing Letters ,52(5):273 – 276, 1994.[20] G. T. Fechner.
Elemente der Psychophysik . Breitkopf und Härtel, Leipzig,1860.[21] M. Fedrizzi and S. Giove. Incomplete pairwise comparison and consistencyoptimization.
European Journal of Operational Research , 183(1):303–313,2007.[22] M. Fedrizzi and S. Giove. Optimal sequencing in incomplete pairwise com-parisons for large-dimensional problems.
International Journal of GeneralSystems , 42(4):366–375, February 2013.
EFERENCES
Mathematical and Computer Modelling , 17(4-5):19–26, February 1993.[24] J. Franek and A. Kresta. Judgment Scales and Consistency Measure inAHP.
Procedia Economics and Finance , 12:164–173, 2014.[25] B. L. Golden and Q. Wang.
An Alternate Measure of Consistency , pages68–81. Springer Berlin Heidelberg, Berlin, Heidelberg, 1989.[26] A. Z. Grzybowski. New results on inconsistency indices and their relation-ship with the quality of priority vector estimation.
Expert Systems withApplications , 43:197–212, January 2016.[27] P. T. Harker. Alternative modes of questioning in the analytic hierarchyprocess.
Mathematical Modelling , 9(3):353 – 360, 1987.[28] P. T. Harker. Incomplete pairwise comparisons in the analytic hierarchyprocess.
Mathematical Modelling , 9(11):837–848, 1987.[29] W. Ho and X. Ma. The state-of-the-art integrations and applications ofthe analytic hierarchy process.
European Journal of Operational Research ,267:399–414, 2018.[30] N. V. Hovanov, J. W. Kolari, and M. V. Sokolov. Deriving weights from gen-eral pairwise comparison matrices.
Mathematical Social Sciences , 55(2):205– 220, 2008.[31] J. Jablonsky. Analysis of selected prioritization methods in the analytichierarchy process.
Journal of Physics: Conference Series , 622(1):1–7, 2015.[32] W. W. Koczkodaj. A new definition of consistency of pairwise comparisons.
Math. Comput. Model. , 18(7):79–84, October 1993.[33] W. W. Koczkodaj and R. Szwarc. On axiomatization of inconsistencyindicators in pairwise comparisons.
Fundamenta Informaticae , 132(485-500), 2014.[34] W. W. Koczkodaj and R. Urban. Axiomatization of inconsistency indica-tors for pairwise comparisons.
International Journal of Approximate Rea-soning , 94:18–29, March 2018.[35] W.W. Koczkodaj and R. Szwarc. On axiomatization of inconsistency in-dicators for pairwise comparisons.
Fundamenta Informaticae , 4(132):485–500, 2014.[36] G. Kou and C. Lin. A cosine maximization method for the priority vectorderivation in AHP.
European Journal of Operational Research , 235(1):225–232, May 2014.
EFERENCES
Fundamenta Informaticae , 139(4):403 – 419,July 2015.[38] K. Kułakowski. Notes on the existence of a solution in the pairwise com-parisons method using the heuristic rating estimation approach.
Annals ofMathematics and Artificial Intelligence , 77(1):105–121, 2016.[39] K. Kułakowski. Inconsistency in the ordinal pairwise comparisons methodwith and without ties.
European Journal of Operational Research ,270(1):314 – 327, 2018.[40] K. Kułakowski. On the geometric mean method for incomplete pairwisecomparisons.
CoRR , abs/1905.04609, 2019.[41] K. Kułakowski and A. Kedzior. Some Remarks on the Mean-Based Priori-tization Methods in AHP. In Ngoc-Thanh Nguyen, Lazaros Iliadis, YannisManolopoulos, and Bogdan Trawiński, editors,
Lecture Notes In ComputerScience, Computational Collective Intelligence: 8th International Confer-ence, ICCCI 2016, Halkidiki, Greece, September 28-30, 2016. Proceedings,Part I , pages 434–443. Springer International Publishing, 2016.[42] K. Kułakowski and J. Szybowski. The new triad based inconsistency indicesfor pairwise comparisons.
Procedia Computer Science , 35(0):1132 – 1137,2014.[43] K. Kułakowski, J. Szybowski, and A. Prusak. Towards quantification ofincompleteness in the pairwise comparisons methods.
International Journalof Approximate Reasoning , 115:221–234, October 2019.[44] M. Lundy, S. Siraj, and S. Greco. The mathematical equivalence of the“spanning tree” and row geometric mean preference vectors and its impli-cations for preference analysis.
European Journal of Operational Research ,pages 1–12, September 2016.[45] R. Merris. Laplacian matrices of graphs: a survey.
Linear Algebra and itsApplications , 197-198:143–176, January 1994.[46] C. Meyer.
Matrix Analysis and Applied Linear Algebra . SIAM: Society forIndustrial and Applied Mathematics, April 2000.[47] G. Oliva, R. Setola, and A. Scala. Sparse and distributed Analytic Hierar-chy Process.
Automatica , 85:211–220, November 2017.[48] D. Pan, X. Liu, J. Liu, and Y. Deng. A Ranking Procedure by IncompletePairwise Comparisons Using Information Entropy and Dempster-ShaferEvidence Theory.
The Scientific World Journal , pages 1–11, August 2014.[49] A. Quarteroni, R. Sacco, and F. Saleri.
Numerical mathematics . SpringerVerlag, 2000.
EFERENCES
Journal of Mathematical Psychology , 15(3):234 – 281, 1977.[51] T. L. Saaty. The analytic hierarchy and analytic network processes for themeasurement of intangible criteria and for decision-making. In
MultipleCriteria Decision Analysis: State of the Art Surveys , volume 78 of
Inter-national Series in Operations Research and Management Science , pages345–405. Springer New York, 2005.[52] T. L. Saaty and G. Hu. Ranking by eigenvector versus other methods inthe analytic hierarchy process.
Applied Mathematics Letters , 11(4):121–125,1998.[53] A. A. Salo and R. P. Hämäläinen. Preference programming through ap-proximate ratio comparisons.
European Journal of Operational Research ,82(3):458 – 475, 1995.[54] Amartya K Sen. Social Choice Theory: A Re-examination.
Econometrica ,45(1):53–89, January 1977.[55] S. Siraj, L. Mikhailov, and J. A. Keane. Enumerating all spanning trees forpairwise comparisons.
Computers and Operations Research , 39(2):191–199,February 2012.[56] B. Srdjevic, Z. Srdjevic, and B. Blagojevic. First-Level Transitivity RuleMethod for Filling in Incomplete Pair-Wise Comparison Matrices in theAnalytic Hierarchy Process.
Applied Mathematics & Information Sciences ,8(2):459–467, 2014.[57] D. Talaga. Inconsistency of incomplete pairwise comparisons matrices.Master’s thesis, AGH University of Science and Technology, 2018.[58] L. L. Thurstone. The Method of Paired Comparisons for Social Values.
Journal of Abnormal and Social Psychology , pages 384–400, 1927.[59] W. C. Wedley. Consistency prediction for incomplete AHP matrices.
Math-ematical and Computer Modelling , 17(4-5):151–161, February 1993.[60] Y. Wind and T. L. Saaty. Marketing Applications of the Analytic HierarchyProcess.
Management Science , 26(7):641–658, 1980.[61] K. K. F. Yuen. Compound Linguistic Scale.
Applied Soft Computing ,21:38–56, August 2014.
Appendix E F E R E N C E S k D ( e I G , k ) D ( e I G , k ) D ( e K, k ) D ( e I , k ) D ( e I , k ) D ( e I α , k ) D ( e I α,β , k ) D ( g GW , k ) D ( e I SH , k ) D ( f RE , k ) D ( f RE , k ) D ( f CI , k ) D ( e L, k ) D ( O, k ) D ( e I G ) D ( e I G ) D ( e K ) D ( e I ) D ( e I ) D ( e I α ) D ( e I α,β ) D ( g GW ) D ( e I SH ) D ( f RE ) D ( f RE ) D ( f CI ))
Appendix E F E R E N C E S k D ( e I G , k ) D ( e I G , k ) D ( e K, k ) D ( e I , k ) D ( e I , k ) D ( e I α , k ) D ( e I α,β , k ) D ( g GW , k ) D ( e I SH , k ) D ( f RE , k ) D ( f RE , k ) D ( f CI , k ) D ( e L, k ) D ( O, k ) D ( e I G ) D ( e I G ) D ( e K ) D ( e I ) D ( e I ) D ( e I α ) D ( e I α,β ) D ( g GW ) D ( e I SH ) D ( f RE ) D ( f RE ) D ( f CI )) D ( e L ))
Appendix E F E R E N C E S k D ( e I G , k ) D ( e I G , k ) D ( e K, k ) D ( e I , k ) D ( e I , k ) D ( e I α , k ) D ( e I α,β , k ) D ( g GW , k ) D ( e I SH , k ) D ( f RE , k ) D ( f RE , k ) D ( f CI , k ) D ( e L, k ) D ( O, k ) D ( e I G ) D ( e I G ) D ( e K ) D ( e I ) D ( e I ) D ( e I α ) D ( e I α,β ) D ( g GW ) D ( e I SH ) D ( f RE ) D ( f RE ) D ( f CI )) D ( e L )) D ( O ))