Inconsistency thresholds for incomplete pairwise comparison matrices
EExtension of Saaty’s inconsistency index to incompletecomparisons: Approximated thresholds
Kolos Csaba ´Agoston a L´aszl´o Csat´o b “ Mathematics is the part of physics where experiments are cheap. ” (Vladimir Igorevich Arnold: On teaching mathematics ) Abstract
Pairwise comparison matrices are increasingly used in settings where some pairs aremissing. However, there exist few inconsistency indices to analyse such incompletedata sets and even fewer measures have an associated threshold. This paper general-ises the inconsistency index proposed by Saaty to incomplete pairwise comparisonmatrices. The extension is based on the approach of filling the missing elements tominimise the eigenvalue of the incomplete matrix. It means that the well-establishedvalues of the random index, a crucial component of the consistency ratio for whichthe famous threshold of 0.1 provides the condition for the acceptable level of incon-sistency, cannot be directly adopted. The inconsistency of random matrices turnsout to be the function of matrix size and the number of missing elements, witha nearly linear dependence in the case of the latter variable. Our results can bedirectly used by practitioners as a statistical criterion for accepting/rejecting anincomplete pairwise comparison matrix.
Keywords : Analytic Hierarchy Process (AHP); decision analysis; inconsistencythreshold; incomplete pairwise comparisons; multi-criteria decision-making
MSC class : 90B50, 91B08
JEL classification number : C44, D71 a Email: [email protected]
Corvinus University of Budapest (BCE), Department of Operations Research and Actuarial Sciences,Budapest, Hungary b Corresponding author. Email: [email protected]
Institute for Computer Science and Control (SZTAKI), E¨otv¨os Lor´and Research Network (ELKH),Laboratory on Engineering and Management Intelligence, Research Group of Operations Research andDecision Systems, Budapest, HungaryCorvinus University of Budapest (BCE), Department of Operations Research and Actuarial Sciences,Budapest, Hungary Source: Arnold (1998, p. 229). a r X i v : . [ m a t h . S T ] F e b Introduction
Pairwise comparisons form an essential part of many decision-making techniques, especiallysince the appearance of the popular
Analytic Hierarchy Process (AHP) methodology (Saaty,1977, 1980). Despite simplifying the issue to comparing objects pair by pair, the tool ofpairwise comparisons presents some challenges. First, the comparisons might not satisfyconsistency: if alternative 𝐴 is two times better than alternative 𝐵 and alternative 𝐵 is three times better than alternative 𝐶 , then alternative 𝐴 is not necessarily six timesbetter than alternative 𝐶 . The origin of similar inconsistencies resides in asking seemingly“redundant” questions. However, this additional information is often required to increaserobustness (Boz´oki et al., 2020) and inconsistency usually does not cause a serious problemuntil it remains at a moderate level.Inconsistent preferences come with two unavoidable consequences, namely, how toderive the non-trivial weight vector and how to quantify the level of inconsistency. The firstand by far the most extensively used inconsistency index has been proposed by the founderof the AHP, Thomas L. Saaty (Saaty, 1977). He has also provided a sharp threshold, awidely accepted rule to decide whether the matrix containing pairwise comparisons has anacceptable level of inconsistency or not.Furthermore, there are at least three arguments why incomplete pairwise comparisonsshould be considered in decision-making models (Harker, 1987):• in the case of a large number 𝑛 of alternatives, completing all 𝑛 ( 𝑛 − / The pairwise comparisons of the alternatives are collected into a matrix A = [ 𝑎 𝑖𝑗 ] suchthat the entry 𝑎 𝑖𝑗 is the numerical answer to the question “How many times alternative 𝑖 is better than alternative 𝑗 ?” Let R + denote the set of positive numbers, R 𝑛 + denote theset of positive vectors of size 𝑛 and R 𝑛 × 𝑛 + denote the set of positive square matrices of size 𝑛 with all elements greater than zero, respectively. Definition 2.1.
Pairwise comparison matrix : Matrix A = [ 𝑎 𝑖𝑗 ] ∈ R 𝑛 × 𝑛 + is a pairwisecomparison matrix if 𝑎 𝑗𝑖 = 1 /𝑎 𝑖𝑗 for all 1 ≤ 𝑖, 𝑗 ≤ 𝑛 .Let 𝒜 denote the set of pairwise comparison matrices and 𝒜 𝑛 × 𝑛 denote the set ofpairwise comparison matrices of size 𝑛 , respectively. Definition 2.2.
Consistency : A pairwise comparison matrix A = [ 𝑎 𝑖𝑗 ] ∈ 𝒜 𝑛 × 𝑛 is consist-ent if 𝑎 𝑖𝑘 = 𝑎 𝑖𝑗 𝑎 𝑗𝑘 for all 1 ≤ 𝑖, 𝑗, 𝑘 ≤ 𝑛 . Otherwise, it is said to be inconsistent . Definition 2.3.
Inconsistency index : The function 𝑓 : 𝒜 → R is an inconsistency index .According to the famous Perron–Frobenius theorem, there exists a unique positiveweight vector w for any pairwise comparison matrix A ∈ 𝒜 satisfying Aw = 𝜆 max ( A ) w ,where 𝜆 max ( A ) is the maximal or Perron eigenvalue of matrix A .Saaty has considered an affine transformation of this eigenvalue. Definition 2.4.
Consistency index : Let A ∈ 𝒜 𝑛 × 𝑛 be any pairwise comparison matrix ofsize 𝑛 . Its consistency index is 𝐶𝐼 ( A ) = 𝜆 max ( A ) − 𝑛𝑛 − . Since 𝐶𝐼 ( A ) = 0 ⇐⇒ 𝜆 max ( A ) = 𝑛 , the consistency index 𝐶𝐼 is a reasonable measureof how far a pairwise comparison matrix is from a consistent one (Saaty, 1977, 1980).Saaty has recommended using a discrete scale for the matrix elements, i.e., for all1 ≤ 𝑖, 𝑗 ≤ 𝑛 : 𝑎 𝑖𝑗 ∈ { / , / , / , . . . , / , , , . . . , , } . (1)Aupetit and Genest (1993) provide a tight upper bound for the value of 𝐶𝐼 when theentries of the pairwise comparison matrix are expressed on a bounded scale.A normalised measure of inconsistency can be obtained as suggested by Saaty.3able 1: The values of the random index for complete pairwise comparison matricesMatrix size 4 5 6 7 8 9 10Random index 𝑅𝐼 𝑛 Definition 2.5.
Random index : Consider the set 𝒜 𝑛 × 𝑛 of pairwise comparison matricesof size 𝑛 . The corresponding random index 𝑅𝐼 is given by the following algorithm (Alonsoand Lamata, 2006):• Generating a large number of pairwise comparison matrices such that each entryabove the diagonal is drawn independently and uniformly from the Saaty scale(1).• Calculating the consistency index 𝐶𝐼 for each random pairwise comparison matrix.• Computing the mean of these values.Several authors have published slightly different random indices depending on thesimulation method and the number of generated matrices involved, see Alonso and Lamata(2006, Table 1). The random indices 𝑅𝐼 𝑛 are reported in Table 1 for 4 ≤ 𝑛 ≤
10 asprovided by Boz´oki and Rapcs´ak (2008) and validated by Csat´o and Petr´oczy (2021).These estimates are close to the ones given in previous works (Alonso and Lamata, 2006;Ozdemir, 2005). Boz´oki and Rapcs´ak (2008, Table 3) also uncovers how 𝑅𝐼 𝑛 depends onthe largest element of the ratio scale. Definition 2.6.
Consistency ratio : Let A ∈ 𝒜 𝑛 × 𝑛 be any pairwise comparison matrix ofsize 𝑛 . Its consistency ratio is 𝐶𝑅 ( A ) = 𝐶𝐼 ( A ) /𝑅𝐼 𝑛 .Saaty has proposed a threshold for the acceptability of inconsistency, too. Definition 2.7.
Acceptable level of inconsistency : Let A ∈ 𝒜 𝑛 × 𝑛 be any pairwise com-parison matrix of size 𝑛 . It is sufficiently close to a consistent matrix and therefore can beaccepted if 𝐶𝑅 ( A ) ≤ . As it has already been mentioned, certain entries of a pairwise comparison matrix aresometimes missing.
Definition 3.1.
Incomplete pairwise comparison matrix : Matrix A = [ 𝑎 𝑖𝑗 ] is an incompletepairwise comparison matrix if 𝑎 𝑖𝑗 ∈ R + ∪ {*} such that for all 1 ≤ 𝑖, 𝑗 ≤ 𝑛 , 𝑎 𝑖𝑗 ∈ R + implies 𝑎 𝑗𝑖 = 1 /𝑎 𝑖𝑗 and 𝑎 𝑖𝑗 = * implies 𝑎 𝑗𝑖 = * .4et 𝒜 𝑛 × 𝑛 * denote the set of incomplete pairwise comparison matrices of size 𝑛 .The graph representation of incomplete pairwise comparison matrices is a convenienttool to visualise the structure of known elements. Definition 3.2.
Graph representation : An incomplete pairwise comparison matrix A ∈𝒜 𝑛 × 𝑛 * can be represented by the undirected graph 𝐺 = ( 𝑉, 𝐸 ), where the vertices 𝑉 = { , , . . . , 𝑛 } correspond to the alternatives and the edges in 𝐸 are associated with theknown matrix entries outside the diagonal, that is, 𝑒 𝑖𝑗 ∈ 𝐸 ⇐⇒ 𝑎 𝑖𝑗 ̸ = * and 𝑖 ̸ = 𝑗 .To summarise, there are no edges for the missing elements ( 𝑎 𝑖𝑗 = * ) as well as for theentries of the diagonal ( 𝑎 𝑖𝑖 ).In the case of an incomplete pairwise comparison matrix A , Shiraishi et al. (1998) andShiraishi and Obata (2002) consider an eigenvalue optimisation problem by substitutingthe 𝑚 missing elements of matrix A above the diagonal with positive values arranged inthe vector x ∈ R 𝑚 + , while the reciprocity condition is preserved:min x ∈ R 𝑚 + 𝜆 max ( A ( x )) . (2)The motivation is clear, all missing entries should be chosen to get a matrix which is asclose to a consistent one as possible in terms of Saaty’s inconsistency index.Boz´oki et al. (2010, Section 3) solve the optimisation problem (2) and give the necessaryand sufficient condition for uniqueness: the graph 𝐺 representing the incomplete pairwisecomparison matrix A should be connected. This is an intuitive and almost obviousrequirement since the relation of two alternatives cannot be established if they are notcompared at least indirectly, through other alternatives. Consider an incomplete pairwise comparison matrix A ∈ 𝒜 𝑛 × 𝑛 * and a complete pairwisecomparison matrix B ∈ 𝒜 𝑛 × 𝑛 , where 𝑏 𝑖𝑗 = 𝑎 𝑖𝑗 if 𝑎 𝑖𝑗 ̸ = * . Let A ( x ) ∈ 𝒜 𝑛 × 𝑛 be the optimalcompletion of A according to (2). Clearly, 𝜆 max ( A ( x )) ≤ 𝜆 max ( B ), hence 𝐶𝐼 ( A ( x )) ≤ 𝐶𝐼 ( B ). It means that the value of the random index 𝑅𝐼 𝑛 , calculated for complete pairwisecomparison matrices, cannot be applied in the case of an incomplete pairwise comparisonmatrix because its consistency index 𝐶𝐼 is obtained through optimising (i.e. minimising)its level of inconsistency.Consequently, by adopting the numbers from Table 1, the ratio of incomplete pairwisecomparison matrices with an acceptable level of inconsistency will exceed the concept ofSaaty, and this discrepancy increases as the number of missing elements grows. In theextreme case when graph 𝐺 is a spanning tree of a complete graph with 𝑛 nodes (thus itis a connected graph consisting of exactly 𝑛 − 𝑚 , thenumber of missing elements. Remark . In the view of the Saaty scale (1), there are at least three different ways tochoose the missing entries 𝑥 𝑘 , 1 ≤ 𝑘 ≤ 𝑚 : 5 234Figure 1: The graph representation of the pairwise comparison matrix A in Example 4.11. Method 1 : 𝑥 𝑘 ∈ R + , namely, each missing entry can be an arbitrary positivenumber;2. Method 2 : 1 / ≤ 𝑥 𝑘 ≤
9, namely, the missing entries cannot be higher (lower)than the theoretical maximum (minimum) of the known elements;3.
Method 3 : 𝑥 𝑘 ∈ { / , / , / , . . . , / , , , . . . , , } , namely, each missingentry is drawn from the discrete Saaty scale.Let us illustrate the three approaches listed in Remark 1. Example 4.1.
Take the following incomplete pairwise comparison matrix: A = ⎡⎢⎢⎢⎣ * ** / / * / / ⎤⎥⎥⎥⎦ . The corresponding undirected graph 𝐺 is depicted in Figure 1. Note that 𝐺 would bea spanning tree without the edge between nodes 2 and 4 and 𝑎 = 8 = 2 × 𝑎 𝑎 .Consequently, A can be filled out consistently in a unique way: A = ⎡⎢⎢⎢⎣ / / / / / / / ⎤⎥⎥⎥⎦ . The first technique (Method 1 in Remark 1) results in A with 𝜆 max ( A ) = 4.On the other hand, A is not valid under Method 2 in Remark 1 because 𝑎 = 36 > A is given by the solution of the convexeigenvalue minimisation problem (2) with the additional constraints 1 / ≤ 𝑥 𝑘 ≤ ≤ 𝑘 ≤ 𝑚 and is as follows: A = ⎡⎢⎢⎢⎣ / / / / / / / ⎤⎥⎥⎥⎦ , where 𝜆 max ( A ) = 4 . A is not valid under Method 3 in Remark 1 because 𝑎 = 9 / / ∈ Z , that is,even though the optimal filling by Method 2 does not contain any value exceeding the6ounds of the Saaty scale (1), some of them are not integers or the reciprocals of integers.Hence, the best possible filling on the Saaty scale (1) is A = ⎡⎢⎢⎢⎣ / / / / / / ⎤⎥⎥⎥⎦ , which leads to 𝜆 max ( A ) = 4 . 𝑅𝐼 𝑛 . The first reason is thatthe algorithm for the 𝜆 max -optimal completion (Boz´oki et al., 2010, Section 5) involves anexogenously given tolerance level to determine how accurate are the coordinates of theeigenvector associated with the dominant eigenvalue as a stopping criterion. Consequently,it cannot be chosen appropriately if the matrix entries and the elements of the weight vectorcan differ substantially: the consistent completion of an incomplete pairwise comparisonmatrix with 𝑛 alternatives may contain (1 / ( 𝑛 − or 9 ( 𝑛 − if the corresponding graph isa chain. Furthermore, it remains questionable why elements below or above the Saatyscale (1) are allowed for the missing entries if they are prohibited in the case of knownelements. On the other hand, Method 3 presents a discrete optimisation problem that ismore difficult to handle than its continuous analogue of Method 2. To summarise, since theprocess is based on generating a large number of random incomplete pairwise comparisonmatrices to be filled out optimally, it is necessary to reduce the complexity of optimisationproblem (2) by using Method 2.A complete pairwise comparison matrix of size 𝑛 can be represented by a complete graphwhere the degree of each node is 𝑛 −
1. Hence, the graph corresponding to an incompletepairwise comparison matrix is certainly connected if 𝑚 ≤ 𝑛 −
2, implying that the solutionof the 𝜆 max -optimal completion is unique. However, the graph might be disconnected if 𝑚 ≥ 𝑛 −
1, in which case it makes no sense to calculate the consistency index 𝐶𝐼 of theincomplete pairwise comparison matrix. Furthermore, if 𝑚 > 𝑛 ( 𝑛 − / − ( 𝑛 − 𝑛 − 𝑚 = 𝑛 ( 𝑛 − / − ( 𝑛 −
1) = ( 𝑛 − 𝑛 − / As we have argued in Section 4, the value of the random index 𝑅𝐼 depends not only onthe size 𝑛 of the incomplete pairwise comparison matrix but on the number of its missingelements 𝑚 , too. Thus the random index is computed according to the following procedure(cf. Definition 2.5):1. Generating an incomplete pairwise comparison matrix A of size 𝑛 with 𝑚 missingentries above the diagonal such that each element above the diagonal is drawn7able 2: The values of the random indexfor incomplete pairwise comparison matricesMissing elements 𝑚 Matrix size 𝑛 𝐺 representing the incomplete pairwise comparisonmatrix A is connected or disconnected.3. If graph 𝐺 is connected, optimisation problem (2) is solved by the algorithm forthe 𝜆 max -optimal completion (Boz´oki et al., 2010, Section 5) with restricting allentries in x ∈ R 𝑚 + according to Method 2 in Remark 1 to obtain the minimumvalue of 𝜆 max ( A ( x )).4. Computing and saving the consistency index 𝐶𝐼 ( A ( x )) based on Definition 2.4.5. Repeating Steps 1–4 to get 1 million random matrices with a connected graphrepresentation, and calculating the mean of the consistency indices 𝐶𝐼 fromStep 4.Our central result is reported in Table 2, which is an extension of Table 1 to the casewhen some pairwise comparisons are unknown. The values in the first row, which coincidewith the ones from Table 1, confirm the integrity of the proposed technique to computethe thresholds for Saaty’s inconsistency index. The role of missing elements cannot beignored at all commonly used significance levels as reinforced by t-test.Recall that the maximal number of missing elements is at most 𝑛 ( 𝑛 − / − ( 𝑛 −
1) =( 𝑛 − 𝑛 − / 𝑛 = 4, 6 if 𝑛 = 5,and 10 if 𝑛 = 6. Some thresholds are lacking from Table 2—for example, 𝑛 = 7 and 𝑚 = 4—due to excessive computation time.However, 𝑅𝐼 𝑛,𝑚 can be easily predicted as follows. Figure 2 reveals that the randomindex is monotonically decreasing as the function of missing values 𝑚 according to commonintuition. Furthermore, the dependence is nearly linear, thus a plausible estimation isprovided by the below formula, which requires only the “omnipresent” Table 1: 𝑅𝐼 𝑛,𝑚 ≈ [︃ − 𝑚 ( 𝑛 − 𝑛 − ]︃ 𝑅𝐼 𝑛 . (3)8 . Number of missing elements 𝑚 Matrix size 5 . Number of missing elements 𝑚 Matrix size 6
Figure 2: The values of the random index in the presence of missing entriesTable 3: Approximation of the random index for incompletepairwise comparison matrices according to equation (3)Matrix size 𝑛 Missing elements 𝑚 Value of 𝑅𝐼 𝑛,𝑚 Computed Approximated by formula (3)7 4 0.998 0.9838 5 1.088 1.0709 6 1.158 1.14010 7 1.215 1.197According to the “case studies” in Table 3, (3) gives at least a reasonable guess of 𝑅𝐼 𝑛,𝑚 without much effort, even though it somewhat underestimates the true value. Thediscrepancy is mainly caused by 𝑅𝐼 𝑛, ( 𝑛 − 𝑛 − / being larger than zero (see Table 2) asincomplete pairwise comparison matrices represented by a spanning tree can be madeconsistent only if the missing elements can be arbitrary, but not if they are bounded tothe interval [1 / , 𝐶𝑅 forany incomplete pairwise comparison matrix. Definition 5.1.
Consistency ratio : Let A ∈ 𝒜 𝑛 × 𝑛 * be any incomplete pairwise comparisonmatrix of size 𝑛 with 𝑚 missing entries above the diagonal. Its consistency ratio is 𝐶𝑅 ( A ) = 𝐶𝐼 ( A ) /𝑅𝐼 𝑛,𝑚 .The popular 10% threshold of Definition 2.7 can be adopted without changes.Finally, a numerical illustration highlights the implications of the calculated thresholdsfor the random index. Example 5.1.
Take the following parametric incomplete pairwise comparison matrix ofsize 𝑛 = 4 with 𝑚 = 2 missing elements: A ( 𝛼, 𝛽 ) = ⎡⎢⎢⎢⎣ 𝛼 * 𝛽 /𝛼 𝛼 ** /𝛼 𝛼 /𝛽 * /𝛼 ⎤⎥⎥⎥⎦ . A ( 𝛼, 𝛽 ) in Example 5.1Value of 𝛽 Value of 𝛼 Bold numbers indicate that the consistency ratio 𝐶𝑅 = 𝐶𝐼/𝑅𝐼 , is below the 10% threshold. Italic numbers indicate that
𝐶𝐼/𝑅𝐼 is below the 10% threshold but the consistency ratio 𝐶𝑅 = 𝐶𝐼/𝑅𝐼 , is above it.Now 𝑅𝐼 = 𝑅𝐼 , ≈ .
884 and 𝑅𝐼 , ≈ .
356 from Table 2. There are three instanceswhere the optimal filling of matrix A ( 𝛼, 𝛽 ) results in a consistent pairwise comparisonmatrix: ( 𝛼, 𝛽 ) ∈ {︂(︂ , )︂ ; (1 ,
1) ; (2 , }︂ . They should be accepted under any circumstances.Examine what happens if 𝛼 = 1 is fixed. Then 𝛽 = 3 implies 𝐶𝐼 (1 , ≈ . < . × 𝑅𝐼 , , which still corresponds to an acceptable level of inconsistency. However, 𝐶𝐼 (1 , ≈ . > . × 𝑅𝐼 , , making it necessary to reduce inconsistency if 𝛽 = 4. Onthe other hand, 𝐶𝐼 (1 , ≈ . < . × 𝑅𝐼 , thus the optimally filled out incompletepairwise comparison matrix might be accepted according to the “standard” threshold forcomplete matrices because the latter does not take into account the automatic reductionof inconsistency due to the optimisation procedure.Table 4 reports the consistency index 𝐶𝐼 of matrix A ( 𝛼, 𝛽 ) for some parameters 𝛼 and 𝛽 . Bold numbers correspond to the cases when inconsistency can be tolerated based onthe approximated thresholds of Table 2, while italic numbers show instances that can beaccepted only if the optimal solution A ( x ) of (2) is wrongly considered as a (complete)pairwise comparison matrix.Example 5.1 reinforces that the use of the extended values of the random indexin Table 2 becomes indispensable in order to generalise Saaty’s inconsistency index toincomplete comparisons. 10 Conclusions
The paper reports approximated thresholds for the most popular inconsistency index,proposed by Saaty, in the case of incomplete pairwise comparison matrices. They aredetermined by the value of the random index, that is, the average consistency ratio ofa large number of random pairwise comparison matrices with missing elements. Thecalculation is far from trivial since a separate convex optimisation problem should besolved for each matrix to find the optimal filling of unknown entries. Numerical resultsuncover that the threshold depends not only on the size of the pairwise comparison matrixbut the number of missing entries, too. However, there exists a plausible linear estimationof the random index.With this rule of acceptability, the decision-maker can decide for any incompletepairwise comparison matrix whether there is a need to revise earlier assessments or not. Itallows the level of inconsistency to be monitored even before all comparisons are given,which may immediately indicate possible mistakes and suspicious entries. Therefore, thepreference revision process can be launched as early as possible. It will be examined infuture studies how this opportunity can be built into the known inconsistency reductionmethods (Abel et al., 2018; Boz´oki et al., 2015; Ergu et al., 2011; Xu and Xu, 2020).
Acknowledgements
We are grateful to
S´andor Boz´oki and
Zsombor Sz´adoczki for useful advice.The research was supported by the MTA Premium Postdoctoral Research Program grantPPD2019-9/2019.
References
Abel, E., Mikhailov, L., and Keane, J. (2018). Inconsistency reduction in decision makingvia multi-objective optimisation.
European Journal of Operational Research , 267(1):212–226.Aguar´on, J. and Moreno-Jim´enez, J. M. (2003). The geometric consistency index: Ap-proximated thresholds.
European Journal of Operational Research , 147(1):137–145.Alonso, J. A. and Lamata, M. T. (2006). Consistency in the analytic hierarchy process: anew approach.
International Journal of Uncertainty, Fuzziness and Knowledge-BasedSystems , 14(4):445–459.Amenta, P., Lucadamo, A., and Marcarelli, G. (2020). On the transitivity and consistencyapproximated thresholds of some consistency indices for pairwise comparison matrices.
Information Sciences , 507:274–287.Arnold, V. I. (1998). On teaching mathematics.
Russian Mathematical Surveys , 53(1):229–236.Aupetit, B. and Genest, C. (1993). On some useful properties of the Perron eigenvalue ofa positive reciprocal matrix in the context of the analytic hierarchy process.
EuropeanJournal of Operational Research , 70(2):263–268.11oz´oki, S., Csat´o, L., and Temesi, J. (2016). An application of incomplete pairwisecomparison matrices for ranking top tennis players.
European Journal of OperationalResearch , 248(1):211–218.Boz´oki, S., F¨ul¨op, J., and Poesz, A. (2015). On reducing inconsistency of pairwisecomparison matrices below an acceptance threshold.
Central European Journal ofOperations Research , 23(4):849–866.Boz´oki, S., F¨ul¨op, J., and R´onyai, L. (2010). On optimal completion of incomplete pairwisecomparison matrices.
Mathematical and Computer Modelling , 52(1-2):318–333.Boz´oki, S. and Rapcs´ak, T. (2008). On Saaty’s and Koczkodaj’s inconsistencies of pairwisecomparison matrices.
Journal of Global Optimization , 42(2):157–175.Boz´oki, S., Sz´adoczki, Zs., and Tekile, H. A. (2020). Filling in pattern designs forincomplete pairwise comparison matrices: (quasi-)regular graphs with minimal diameter.Manuscript. arXiv: 2006.01127.Brunelli, M. (2018). A survey of inconsistency indices for pairwise comparisons.
Interna-tional Journal of General Systems , 47(8):751–771.Chao, X., Kou, G., Li, T., and Peng, Y. (2018). Jie Ke versus AlphaGo: A rankingapproach using decision making method for large-scale data with incomplete information.
European Journal of Operational Research , 265(1):239–247.Csat´o, L. (2013). Ranking by pairwise comparisons for Swiss-system tournaments.
CentralEuropean Journal of Operations Research , 21(4):783–803.Csat´o, L. (2017). On the ranking of a Swiss system chess team tournament.
Annals ofOperations Research , 254(1-2):17–36.Csat´o, L. and Petr´oczy, D. G. (2021). On the monotonicity of the eigenvector method.
European Journal of Operational Research , 292(1):230–237.Csat´o, L. and T´oth, Cs. (2020). University rankings from the revealed preferences of theapplicants.
European Journal of Operational Research , 286(1):309–320.Ergu, D., Kou, G., Peng, Y., and Shi, Y. (2011). A simple method to improve theconsistency ratio of the pair-wise comparison matrix in ANP.
European Journal ofOperational Research , 213(1):246–259.Harker, P. T. (1987). Alternative modes of questioning in the Analytic Hierarchy Process.
Mathematical Modelling , 9(3-5):353–360.Ku lakowski, K. and Talaga, D. (2020). Inconsistency indices for incomplete pairwisecomparisons matrices.
International Journal of General Systems , 49(2):174–200.Liang, F., Brunelli, M., and Rezaei, J. (2019). Consistency issues in the best worst method:Measurements and thresholds.
Omega , 96:102175.Lin, C., Kou, G., and Ergu, D. (2013). An improved statistical approach for consistencytest in AHP.
Annals of Operations Research , 211(1):289–299.12in, C., Kou, G., and Ergu, D. (2014). A statistical approach to measure the consistencylevel of the pairwise comparison matrix.
Journal of the Operational Research Society ,65(9):1380–1386.Ozdemir, M. S. (2005). Validity and inconsistency in the analytic hierarchy process.
Applied Mathematics and Computation , 161(3):707–720.Petr´oczy, D. G. (2021). An alternative quality of life ranking on the basis of remittances.Manuscript. arXiv: 1809.03977.Petr´oczy, D. G. and Csat´o, L. (2020). Revenue allocation in Formula One: A pair-wise comparison approach.
International Journal of General Systems , in press. DOI:10.1080/03081079.2020.1870224.Saaty, T. L. (1977). A scaling method for priorities in hierarchical structures.
Journal ofMathematical Psychology , 15(3):234–281.Saaty, T. L. (1980).
The Analytic Hierarchy Process: Planning, Priority Setting, ResourceAllocation . McGraw-Hill, New York.Shiraishi, S. and Obata, T. (2002). On a maximization problem arising from a positivereciprocal matrix in AHP.
Bulletin of Informatics and Cybernetics , 34(2):91–96.Shiraishi, S., Obata, T., and Daigo, M. (1998). Properties of a positive reciprocal matrixand their application to AHP.
Journal of the Operations Research Society of Japan-KeieiKagaku , 41(3):404–414.Szybowski, J., Ku lakowski, K., and Prusak, A. (2020). New inconsistency indicators forincomplete pairwise comparisons matrices.
Mathematical Social Sciences , 108:138–145.Xu, K. and Xu, J. (2020). A direct consistency test and improvement method for theanalytic hierarchy process.