Inconsistency thresholds for incomplete pairwise comparison matrices

Pairwise comparison matrices are increasingly used in settings where some pairs are missing. However, there exist few inconsistency indices for similar incomplete data sets and no reasonable measure has an associated threshold. This paper generalises the famous rule of thumb for the acceptable level of inconsistency, proposed by Saaty, to incomplete pairwise comparison matrices. The extension is based on choosing the missing elements such that the maximal eigenvalue of the incomplete matrix is minimised. Consequently, the well-established values of the random index cannot be adopted: the inconsistency of random matrices is found to be the function of matrix size and the number of missing elements, with a nearly linear dependence in the case of the latter variable. Our results can be directly built into decision-making software and used by practitioners as a statistical criterion for accepting or rejecting an incomplete pairwise comparison matrix.


Introduction
Pairwise comparisons form an essential part of many decision-making techniques, especially since the appearance of the popular Analytic Hierarchy Process (AHP) methodology (Saaty, 1977(Saaty, , 1980. Despite simplifying the issue to evaluating objects pair by pair, the tool of pairwise comparisons presents some challenges due to the possible lack of consistency: if alternative is two times better than alternative and alternative is three times better than alternative , then alternative is not necessarily six times better than alternative . The origin of similar inconsistencies resides in asking seemingly "redundant" questions. Nonetheless, additional information is often required to increase robustness (Szádoczki et al., 2022), and inconsistency usually does not cause a serious problem until it remains at a moderate level.
Inconsistent preferences call for quantifying the level of inconsistency. The first and by far the most extensively used index has been proposed by the founder of the AHP, Thomas L. Saaty (Saaty, 1977). He has also provided a sharp threshold to decide whether a pairwise comparison matrix has an acceptable level of inconsistency or not.
This widely accepted rule of inconsistency has been constructed for the case when all comparisons are known. However, there are at least three arguments why incomplete pairwise comparisons should be considered in decision-making models (Harker, 1987): • in the case of a large number of alternatives, completing all ( − 1)/2 pairwise comparisons is resource-intensive and might require much effort from experts suffering from a lack of time; • unwillingness to make a direct comparison between two alternatives for ethical, moral, or psychological reasons; • the decision-makers may be unsure of some of the comparisons, for instance, due to limited knowledge on the particular issue.
In certain settings, both incompleteness and inconsistency are inherent features of the data. The beating relation in sports is rarely transitive and some players/teams have never played against each other (Bozóki et al., 2016;Csató, 2013Csató, , 2017Petróczy and Csató, 2021;Chao et al., 2018). Analogously, there exists no guarantee for consistency when the pairwise comparisons are given by the bilateral remittances between countries (Petróczy, 2021), or by the preferences of students between universities (Csató and Tóth, 2020). Finally, note that pairwise comparison matrices are usually filled sequentially by the decision-makers, see e.g. the empirical research conducted by Bozóki et al. (2013). If the degree of inconsistency is monitored continuously during this process, the decision-maker might be warned immediately after the appearance of an unexpected value (Bozóki et al., 2011). Consequently, there is a higher chance that the problem can be solved easily compared to the usual case when the supervision of the comparisons is asked only after all pairwise comparisons are given. This is especially important as these values are often provided by experts who suffer from a lack of time.
Let us see an example, where the missing elements are denoted by *: Pairwise comparison matrix A is inconsistent because 12 × 23 × 34 = 2 × 2 × 2 = 8 ̸ = 4 = 14 . But it remains unknown whether this deviation can be tolerated or not. The current paper aims to provide thresholds of acceptability for pairwise comparison matrices with missing entries. We want to follow the concept of Saaty as closely as possible. Therefore, the unknown elements are considered as variables to be chosen to reduce the inconsistency of the parametric complete pairwise comparison matrix, that is, to minimise its maximal eigenvalue as suggested by Shiraishi et al. (1998) and Shiraishi and Obata (2002). The main challenge resides in the calculation of the random index, a key component of Saaty's threshold: the optimal completion of each randomly generated incomplete pairwise comparison matrix should be found separately in order to obtain the minimal value of the Perron root of the completed matrix (Bozóki et al., 2010).
On the other hand, the study of inconsistency indices for incomplete pairwise comparisons has been started only recently. Szybowski et al. (2020) introduce two new inconsistency measures based on spanning trees. Ku lakowski and Talaga (2020) adapt several existing indices to analyse incomplete data sets but do not provide any threshold. To conclude, without the present contribution, one cannot decide whether the inconsistency of the above incomplete pairwise comparison matrix A is excessive or not. Thus our work fills a substantial research gap.
Even though Forman (1990) computes random indices for incomplete pairwise comparison matrices, his solution is based on the proposal of Harker (1987). That introduces an auxiliary matrix for any incomplete pairwise comparison matrix instead of filling it by optimising an objective function as we do. Our approach is probably closer to Saaty's concept since the auxiliary matrix of Harker (1987) is not a pairwise comparison matrix.
The paper is structured as follows. Section 2 presents the fundamentals of pairwise comparison matrices and inconsistency measures. Incomplete pairwise comparison matrices and the eigenvalue minimisation problem are introduced in Section 3. Section 4 discusses the details of computing the random index. The inconsistency thresholds are reported in Section 5. A numerical example is provided in Section 6, and a real life application in Section 7. Finally, Section 8 offers a summary and directions for future research.

Pairwise comparison matrices and inconsistency
The pairwise comparisons of the alternatives are collected into a matrix A = [ ] such that the entry is the numerical answer to the question "How many times alternative is better than alternative ?" Let R + denote the set of positive numbers, R + denote the set of positive vectors of size and R × + denote the set of positive square matrices of size with all elements greater than zero, respectively. Let denote the set of pairwise comparison matrices and × denote the set of pairwise comparison matrices of size , respectively. According to the famous Perron-Frobenius theorem, for any pairwise comparison matrix A ∈ , there exists a unique positive weight vector w satisfying Aw = max (A)w and ∑︀ =1 = 1, where max (A) is the maximal or Perron eigenvalue of matrix A. Saaty has considered an affine transformation of this eigenvalue. Definition 2.3. Consistency index: Let A ∈ × be any pairwise comparison matrix of size . Its consistency index is Since (A) = 0 ⇐⇒ max (A) = , the consistency index is a reasonable measure of how far a pairwise comparison matrix is from a consistent one (Saaty, 1977(Saaty, , 1980. Aupetit and Genest (1993) provide a tight upper bound for the value of when the entries of the pairwise comparison matrix are expressed on a bounded scale.

Definition 2.4. Random index:
Consider the set × of pairwise comparison matrices of size . The corresponding random index is provided by the following algorithm (Alonso and Lamata, 2006): • Generating a large number of pairwise comparison matrices such that each entry above the diagonal is drawn independently and uniformly from the Saaty scale (1).
• Calculating the consistency index for each random pairwise comparison matrix.
• Computing the mean of these values.
Several authors have published slightly different random indices depending on the simulation method and the number of generated matrices involved, see Alonso and Lamata (2006, Table 1). The random indices are reported in Table 1 for 4 ≤ ≤ 10 as provided by Bozóki and Rapcsák (2008) and validated by Csató and Petróczy (2021). These estimates are close to the ones given in previous works (Alonso and Lamata, 2006;Ozdemir, 2005). Bozóki and Rapcsák (2008 , Table 3) uncovers how depends on the largest element of the ratio scale.
Definition 2.5. Consistency ratio: Let A ∈ × be any pairwise comparison matrix of size . Its consistency ratio is Saaty has proposed a threshold for the acceptability of inconsistency, too.
Definition 2.6. Acceptable level of inconsistency: Let A ∈ × be any pairwise comparison matrix of size . It is sufficiently close to a consistent matrix and therefore can be accepted if (A) ≤ 0.1.
Even though applying a crisp decision rule on the fuzzy concept of "large inconsistency" is strange (Brunelli, 2018) and there exist sophisticated statistical studies to test consistency (Lin et al., 2013(Lin et al., , 2014, it is assumed throughout the paper that the 10% rule is a wellestablished standard worth generalising to incomplete pairwise comparison matrices.

The eigenvalue minimisation problem for incomplete pairwise comparison matrices
Certain entries of a pairwise comparison matrix are sometimes missing.

Definition 3.1. Incomplete pairwise comparison matrix: Matrix
Let × * denote the set of incomplete pairwise comparison matrices of size . The graph representation of incomplete pairwise comparison matrices is a convenient tool to visualise the structure of known elements. To summarise, there are no edges for the missing elements ( = *) as well as for the entries of the diagonal ( ).
In the case of an incomplete pairwise comparison matrix A, Shiraishi et al. (1998) and Shiraishi and Obata (2002) consider an eigenvalue optimisation problem by substituting the missing elements of matrix A above the diagonal with positive values arranged in the vector x ∈ R + , while the reciprocity condition is preserved: The motivation is clear, all missing entries should be chosen to get a matrix that is as close to a consistent one as possible in terms of the consistency index . According to Bozóki et al. (2010, Section 3), (2) can be transformed into a convex optimisation problem. The authors also give the necessary and sufficient condition for the uniqueness of the solution: the graph representing the incomplete pairwise comparison matrix A should be connected. This is an intuitive and almost obvious requirement since the relation of two alternatives cannot be established if they are not compared at least indirectly, through other alternatives.

The calculation of the random index for incomplete pairwise comparison matrices
Consider an incomplete pairwise comparison matrix A ∈ × * and a complete pairwise . This implies that the value of the random index , calculated for complete pairwise comparison matrices, cannot be applied in the case of an incomplete pairwise comparison matrix because its consistency index is obtained through optimising (i.e. minimising) its level of inconsistency.
Consequently, by adopting the numbers from Table 1, the ratio of incomplete pairwise comparison matrices with an acceptable level of inconsistency will exceed the concept of Saaty and this discrepancy increases as the number of missing elements grows. In the extreme case when graph is a spanning tree of a complete graph with nodes (thus it is a connected graph consisting of exactly − 1 edges without cycles), the corresponding incomplete matrix can be filled out such that consistency is achieved.
Therefore, the random index needs to be recomputed for incomplete pairwise comparison matrices, and its value will supposedly be a monotonically decreasing function of , the number of missing elements.
Let us illustrate the three approaches listed in Remark 1.
Among the three ideas in Remark 1, Method 1 always leads to the smallest dominant eigenvalue, followed by Method 2, whereas Method 3 provides the greatest optimum of problem (2) as can be seen from the restrictions in Remark 1.
We implement Method 2 to calculate the random indices . The first reason is that the algorithm for the max -optimal completion (Bozóki et al., 2010, Section 5) involves an exogenously given tolerance level to determine how accurate are the coordinates of the eigenvector associated with the dominant eigenvalue as a stopping criterion. Consequently, it cannot be chosen appropriately if the matrix entries and the elements of the weight vector can differ substantially: the consistent completion of an incomplete pairwise comparison matrix with alternatives may contain (1/9) ( −1) or 9 ( −1) as an element if the corresponding graph is a chain. Furthermore, it remains questionable why elements below or above the Saaty scale (1) are allowed for the missing entries if they are prohibited in the case of known elements. On the other hand, Method 3 presents a discrete optimisation problem that is more difficult to handle than its continuous analogue of Method 2. To summarise, since the process is based on generating a large number of random incomplete pairwise comparison matrices to be filled out optimally, it is necessary to reduce the complexity of optimisation problem (2) by using Method 2.
A complete pairwise comparison matrix of size can be represented by a complete graph where the degree of each node is − 1. Hence, the graph corresponding to an incomplete pairwise comparison matrix is certainly connected if ≤ − 2, implying that the solution of the max -optimal completion is unique. However, the graph might be disconnected if ≥ − 1, in which case it makes no sense to calculate the consistency index of the incomplete pairwise comparison matrix. Furthermore, if > ( − 1)/2 − ( − 1), then there are less than − 1 known elements, and the graph is always disconnected.
If the number of missing entries is exactly = ( − 1)/2 − ( − 1) = ( − 1)( − 2)/2, then the graph is connected if and only if it is a spanning tree. Even though these incomplete pairwise comparison matrices certainly have a consistent completion under Method 1, this does not necessarily hold under Method 2 when the missing entries cannot be arbitrarily large/small.

Generalised thresholds for the consistency ratio
As we have argued in Section 4, the value of the random index , probably depends not only on the size of the incomplete pairwise comparison matrix but on the number of its missing elements , too. Thus the random index is computed according to the following procedure (cf. Definition 2.4): 1. Generating an incomplete pairwise comparison matrix A of size with missing entries above the diagonal such that each element above the diagonal is drawn independently and uniformly from the Saaty scale (1), while the place of the unknown elements above the diagonal is chosen randomly.
2. Checking whether the graph representing the incomplete pairwise comparison matrix A is connected or disconnected.
3. If graph is connected, optimisation problem (2) is solved by the algorithm for the max -optimal completion (Bozóki et al., 2010, Section 5) with restricting all entries in x ∈ R + according to Method 2 in Remark 1 to obtain the minimum of max (A(x)) and the corresponding complete pairwise comparison matrixÂ.

Computing and saving the consistency index
(︁Â)︁ based on Definition 2.3.

Repeating
Steps 1-4 to get 1 million random matrices with a connected graph representation, and calculating the mean of the consistency indices from Step 4. Table 2, which is an extension of Table 1 to the case when some pairwise comparisons are unknown. The values in the first row, which coincide with the ones from Table 1, confirm the integrity of the proposed technique to compute the thresholds for the consistency index . The role of missing elements cannot be ignored at all commonly used significance levels as reinforced by the t-test: for any given , the values of , are statistically different from each other. Recall that the maximal number of missing elements is at most ( − 1)/2 − ( − 1) = ( − 1)( − 2)/2 if connectedness is not violated, and this value is 3 if = 4, 6 if = 5,   Table 2-for example, the pair = 7 and = 4-due to excessive computation time (> 48 hours). However, , can be easily predicted as follows. Figure 2 reveals that the random index is monotonically decreasing as the function of missing values according to common intuition. Furthermore, the dependence is nearly linear, thus a plausible estimation is provided by the below formula, which requires only the "omnipresent" Table 1:

Our central result is reported in
Obviously, (3) returns ,0 if there are no missing elements ( = 0). On the other hand, = ( − 1)( − 2)/2 means that the graph representing the incomplete pairwise comparison matrix is either unconnected, or it is a spanning tree, thus the matrix can be filled consistently if there is no restriction on its elements. Formula (3) immediately follows by assuming a linear function for intermediate values of . According to the "case studies" in Table 3, (3) gives at least a reasonable guess of , without much effort, even though it somewhat underestimates the true value. The discrepancy is mainly caused by ,( −1)( −2)/2 being larger than zero (see Table 2) as incomplete pairwise comparison matrices represented by a spanning tree can be made consistent only if the missing elements can be arbitrary, but not if they are bounded to the interval [1/9, 9].
Definition 2.5 can be modified straightforwardly to derive the consistency ratio for any incomplete pairwise comparison matrix.
Definition 5.1. Consistency ratio: Let A ∈ × * be any incomplete pairwise comparison matrix of size with missing entries above the diagonal andÂ be the complete pairwise comparison matrix given by the optimal filling of A. The consistency ratio of the incomplete matrix A is (A) = (Â)/ , .
The popular 10% threshold of Definition 2.6 can be adopted without any changes.
In the applications of the AHP methodology, the optimal number of alternatives does not exceed nine (Saaty and Ozdemir, 2003). Random indices for complete pairwise comparison matrices have been determined for ≤ 16 in Aguarón and Moreno-Jiménez (2003) and for ≤ 15 in Alonso and Lamata (2006). The corresponding thresholds for incomplete pairwise comparison matrices can be calculated offline by a supercomputer and built into any software used by practitioners. If these are not available, formula (3) provides a good approximation for any number of alternatives and missing elements , see Table 3.

An illustrative example
In this section, we highlight the implications of the calculated thresholds for the random index by a numerical illustration. It has been chosen to be simple but expressive. With three alternatives and one missing entry, the matrix can be filled out consistently. Therefore, the number of alternatives is four. Again, there exists a consistent filling if there are three missing elements, hence their number is two. Furthermore, they are in different rows, which is the more likely case.  Bold numbers indicate that the consistency ratio Italic numbers indicate that (︁Â ( , ) )︁ / 4,0 is below the 10% threshold but the consistency ratio (︁Â ( , ) )︁ = (︁Â ( , ) )︁ / 4,2 is above it.
Example 6.1. Take the following parametric incomplete pairwise comparison matrix of size = 4 with = 2 missing elements: . Now 4,0 ≈ 0.884 and 4,2 ≈ 0.356 from Table 2. There are three instances where the optimal filling of matrix A( , ) results in a consistent pairwise comparison matrix:  (1, 4) )︁ ≈ 0.0404 < 0.1 × 4,0 , thus the optimally filled out incomplete pairwise comparison matrix might be accepted according to the "standard" threshold for complete matrices because the latter does not take into account the automatic reduction of inconsistency due to the optimisation procedure. Table 4 reports the consistency index of matrix A( , ) for some parameters and . is restricted between 1/5 and 5 because 12 ( , ) × 23 ( , ) × 34 ( , ) = 3 but 14 ( , ) = . Bold numbers correspond to the cases when inconsistency can be tolerated based on the approximated thresholds of Table 2, while italic numbers show instances that can be accepted only if the optimal solution A(x) of (2) is considered as a (complete) pairwise comparison matrix and the threshold of 10% is used for (A(x)) / 4,0 . Bozóki et al. (2013) carried out a controlled experiment, where university students were divided into subgroups to make pairwise comparisons from different types of problems, with different number of alternatives in different questioning orders. Consequently, not only the complete pairwise comparison matrices are known but their incomplete submatrices obtained after a given number of comparisons was asked. We have picked one interesting matrix from this dataset.

A real life application: continuous monitoring of inconsistency
Example 7.1. The following pairwise comparison matrix reflects the opinion of a decisionmaker on how much more a summer house is liked compared to another summer house on a numerical scale:  Ross (1934), optimises two objective functions: it maximises the distances for the same alternatives to reappear and aims to balance the number of the first and second positions in the comparison for every alternative. Figure 3 shows how inconsistency changes as more and more comparisons are given by the decision-maker. Following Bozóki et al. (2013, Figure 2), the solid red line uses the random index associated with a complete 6 × 6 pairwise comparison matrix, which is not a valid approach according to Section 4. On the other hand, the dashed blue line is obtained by the values of the random index according to our computations, see Table 2. The naïve approach indicates no problem around inconsistency, its level remains below the 10% threshold during the filling in process. However, accounting for the number of missing elements reveals that inconsistency is substantially increased when the seventh comparison ( 24 ) is made. Even though the complete pairwise comparison matrix can be accepted with respect to inconsistency, continuous monitoring warns the decision-maker that this particular comparison is worth reconsidering.

Conclusions
The paper reports approximated thresholds for the most popular measure of inconsistency, proposed by Saaty, in the case of incomplete pairwise comparison matrices. They are  determined by the value of the random index, that is, the average consistency ratio of a large number of random pairwise comparison matrices with missing elements. The calculation is far from trivial since a separate convex optimisation problem should be solved for each matrix to find the optimal filling of unknown entries. Numerical results uncover that the threshold depends not only on the size of the pairwise comparison matrix but on the number of missing entries, too. A plausible linear estimation of the random index has also been provided. According to Table 2 and two examples, the extended values of the random index become indispensable in order to generalise Saaty's concept to incomplete comparisons. The associated thresholds can be directly programmed into decision-making software.
With the suggested rule of acceptability, the decision-maker can decide for any incomplete pairwise comparison matrix whether there is a need to revise earlier assessments or not. It allows the level of inconsistency to be monitored even before all comparisons are given, which may immediately indicate possible mistakes and suspicious entries. Therefore, the preference revision process can be launched as early as possible. It will be examined in future studies how this opportunity can be built into the known inconsistency reduction methods (Abel et al., 2018;Bozóki et al., 2015;Ergu et al., 2011;Xu and Xu, 2020).