An experimental approach: converting verbal expressions to numerical scales

Abstract

A key issue in decision problems is the selection and use of the appropriate response scale. In this paper verbal expressions are converted into numerical scales for a subjective problem instance. In our experiment, we conducted a color selection test with 462 subjects by testing six colors on color-calibrated tablets in ISO standardized sensory test booths. The colors were evaluated both in a pairwise comparison matrix (indirect ranking with four-item verbal category scale) and on a direct scoring basis. We determined scales that provide the closest results on average and individually to the direct scoring, based on the eigenvector and the logarithmic least squares methods. All results show that the difference between verbal expressions is much smaller than the one used by most of the common numerical scales. As a sensitivity analysis, different discrepancy measures were investigated, and further experiments were also carried out. All the main findings were confirmed to be robust regarding the weight calculation technique, the discrepancy measure, and the problem instance as well. The results suggest that several methods (e.g., the AHP, and different ISO standards) need substantial revision. The respondents’ inconsistency was also analyzed with a repeated question regarding their preference between a given pair of colors. It is shown that most decision makers answer similarly for the second time, but there can be significant (even ordinal) differences. The respondents whose answers are further from the original tend to be more inconsistent in general.