Five-minute technical talk | Application of analytic hierarchy process (AHP) in user experience design evaluation

Part 01 What is AHP?

AHP, or analytic hierarchy process, is a hierarchical weighted decision analysis method proposed by Thomas Setty, an American operations researcher. It is suitable for decision-making problems with target systems that have hierarchical and staggered evaluation indicators and whose target values ​​are difficult to describe quantitatively. It will decompose our goals into multiple goals or criterion layers, and then reasonably calculate the weighted values ​​between each criterion based on the relative importance of the measurement criteria to assist the decision-making process. It can not only simplify system analysis and calculation work, greatly reduce many uncertain factors, but also normalize and standardize people's subjective judgments.

Part 02 Basic principles of AHP

The basic principle of AHP is to decompose our goals into different components, and generate a multi-level analysis structure model in which each element is interconnected through analysis of these elements; then make a more objective judgment on the elements of each layer, quantitatively give a relative importance expression, and calculate the relative importance weights of the factors at each level through a mathematical model; finally, the problem solution or decision plan can be selected based on the calculation results of the weight ranking.

Part 03 Examples of AHP modeling steps

The AHP modeling steps (Figure 1) mainly include: building a hierarchical indicator model, constructing a judgment matrix, single-level sorting and consistency verification, hierarchical total sorting and consistency verification, etc. In this paper, we will take a certain type of smart home product experience scoring model as an example to explain and analyze.

3.1 Building a hierarchical indicator model

When applying AHP to analyze decision-making problems, we first need to hierarchize the problem and build a hierarchical structural indicator model. The hierarchical elements in the model serve as criteria to dominate the relevant elements of the next level. For example, when building a product experience scoring model for a certain type of smart furniture, our first-level indicators are defined as functional completeness, hardware reliability, user experience, and terminal performance. Each indicator is further divided into second-level indicators as shown in Figure 2.

The number of levels in our hierarchical indicator model is mainly determined by the complexity of the problem and the level of detail required for analysis. It should be noted that although the number of levels is generally unlimited, the number of elements controlled by each element in each level should generally not exceed 9, because when there are too many controllable elements, it will be difficult to make pairwise comparisons. Elements also need to have relatively independent characteristics. If the correlation is high, it will affect the accuracy of the results.

3.2 Constructing the Judgment Matrix

The hierarchical indicator structure obtained in the first step reflects the relationship between factors, but the proportion of element indicators in the minds of different decision makers is definitely different. Moreover, when there are many factors affecting a certain factor, if the degree of influence of each factor on the factor is directly considered, some inconsistent or contradictory results may be obtained due to incomplete consideration.

To provide more reliable data, suppose we want to compare the influence of n factors on a certain factor. We use the method of comparing the factors in pairs and finally establish a judgment matrix. Each time we take two factors, we use the method in Table 1 below to compare them, and use numbers 1 to 9 and their reciprocals as scales. All comparison results are represented by matrix A.

This is still a bit abstract. We use the above scale as a rule to construct the judgment matrix table as follows:

The table describes the relative importance of the factors. The values ​​can be determined by the subjective judgment of the decision maker, based on surveys or literature, or by expert discussion. The values ​​in the table above are based on subjective judgment. It is not difficult to understand that the diagonal symmetrical elements of the judgment matrix should be reciprocal to each other, and the values ​​of the judgment matrix should also comply with logical specifications, otherwise they will not pass the subsequent consistency check. For example, in Table 2, high temperature and low temperature resistance is more important than waterproofness, and waterproofness is more important than drop resistance. If we fill in drop resistance as more important than high temperature and low temperature resistance, it must be a logical error.

3.3 Single-level sorting and consistency check

Hierarchical single sorting is to calculate the weight values ​​of the importance order of all factors related to an indicator element of the previous layer according to the judgment matrix, and sort them according to the weight. The weight value calculation includes summation method, square root method and eigenvector method. We use Table 2 as an example of the summation method. First, each column of the matrix is ​​standardized, and then the standardized elements are summed by row. Finally, the summation result is standardized to obtain the weight value of each factor. Figure 3 shows our calculation process.

In order to check whether the values ​​of the judgment matrix conform to the logical specification, we need to perform consistency check. We need to find the maximum characteristic root, and then use the following consistency index CI to check the consistency index of the judgment, where n is the order of the judgment matrix:

CI=0 means that the judgment matrix is ​​completely consistent. The larger the CI, the more serious the inconsistency of the judgment matrix. Then we solve the CR value based on the CI and RI values ​​to determine whether its consistency is passed.

The value of RI should be determined by referring to the following average random consistency index table, and its value n is the order of the judgment matrix.

If CR < 0.1, the judgment matrix is ​​considered to have passed the consistency test. If the condition is not met, the judgment matrix needs to be checked and its value adjusted.

We take Table 2 as an example, the value of n is 3, and we calculate the maximum characteristic root. The formula is:

AW is: judgment matrix * standardized weight, and then the cumulative value of the row. According to the above formulas, CR < 0.1 can be obtained, so it passes the consistency test. The other judgment matrices can also be solved in the same way and consistency checked.

3.4 Total sorting and consistency check of levels

From the above steps, we get a weight vector of a group of elements to an element in the previous layer. We ultimately want to get the ranking weight of each element, especially the lowest layer, for the target. The total ranking weight should synthesize the weights under the single criterion from top to bottom. For example, in Figure 1, the weight of the second layer "user experience" is multiplied by the weight of the third layer "hardware user experience" and the weight of "software user experience", and the weight values ​​relative to the target are obtained in turn, and the weights are finally sorted. The total hierarchical ranking of each scheme in the lowest layer needs to be checked for consistency, and the test can be performed layer by layer from high to low. When the total ranking random consistency ratio CR<0.1, it is considered that the hierarchical total ranking result has a satisfactory consistency and the analysis result is accepted.

Part 04 Application direction of relevant user experience design evaluation

From the above analysis, we can see that AHP (Hierarchy Analysis Process) is a multi-criteria decision-making method that can be used to help evaluate and compare the importance of different factors. In user experience design evaluation, AHP can also be applied to the following aspects:

1) Allocate the importance of related functional requirements: AHP can help determine the relative importance of functional requirements, identify key areas of focus in product design, and provide certain guidance for the design of products or services.

2) Help make decisions on products or design solutions: AHP can help compare and decide on different products or design solutions. By finding the criteria that affect the design or product solution and using AHP to calculate the relative weights between them, the decision maker can finally find the best product or design solution.

3) Determine the priority direction of product improvement: By applying AHP, different aspects of user experience can be evaluated and ranked to determine the areas that need the most improvement. User experience can be broken down into multiple dimensions such as ease of use, efficiency, and satisfaction. By comparing and weighting these dimensions, decision makers can be helped to determine the priority direction of improvement during product development.

4) Evaluate user satisfaction: AHP can also be used to evaluate user satisfaction. By breaking down satisfaction into different influencing factors and performing comparisons and weight calculations, we can find out the factors that have the greatest impact on user satisfaction and what aspects need to be improved to improve user satisfaction.

It should be pointed out that when using AHP for user experience evaluation, the comparison matrix should be made as objectively and accurately as possible to avoid the influence of subjective bias on the results. At the same time, the evaluator and data source should be carefully selected. In general, AHP provides a systematic and quantifiable method for improving user experience design, which helps corporate products provide a more satisfactory user experience.

➺ References

[1] Ye Zhen. Research and application of fuzzy comprehensive evaluation method based on AHP[D].

[2] https://blog.csdn.net/weixin_43095238/article/details/108055579.

[3] https://zhuanlan.zhihu.com/p/448412538.