This is the image quality assessment tool used by the TikTok group. Are you sure you don’t want to try it?

Introduction

This article starts with the construction process of the internal image quality evaluation system of Douyin Group, mainly sharing the importance of image quality evaluation to the business , the main application scenarios and some typical practical cases of internal products . By sharing some problems encountered from the business perspective and our solutions, we hope to provide valuable reference for partners who encounter similar problems.

The construction process of the image quality assessment system

Why is it so important to evaluate image quality?

Through a large number of online business experiments, we found that the quality of pictures has a great impact on click-through rate,   It has a positive correlation with consumption indicators such as dwell time , which indirectly affects user revenue indicators. Therefore, it is very necessary to build an effective image quality evaluation system to ensure the user's image quality experience .

Intuitively speaking, improving picture quality can bring a better viewing experience, but the comprehensive QoE experience also needs to consider other factors such as user equipment, network conditions, viewing environment, and other factors. Whether improving picture quality at any cost can continue to bring QoE benefits to users requires rigorous experimental plans to verify the effect in business scenarios.

Through the accumulation of data analysis in multiple businesses such as low-quality image suppression and image quality-based recommendation optimization, we have obtained a clear relationship between image quality ratings and user subjective experience. Data statistics show that users have different trends in their sensitivity to content with different image qualities. By continuously improving image quality in the mid-range image quality range, users' QoE experience will also be significantly improved. However, when the image quality is lower or higher than a certain threshold, users will no longer be sensitive to image quality, and the impact of improving/reducing image quality on users will be reduced.

The expected sweet spot relationship between image quality and the image quality improvement in the middle range will continue to bring QoE benefits

In actual business scenarios, we analyze the relationship between image quality and average viewing time of users. Medium to high image quality can bring continuous viewing revenue.

The following figure specifically describes the main value of the image quality assessment system in business practice under two typical application scenarios:

Why do we develop our own image quality assessment system?

The end users of image services are humans, and image quality assessment is committed to becoming an objective calculation method that can measure the quality requirements of human eye perception of images .

Industry status

• Subjective quality assessment: the most accurate, but time-consuming, labor-intensive, and costly, and difficult to apply in large quantities. Examples include expert evaluation and crowdsourcing testing.
• Objective evaluation algorithm: It saves time and effort and can be applied on a large scale. However, there is a certain GAP between the full-reference/no-reference algorithm and subjective evaluation. In the UGC scenario, the gap will be more obvious.

The commonly used image quality assessment algorithms in the industry include PSNR, SSIM, and VMAF:

Pain Points

• It is difficult to quantify the effect of image quality enhancement : Common industry indicators (PSNR, SSIM, VMAF, etc.) are all reference image quality indicators, which are mainly suitable for image quality evaluation of compression distortion. It is difficult to quantify the effect of image quality enhancement.
• Not suitable   UGC   Scenario scoring : There are certain limitations in the application scenarios of industry-wide general indicators. Their training data sets are mainly PGC content, and their generalization effect in UGC scenarios is poor.
• Limited evaluation dimensions : In UGC scenarios, the image content is complex and the factors affecting image quality are diverse, so more dimensional evaluation indicators are needed for image quality analysis and optimization guidance.

How do we build an image quality assessment system?

According to the needs of different business forms such as on-demand, live broadcast and pictures, the VQScore image quality system developed by the Video Architecture Multimedia Laboratory provides the best full-link image quality scoring capability, providing asynchronous or real-time image quality scoring data, and providing capability support for subsequent transcoding, enhancement, recommendation strategies and overall monitoring.

The specific image quality analysis and scoring capabilities are divided into two parts:

1. Content analysis and understanding : mainly includes basic classification and detection capabilities such as ROI detection, CG content detection, face detection, and content classification, providing subdivided dimension decomposition capabilities and key content recognition capabilities for subsequent image quality scoring and enhanced transcoding, and achieving precise end-to-end adaptive enhanced transcoding combination capabilities
2. Image quality scoring capability : mainly includes general clarity scoring algorithm, aesthetic index, high-order color index, portrait quality and other evaluation indicators, noise, block effect, overexposure, dirty lens, blur and pseudo-HD and other segmented attribution indicators, as well as pre-processing image quality improvement capability evaluation indicators such as super-resolution quality, sharpening quality and enhanced combination evaluation. The combination of general + attribution + enhancement multiple dimensions provides a full range of image quality scoring capabilities such as monitoring, analysis, and strategy recommendation to meet the image quality optimization needs of different business scenarios.

The general picture quality clarity assessment algorithm is based on a lightweight transformer-based deep learning solution driven by subjectively annotated samples from diverse multi-business scenarios, open source datasets, and diverse distorted synthetic datasets. It provides more stable and accurate objective clarity prediction capabilities in UGC video/image scenarios.

In various business scenarios, according to the different business needs of on-demand, live broadcast and picture, it supports source image quality analysis of different contributed content within the highest 4K resolution, provides in-depth and detailed image quality dimension analysis in combination with business attribute dimensions, provides encoding optimization comparison and image quality monitoring at different time scales for adaptive transcoding, and provides effective QoE dimension data for business processes such as AB experiments and version iteration. At the same time, it can also provide adaptive playback distribution optimization capabilities based on a combination of image quality, QoS network, equipment and other factors for multi-resolution/bitrate playback.

What are the advantages of Tik Tok’s image quality assessment system?

Wide range of applications

• A high-quality and large-scale training dataset that covers both PGC and UGC content and has a wide range of applications (especially for UGC scenarios).
• The algorithm model has been continuously refined and optimized through products with hundreds of millions of DAU, and has strong generalization capabilities.

Multiple evaluation dimensions

It includes two types of comprehensive indicators, such as subjective clarity and public aesthetic quality, and more than ten types of detailed indicators, such as noise and brightness. It supports multi-dimensional and fine-grained analysis of image quality issues, making it easier for businesses to optimize and adjust strategies in a targeted manner.

Online verification of multiple businesses brings significant benefits

After being verified online by dozens of large-scale businesses such as TikTok, Toutiao, and Tomato Novels, the evaluation results are reliable and can effectively support the business in improving the picture quality experience, thereby bringing about an improvement in user consumption indicators and significant benefits.

Industry-leading algorithm capabilities

The algorithm models involved in the image quality assessment system have applied for multiple patents, such as a method for detecting pseudo-HD videos, a high-order video color quality evaluation model based on a multi-task twin neural network, and a sandwich video adaptive playback method.

In the "Compressed UGC Video Quality Assessment" competition at ICME 2021, Volcano Engine-Multimedia Lab won the first place in the No-Reference Video Quality Assessment (NR-VQA) MOS track with its self-developed VQScore algorithm. (Detailed introduction)

The competition mainly focuses on the research of UGC source video quality and the impact of H.264/AVC compression distortion on the subjective quality of the video.

In what scenarios is image quality assessment mainly used?

To make a simple analogy using the relationship between a weight loss plan and a weight scale, the image quality assessment system is a relatively objective and effective evaluation tool that has a wide range of applications in helping products understand the current status of business image quality, understanding the industry and market status, monitoring changes in online image quality, and supporting the improvement of user experience.

1. Understand the current status of business image quality

Business teams can use the image quality evaluation tools provided by veImageX to efficiently complete the image quality assessment of business products through offline and online evaluation. At the same time, the image quality evaluation system contains a variety of evaluation dimensions (such as noise intensity, color quality, block effect detection, overexposure detection, etc.), and dozens of detailed evaluation indicators can efficiently help business teams complete low-quality image attribution analysis and quickly identify the problem.

2. Understand the current status of the industry/market

With the help of image quality assessment tools, business teams can evaluate the image quality of mainstream products or similar businesses in the market in order to set reasonable image quality improvement goals; at the same time, by integrating the correspondence between user subjective evaluations and objective indicators, business teams can efficiently determine image quality assessment standards suitable for their own business.

3. Monitor online image quality changes

For a product that focuses on user image quality experience, online image quality monitoring tools are essential. veImageX provides end-to-end image quality indicator monitoring tools that can help business teams monitor online image quality changes in a long-term and efficient manner; through before-and-after data comparison and analysis, it helps businesses effectively verify the effectiveness of image quality optimization measures; at the same time, online low-quality problem alerts can also help business teams discover problems in a timely manner and ensure online user browsing experience.

4. Support to improve user experience

With the evaluation results provided by the image quality assessment system, the business team can suppress low-quality content by downgrading the search/recommendation of low-quality images, or improve the image quality with the help of image quality enhancement capabilities, effectively improving the user's browsing experience, thereby bringing about positive improvements in business indicators such as click-through rate, average reading/consumption time per person, and user retention.

Typical case practice sharing

At present, the image quality assessment tool provided by Volcano Engine veImageX has served dozens of business lines such as Douyin, Toutiao, Xigua, Fanqie Novels, and Dongqiudi, playing an important role in ensuring the user's image quality experience. Next, we selected several typical cases to briefly share our practical experience with you.

A short video/community platform

Demand background

A short video/community platform has users mainly distributed in multiple countries and regions, and publishes content covering multiple vertical categories. The business team received feedback from some users that there are certain differences in image quality between different countries and content vertical categories, which affects the user's browsing experience, so a special project was set up to solve the problem.

Practice plan

The business team first used image quality assessment tools to conduct an offline survey and analysis of image quality in the entire region, and found that there were large differences in image quality between some countries and between certain key vertical categories. Therefore, an adaptive enhancement model was used to improve image quality in a targeted manner while saving bit rate as much as possible.

Overall Benefit

After the optimization, the image quality between regions and key vertical categories on the platform is basically aligned and has reached the [good] level or above. The image size has been significantly reduced, and consumption indicators such as average stay time, average interaction, average reading time, and average number of sessions have all been significantly positive .

Tomato Novel

1. Demand Background

Compared with online articles, comics have more exquisite book covers and contain more information. Therefore, in terms of product form, the Tomato Novel Channel adopts a large-screen display. However, after the comic function was launched, the business team found that the original book covers of some comics were blurry, which seriously affected the user's browsing experience. As shown in the following figure:

In order to improve the image quality of these pictures, the business team came up with the idea of ​​screening low-quality pictures through image quality assessment, using image quality enhancement capabilities to build an automated processing flow, and processing low-quality pictures in a targeted manner to obtain high-definition pictures to improve the overall viewing experience.

2. Practice plan

The business team uses the veImageX image quality assessment tool to evaluate the image quality of publications (such as novel covers, illustrations, e-book covers, audio player covers, etc.)   and comics (comic covers, horizontal illustrations, etc.)   We conduct offline image quality evaluation in different scenarios, and conduct a quality survey of images of different resolutions. We analyze the causes of low quality and comprehensively evaluate the benefits of the enhancement algorithm on subjective image quality improvement, and identify differentiated processing solutions. In the end, the business team chose to build an automated processing flow, and perform optimization processing such as adaptive enhancement and super-resolution on images of different image quality levels based on the evaluation results, so as to improve the user's browsing experience in a targeted manner.

The comparison of low-quality images before and after optimization is as follows:

3. Overall benefits

With the help of veImageX's image quality assessment and enhancement capabilities, the Tomato Novel team has improved the image quality in a targeted manner, effectively improving user image quality experience and user consumption indicators such as click-through rate, average reading/consumption time per person, and retention .

Today's headlines

Demand background

Toutiao's short video channel is mainly displayed in two columns, and the display format of the two-column stream channel is mainly cover images. Combining online experimental results and practical experience, it is found that the quality of the cover image not only affects the user's browsing experience, but also affects business indicators such as click-through conversion rate and user retention. How to effectively identify content with blurred covers and suppress and regulate it has become a more difficult task.

Practice plan

With the help of image quality assessment tools, the business team scores the image quality of cover images, efficiently identifies low-quality covers (blockiness ≥ X and vqscore < Y) and implements suppression and regulation strategies; at the same time, vqscore is included in the reference indicators of the recommendation model to provide more priority exposure opportunities for high-quality content.

Overall Benefit

The business team suppressed and regulated low-quality cover images, and the rate of high-quality covers evaluated by manual evaluation increased by about 3 times , and the rate of low-quality covers decreased by about 36.7%.   , the proportion of blurred cover images decreased by about 51.4%   , average number of readings per capita,   Length of stay   , click-through conversion rate and other business indicators have also been significantly improved . (Data from business AB experiment)

Happiness VR

Demand background

In the early stages of building Xingfuli Real Estate's VR capabilities, due to the diverse sources and channels of material supply, uneven quality, and frequent online user feedback, image quality control mainly relied on manual review, regular spot checks, and online feedback, which was not only labor-intensive but also subjective in evaluation. There was a lack of differentiated data indicators for panoramic images to quantify the gap between image quality and industry-leading levels, which made it difficult for the business team to efficiently locate image quality issues and make targeted improvements and evaluate optimization effects.

Practice plan

The business team finally selected the resolution by conducting an offline image quality survey of online sample data and combining the suggestions of algorithm experts.   （   VQScore   ）、Noise、Brightness、Overexposure   Four indicators are used as quantitative evaluation indicators for panoramic images. The evaluation found that there are significant differences in image quality among the three types of decoration, namely, hardcover, simple decoration, and rough decoration. The key differences are highly correlated with factors such as ambient light and lighting . The business team carried out targeted iterative optimization and monitored changes in image quality indicators, which significantly improved the effect of VR house viewing.

Overall Benefit

The business team used image quality assessment tools to locate specific image quality issues and carried out targeted iterative optimization to narrow the gap with industry-leading levels . At the same time, with the help of the VR image quality enhancement capabilities provided by veImgaeX, the image quality of panoramic images was significantly improved , achieving zero user complaints in stages and making up for problems such as the uneven quality of front-end acquisition equipment .

Final Thoughts

This article briefly introduces the business thinking, construction process, application scenarios and some practical experience of the Douyin Group on the image quality evaluation system. Due to space limitations, this article omits some details such as the exploration process and specific implementation, but still hopes to provide some inspiration or reference for colleagues in the industry.