Hundreds of millions of video views per day, Miaopai playback link optimization practice

[51CTO.com original article] Currently, we are in an era of fragmented reading with rapid development of information. Short videos with the advantages of integrated graphics and text are becoming more and more popular among the public. In 2017, major Internet giants successively entered the field of short videos, and the popularity of short videos exceeded many people's expectations. The real competition of short videos is content and user experience. As pioneers in the short video industry, Miaopai and Xiaokaxiu, both owned by Yixia Technology, have so far reached a peak of over 3 billion daily views and 2 million daily uploads. How do they ensure that the user experience is not affected by the quality of uploads and playbacks?

Recently, the 15th 51CTO Technology Salon with the theme of "Tech Neo" was held in Beijing. Jia Chaoteng, an architect from Yixia Technology, was invited to the event. He joined Miaopai in 2014 and promoted Miaopai's cloud migration, visual log analysis, infrastructure transformation and service transformation, service tuning, etc. He is also keen on various infrastructure practices, performance tuning, and availability assurance of the entire service link in high-concurrency scenarios. From his sharing, we can learn about Miaopai's optimization practices in the playback link.

Technical architecture of Miaopai backend services

Let's first take a look at the basic technical architecture of Miaopai's backend service , which is divided into four layers from top to bottom, as shown in the following figure:

The top layer is the cooperation between Web, APP, Weibo and other partners. They are directly facing users and provide users with the most direct product services.

The second layer is the business logic API directly oriented to services, which provides data presentation such as activity and video lists, user lists, etc. that intuitively reflect product and business logic.

The third layer is a series of segmented services such as users and videos that are relatively independent of business logic.

The fourth layer is basic services, such as relationship services, counting services, search services, Push services, recommendation services, sensitive word filtering services, Feed, storage, etc.

In the entire service process, uploading and playing are the core businesses. The following figure shows the process of uploading resources before playing :

• The client sends an upload request to the business logic server, and the business logic server tells which upload point to use for uploading.
• Qiniu or Sina S3 are responsible for storing the uploaded segments respectively, and then the upload end requests to merge them. The business server receives the callback after the video is uploaded and updates the status. At this time, the entire video is processed from physical resources to business status.
• Then transcoding and screenshot processing are carried out, and the service processing flow after uploading is entered.

Among them, storage mainly relies on third parties, while services such as video screenshot and video transcoding mainly use self-developed services.

It is worth noting here that a lot of optimization needs to be done during resource uploading.

Currently, the functions of the mobile terminal are very powerful, enough to support users to preview and change various special effects at will. For example, the preview of special effects, video special effects editing and preview are carried out on the mobile terminal. On the one hand, customers can intuitively experience the video effects, and on the other hand, they can make full use of the client computing resources and reduce the processing pressure of service transcoding. In addition, the transcoding compression after video shooting and editing will also be preliminarily processed on the client, and then uploaded to the business server, and then the transcoding service will perform secondary processing to slim down the video file, reduce user traffic consumption, and increase user upload speed.

On the video production side, we have relevant statistics such as shooting, editing, client transcoding, and uploading, which are used to monitor the efficiency of each link on the production side, promote the continuous optimization of each link, and improve the results visible in the data after the optimization of each link.

Playback & Scheduling History

The above are some optimization methods for uploading resources before playback. Next, let's talk about playback & scheduling. Miaopai's playback & scheduling has gone through three major versions:

The business logic directly returns the playback address, which is the most simplified version of the playback process.

This is also the initial version. In the early stage of the service launch, both the video upload volume and the playback volume are in the accumulation stage. The upload and playback efficiency are issues we are more concerned about, but the current biggest pain point is that the product form must meet market demand. This version is mainly based on the HTTP protocol, where the client directly requests the video resource address returned by the service API, and plays it directly after downloading.

Based on the detection points, CDN logs are used to calculate CDN quality offline and schedule the volume, which is an evolution of the playback process.

As the business develops, the daily video upload and playback volume increases, and playback efficiency becomes particularly important because it directly affects the user experience of the product.

At this time, we added detection points in different regions to detect the playback and download quality, and collected data points for quality distribution. At the same time, we also took the CDN download request log for analysis and scheduling. This version of scheduling uses offline calculations to determine the best result, and then the business logic API returns the playback address, and the client plays it directly.

The third version is an independent scheduling service, which judges the playback quality in real time and offline and performs scheduling.

Based on the research of the previous version, we extracted a relatively independent scheduling service based on Nginx+Lua, with very light processing logic, and collected the playback quality on the end. We compared the quality of different CDNs in the province, city, region, and operator where the user is located, and then gave the user the final best CDN address.

In addition to playback quality, a series of parameters and indicators such as failure rate and playback volume are also collected for analysis or rapid fault recovery and transfer to prevent the node load of the cooperating CDN manufacturer from being too high, which may affect the playback speed.

In addition, the scheduling service will also make some business policy processing. For example, for videos suspected of violating regulations, it will make a judgment when returning the playback address, delete the source site video, and clean up the CDN cache.

The evolved version also carries out iterative tuning in terms of its own IP library and IP library data update and maintenance, anti-hijacking, data reporting, and log analysis.

The following is a simplified diagram of the scheduling system currently used by Miaopai :

In terms of playback scheduling, we cooperate with a number of CDN vendors to conduct quality scheduling and achieve fast access, thereby achieving fast return, weak business logic, lightweight, high efficiency, and flexible controllability.

Unpredictable Black Swan

In the process of business development and service evolution, unpredictable black swans will inevitably be encountered, such as the interruption of the business intranet and the cutting of Qiniu Ningbo optical fiber.

In order to prevent the failure of the source station from affecting the normal operation of the service, we have adopted a multi-source station solution. When a problem occurs in a source station, it can be switched at any time. At the same time, cross-source station backup can be achieved between source stations.

In terms of uploading, multiple upload points are deployed and flexibly allocated to ensure that users can upload normally.

In terms of playback, the service availability is checked to schedule node feedback in a more timely manner. For playback quality scheduling, the best CDN can be called based on the quality of service.

Massive log analysis and monitoring

Miaopai logs mainly consist of server access log, error log, service log, app log, user log, etc.

The pain points of traditional log analysis are as follows:

• The log volume is too large, single log analysis is too slow, logs are distributed in different service clusters, and different nodes cannot quickly locate service nodes.
• The upstream and downstream status is unknown, there are positioning issues, and the efficiency of scenario analysis is too low.
• After a failure, there is a herd effect, feedback from one place, and multiple groups (departments) respond to investigate, which wastes a lot of energy.

To address the pain points of traditional log analysis, Miaopai has deployed a real-time log analysis system . The following figure shows some details:

The main function of the ELK real-time log analysis system is to analyze the business logs directly after they come in, and quickly provide feedback on the status of each service and some statistical information in a graphical way. When problems occur, they can be quickly perceived, and there is also service monitoring based on ELK.

Combined with ELK, the system integrates the internal monitoring and alarm platform to achieve visual indicator comparison of the entire video shooting, editing, uploading, transcoding, distribution, and playback, ensuring the efficiency of the entire link and real-time alarms for related metrics to ensure the stability and availability of the service.

An indispensable part of broadcasting business

The quality of video and image encoding and transcoding will have a direct impact on playback. Transcoding service is an indispensable part of the playback business. The following figure is a flow chart of the entire transcoding service :

After users upload videos, different business terminals will submit transcoding tasks online or offline, and the transcoding service will perform asynchronous transcoding to produce videos or covers of different specifications, which will be provided to different partners and displayed in different list page formats.

Video specifications include multi-stream, 16:9, webp, hidden watermark, etc. The transcoding process should control the file size, bit rate and other parameters to the minimum without affecting the video clarity and quality. In this way, the flow rate, loading speed and other aspects can be continuously optimized, allowing users to watch videos smoothly and clearly, and also saving a lot of flow.

In terms of multi-stream, Miaopai adopts a variable bit rate solution to adapt the bit rate according to the video content to control video encoding.

Written in ***:

Miaopai was launched in August 2013 and has occupied a certain domestic market since then. In the future, Miaopai will focus on data nodes, deploy multi-site active-active, and consider hybrid cloud transformation to address the problem that cloud vendors cannot be relied on, hide a lot of data information, and it is difficult to find the source of problems. It will also improve the integration of self-built CDN nodes, disaster recovery construction, and monitoring statistics. At present, with the increasingly fierce competition in the short video industry, I hope Miaopai will continue to break through and innovate in the future and go further and further. Let us look forward to the future of the fast-growing short video industry.

【Guest Profile】

[[207669]]

Jia Chaoteng is an architect at Xia Technology (Miaopai, Xiaokaxiu, Yizhibo). He joined Miaopai in 2014 and has promoted Miaopai's cloud migration, visual log analysis, infrastructure transformation and service-oriented transformation, and service optimization. He is keen on various infrastructure practices, performance tuning, and full-link service availability assurance in high-concurrency scenarios.

Other Salon Lecturer Articles

A thorough understanding of CDN pain points, Internet veterans talk about CDN!

Cooling down high concurrency: Meituan’s high-performance, high-reliability four-layer load balancing MGW optimization practice

October 28 / Beijing, the 16th "Tech Neo" salon, theme: "Automated Operation and Maintenance and DevOps", click on the picture to register immediately.

[51CTO original article, please indicate the original author and source as 51CTO.com when reprinting on partner sites]