Practical information! The key points and technologies for building a high-quality communication cloud platform!

On December 19, the Shenzhen session of UPYUN Architecture and Operation Conference (Arch&Ops Conf) was held in Kexing Science Park. This large-scale technical conference for operation and architecture practitioners is committed to promoting the development and application of various operation and maintenance technologies, product architectures, etc. in the Internet and mobile Internet. The Shenzhen session was the finale of the conference. Mr. Jia Junjie, CTO of Yunzhixun, was the most distinguished guest of the conference and gave a keynote speech entitled "Architecture Points of High-Quality Communication PAAS Platform".

Yunzhixun CTO Jia Junjie's speech (abstract version):

(Cloud News Chief Technology Officer, Mr. Jia Junjie, please call me the most handsome CTO~)

Jia Junjie: My topic is the key points and key technologies for building a high-quality communication cloud platform. The concept of communication is very broad. Any information exchange can be considered communication, and all visible Internet services can be called communication. The concept I am talking about belongs to the traditional communication concept in the telecommunications network, including the familiar phone calls, video calls, text messages, conferences, and social centers. Cloud platforms should be very familiar to everyone. We are now using them very widely. From IaaS, PaaS, and SaaS, everyone often uses these three layers. PaaS includes push and other cloud platforms. I guess everyone has used SMS services. I think this belongs to the communication field. For example, when we send verification codes, how to integrate traditional phone call capabilities into APPs, and concentrate video calls, voice conferences, and video conferences into APPs, this is the concept of communication cloud platforms. We are a cloud platform service provider for integrated communications. Today, we will provide you with a direction for the problems encountered in the construction process and the solutions to these problems.

The way we look at the construction of the communication cloud platform depends on the development trend of the communication business. If we know something about communication, we will find that communication is developing towards IP and mobile Internet. Very traditional narrowband communication transmission is completely IP-based transmission regardless of the telecommunications network. Not to mention the mobile Internet, which is definitely also transmitted on the basis of IP, which is a very big development direction. First, the communication service providers of the traditional telecommunications network have very single communication capabilities. The traditional communication capabilities can only make voice calls and send text messages, which cannot meet the development needs of the Internet. Second, WeChat and QQ have done this and can communicate in the context of mobile Internet.

The development direction of cloud computing mainly comes from the business development needs of the Internet and mobile Internet, including the traditional Internet and the current mainstream traditional telecommunications network and mainstream IP network communication service providers, which are quite closed. Now, communication capabilities are needed in all walks of life, such as online medical care, online education, smart hardware, and other industries. In the past two years, communication cloud services have gradually emerged. Under such a development trend, how do we build a communication cloud platform? We have summarized the following points: first, exquisiteness and efficiency. In Internet-based communication, due to its own characteristics, the requirements for real-time performance are particularly high. Second, this communication is two-way, which places great demands on the architecture of our platform.

In terms of efficiency, we have designed a layered architecture. We have separated the service layer and business layer, and we have OTP network support at the bottom. Different capabilities are responsible for different layers. In terms of access layer design, the network layer does not understand the business at all. We call it a transparent forwarding network. For the access layer, it is consistent with the business and does not fully understand the business. For the business layer, it completes various communication services to achieve the division of capabilities at different levels, which is convenient for future development. In terms of expansion isolation, the access layer does not understand the business. It needs to route through the business ID or the backup ID in the communication process. According to the requests of different IDs and routes, it is divided into different modules. We have conferences and point-to-point voice for segmentation, and there will be no coupling relationship. Heterogeneous diversification is in the access layer and network layer. According to our current understanding, our trend is in the era of integrated communication of traditional telecommunications networks and the Internet, which complement each other.

The access layer has rich access capabilities. For Internet access capabilities, we need to support technologies such as various mobile terminals, PC fixed terminals, web calls, etc. Its connection method is mainly implemented through TCP. There are many access methods for SMS, which may be HTTP or traditional telecommunications SMS protocols. In terms of stability and high availability, we mainly build an OTT communication network. This network is an accelerated network based on the public Internet. I run my business on this network to avoid the impact of the general Internet on my real-time communication. We will also introduce it to you later.

When it comes to the network, we need to understand the current status of VOIP on the Internet. The network is very complex, both in terms of user access network and backbone transmission network. User networks are Wifi, 3G, and 4G, and the transmission network is a variety of operators, involving cross-operator and cross-regional communications, which has a very important impact on real-time communications. Real-time communication mainly depends on packet loss, delay, and jitter. This is very different from our general Internet and mobile Internet. In real-time communication, the delay requirement is tens of milliseconds or hundreds of milliseconds, and many Internet services do not have high requirements. Packet loss, sometimes when the packet loss is 1-2%, it greatly affects the call experience, which is basically not perceived in normal business.

All these problems led to various difficulties when we were building the communication cloud platform. What should we do? This is also back to the OTT network construction we just mentioned, a transparent intelligent routing network. This is my self-built acceleration platform. The Internet communications of major companies including Google, Tencent, YY, etc. have networks. This network can greatly accelerate my communication services, ensure the quality of communication, and solve the problems of network complexity and the last kilometer.

This is similar to the mainstream SDN technology. By managing the routing by itself, the core deployed nodes are associated. If two users are communicating, I only need to identify the user by its IP address and connect it to the node closest to him. The called party is also identified by IP. The communication between the two nodes must be realized through its own detection technology and monitoring technology, and dynamic calculations are performed to ensure that the fastest, highest quality and most stable channel is selected between the two users to transmit voice data. Tunnel technology and OSFP protocol routing technology are mainly used here. In terms of performance, if we consider that media forwarding has a great impact on the computer room and network, we have a forwarding module to accelerate and improve performance.

The core issues in building an OTT network are routing mechanism, strategy, and how to deploy these points. If the user chooses a node that is not the closest to the user, or the node is not the best, even if the strategy is very good, the user effect cannot be guaranteed. We base this on two points: one is the distribution of your own users, where are your users distributed, what are the user types, and whether there are more Wifi or 3G in a certain area, or more Unicom or Telecom. We decide the points based on the distribution of users. We will divide the regions according to the national situation and select points for different regions. This is closely related to the regional distribution of users and a certain capacity node. In the process of selecting points, we use some third-party or our own business testing methods to monitor.

The point I choose must be the most effective for me, and I guarantee that the point I choose is the most effective. This is the network topology of our network, including hierarchical network architecture, distributed deployment, and business division by region. The front end is user deployment, which must be deployed in the place closest to the user to ensure that the user chooses a point nearby for access. The other is the core forwarding node. This access node may be a single-line one. This point only accesses China Unicom users, and this point only accesses China Telecom users. I can go through the middle forwarding node, which must be a three-line or BGP computer room. Distributed deployment is also reflected in the network topology. It is more widely distributed than the traditional communication network. Traditional e-commerce, on-demand, and live broadcasting may not require so many distribution points. We are more widely distributed, covering the prefecture-level level. Business is divided by region. Each region is only responsible for accessing the business in the region. The core point is also that a certain node is only responsible for the local, the transfer of certain access points, rather than the global, it has a distribution strategy.

While building the OTT network, how to monitor and manage the network quality? First, the network status monitoring of all network nodes; second, real-time warning of all network node failures; third, continuous optimization based on business operations. I will dynamically adjust according to its monitoring situation, and in some cases, it can achieve automatic switching. Users generally get more than two nodes dynamically. If the node closest to it fails, it will dynamically select the next closest node, which will not cause access failure. We will continue to make adjustments according to the business situation, which is very critical. In particular, we have selected so many points, and each point we selected may not be the best locally. Even if it is the best at the time, it may not be very good in the long run, and it may deteriorate.

Judging by the data of business operations, we have some value in voice communication. If this value continues to decline, I will remove this point and choose a better and newer node to replace it. There are still some things that need to be optimized through the network, which is different from what I just said. I just talked about HTTP and other protocols. HTTP is basically not used here. We use SIP, XMPP, and private protocols more. We choose private protocols. Private protocols are similar to completely binary, and they are implemented by encoding and decoding. This efficiency is the best, especially for mobile Internet scenarios.

Media transmission quality assurance is different from general on-demand. It is bidirectional. The packet loss rate has a great impact and the real-time requirements are very high. We will take a lot of error correction and quality assurance measures. When I transmit a piece of data, it may be transmitted 2 or 1.5 times, depending on the network. After a packet is lost, I can repair it through other packets. In some packet loss scenarios, the effect of my data transmission can be guaranteed. We are currently losing 5-25%. When it is greater than 30%, the Mos value will be lower than normal and cannot be guaranteed. The effect will be affected to a certain extent.

We use ARQ technology, which we call automatic simultaneous interpretation. Especially for videos, the size of each frame is different. We will perform simultaneous interpretation on the key values. If some I frames are lost, we will keep some.

We believe that unified policy control is very critical. As far as communication is concerned, it is not very adaptable to terminals. Different mobile phones have very different parameters for voice and video communications. Our strategy is to configure all policies in a unified manner. For example, when the client sets the language, the size of the stream, connection method, routing method, and preferred encoding and decoding are all controlled through the backend center. The idea of ​​policy control mainly includes backend, unified policy formulation; dynamic, intelligent mobilization; front-end, enabling controllability.

Whether it is the network factor or the different network types of the mobile phone itself, there are often changes. If the strategy is hard-coded, it will have a great impact on the communication guide in various situations. This is a unified configuration in the background. This is dynamic and intelligently adjusted. We say that the parameters of each call are different. Before making a call or communicating, I dynamically obtain information. I will determine what network this call is using and what your network quality is like. I will dynamically determine what parameters you need under such parameters to ensure dynamic acquisition of configuration and choose the most appropriate communication method to communicate with the background.

In terms of enabling, many communication parameter configurations are divided into different levels. As mentioned later, some are global, and some are business-based. Different services use different parameters. The highest level is a problem that only some users will encounter. We will customize parameters for them separately. This is the strategy and parameters for flow control. The first is a global configuration, a heartbeat technology, and the minimum flow consumption. In media data compression, we will choose a private compression method to compress the information. In terms of bit rate, we make dynamic adjustments. For example, in the case of 3G, the bit rate is generally between 150 and 200Kbps, which is the most suitable effect, otherwise there may be problems.

In the case of Wifi, the bit rate is between 200 and 300. If the setting is not reasonable, you will use a very high bit rate, which will lead to insufficient bandwidth. The more data is transmitted, the greater the packet loss rate. In the case of Wifi, the bandwidth utilization rate is not good, and a lot of bandwidth is not used.

We can dynamically adjust the client connection strategy. There are several main points. First, access the nearest network. Second, do the same network, especially for communication services. According to our experience, the packet loss rate and other effects are very poor. Third, pre-embed the information. The access layer obtained in history is pre-embedded. You can select multiple nodes at the same time and keep the poor node as the best node according to the actual situation. This is the media routing strategy. We will choose the media multiple node selection strategy. I will use the minimum path algorithm. I will choose how to jump. If both are in the same network, it may be P2P or one-hop transfer. The other is the dynamic strategy adjustment technology. This is our strategy control mechanism. When we control the strategy, we will count all the terminal types, version numbers, and network status. According to different mobile phone models, what parameters may be used to optimize, I will customize a set of parameters in the back to ensure that it is under this parameter. The quality and communication effect are the best. This is the specific client control parameter. Do I need to do redundancy? Do I need to do a log detection mechanism? These backgrounds can control it.

This is the IM message synchronization mechanism, which is applicable to IM communication technology. Traditionally, it may be pushed directly, but this method may be better. At the beginning, it is this kind of notification, digging out all notifications to ensure that 100% of the messages are not lost. This is a very good strategy. Real-time communication is the current niche business direction. I hope it will be helpful to everyone. If you want to learn more, we can communicate offline. Thank you!