From 0 to 100——The History of Zhihu’s Architecture Changes

Perhaps many people don’t know that Zhihu is the largest UGC (user-generated content) community on the Chinese Internet, second only to Baidu Tieba and Douban. Zhihu has been established for three years, starting from scratch, and now has more than 100 servers. Currently, Zhihu has more than 11 million registered users, with more than 80 million users per month; the website has more than 220 million page views per month, and more than 2,500 dynamic requests per second.

At ArchSummit Beijing 2014, Zhihu co-founder and CTO Li Shenshen gave the first comprehensive technical presentation of Zhihu in more than three years (download slides). This article is compiled based on the speech.

Initial architecture selection

When we actually started working on Zhihu in October 2010, there were only two engineers, including Li Shenshen. By the time it went online in December 2010, there were four engineers.

Zhihu's main development language is Python. Because Python is simple and powerful, it can be quickly learned, has high development efficiency, and has an active community, team members also like it.

Zhihu uses the Tornado framework. Because it supports asynchrony, it is very suitable for real-time Comet applications, and it is simple and lightweight, with low learning costs. In addition, it has mature cases of FriendFeed and community support from Facebook. One feature of Zhihu's products is that it hopes to establish a long connection with the browser to facilitate real-time push of feeds and notifications, so Tornado is more suitable.

Initially, the entire team focused all their energy on the development of product functions, and in other aspects, basically they used the simplest methods to save time and save money. Of course, this also brought some problems later.

The original idea was to use a cloud host to save costs. Zhihu's first server was a Linode host with 512MB of memory . However, after the website went online, the popularity of the internal test exceeded expectations, and many users reported that the website was very slow. The cross-border network delay is greater than expected, especially the uneven domestic network, and the access conditions of users across the country are not the same. This problem, coupled with the need to register the domain name at the time, made Zhihu go back to the old way of buying machines and looking for computer rooms .

After buying the machine and finding the computer room, we encountered a new problem: the service often went down. At that time, the service provider's machine memory always had problems and had to be restarted at any time. Finally, one time the machine went down and couldn't be started. At this time, Zhihu made the Web and database highly available . Entrepreneurship is like this. You never know what kind of problems you will face when you wake up tomorrow morning.

This is the architecture diagram at that stage. Both the Web and database were set up as master-slave. The image service at that time was hosted on Youpai Cloud. In addition to the master-slave, read-write separation was also done for better performance. To solve the synchronization problem, a server was added to run offline scripts to avoid response delays to online services. In addition, to improve the throughput delay of the intranet, the equipment was replaced, which increased the throughput of the entire intranet by 20 times.

In the first half of 2011, Zhihu was already very dependent on Redis. In addition to the initial use of queues and searches, it later began to use cache. Single-machine storage became a bottleneck, so sharding was introduced and consistency was achieved.

The Zhihu team believes in tools and believes that tools can improve efficiency. Tools are actually a process. There is no so-called best tool, only the most suitable tool. And it is constantly changing throughout the process, with the changes in the entire state and environment. The tools developed or used by Zhihu include Profiling (function-level request tracking, analysis and tuning), Werkzeug (tools for easy debugging), Puppet (configuration management) and Shipit (one-click launch or rollback).

Logging system

Zhihu was originally an invitation-only platform. In the second half of 2011, Zhihu launched an application registration system. Users without an invitation code could also apply to register on Zhihu by filling out some information. The number of users increased again, and some accounts posted advertisements, which needed to be removed. The need for a log system was put on the agenda.

This log system must support distributed collection, centralized storage, real-time, subscribeable and simple features. At that time, we investigated some open source systems, such as Scribe, which was generally good, but did not support subscription. Kafka was developed in Scala, but the team had little experience in Scala. Flume was similar and heavier. So the development team chose to develop a log system by themselves - Kids (Kids Is Data Stream) . As the name suggests, Kids is used to aggregate various data streams.

Kids refers to the idea of ​​Scribe. Kdis can be configured as an agent or server on each server. The agent directly receives messages from the application, and after collecting the messages, it can call the next agent or directly call the central server. When subscribing to logs, you can get them from the server or from some agents on the central node.

The details are shown in the figure below:

Zhihu also developed a web gadget (Kids Explorer) based on Kids , which supports real-time viewing of online logs and has now become the most important tool for debugging online problems.

Kids has been open sourced and put on Github .

Event-driven architecture

Zhihu has a feature that after adding an answer, the only subsequent operations are updating notifications and dynamics. However, as the entire function increases, there are more operations such as updating indexes, updating counts, and content review, and the subsequent operations are varied. If the traditional method is used, the maintenance logic will become more and more complex and the maintainability will be very poor. This scenario is very suitable for event-driven methods, so the development team adjusted the entire architecture and made an event-driven architecture.

At this time, the first thing you need is a message queue, which should be able to obtain a variety of events and has high requirements for consistency. To meet this demand, Zhihu developed a small tool called Sink. After it gets the message, it first saves and persists it locally, and then distributes the message. If the machine crashes, it can be fully restored when it restarts to ensure that the message is not lost. Then it puts the message into the task queue through the Miller development framework. Sink is more like a serial message subscription service, but the tasks need to be processed in parallel, so Beanstalkd comes in handy to manage the tasks throughout the entire cycle. The architecture is shown in the figure below:

For example, if a user answers a question, the system will first write the question to MySQL, insert the message into the Sink, and then return the question to the user. The Sink sends the task to Beanstalkd through Miller, and the Worker can find the task and process it by itself.

When it was first launched, there were 10 messages per second and 70 tasks generated. Now there are 100 events per second and 1,500 tasks generated, which are supported by the current event-driven architecture.

Page rendering optimization

In 2013, Zhihu had millions of page views per day. Page rendering is actually computationally intensive. In addition, because it needs to obtain data, it is also IO intensive. At this time, the development team componentized the page and upgraded the data acquisition mechanism. Zhihu obtains data in layers from top to bottom according to the structure of the entire page component tree. When the data of the upper layer has been obtained, the data of the lower layer does not need to go down. Basically, there are several layers with several data acquisitions.

Based on this idea, Zhihu developed its own template rendering development framework - ZhihuNode .

After a series of improvements, the performance of the page has been greatly improved. The question page has been reduced from 500ms to 150ms, and the feed page has been reduced from 1s to 600ms.

Service-Oriented Architecture (SOA)

As Zhihu's functions become more and more complex, the entire system is also getting larger. How does Zhihu do service-oriented work?

First of all, we need a basic RPC framework, which has also undergone several versions of evolution.

The first version was Wish, which was a strictly defined serialization model. The transport layer used STP, which was a very simple transport protocol written by myself and ran on TCP. It worked well at first because only one or two services were written at the beginning. However, as the number of services increased, some problems began to emerge. First, ProtocolBuffer would generate some description codes, which were very lengthy and looked ugly when placed in the entire library. In addition, the strict definition made it inconvenient to use. At this time, an engineer developed a new RPC framework - Snow. It uses simple JSON for data serialization. However, the problem faced by loose data definition is that, for example, if a service needs to be upgraded and the data structure needs to be rewritten, it is difficult to know which services are in use and it is difficult to notify them, and errors often occur. So a third RPC framework was born. The engineer who wrote the RPC framework hoped to combine the characteristics of the previous two frameworks. First, keep Snow simple, and second, require a relatively strict serialization protocol. This version introduced Apache Avro. At the same time, a special mechanism was added, making the transport layer and the serialization protocol layer pluggable. You can use either JSON or Avro, and the transport layer can use STP or binary protocol.

Then we built a service registration discovery, which allows us to find out which machine the service is on by simply defining the service name. At the same time, Zhihu also has corresponding tuning tools and has developed its own Tracing system based on Zipkin.

According to the calling relationship, Zhihu's services are divided into three layers: aggregation layer, content layer and basic layer . According to the attributes, they can be divided into three categories: data services, logic services and channel services. Data services are mainly for the storage of special data types, such as image services. Logical services are more CPU-intensive and computationally intensive operations, such as the definition and parsing of answer formats. The characteristic of channel services is that there is no storage, and they are more of a forwarding, such as Sink.

This is the overall architecture after the introduction of service-oriented computing.

The speech also introduced the new practice of developing Zhihu columns based on AngularJS. We will release the speech video on the website later, so stay tuned.