2 pictures to help you understand Toutiao's recommendation system

The “predecessors” of recommendation systems

In 2016, Tencent invested in Toutiao at a valuation of US$8 billion. As we all know, Zhang Yiming rejected Tencent's investment. Now we also know that ByteDance is valued at US$75 billion. The recommendation system has contributed greatly to all of this.

Because search engines and recommendation systems are so similar and relatively simpler (please don’t criticize me), let’s first understand search engines. As for whether search engines are the predecessor of recommendation systems, I am too lazy to investigate.

As shown in the figure above, the search engine is divided into an offline part and an online part, and each part has a different mission.

Simply put, the offline part of the search engine focuses on collecting and processing content. Search engines use web crawlers to crawl the original content on the website and index the content. These contents will establish different index systems according to the different requirements of the search system. For example, for news-type content, timely index data will be established.

The online part of the search engine is responsible for responding to users' search requests, completing content screening and sorting, and returning the final results to users. Let's take an example to illustrate this process:

When a user enters a keyword NBA into a search engine, the search engine will analyze, transform, expand, and correct the keyword. For example, if it finds that the NBA and the National Basketball Association are synonyms, it will expand them. Next, the search engine will obtain candidate sets from different index data in a variety of ways. This step is called recall. After obtaining the candidate set, the search engine uses a more sophisticated calculation model to calculate the score of each candidate content and rank each item in the candidate set. At this time, the results cannot be displayed to the user, and this process requires rule intervention. This process serves a specific product purpose. If there is such a rule as "official website protection rule, ensuring that all brand search terms can return to the official website first", then the official website will be inserted and pinned to the top, and finally the results will be displayed to the user. At this point, the search engine's work is not over yet. Search engines will optimize the ranking model based on user click feedback. For example, if most users do not click on article 10, then article 10 will not be displayed in a higher position subsequently.

The summary of the above two pictures is the following picture, which is the first picture I want you to see:

Toutiao’s recommendation system

Through the previous picture, we understand the principle of search engines (I will pretend that you understand it no matter what), and this picture of Toutiao has one more stroke than the previous one. Considering that the two pictures are highly similar, of course I, such a lazy person, will not draw it. You can use your imagination.

In fact, the recommendation system also has offline and online parts. The picture above (that’s not a picture, it’s a PNG) is the offline part of the recommendation system, which is similar to a search engine.

Like search engines, recommendation systems also need to obtain content. The recommendation system obtains recommended content through database import, protocol synchronization and user submission. Unlike search engines, recommendation systems have more ways to obtain content, and the content is much more structured than what search engine crawlers can capture. The recommendation system also needs to index the content to be recommended, which is similar to a search engine. The dimensions of the recommendation system will be more.

Next, it is the online part of the recommendation system. Oh my god, when I saw the picture above, I found that the recommendation system is really very similar to the search engine, with one more thing.

The input of the search engine is the user's search keywords. The recommendation system also requires input, but the user is not aware of this process. For the recommendation system, its input is scene information, such as time, place, and equipment. After the search engine receives the input, it will process the keywords. For the recommendation system, it will perform user portrait queries. In this case, the recommendation system learns that the user is interested in NBA in terms of entity words, and in sports and technology in terms of classification. After querying the user portrait, the recommendation system enters the recall phase. It obtains candidate sets from different index data in a variety of ways based on the user portrait query results "NBA, Sports and Technology". After the recall is completed, like a search engine, the recommendation system ranks the candidate sets according to the predetermined estimated targets. Similarly, the recommendation system also needs to go through the rule intervention step before the final result is displayed to the user. For the last step, various user actions and behaviors will continuously optimize the ranking model in the search engine and continuously improve their own portraits in the recommendation system.

The summary of the above two pictures is the following picture, which is the second PNG (picture) I want to show you:

The Nature of Recommender Systems

Through the two pictures of search engines and recommendation systems, we have a general understanding of what the recommendation system is all about. In fact, the recommendation system is a strategic behavior. For strategy, he has four elements, namely:

Problem to be solved Input (factors that affect the solution) Computational logic (rules for converting input into output) Output (specific solution)

For Toutiao today:

The problem it needs to solve is "finding content that users like from a huge amount of content";

His input is “user profile and content features”;

Calculation logic: convert these content features into likes according to certain rules;

Output: Sort the content by popularity from high to low.

Since my course on recommendation systems has not yet been completed, I will not expand on this part for now. The above article will be expanded in the following blogs. Students who are interested are welcome to follow.

Note: The above content is summarized from personal reading and paid courses.

Author: Belt

Source: Belt