Revealing the secret of Toutiao’s user growth!

Toutiao has always been criticized for its vulgar content.

But we have to admit: Toutiao has done a great job in user growth !

According to the survey, the usage time of independent Toutiao App users has soared from 3.9% to 10.1% since 2018, a 1.6-fold increase, surpassing Baidu and Alibaba to rank second in total usage time.

What’s even more amazing is that the rapid growth rate of Toutiao products is still continuing, with the Toutiao News App alone still maintaining more than 1 million new users every day.

Rabbit said so much, not to brag about the headlines, but just to use this to introduce the following content.

Behind Toutiao’s “insane” user growth, what are some tips worth learning?

How did Toutiao achieve rapid growth?

In fact, the reason why Toutiao can achieve rapid growth is mainly due to its powerful data monitoring system (spending a lot of money to buy data). The daily activity and growth data of any product are under their monitoring.

In addition, in order to help improve the success rate of innovative products, Toutiao has even developed a growth engine. While we are still struggling with which creative idea, landing page, or color button to use, they are already conducting dozens or even hundreds of A/B tests at the same time to help product managers and operators find the best solution.

Although we cannot imitate Toutiao’s powerful data monitoring system, we are unable to develop their growth engine for the time being. However , we can still learn from A/B testing to help screen the best solution.

A/B testing of Toutiao

Seeing this, some SEM ers may start to get impatient, thinking that Rabbit is talking about something that is commonplace.

Indeed, A/B testing is something that is very familiar to bidders or optimizers. So, without going into some conceptual stuff, let’s take a look at: How does Toutiao conduct A/B testing?

Toutiao’s headlines have a “double title ” feature, which is actually a practice of A/B testing.

An article can have two titles. If one title becomes useless, there is another one to support it. Unlike WeChat public accounts , which only have one title, once the title is useless, the number of readers is basically gone.

How does this “dynamic” content recommendation mechanism work?

For the same solution, Toutiao will first recommend it to a small group of people:

For example, if there are 100 people and these 100 people give good feedback on the title and content, then recommend the plan to a larger group of people, such as 500 people. If these 500 people give very good feedback on the title and content, then recommend it to a larger group of people, such as 2,000 people, and so on.

Users' behavioral actions will be collected. According to the "Principles of Toutiao Recommendation System", users' feedback on content can be seen basically every hour. However, because there are fluctuations in data every hour, Toutiao usually uses days as the time node to view user behavior data.

After collecting user behavior actions, Toutiao will process logs, distribute statistics, and write them into the database.

Because Toutiao has set up the "double title" function, it can understand users' feedback on titles more accurately and thus grasp users' behavioral data.

Of course, Toutiao’s most powerful A/B test is not the “double title” feature, because testing only the title will lead to the rampant spread of “ clickbait ”. Based on this, Toutiao A/B has a "dynamic" content recommendation mechanism. The "dynamic" here refers to real-time updates and adjustments based on feedback results.

The Toutiao system can automatically generate: experimental data comparison, experimental data confidence, experimental conclusion summary and experimental optimization suggestions.

In other words: Through A/B testing , Toutiao not only completed the solution research, but also tested user tastes and achieved refined content recommendations, thereby better retaining users.

A/B testing pitfalls

A/B testing may seem simple, but it actually involves many complications. If you are not careful, the test results may deviate from the scientific track.

1) Ignore differences in test environments

If you were asked to A/B test your creative ideas, which of the following methods would you use to test your headline?

a. Divide users in Shanghai into 3 groups and push 3 different titles to each of them at the same time.

b. Divide users in Shanghai into three groups and push three different titles to each group at different time points.

If you chose option b, congratulations, you are out of luck!

To give an inappropriate example, the testing method of plan B is like placing an advertisement on TV, selecting 3 pm on weekdays and prime time in the evening for test collection.

Since the test environments during the rotation presentations are not the same and the target audiences vary greatly, the data results after the final experiment will inevitably have certain deviations, making them less convincing.

2) It is easy to generalize

If you give up directly when the test results do not show the ideal data improvement, you may fall into the trap again.

As an international short-term rental platform, search is a very basic component of the Airbnb ecosystem. Airbnb once conducted an A/B test on search page optimization, and the new version emphasized the listing pictures and the location of the house (as shown in the figure below).

After waiting long enough, the test results showed that the overall data of the new and old versions were almost the same, and it seemed that this optimization did not have a very good effect.

If Airbnb had given up on this optimization based on the overall data performance at this point, then this project, which had taken a lot of effort to design, would have been wasted.

On the contrary, after careful study, they found that the new version performed well in different browsers except IE. After realizing that the new design restricted clicks using older versions of IE (which obviously had a very negative impact on global results), Airbnb immediately patched it.

Since then, IE has returned to displaying the same results as other browsers, and the overall data of the experiment has increased by more than 2%.

Through the example of Airbnb, we can learn that the correct approach is: when the overall effect is not good, do not judge it simply, but observe the individual situation from multiple dimensions to avoid decision-making bias caused by the herd fallacy.

3) Only local optimum is achieved

After avoiding the above two pitfalls, you may get a relatively good test result. When you are ecstatic and are about to announce the results, you may have stepped into another pit - "local optimum"

Taking the A/B test of a financial platform to increase the registration rate of new users as an example, through continuous optimization of the registration button copy , it was found that compared with "Register now", "Register for free" and other copy, the registration rate of "Receive a 100 yuan new user red envelope" was the highest.

However, if he only focuses on testing the copy, he may miss other more effective hypotheses to increase user registration rate.

The correct approach is to conduct user surveys to understand why users do not register. Usually, the reasons why financial platforms make users give up registration are complicated registration processes, trust issues, no matching financial products, etc. Therefore, after completing the copywriting test of the registration button, we also need to conduct the desired experiments in these aspects.

Summarize

In today’s article, we analyzed Toutiao to show you the powerful effects of A/B testing.

Although A/B testing has long been an indispensable tool for us marketers in the promotion process, I hope that through this sharing, everyone can discover things that they have not noticed before.

Source: Operation Research Society