Uncovering the secret of Toutiao’s growth — A/B testing

In 2018, the growth of China's mobile Internet users slowed down, with only an increase of 20 million in the first half of the year.

But there was a dark horse that managed to go against the current in such adverse circumstances.

The usage time of independent Toutiao App users increased from 3.9% to 10.1%, a 1.6-fold increase, surpassing Baidu and Alibaba to rank second in total usage time.

What’s even more amazing is that the rapid growth rate of Toutiao products is still continuing, with the Toutiao News App alone still maintaining more than 1 million new users every day.

There are actually some little-known secrets behind how Toutiao achieved such "crazy" user growth.

Toutiao will innovate (or copy) incubated products internally and set a retention life-or-death line (RIO) for each product. Products that exceed the life-or-death line can receive support from Toutiao’s internal traffic , allowing them to quickly reach tens of millions of daily active users.

Toutiao is involved in nearly 60 products

Toutiao has a very powerful data monitoring system (it spends a lot of money to buy data), and the daily activity and growth data of any product are under their monitoring.

In addition, in order to help improve the success rate of innovative products, Toutiao has even developed a growth engine internally. While we were still struggling with the order of function buttons, they were conducting dozens or even hundreds of A/B tests at the same time to help product managers and operators find the best solution.

We cannot imitate Toutiao’s powerful life and death line and data monitoring system, and we cannot develop their growth engine for the time being. However, we can still learn from A/B testing to help screen the best solution.

Although A/B testing used to be more in the product field, it was used to judge the value of a feature. In fact, A/B testing has now penetrated into operations. If the official account can launch a small-traffic opening rate testing function one day, I think you will be very happy!

01 What is A/B testing

A/B testing, also called controlled experiments and randomized experiments. Simply put, it means designing Plan A and Plan B for the same goal, and letting some users use Plan A and some users use Plan B. Record user usage and compare which solution is better based on user feedback.

Don’t think that the “ A/B test ” here really only has plan A and plan B. “A/B test” is just a habitual name. You can also have plans C, D, and E for testing.

After reading the definition, you may think that the concept of "A/B testing" is already common. Isn't it just proposing multiple options and then choosing the one with the best effect?

In fact, it is very simple to talk about in theory, but it still requires a lot of skills to actually do it. Let’s first take a look at how Toutiao conducts A/B testing?

Toutiao’s headlines have a “double title ” feature, which is actually a practice of A/B testing.

Maybe you will say, isn't this to give users more room for title creation? An article can have two titles. If one title becomes useless, there is another one to support it. Unlike WeChat public accounts , which only have one title, once the title is useless, the number of readers is basically gone.

In fact, Toutiao set up the "double title" function in order to more accurately understand users' feedback on titles and thus grasp users' behavioral data.

Of course, Toutiao’s most powerful A/B test is not the “double title” feature, because testing only the title will lead to the rampant spread of “ clickbait ”. Based on this, Toutiao A/B has a "dynamic" content recommendation mechanism. The "dynamic" here refers to real-time updates and adjustments based on feedback results.

How does this “dynamic” content recommendation mechanism work?

For the same solution, Toutiao will first recommend it to a small group of people:

Users' behavioral actions will be collected. According to the "Principles of Toutiao Recommendation System", users' feedback on content can be seen basically every hour. However, because there are fluctuations in data every hour, Toutiao usually uses days as the time node to view user behavior data.

After collecting user behavior actions, Toutiao will process logs, distribute statistics, and write them into the database.

The Toutiao system can automatically generate: experimental data comparison, experimental data confidence, experimental conclusion summary and experimental optimization suggestions.

From this perspective, don’t you think that A/B testing is really powerful? It not only completes solution research, but also helps you understand user tastes through testing, make refined content recommendations, and thus better retain users.

02 Application of A/B Testing

You may think that A/B testing is only affordable for big companies and has nothing to do with us. You may also think that this is just a product issue and has nothing to do with our operations.

Then you have a big misunderstanding. Not only can small players play it, but it is also closely related to our operations. If you are an operator who can do growth, you will most likely become the leader of your company.

For example:

We previously invited Afu, the editor-in-chief of " Late Night Hair ", to give us a micro-class sharing. Since we are talking about micro-courses, we must polish the micro-course posters to enhance the publicity effect.

So we asked our chief designer to make a version. As soon as it came out, the ladies in the editorial department disagreed and started arguing.

One person thinks that the topic “How to write marketing copy that users like” is not attractive and should be changed to “10 tips to write marketing copy that users like”.

Another person agreed, but said it should be further optimized and changed to “10 tips for writing conversion copy that will make users go viral.”

At this time, Brother Xian said very cleverly, can't you make 3 versions of the poster? Only one baby account is allowed to forward each version of the poster at the same time (all baby accounts have 5,000 friends). In this way, wouldn't it be possible to test which version has a better effect?

We didn't realize at the time that this was actually the thinking behind A/B testing.

Later, we implemented Xian Ge’s idea and tested the forwarding rates of three posters (users who forwarded the poster/new users), which were 30%, 35%, and 44% respectively. The one with the best effect was: 10 tips to write conversion copy that will go viral among users.

Next, we mobilized all baby accounts and all employees of the company to share the posters, which eventually attracted 3,000 people to attend the class. This is also our most effective micro-class sharing so far.

In addition to the application of micro-courses, A/B testing can also be used in many other places.

For example, App push can do A/B testing. Ele.me wanted to test how effective different promotional activities were in retaining users, so the following scenario occurred.

Yesterday, Xiao Songguo excitedly told me that Ele.me sent her a push notification: "You have a 15 yuan coupon to be collected", but when she clicked on it, she found out it was "15 off for purchases over 40 yuan". Despite this, she still ordered a lot of things.

Based on past experience, Ele.me likes to push notifications before we eat. So I looked at my Ele.me push notification: "Enjoy 25% off on orders over 45". I lost interest at first sight because the starting point was so high.

In fact, this is the A/B test that Ele.me operations did for us.

Through this test, they were able to find out through the order rate that, for both offering a 15 yuan discount (assuming both spend 40 yuan), directly highlighting the number “15 yuan discount” would be more effective than “15 yuan off for orders over 40 yuan”.

In short, through A/B testing, we can indeed try out the best growth method. In addition to App push and micro-course themes, there are many scenarios where A/B testing can be used, such as paid advertising, app stores , landing pages, new user guidance processes, etc.

03 Common pitfalls of A/B testing

A/B testing may seem simple, but it actually involves many complications. If you are not careful, the test results may deviate from the scientific track.

1) Ignore differences in test environments

If one day a public account develops the function of A/B testing for titles, which of the following solutions would you use to test your titles?

If you chose option b, congratulations, you are out of luck!

To give an inappropriate example, the testing method of plan B is like placing an advertisement on TV, selecting 3 pm on weekdays and prime time in the evening for test collection.

Since the test environments during the rotation presentations are not the same and the target audiences vary greatly, the data results after the final experiment will inevitably have certain deviations , making them less convincing.

2) It is easy to generalize

If you give up directly when the test results do not show the ideal data improvement, you may fall into the trap again.

As an international short-term rental platform, search is a very basic component of the Airbnb ecosystem. Airbnb once conducted an A/B test on search page optimization, and the new version emphasized the listing pictures and the location of the house (as shown in the figure below).

After waiting long enough, the test results showed that the overall data of the new and old versions were almost the same, and it seemed that this optimization did not have a very good effect.

If Airbnb had given up on this optimization based on the overall data performance at this point, then this project, which had taken a lot of effort to design, would have been wasted.

On the contrary, after careful study, they found that the new version performed well in different browsers except IE. After realizing that the new design restricted clicks using older versions of IE (which obviously had a very negative impact on global results), Airbnb immediately patched it.

Since then, IE has returned to displaying the same results as other browsers, and the overall data of the experiment has increased by more than 2%.

Through the example of Airbnb, we can learn that the correct approach is: when the overall effect is not good, do not judge it simply, but observe the individual situation from multiple dimensions to avoid decision-making bias caused by the herd fallacy.

3) Only local optimum is achieved

After avoiding the above two pitfalls, you may get a relatively good test result. When you are ecstatic and are about to announce the results, you may have stepped into another pit - "local optimum"

Taking a financial platform's A/B test to increase the registration rate of new users as an example, the operation team continuously optimized the copy of the registration button and found that compared with "Register now" and "Register for free", the registration rate of "Receive a 100 yuan red packet for new users" was the highest.

However, if he only focuses on testing the copy, he may miss other more effective hypotheses to increase user registration rate.

The correct approach is to conduct user surveys to understand why users do not register. Usually, the reasons why financial platforms make users give up registration are complicated registration processes, trust issues, no matching financial products, etc. Therefore, after completing the copywriting test of the registration button, we also need to conduct the desired experiments in these aspects.

04 Conclusion

In today’s article, we analyzed Toutiao to show you the powerful effect of A/B testing. Using A/B testing can not only select the best solution from many options, but also continuously iterate and optimize the product to achieve user retention .

Nowadays, A/B testing has penetrated into operational work, and operational agencies have also found that this theory is really useful in actual work.

1) When there are multiple options to choose from but opinions differ, A/B testing can be used to find the optimal solution. For example, article titles, fission poster copywriting, and App Push, etc.

2) A/B testing may seem simple, but it actually has many pitfalls:

① Ignoring the differences in test environments and not controlling variables (there can only be one variable), resulting in data deviation;

② Generalizing from a single case. When the test results do not achieve the ideal data improvement, giving up on product optimization will lead to the failure of the project;

③ Only local optimization is performed while ignoring the adjustment and update of other parts, making it impossible to find loopholes in other aspects.

Source: Routine Editorial Department