Refined user growth case!

ABtest is gaining more and more attention. Fast and flexible comparison experiments can quickly identify problems and avoid large-scale waste of resources. So designing a good experiment is crucial.

1. What is A/B testing?

Simply put, the application method of A/B testing in product optimization is: before the product is officially iterated and released, two (or more) plans are formulated for the same goal, and user traffic is divided into several groups. On the premise of ensuring that each group of users has the same characteristics, users are allowed to see different plan designs separately. Based on the real data feedback from several groups of users, scientific help is provided to help product decisions.

The application method of A/B testing determines its three major characteristics: a priori, parallel and scientific.

A priori: A/B testing is actually an "a priori" experimental system, which is a predictive conclusion and is very different from the "post-hoc" inductive conclusion. The same method of using data to count and analyze the quality of versions is to release the version first and then verify the effect through data. However, A/B testing obtains representative test conclusions through scientific test design, representative sampling samples, traffic segmentation and small traffic testing. In this way, a very small sample size can be used to generalize to all traffic and make it reliable.

Parallelism: A/B testing is to test two or more solutions online at the same time. The advantage of this is that it ensures the consistency of the environment of each version, making it easier to compare the advantages and disadvantages in a more scientific and objective manner. At the same time, it also saves verification time, and there is no need to test another version after verifying one version.

Scientificity: What is emphasized here is the scientific nature of traffic distribution. The correct approach to A/B testing is to evenly distribute users with similar characteristics into the test groups to ensure the similarity of user characteristics in each group, thereby avoiding data bias and making the test results more representative.

2. Misunderstandings of A/B Testing

Myth 1: Showing different versions in rotation

First of all, it needs to be made clear that this approach is not a true A/B test. This phenomenon often occurs in today's advertising . In order to improve the conversion rate of landing pages, advertisers will choose to display different versions of ads in rotation.

However, this approach cannot guarantee that each version will be in the same environment. For example, if the prime time slots are 7 p.m. on weekdays or 3 p.m., the audience groups will be significantly different, so it is difficult to determine whether the final effects will be different, or even the reasons for the different effects.

Correct approach: Test different versions of the solution online in parallel (simultaneously) to minimize the differences in the test environment for all versions.

Misconception 2: Choose different app markets to launch (randomly select users for testing)

For some companies that have realized the importance of data a priori, in order to verify the real impact of the new version on user usage, they may choose to package different versions and release them to different application markets. When they find that the data performance of a certain version is the best, they decide to launch the entire version online.

What's more, a number of users (even internal company employees) will be randomly selected for preliminary trials, and the iterative version will be decided based on data feedback. This violates the principle of scientific traffic allocation of A/B testing and can easily lead to Simpson's paradox (that is, two sets of data under a certain condition will satisfy a certain property or trend when discussed separately, but once considered together, they may lead to opposite conclusions).

Correct approach: Allocate traffic scientifically to ensure that the user characteristics of each trial version are similar.

Myth 3: Let users choose their own version

Many companies will leave an entrance to return to the old version on the new version of the page, allowing users to choose which version to use and determine the best version by collecting the click rate of the return button. However, this approach is not conducive to statistical analysis of user behavior data on the new version, because users may leave the new version simply because they are used to using the old version, rather than because they think the new version experience is bad, which ultimately leads to inaccurate test results.

Correct approach: Allow users to demonstrate their real usage experience of different versions. Enterprises should pay real-time attention to the data performance of each version and adjust the test traffic in a timely manner based on data feedback.

Misunderstanding 4: Too shallow understanding and analysis of test results

This misunderstanding includes two different contents:

First, they believe that an experiment is successful only if the experimental version performs better than the original version. In fact, A/B testing is a tool used to select the best version. There are three possible results of the experiment: the experimental version is improved (the experimental version is the best), there is no significant difference (both versions are acceptable), and the experimental version performs worse than the original version (the original version is the best). These three results actually indicate the success of the experiment.

Second, the performance of all scenarios cannot be inferred based solely on the overall data results of the experiment. For example, when the results of an A/B test show that the experimental version performs worse than the original version, it is assumed that the effect is negative in all regions or channels.

However, if you break down the data of different browsers in each version, you may find that the overall test data is poor due to the obvious disadvantage of a certain browser. Therefore, do not focus only on the overall performance of the test data and ignore the possible deviations in the results caused by segmented scenarios.

Correct approach: While analyzing the overall test data, it is necessary to consider the test data results from multiple dimensions.

3. Where can A/B testing be used?

Although A/B testing can make up for deficiencies in product optimization, it is not completely suitable for all products. Because the results of A/B testing require a lot of data support, the more daily traffic a website has, the more accurate the results will be. Generally speaking, we recommend that when conducting A/B testing, you ensure that the daily traffic of each version is above 1,000 UVs . Otherwise, the test cycle will be very long, or it will be difficult to obtain accurate (results converge) data results.

After talking about what kind of products are suitable for A/B testing, we will now explain where A/B testing can be used from the two aspects of content optimization and application scenarios. I hope this will give you some inspiration.

Optimize content

Product UI

Products in different industries require different styles, while also complementing the company's brand. Using A/B testing to optimize the UI can give users a better interactive experience and visual experience.

Content

As the name suggests, it refers to the text content that users read. It runs through all parts of a product, from picture captions and button text to article titles and even section themes. You can try changing the copy content in these parts and test the data effects of different solutions.

Page Layout

Sometimes, there may be no need to adjust the product's UI or copywriting content; just changes in the layout can produce growth effects.

Product Features

I want to add a new feature to the product, but it is difficult to determine whether it can meet user expectations. If I launch it blindly, it may cause some losses. Use A/B testing and be truly accountable to your users. For example, before a new paid photo viewing feature is officially launched, a social product needs to conduct an A/B test to verify the usage and effectiveness of the feature.

Recommendation Algorithm

These include content-based recommendation algorithms (recommending similar content based on the user's historical records), collaborative filtering-based recommendation algorithms (recommending related content based on the behavior of users with similar interests), and association rule-based recommendation algorithms (recommending content to users based on the relevance of the content itself), ultimately increasing user stickiness.

IV. Application Scenarios

Ad landing page

Landing pages are a key step in attracting and converting traffic in Internet marketing. It is very important to get visitors to click on the landing page after seeing the advertisement (or marketing page), continue to maintain their interest in your products or services, and even develop a favorable impression, and finally complete conversion behaviors such as registration, purchase, and sharing. A/B testing can help you maximize your marketing ROI.

Web/H5 pages

In addition to advertising and marketing, the product's official website page (whether PC or mobile) is always an important channel for users to learn about the product. Therefore, how to make users better understand product information and stimulate their willingness to take further actions, thereby obtaining higher registration rates, purchase rates, download rates, etc., is the primary goal of web page optimization. By using A/B testing, you can find the best way to display the page at a lower cost.

APP User Experience

With the massive influx of C-end users, the complexity of products is increasing, and the decision-making risk of new versions is also increasing dramatically. Maintaining a steady growth in the core business data of the product is the version goal of each App. Through A/B testing, the data performance of each version is verified before the official release of each version, so that each iteration can achieve deterministic growth.

Media advertising placement and management

For media and advertising technology companies, A/B testing can be used to optimize innovative advertising products driven by design and data.

On the one hand, testing can be used to optimize advertising effectiveness and advertising resource fill rate to achieve the goal of increasing advertising unit prices; on the other hand, it can also measure the impact of existing advertising products on user experience, and by continuously improving user experience advertising products, it can drive higher mobile advertising business revenue.

Grayscale release

The current method of product optimization and iteration is usually to directly release a version online to all users. Once an online accident (or bug) occurs, it has a huge impact on users, and the problem-solving cycle is long. Sometimes, it even has to be rolled back to the previous version, which seriously affects the user experience. A/B testing effectively reduces the probability of online accidents/major bugs occurring among all users by releasing versions to small batches of users. The vast majority of users are unaware of bugs, thus ensuring a good user experience to the greatest extent possible.

V. Conclusion

AB test is suitable for products with a large number of users. It only provides one of our working ideas. It is not a god-like existence and should neither be overestimated nor underestimated.

Author: Bai Gaoliang

Source: Public Account: Baigaoliang