Growth Strategy: How to use AB testing to evaluate and optimize activities?

The e-commerce industry is full of promotional activities of all sizes, and subsidy wars are often staged in the O2O field. In the initial stage of expansion, spending money and making concessions to attract users is naturally the most effective method, but when a certain scale is reached, is the model of crazy promotional subsidies still efficient? Is a personalized operation strategy necessary? What is the real effect of the activity?

In order to solve the above problems and establish a scientific activity effect evaluation system, the simplest method at present is AB testing.

1. How to establish an evaluation system?

AB testing is currently widely used in the grayscale release stage of products. In addition, AB testing has more extensive uses, such as growth strategies such as precision marketing.

To verify a certain type of plan, the concept is very simple, which is nothing more than controlling variables. However, the business world is particularly complex, and it is extremely difficult to control all irrelevant variables. Therefore, AB testing is very popular and has become a common standard for testing "truth".

Based on the idea of ​​AB testing, random grouping can ensure that other variables of the two groups are basically consistent. By exerting influence on the experimental group, the performance difference between the experimental group and the control group can be observed, thereby evaluating the effect of the influence.

Next, we will take the promotional activities in the e-commerce industry as an example and try to build an evaluation system for promotions.

First, potential groups are screened out through labels, and a portion of users are randomly selected as the test group. Without any intervention, the conversion of users is secretly observed.

• For users in the experimental group, we intervened through targeted coupons and text message outreach, and then waited quietly for users to convert.
• For the control group, users are all converted naturally, while users in the experimental group are converted under intervention. In the experimental group, some users were indeed attracted by the discounts and placed orders, but some other users were likely to convert naturally even without coupons.

Therefore, we use the conversion rate of the control group as the natural conversion rate without coupons, then we can get the following user-related indicators:

• User baseline = total number of interventions * natural conversion rate
• User increase = total number of interventions * (intervention conversion rate - natural conversion rate)
• User promotion rate = user promotion / user baseline

Similarly, from the perspective of business analysis, we can evaluate the actual sales results:

1. GMV baseline = user baseline * average order value
2. GMV increase = user increase * average order value
3. Cost = discount amount per order * number of users placing orders + charge for each SMS message * total number of people involved
4. Net GMV increase = GMV increase - cost

Finally, we can get comprehensive indicators to evaluate business performance:

1. GMV net increase rate = GMV net increase / GMV baseline
2. ROI = Net GMV Increase / Cost

There are so many indicators above, which one is the most important?

The purpose of listing the above indicators is to facilitate understanding of the indicator decomposition and calculation process. But for different roles, ultimately you only need to focus on the outcome indicators that are relevant to them.

• For operations or marketing, their assessment goals may focus on the number of monthly active users, so they will be more concerned about the user improvement rate . Through horizontal comparison of multiple activities, they can identify the effectiveness of the activities and then gradually explore the root causes.
• For colleagues in charge of sales, the factors that need to be considered are relatively complex, but they can also directly evaluate the overall effect of the activity through the two indicators of GMV net increase rate and ROI.

Among them, the GMV net increase rate reflects the effect of the activity on GMV. If the current goal is to increase GMV at all costs, then you should choose an activity form or intensity with a high GMV net increase rate. If you need to weigh the profits, you should also include ROI in the analysis and select activities that have a high net increase in GMV and a relatively impressive ROI.

2. How to build an optimization strategy?

It was mentioned above that operations or sales can optimize activity plans through AB testing, so how should it be implemented specifically?

If you need to test the effects of different promotion forms and different activity intensities, it is necessary to further subdivide the experimental group.

For example: for experimental group 1, you can give out coupons worth 20 off for purchases over 100, and for experimental group 2, you can give out coupons worth 10 off for purchases over 100. Finally, compare and analyze the differences in activity effects with the control group. The GMV increase of a coupon with a 20 yuan discount on purchases over 100 yuan may be high, but due to the high cost, the net increase in GMV and ROI will not be high.

Similarly, for different forms of promotions, such as targeted coupons, targeted preferential prices, etc., different thresholds for the same discount, such as 100 minus 20 and 200 minus 40, can also be tested in a similar way.

After conducting multiple AB tests, you can make a preliminary judgment based on historical test data and identify the best promotion plan that meets your goals.

3. What are the pitfalls of AB testing?

Methodology always looks good, but when it comes to implementation, it will face many unexpected problems. Let’s take a look at the pitfalls of using the above-mentioned AB testing method to evaluate and optimize activity effects.

1. The number of people in the test group is too small: the number of people in the test group and the experimental group does not need to be exactly the same, but the number of people in each group must be guaranteed to be statistically significant. If the experimental group has 100,000 people and the test group has only 10 people, then the results of the test group will be greatly affected by the individual, which may eventually lead to abnormal results.
2. There are too few people in the experimental group: If you want to test which threshold of coupons is more effective, you need to split into multiple experimental groups and issue different coupons. If there are too many groups, the number of people in each experimental group will be too small, resulting in inaccurate results.
3. Your test group is someone else's experimental group: In reality, there is no absolutely clean test field. You think the test group can represent the natural conversion of users without interference, but you don't know that other activities have already classified them into the experimental group. If it is an internal conflict within the company, then users participating in other experiments can be eliminated after the data flows back. However, if it is interfered with by external competitors, then we will have no way of knowing. Only through comprehensive evaluation of multiple experiments can we hedge against special situations that may be faced in a single activity.
4. Your experimental group is someone else's experimental group: Similarly, external competitors may have more intensive activities that just happen to target your experimental group users, so the results can be imagined.
5. Intervention time lag: There is a certain lag from identifying users, applying for coupons to the final SMS delivery. If the system does not determine whether the user has placed an order, the user will receive the coupon only after placing the order. On the one hand, it will affect the experience. On the other hand, the user may cancel the order and then place another order, resulting in an increase in fulfillment costs. If users who have placed orders are excluded before sending text messages, the actual intervention will be on people with relatively weak purchasing tendencies, which will lead to inaccurate results. Therefore, the completeness of the marketing system and the coordinated execution of various departments are crucial.

Conclusion

The real world is complicated and it is not easy to unravel it and get a clue. It is always better to experiment than to stand still. In practice, we can learn to avoid one pitfall after another.

We are all trying to build a framework, explore repeatedly within this framework, and find a possible clue. Eventually, these clues will weave a network.

Source: Moji Data Products