How to iterate community operation model through A/B testing?

Whether we are developing products or operating operations, we will go through some iterative processes of hypothesis-test-verification-definition on a daily basis. Only through continuous iterative exploration and experimentation can we find more growth points. This article mainly talks about how to iterate the community operation gameplay model through A/B testing. Let’s take a look~

Article Outline:

1. User growth theory
2. AB test operation process
3. Product iteration

When we are doing growth now, such as attracting new customers, we may learn from others' methods or come up with an idea on our own. We may not necessarily do it directly without verifiable logical support or whether it is suitable for your products and users. The result is that your activities are generally not very effective, and you cannot produce a stable operational strategy. This requires us to develop our own growth methods that can be sustainably iterated.

Below I share a case I worked on, and I look forward to communicating with you and learning more growth thinking models.

1. User Growth Theory

Whether we are developing products or operating operations, we will go through some iterative processes of hypothesis-test-verification-definition on a daily basis. Only through continuous iterative exploration and experimentation can we find more growth points.

User growth methodology: determine the North Star indicator - break it down into specific and actionable optimization scenarios and indicators - plan and sort A/B testing experiments - conduct more exploration and experiments on effective directions.

Next, I will give an example of a real case, an iterative experiment in which I personally participated, to illustrate the above sentence. I hope that everyone can understand from the case that this is what growth and iteration are all about.

First, let me introduce the background of the event:

We have a small team responsible for the CET-4 and CET-6 projects for college students. We attract college students by promoting free traffic courses, and then promote other high-priced courses on our platform. It has been verified in the early stages that adding community services can increase the completion rate and user renewal conversion rate. Therefore, our future circle classes will add community services. Next, we need to continuously iterate and test the community services to improve the efficiency of community operations.

We have sorted out that our North Star indicator is to increase user value (value of a single class = total turnover/number of class leaders), and the total turnover is related to the number of people who renew their classes. Further breaking down the number of people who renew their classes is related to the number of friends and the class-taking rate, and the class-taking rate is related to the improvement of user trust in the community's services. The community’s services include: answering questions, live broadcasts, and learning reminders. Therefore, what we ultimately want to increase is the number of users who re-enroll in courses in scenarios such as Q&A, live broadcasts, and learning promotion.

You can directly look at the flowchart of this North Star indicator, as shown below:

By breaking down the North Star Indicator, we learned that the original operational action to be implemented was to increase the proportion of participants in community services. We began to sort out the entire operational process, and then we found some variables that might affect our final North Star Indicator. At this time, we need to do AB test verification and fix a variable to test the positive and negative impact on the North Star Indicator.

2. A/B test operation process

When we were sorting out the operational processes, we found that two variables appeared in the user learning behavior process: the academic system (classes started on a fixed schedule and on-demand learning) and grouping (with grouping and without grouping). The scheduled class starts and grouping have relatively high personnel operating costs. Whether to cancel them or not, how much impact these two variables have on the course completion rate and the value of a single class entry, all require us to conduct experiments to verify.

Let’s take one of the AB tests as an example:

(1) Clarify the purpose of the trial

For example, one of our experiments is to compare the impact of scheduled class start and on-demand learning modes on the value of a single class. If the value of a study-as-you-enroll model is similar to or higher than that of a single-entry class with scheduled classes, then this model can be adopted directly to reduce labor costs, and vice versa.

(2) Set up an A/B control group

Verification method:

• Control group A: users who receive the regular class topics of the Level 4 vocabulary circle class;
• Control group B: users who registered for the Level 4 Vocabulary Circle class and received the on-demand learning special topic.

In order to ensure the accuracy of the test, the two samples A and B need to meet the following conditions:

1. The user scale is similar, but not too small, at least 600 or more, and must be able to represent global characteristics.
2. User attributes are similar and representative. For example, our target users are mainly college students.
3. The test environment can represent global characteristics. For example, the previous user course path is used, only one variable is retained, and other environments are the same.

Develop a community operation AB testing process plan to determine the course renewal contact method and user renewal path, as shown below:

(Community operation ABtest process plan)

It is very important to sort out this process plan and schedule your entire operational actions into it to help you follow your own operational rhythm.

Now, have we completed the preparations before the launch of the event and are now waiting to see the results? There is still a very important job to do.

Data collection requirements:

After the test plan is determined, before the activity goes online, it is very important to raise data requirements with BI to ensure that the data can meet the following points. Otherwise, if you raise requirements with data experts after the experiment is over, you may end up in an awkward situation where the data cannot be obtained.

Ensure that activity data can meet:

1. It can evenly divide independent users into two sample groups A and B with consistent attributes. For example: Our courses are divided into two topics, and users are divided into A and B.
2. Each independent user can be identified, which means that the user data of each sample can be tracked. For example: the data of the user's path from taking a course - adding a personal account - learning behavior - renewing the course can be tracked.
3. The variables of A and B can be adjusted online. For example: One of our experimental variables this time is the different school systems. We directly designed two class start models on the special page: study on demand and start class on a scheduled basis.
4. It can separate the data differences between A and B caused by variables. For example, for the renewal of two sample users, we set up course links with sources respectively, so that we can obtain the number of renewal conversions from different channels under each sample.

After completing the above preparations, you can start the activity online and then look at the final data results.

(3) Analyze data

After the event is over, based on the conversion cycle of renewed courses, we can start looking for BI to pull data 3 days after the event ends. Still using the above example, let's analyze the data of the two AB tests:

Comparison of learning behavior data (the following data has been modified, not real data, for demonstration purposes only):

User learning behavior data (retention and completion rate)

From the data, we can see that the 7-day TAD (7-day cumulative active days) and completion rate of users in Group B's scheduled class start mode are significantly higher than those of Group A's drop-in study system; the 7-day TAD and completion rate of Group A with grouping are significantly higher than those of Group B without grouping.

Comparison of single-class value data:

(Continued reporting of conversion data)

From the data, it can be seen that the value of a single class (total turnover/users taking the course) in Group A's study-as-you-go model is significantly higher than that in Group B's scheduled classes; the value of a single class for users in Group A's grouped model is significantly higher than that for users in Group B's scheduled classes.

(4) Conclusion

Looking back at the North Star Indicator of our community operations, the single-class value of users is higher under the two academic systems of enrollment and grouping, and the subsequent community operation process can continue to be used.

3. Product iteration

The above conclusion is that the community gameplay model that adopts on-the-spot learning and group PK is more effective. However, since the manual operation cost of group PK in WeChat groups is very high, if the community user scale is large, it is necessary to adopt a product-side grouping model to free up manpower and improve community operation efficiency.

In addition to using experiments to verify which learning model is more effective for users, we can also do more other experiments, such as setting up a control group to test which community conversion model is more effective, and continuously iterate and optimize to improve operational efficiency.

Source: Li Junlong (Alger)