How to discover user growth points through growth experiments?

Now that the traffic dividend has disappeared, refined operations are needed to achieve sustained and effective user growth. Growth experiments are a very important way to drive growth. Its function is to verify the causality between products, operational actions and growth data. A/B testing methods are often used to verify whether a solution can really drive user growth. Many product and operations personnel know about growth (experiment) hacking, but they still don’t understand or are not familiar with the specific operational applications. The following is an explanation based on actual cases.

First, we need to understand the entire process of growth experimentation.

Before conducting a growth experiment, we need to understand the process of the growth experiment in order to effectively promote the experiment. Many of my friends’ company teams usually make decisions based on experience and subjective judgment when doing projects, and implement them directly once a decision is made. This will cause the team to execute blindly, and it is easy to spend a lot of time on the project, but the final effect will not meet expectations.

1. The whole process of growth experiment

• To clarify the purpose (goal) of our experiment, we need to clarify the key indicators of our work at this stage, which are often referred to as the "North Star Indicator", to guide the direction of our work at this stage.
• Common growth model applications include full-chain funnel model and factor decomposition model. The growth model is that we need to find all the segmented indicators that affect the North Star indicator and connect these segmented indicators. This is the growth model we build.
• Find the focus area. The so-called focus area is to find the explosion point that can significantly increase growth through the constructed growth model. We verify the mid- to long-term (0-120 days) growth of this focus area and, through continuous experimental verification, we find the exact outbreak point and amplify it, ultimately achieving the effect of product growth.

1. Growth Experiment Process

Step 1: Generate Experimental Ideas

Starting from the gathering field, high-quality experimental ideas are generated. Help improve the success rate of experiments by gaining insights from data. This will form a clearer experimental hypothesis. and forming a library of experimental ideas.

Step 2: Prioritize

For various ideas in the experimental idea library, the experimental priorities are sorted through the sorting model.

Step 3: Design a Growth Experiment

Once you have an idea for a specific experiment, refine it into a feasible experimental plan (develop PRD). This step mainly talks about formulating experimental indicators, determining the experimental audience, and designing experimental versions to avoid missing important aspects that affect the experimental results.

Step 4: Experimental development and launch

After planning the data tracking, developing the experiment and launching it online, you can clearly measure the experimental results based on the tracking data.

Step 5: Analyze the experimental results

After obtaining the results, how to analyze the experimental results through a systematic method, evaluate the credibility of the experimental results and draw credible conclusions, and amplify the experimental impact of the conclusions.

2. Design Growth Experiments

The editor will tell you how to carry out growth experiments based on previous projects, which can help you better understand. The editor was once in charge of a smart parking public account growth project, which required increasing the number of fans to 14 million within one year. The goals were broken down, and my project team was responsible for improving the retention rate of the official account.

Experimental background & purpose:

Before designing the growth experiment, we analyzed the data and found that after users followed the official account to pay for temporary parking fees, most users would unfollow the official account within 1-2 days. Therefore, we need to improve user retention rate , but then the question arises: what are the factors and corresponding strategies that affect user retention rate? Our team brainstormed and came up with various ideas and hypotheses, hoping to verify them through experiments. This is the most basic idea behind our experiment.

1) Improve user retention rate for official accounts

It is known that the average number of people using the official account to pay for temporary parking is 100,000 per day

It is known that the current average number of followers of the public account is 1,000 people per day, and the retention rate is 60%. The goal is to increase it to 80%.

Statistical significance must reach 95%

2) Preparation before the experiment - define the indicators

[(Next day/7th day/14th day) retention rate] Definition: Number of followers of a user (next day/7th day/14th day) / Cumulative number of followers of a user (first day)

Core indicators: Next-day/7-day/14-day retention rate

Reverse indicator: next day/7 day/14 day unfollow rate

Auxiliary indicators: Assumption 1 focuses on the number of clicks on the function link, the length of time on the page, and the number of functions completed

Assumption 2: Number of people visiting the link, number of people receiving it, and number of people using it

3) Growth Experiment

Step 1: Formulate an experimental hypothesis

Assumption 1:

[If] Push product function introduction in text form after following the official account

[Expected] The retention rate of one day/7 days/14 days can be increased by 40%

【because】

Users passively follow the official account and complete payment through the official account in a very short time without knowing much about the product.

The core goal of following the official account is to establish connections with users and push more parking scenario services to users. If users cannot understand or experience the value that the product brings to them, following an additional public account will inexplicably cause them to be disturbed by messages.

Users have to spend more time to understand and learn the functional value of the product, and subconsciously they will regard attention as a relatively heavy operation. Pushing introductory information after following can shorten the user's self-exploration cost and help them understand the value of the product in a simple and clear way.

Quantitative analysis:

The number of clicks on the function links in the public account push information

The number of users using the public account function link

Assumption 2

[If] Push a link to receive parking tickets to users after following the official account

[Expected] The retention rate of one day/7 days/14 days can be increased by 60%

【because】

After users receive parking tickets, they will be forced to abandon their accounts and not cancel them easily.

3 days and 1 day before the parking ticket expires, the system will automatically trigger a message to remind users to use it, which will activate these silent users.

Parking tickets can be used multiple times by users, forming brand awareness in the minds of users, thereby increasing user stickiness.

Quantitative analysis:

The number of parking tickets that users receive through public account message push

The number of parking tickets used by the user

Experimental audience:

Follow the public account user for the first time

4) Experimental design (AB testing)

The experiment was divided into 3 groups for testing, 1 group remained unchanged, and the other 2 groups were the first to pay attention to the product function introduction information and the link to receive the parking ticket respectively.

Experiment time:

30 days is 2 test cycles

Experimental results analysis:

Referring to the data in the above table, it can basically be shown that this strategy can effectively improve the retention rate of official accounts. Subsequent experiments can test whether parking tickets with different validity periods can significantly increase the length of time users stay. (Due to data confidentiality, the specific data will not be displayed)

Author:Ja

Source: Ja