A growth experiment that increased product conversion rate by 30%

The steps for conducting growth experiments are generally divided into: generating experimental ideas, sorting experimental ideas, designing experiments, developing experiments, analyzing experimental results, and producing growth reports. We will introduce them one by one below.

1. Generate experimental ideas

1. Clarify the goals of the experiment

We conduct growth experiments based on what problems we have discovered.

Is the conversion rate of a certain button too low? Or is the retention rate of a certain feature too low? Only after finding the problem can we know the direction of optimization. For example, if we find that the conversion rate of the registration function is only 30%, then we can set our experimental goal as: to increase the conversion rate of the registration function from 30% to 50%. Below we will take the case of low conversion rate of registration function as an example.

2. Find the cause of the problem

In order to achieve the experimental goal, we need to find out the cause of the problem.

We can find the cause of the problem through both qualitative and quantitative methods.

On the qualitative side, we can use user surveys, user usability studies, and other methods to understand why the registration function conversion rate is so low.

On the quantitative side, we can gain insight through data. For example, can we use the funnel model to see where the user churn rate is the highest? Through user segmentation, can we understand which users have a high registration conversion rate and which type of users have a low registration conversion rate, etc.

3. Find solutions based on the causes

Knowing the cause of the problem makes it easier to find a solution.

We can find solutions through qualitative, quantitative and best practice methods.

I won’t introduce qualitative and quantitative methods, as they are similar to “finding the cause of the problem”.

Let’s talk about best practices here. Best practices are those methods that are generally recognized as likely to increase conversion rates. For example, B=MAT model and Lift model.

In the B=MAT model, if we want to improve the user's conversion rate behavior, we need to give users enough motivation, reduce user resistance and improve ability, and provide trigger reminders in appropriate situations.

The Lift model is mainly divided into value proposition, relevance, clarity, anxiety, distraction, and urgency.

• Value proposition. There must be a clear and powerful marketing slogan so that users can accurately perceive what benefits they can get.
• Relevance. Landing pages and conversion pages meet user expectations and are closely linked to your value proposition.
• Clarity. The experience process is smooth and users know clearly what to do next.
• Feeling of anxiety. Do subtraction and don’t give users too many choices; don’t do things that don’t meet user expectations and habits.
• Distracted attention. Reduce visual infection and information noise.
• Urgency. Create a sense of hunger to make users make quick decisions, and be good at utilizing the endowment effect and loss aversion.

Here we use the above method and assume that the solution to the "low conversion rate of registration function" is as follows:

1. Shorten unnecessary registration processes on the page, for example, do not require users to fill in personal information such as email addresses when registering. Just enter your mobile phone number and verification code to log in and register.
2. Highlight the Register button.

4. Forming an Experimental Hypothesis

Finally, we need to form our experimental hypothesis based on the previous steps. Here I would like to share with you a template for experimental hypothesis.

“If [a specific change is made], it is expected that [a certain indicator] will improve by [X%] because [underlying reasons, supported by data]”

Based on the above, our experimental hypothesis can be as follows:

1. If [the user is asked to enter personal information such as email address when registering], it is expected that the [registration conversion rate] can be increased by [30%], because [through funnel analysis, nearly half of the users are lost when entering personal information such as email address].
2. If [the registration button is increased to 1.3 times its original size], it is expected that the [registration conversion rate] can be increased by [10%] because [the core process is highlighted].

2. Sorting Experiment Ideas

ICE helps us prioritize our experimental ideas. ICE is divided into expected impact (Impact), probability of success (Confidence), and ease (Ease).

• The expected impact refers to how many users this experiment will affect; if the experiment is successful, how much the indicator is expected to improve.
• The probability of success refers to what data or evidence is available to prove that the experiment will be successful.
• The ease refers to how much manpower, material resources and other resources we need to invest to complete the experiment.

Assign scores of 0-10 to each of the three indicators. The final comprehensive score = expected impact score + success probability score + ease score.

3. Design the Experiment

1. Selected Experimental Indicators

Here we need to pay attention to three types of indicators: core indicators, auxiliary indicators, and reverse indicators.

• Core indicators are the key indicators that determine the experiment and are also the most direct indicators. In this experiment of optimizing registration conversion rate, what we need to focus on is whether the registration conversion rate has increased/decreased.
• Auxiliary indicators are indicators that may be indirectly affected by the experiment. For example, what impact does optimizing the registration conversion rate have on daily activity and activation rate?
• Negative indicators are those that measure the negative impact of an experiment. For example, page exit rate, order cancellation rate, app uninstall rate, etc. This experiment of optimizing registration conversion rate does not involve reverse indicators.

2. Identify the audience for your experiment

What kind of people are we targeting in this growth experiment? We can group them by operating system, browser type, region, source channel, etc. For example, in the experiment to increase registration conversion rate, our audience is new Android users who come through Baidu search.

When we do A/B testing, we need to have an experimental group and a control group. Here we assume a premise that all new Android users are found through Baidu search.

Suppose we conduct an experiment here to remove personal information such as email addresses when new users register.

• Experimental group: users who used the modified version.
• Control group: users who used the version before modification.

The advantage of doing this is that it avoids the influence of other variables on the experiment.

3. Estimating the trial sample size

After clarifying the audience, we also need to know how much sample size is needed to prove that our experiment is valid.

Here we need to introduce the concept of "statistical significance". Statistical significance means that any differences between the attitudes of the two groups are due to systematic factors rather than chance. If we use A/B testing to do this experiment and find that there is a difference in conversion rate, when the statistical significance is 95%, it means that there is a 95% probability that the difference in conversion rate is real, and there is a 5% probability that the difference is caused by error.

It is generally recommended that the statistical significance be above 95%.

Assuming that the registration conversion rate of our previous version is 20%, the registration conversion rate of the new version is 25%, and the statistical significance is 95%, we need to allocate at least 670 samples to each version.

If we want to increase the statistical significance to 99%, we need a sample size of at least 700.

If you need this tool, just reply “growth tool” on my official account to get it.

4. Estimated duration of the experiment

Now that we know the sample size, we need to estimate how long the experiment will last. For some experiments, the true results of the experiment may be seen in a very short time, but for some experiments, it may take more time to get the true results.

Here we use a public notice:

Experiment time = total number of samples required / number of page visits per day

Assuming that for the experiment of "removing personal information such as email address during registration", we need a sample size of 1400 (700 samples for the new and old versions respectively), and the number of people visiting the registration page every day is 140. Then the experimental duration = 1400/140 = 10 days.

4. Development Experiment

First, we need to bury the data.

If you are using a third-party A/B testing tool, we need to determine the experimental indicators (described above), find the user behavior corresponding to this indicator, and then write down our tracking requirements. The data is automatically returned to the A/B testing software, and the A/B testing software automatically calculates the experiment's metrics.

If you are manually analyzing the experiment, you also need to confirm the experiment indicators and then establish the user behavior. After the technology is developed and the tracking points are completed, the data will be returned to the database, and we can then export the data for analysis.

Secondly, the buried points must be tested. Ensuring our data is reliable.

Finally, we develop and launch the A/B test version.

5. Analysis of experimental results

1. Credibility of evaluation results

We can input the number of visits and conversions for each version into the tool, and the tool will help us calculate the statistical significance. If there is a significant statistical difference, it proves that our experimental results are reliable.

Suppose our control group has 1,000 visits and 60 conversions; the experimental group has 1,000 visits and 150 conversions. When we input this information into the tool, we concluded that this experiment has a significant difference, which means that the results of this experiment are credible.

2. Focus on three types of indicators

We need to pay attention to the core indicators, auxiliary indicators and reverse indicators mentioned earlier.

• Core indicators: Are the core indicators of the experimental group improved compared to the control group?
• Auxiliary indicators: Do the changes in auxiliary indicators meet our expectations and are they consistent with the trends of core indicators?
• Contrary indicators: Are there no obvious negative impacts and can we accept the change in the contrarian indicators?

3. Observe four states

• If the indicators of the experimental group are significantly improved compared with those of the control group, the experimental group wins.
• If the experimental group's indicators are slightly improved compared to the control group, the experimental group wins.
• If the experimental group's indicators decrease compared to the control group, the control group wins. After the change, the indicator does not increase but decreases. At this time, we need to find the reason. We can find the reason by segmenting the funnel, user grouping, etc.
• If there is no significant change in the indicators of the experimental group compared with the control group, it means that the control group wins. If the indicators do not change, there is no need for us to optimize. Just keep it as it is.

4. Clarify the next step

1. The hypothesis of this experiment was successful. If successful, we can think about whether we can conduct more other experiments on this point? Can the ideas from this experiment be applied to other experiments?
2. This experiment assumes slight success. If the metric has only moved a small amount, we can iterate until we get a level we are happy with.
3. This experimental assumption fails. If our expectations are not met, we should reflect and review what we have learned from this growth experiment. And clean up the code.

6. Output Growth Report

Based on the above experiments, we can finally produce a growth report. The growth report mainly records the experiment's objectives, experimental hypotheses, experimental scores, experimental indicators, experimental audiences, experimental design, experimental results, experimental insights, and follow-up plans. This is of great help to us in accumulating knowledge and reviewing past events.

This is the end of the content sharing for now. We will continue to update the content about growth in the future, and we hope to communicate with you more.

Author: Fu Xiaohu

Source: Little Tiger Speak