Growth Case丨Designing a growth experiment for Keep

Growth is a process of continuous experimentation, just like product iteration. The most scientific approach is to ask for data from iteration and iterate based on the data. The same is true for growth, that is, data drives growth.

Strategy and growth have their own methodologies, but when abstracted, they actually express similar meanings.

Discover problems through data → propose hypotheses → design solutions → obtain more data → verify hypotheses → discover problems with finer granularity → propose more accurate hypotheses.

It is a step-by-step process of countless sets of closed-loop logic. According to the actual results and costs, it is decided which step can meet the goal or stop loss in time.

Each set of methodology has its own logical closed loop and different applicable scenarios. There are also confusions that still exist during execution. The most appropriate approach is to disassemble the output methodology, understand and digest it, and then output your own methodology based on the confusion you encounter.

Growth experiments are usually conducted in the form of A/B testing. It is important to understand that A/B testing is only a means, not an end, and it can also be used in daily life.

2. How to identify problems before designing a growth experiment

2.1 Have a clear goal (find data based on the goal)

Having a clear goal means that the source of demand for initiating the experiment is not the problem discovered after analyzing the data, but a clear and measurable goal proposed by other sources of demand.

For example, the boss assigns a task to increase GMV by 30% this quarter. Then the key indicator can be defined as 2020Q1GMV, and related indicators can be obtained from the already established growth model or formula.

GMV equals traffic * conversion rate * average order value. Traffic includes natural traffic and channel traffic. Natural traffic includes the product’s own APP, mini-programs, etc., and channel traffic includes various external channels.

Select the corresponding particle size analysis according to the actual situation.

2.2 No clear goal (find the goal based on data)

Having no clear goal means that the source of demand that prompts the experiment is not focused on a specific goal. We want to achieve the ultimate goal of improving the overall effect by optimizing a certain content that currently needs to be optimized.

At this time, in order to avoid making plans on the spur of the moment and designing low-quality experiments that waste resources, we need to use some scientific methods to find relatively reasonable goals.

a. Collect user feedback

Collect user feedback through product user feedback portals, app store reviews, social platforms and other channels.

By filtering, organizing, and analyzing user feedback, we can identify growth-related issues and design growth experiments with the goal of solving these issues.

User feedback is relatively random, and it is difficult to determine the impact. It has no direct relationship with the overall product, so other means may be needed to further verify it and focus on the order of priorities.

b. System monitoring

In a system with a more complete data system, problems can be discovered through monitoring.

Setting monitoring indicators and alarm rules in advance is a one-time solution for a period of time, which improves the efficiency of problem discovery and reduces the cost of demand analysis.

For example, the average daily retention rate of a mature community product is 30%. If the daily retention rate is within the normal fluctuation range, we can focus our energy on other places. If the retention rate is not within the normal fluctuation range, it means that an abnormality has occurred (generally referring to a low retention rate).

First, we need to determine whether it is really an abnormal situation based on the actual business situation. If it is determined to be an abnormal situation, we may use growth experiments to bring the abnormal data back to the normal range; if it is not an abnormal situation, we may need to adjust the alarm rules.

For example, it is usually normal for K12 products to have lower activity on weekends compared to Monday to Friday, or for ticket purchasing products to have lower activity after a holiday.

c. Regular review

The review method can be concretely expressed as effect regression. Effectiveness regression is to verify whether the goals have been achieved in the previous growth actions. If it is achieved, explore the space for further optimization; if it is not achieved, analyze the reasons and try to achieve the goal through the next growth action.

3. Growth Experiment Case丨KEEP Community

3.1 Experimental Background & Purpose

(1) Increase users’ attention to other users

• It is known that the average daily PV of the current dynamic details page (dynamics, articles, videos) is 10,000
• It is known that the current attention rate of dynamic details pages (dynamics, articles, videos) is 5%, and the goal is to increase it to 6%
• Statistical significance must reach 95%

Statistical significance is not discussed in depth here. There are specialized statistical tools on the market that can be found through search engines.

(2) Preparation before the experiment - defining indicators

【(Year/Month/Day) Follow Rate】Definition: Cumulative number of followers of a user (Year/Month/Day) / Cumulative number of other users who viewed content from the user (Year/Month/Day). (Deduplicate multiple pieces of content produced by a user)

[Browse] definition: The corresponding content is exposed 60% on one screen. Stay on the page for ≥5s.

3.2 Growth Experiment

Step 1: Formulate an experimental hypothesis

Assumption 1:

【If】Add a follow button to the community feed

[Expected] Monthly attention rate can be increased by 20%

【because】

• The community feed is the first-level page with the lowest bounce rate, shortening the conversion path.
• The core goal of the feed flow is to attract users to immerse themselves in the experience for a longer period of time. Currently, following requires going to a deeper level of [Add Friends] or [Dynamic Details], which interrupts the user's flow to some extent. Buttons placed on the feed can make the experience smoother.
• Users take more actions to follow a user, and subconsciously they will regard following as a relatively important operation. Putting the follow button in the feed stream means that users do not need to filter in [Add Friends] or view more content in [Dynamic Details]. They can determine their willingness to follow just by one dynamic in the feed stream, which refines the following scenarios and satisfies users who pursue diversity and do not attach much importance to following. This is also the direction in which the product needs to subtly influence the user's mind.

Best Practices:

You can follow the publisher directly in the Tik Tok feed

Quantitative analysis

• The number of users who clicked to follow in the [feed stream > dynamic details > feed stream > dynamic details] (loop) path
• After a user likes a post from a non-following user in the feed, the number of people who follow that user

Assumption 2

[If] Add logic: Like is automatically followed by default, the personal center can set whether to like or follow at the same time

[Expected] Monthly attention rate can be increased by 30%

【because】

• Similar to the previous point, the product needs to guide users not to regard following as a heavy operation, so that users can pay more attention.
• The meanings of likes and follows evolve differently in different community operations. The configurable logic of whether to synchronize likes with follows gives the choice back to users and meets the needs of various user groups with different understandings of the two.
• For users who decide to follow based on a post in the feed, they currently need to complete three steps (like, click on the post, and follow) to complete the like + follow. After using the hypothesis, you only need to click one action, and when you perform multiple operations, the cost can be reduced and the efficiency can be increased exponentially.

Best Practices

Generally speaking, shortening the conversion path can increase conversion rate.

Weibo has a similar logic, attracting users to follow publishers through content.

Quantitative analysis

• The number of users who clicked to follow in the [feed stream > dynamic details > feed stream > dynamic details] (loop) path
• After a user likes a post from a non-following user in the feed, the number of people who follow that user

Assumption 3

[If] Open the permission for visitors to browse the community and follow users and topics. (After logging in, choose whether to synchronize the data you follow, and synchronize according to the terminal)

[Expected] Monthly attention rate can be increased by 5%

【because】

• Users do not need to log in to browse, shortening the conversion path
• However, for a product, user login represents the most basic interest and trust. Users who are always in the guest state are not many and are not very active.

Best Practices

For tourist users, you can also create stickiness and silent costs for them, show sincerity, give users value first, and seek future registration from users.

Curiosity Daily, you can browse, collect, participate in research, etc. when you are not logged in.

Quantitative analysis

• Downloads
• Login page UV
• Number of logged-in users

Assumption 4

[If] Similar to Weibo, after successful login, guide users to fill out a simple user questionnaire (fitness-focused content) and recommend a form to follow based on their personal circumstances.

[Expected] Monthly attention rate can be increased by 25%

【because】

• Community apps in various fields such as Weibo, ZCOOL, and Haohaozhu have educated users on habits and are more receptive. User questionnaire feedback can be applied to content recommendations and push strategies to bring more accurate promotion.
• However, according to Keep's business model and the current first-level page C position (sports), it can be clearly seen that the core of KEEP is still a tool attribute;
• Therefore, whether to increase the weight of the community requires strategic considerations.

Best Practices

Weibo's various guided attention and automatic attention mechanisms need to weigh the differences in selective reference.

Quantitative analysis

• User segmentation: The ratio of time spent browsing content to time spent using sports functions is greater than 2:1

Step 2: ICE model scoring and prioritization

Priority: Assumption 1 > Assumption 2 > Assumption 4 > Assumption 3

Scoring Basis - Expected Impact

Scoring basis - probability of success

Scoring basis - ease

Step 3: Experimental Design

To avoid duplication of content, a new experimental hypothesis is introduced here to explain the subsequent content

1) Experimental hypothesis

[If] the user stays in the dynamic details for more than 5 seconds, the follow button will become brighter and larger.

[Estimate] Monthly attention rate can be increased by 20% (5%→6%).

[Because] staying can reflect the user's interest in the content, and thus indicate the interest in the publisher; at 5 seconds, the user's interest is relatively strong but not completely immersed in the content. At this time, the visual change of the follow button can both remind the user to pay attention without being overly intrusive and causing disgust.

Best Practices

(Same as Hypothesis 2) Weibo has a similar logic, attracting users (strong reminders) to follow the publisher through content.

Quantitative analysis

• The number and proportion of users who follow publishers after staying in article details and dynamic details for more than 5 seconds.
• The average time users spend on a publisher's details when following them.

2) Experimental indicators

【Monthly attention rate】Definition: the total number of users followed in a month / the total number of other users who browsed the content of the user in that month. (Deduplicate multiple pieces of content produced by a user)

[Browse] definition: The corresponding content is exposed 60% on one screen. Stay on the dynamic details page for ≥5s.

Because the changes in the experiment are on three pages in the dynamic details (dynamics, videos, and articles), the data comparison before and after the experiment only counts the differences in this part of the data to ensure that it will not be interfered by other factors.

Core indicators: (the three pages above are numbered 1, 2, and 3 respectively) Page attention rate in January, page attention rate in February, and page attention rate in March.

Reverse indicators : page unfollow rate in January, page unfollow rate in February, page unfollow rate in March.

[Unfollow Rate] Definition: The number of users who followed and then unfollowed the corresponding page / the number of users who followed the corresponding page.

Auxiliary indicators: (The above three pages correspond to 1, 2, and 3 respectively) Page 1 focuses on the average stay time, page 2 focuses on the average stay time, and page 3 focuses on the average stay time.

[Duration/Attention Ratio] The average length of time a user stays on a page to pay attention to it.

Compared with the 5s of the experiment, measure the impact of the button style change at the 5s time point on the user's final attention.

3) Experimental audience

• Users with attention habits
• No specific user habits
• Users with no attention habits

[Users with following habits] Definition: Number of followings/Number of active (entering the community) days ≥ 1 (one new following for every day of active)

[Users without specific habits] Definition: 0.3＜Number of followers/Number of active days (entering the community)＜1

[Users with no following habits] Definition: Number of followers/Number of active days (entering the community) ≤ 0.3

4) Experimental Design - A/B Testing

a. Version division and reasons

3 versions:

• Breaking down the details at the strategic level, we can consider 27 combinations (3 groups of users, 3 pages, 3 styles), and then evaluate the operational costs. In this case, only the article detail page is considered.
• Styles with large changes correspond to strong reminders and focus on target conversion. For example, the button becomes 50% larger, the color changes significantly, and the animation is very vivid.
• Styles with small changes correspond to weak reminders, focusing on experience continuity. Style example: the button becomes 20% larger and the color changes slightly.

(The divergent thinking in this stage is only for demonstration of ideas, and the calculation of the next two times does not take into account the disassembly user groups involved in the middle)

b. If the monthly attention rate of article details is increased to 6%

It is known that the current attention rate is 5%, the goal is to increase it to 6%, the statistical significance needs to reach 95%, and the article details have a daily UV of 10,000.

• [Users with attention habits] [No reminder]
• [Users with attention habits] [Weak reminder]
• [Users with attention habits] [Strong reminder]
• [No specific habit user] [No reminder]
• [Users without specific habits] [Weak reminder]
• [Users without specific habits] [Strong reminder]
• [No habit of following users] [No reminder]
• [Users with no attention habits] [Weak reminder]
• [Users with no attention habits] [Strong reminder]

Traffic distribution per group: 10000/9≈1111

The remaining places are all allocated to [Users with attention habits] [No reminder] to reduce the impact on active users before the experiment reaches a conclusion.

Experiment time

Calculated by the tool, it is: 6900/1111≈6.2 days, rounded to the integer of 7 days.

5) Analyze the experimental results

Experimental group: is the group of subjects that receive the experimental variable treatment.

Control group: For the experimental hypothesis, it is the subject group that does not receive the experimental variable treatment.

Focus:

• Whether the traffic of the experimental group and the control group is evenly distributed and whether enough samples are obtained
• Is the experimental period long enough? Is it affected by weekends and holidays? Is it affected by the user's novelty?
• Are all user types (high, medium, and low frequency) covered in each group?
• Compare the core indicators of the experimental group with those of the control group to see whether they have improved, whether the auxiliary indicators have improved or remained the same, and whether the reverse indicators have improved and caused negative effects.
• Whether the core indicators meet the statistical significance requirements (third-party statistical tools can be found in search engines)

6) Next step of the experiment

Start a new cycle

Author: Murasakihara Shinnosuke

Source: Xiao Zi original product notebook