How to quantify and analyze the effectiveness of operational activities

I believe that for many analysts who are just starting out, evaluating activity effectiveness and gaining insight into business opportunities are the most valuable aspects of their work, but they can also be the most troublesome.

1. Activity Background

As the growth rate of mobile Internet users is approaching saturation, the solution to user growth has to shift from attracting new customers to activating existing users.

Using third-party advertising media apps (such as WeChat, TikTok, etc.) to deliver materials targeting old users to promote user activation has become an effective method for many companies to increase the activity of existing old users (hereinafter collectively referred to as "channel activation")

The marketing department of a certain company also began to invest in the budget to test the "channel activation" project. After the project was launched for a period of time, a large amount of user data has been collected and accumulated, but:

• How much does channel activation contribute to increasing DAU?
• Is it worth continuing to invest more resources?
• What is the ROI of the activity? Does it meet expectations?

These issues, which are of great concern to leaders and business parties, require analysts to give fair and objective responses based on data.

2. Analytical Framework and Indicator System

1. Analytical framework

• Evaluation of the overall incremental effect of the activity
• ROI Calculation
• Quality assessment of active users

2. Indicator system

(1) Traffic scale

Data indicators:

• DAU
• The number of users who participated in the activity (for example, the number of users successfully recalled by channel activation)
• The first time the app’s UV is called up through an activity (for example, the first time the app’s UV is called up through a channel)
• The proportion of daily users who first call up the app through activities (for example, the proportion of daily users who first call up the app through channel activation)

Problems that can be solved:

• Observe whether the core indicator DAU of activation has reached the expected target;
• Evaluate the scale of users that can be recalled by channel activation;
• Assess the contribution to the net increase in DAU;

(2) User quality

Data indicators:

• Retention rate (next-day return visit rate, 7-day return visit rate, 30-day return visit rate)
• Average daily usage time
• Core function penetration rate
• Core function average PV
• Crowd profile (gender, city, spending power)

Problems that can be solved:

• Evaluate the quality of channel recall users
• Monitor whether there are cheating channels

(3) User Behavior

Data indicators:

• Off-site conversion funnel (for example: ad exposure-ad click-successful app call-deeplink to a specific page)
• Conversion funnel of core behaviors within the site (for example: activity page - list page - details page)

Problems that can be solved:

• Evaluate whether the user's path from off-site channels to the App is smooth, and discover product bugs or opportunities for improvement
• Evaluate whether the on-site hosting strategy for the activity is reasonable

3. Analysis Process

1. Activity effect evaluation and activity ROI analysis

When quantifying the contribution of DAU (or active days), it is necessary to subtract the user's natural activity, that is, to calculate the "net incremental" contribution. The contribution can be divided into daily contribution and long-term contribution.

• Daily contribution refers to the incremental contribution of the recalled users on that day to the DAU on that day.
• Long-term contribution refers to the increase in the number of user days that will continue to contribute within a specific subsequent time frame due to the subsequent return of recalled users. For example, after the event, the 50 participating users have an average of 10 more active days per person in the subsequent 30 days than before the event, so the incremental contribution of the activation is 1,500 days.

It has to be admitted that AB experiments are best at dealing with attribution and quantification issues. The idea is to randomly divide the traffic into two groups (i.e., control group and experimental group) with uniform quantity and characteristics. The users in the experimental group differ from those in the control group only in product strategy. Therefore, we can assume that the difference in indicators between the two groups of users in the same time dimension can be completely attributed to the difference in strategy.

However, it is impossible to design a corresponding AB experiment for this advertising activation project, but we can construct a user group that is "similar" to the experimental group as a control group based on the idea of ​​AB testing. The specific process is as follows:

1. The users who were called up by the activation channel were used as the experimental group, and the existing users who were not called back by the activation channel were used as the control group.
2. Select features that may affect users' future activity (such as model, new channels, historical activity, etc.), and divide the two groups of users into N pairs of experimental and control groups based on the principle of "same features". Pay attention to discretizing the features through intervals as much as possible to avoid too few samples falling into a certain group, which will lead to unreliable index differences between the two groups of samples. For example, the feature "new date interval" can be discretized into: within 7 days, 8-14 days, and more than 14 days;
3. Calculate the difference in indicators between each group of the experimental group and the control group, as well as the total indicator difference of the experimental group (equal to the sum of the product of the indicator difference of each group * the proportion of the population)

Through the above method, we can calculate the contribution of live streaming to the DAU of the day, as well as the total incremental contribution of live streaming to the DAU of the next 30 days.

In fact, there is a simpler method for the single short-term contribution of activation to DAU, which is based on the idea of ​​"first attribution" and quantitatively evaluated through "the UV that first calls up the app through activation." That is, if a user has launched the app multiple times, then the credit of the activation ad will only be counted when the app is called up for the first time through the activation ad.

It is worth mentioning that the first attribution method can also be applied to the quantification of the effect of "evaluation of new product functions". Usually, we can use "the number of users who access the function for the first time after launching the app" as the net contribution of the function to DAU.

For activity cost accounting, we can use "total cost consumption/total DAU increment" to calculate the cost of each DAU increment to evaluate whether the ROI meets expectations.

2. User behavior analysis and user quality assessment

You can use "general non-activation users", "similar activities in the same period" and "similar activities in the past" as comparison benchmarks, and based on indicators such as user behavior funnel, retention rate, core behavior PV, and average usage time per person, identify whether this activation strategy has channels for wool-pulling or serious cheating, and evaluate the quality of users attracted by the activity. But this is not the focus of this sharing, so I will not elaborate on it.

IV. Conclusion

As a data analyst, the activation strategies encountered in actual work are often varied, but the evaluation process of the effectiveness of the activities still follows certain rules. Finally, let’s briefly summarize the reusability of this article for subsequent activity evaluation:

• How to construct an indicator system for activity evaluation;
• How to quantify the short-term contribution of attributed activities (i.e., the “first attribution” approach);
• How to quickly quantify and evaluate the long-term incremental contribution by constructing a control group when AB testing is not possible;

Author: Hao Xiaoxiao

Source: Hao Xiaoxiao