Scenario restoration: A music APP offers a 7-day membership bonus to new users upon registration. After registration, users can decide whether to click to receive the bonus for a period of one month. Question: How would you evaluate this activity?

1. Common mistakes in activity evaluation

First of all, remember that for all questions that use evaluate/evaluate/judge as verbs, there is only one answer: "good or bad" .

For example, how would you evaluate this activity? You can answer:

1. This activity is very good and should be continued;
2. This activity is not good and cannot be done
3. This activity is neither good nor bad, it's useless
4. This activity has not changed anything, so it is useless to do it

This is the core conclusion of the evaluation analysis. Without these four sentences, everything else is nonsense. for example:

• 40,000 new users registered during the event
• The number of registered people during the event was 10,000 more than before the event
• The click-through rate of new users during the event was 80%
• During the event, new users will enjoy a 30% interest rate

These are not conclusions, but just analysis processes. If you don't have a conclusion and just throw out these process indicators, you will easily be asked back by the business side: "So what? So what? What did you analyze? Where is the conclusion!" In the end, you will be embarrassed.

II. Key issues in activity evaluation

To evaluate an activity, we must first come up with a good/bad evaluation . If the evaluation is good, then see if you can continue to do it and how many times you can do it; if the evaluation is bad, then see where the problem is, whether it is hopelessly bad or can be saved and continued to use.

Data + annotation = judgment. Therefore, to make a good/bad judgment, you need two things:

1. Clear assessment indicators.
2. Clarify the target value.

If the standard is met, it is good; if the standard is not met, it is not good. It’s that simple (see the picture below).

It seems simple, but in fact, operations often do:

1. Confused: I do it because my boss tells me to do it, why? I don’t know either, and I don’t dare to ask.
2. Dai Toudainao: I just want to recruit new people, and that’s all, OMG!
3. Taking shortcuts: It has been done before/others are doing it, so just do it.
4. Fishing in troubled waters: This is about changing users’ mental resources, which cannot be measured by data!

In short, only two out of ten operators can accurately describe the current situation and goals. At this time, data analysts need to have independent judgment ability. Only by analyzing business logic and sorting out business processes can we draw objective conclusions. Here we take a completely confusing scenario as an example to see how to help operations clarify their purpose.

3. How to build an evaluation model from scratch

The first step is to sort out the activity process

Operational activities will change user behavior, which in turn will be reflected in changes in data indicators. When building an evaluation model from scratch, the first step is to understand the specific process of the activity and the changes in user behavior that may be caused by the activity. For example, the new user benefits in the question can be sorted out as follows (as shown below):

After understanding the behavioral changes, we can further look at what data can be used to record these behaviors and what indicators can reflect the changes. After sorting out, we can see clearly: the indicators for measuring the results of the activity . These tasks should be done by operations during the planning stage. If they are not done well beforehand, they will have to be made up afterwards.

Step 2: Screening the main indicators

An activity may affect many aspects. For example, in the question above, there is a gift giving event. You said:

1. Can increase new user registrations - no problem;
2. Can increase the chances of membership purchases - seems to have a chance
3. Can increase loyalty and reduce churn - seems to make sense
4. Can increase DAU! ——Well, theoretically, if there are more new registrations, DAU will also increase.

If you don't look at the data and just listen to what people say, of course the above makes sense. But if we really want to throw everything into a tumble, and ask you to calculate the users who have not left, what percentage of them are free memberships, what percentage of them are the products themselves, and what percentage of them are the number of songs, it would be a miracle if you can figure it out clearly. Therefore , the evaluation indicators should be prioritized to make it easier to clarify the problem.

If the goal is set in advance, the main indicators of the activity should be closely linked to the goal, and the indicators that are directly affected should be preferred . For example, if the activity is to attract new users, then the main indicator is the number of new registered users; if the activity is to increase the retention rate of new users, then the main consideration is the retention within 1-7 days.

This seems simple, but in fact it is easy for the operator to take advantage of the situation. Operations often like to talk about a bunch of influencing indicators, and even say things like "my activities have changed users' mental cognition at a deep level, thereby achieving far-reaching impacts that cannot be measured by data." In short, they bring in a bunch of indicators, talk about the ones that look good, and don't talk about the ones that don't look good.

When doing data evaluation, the most taboo is to create hundreds of indicators and then make a hugely complicated evaluation formula. The more things that are mixed in, the more room there is for word games , and the easier it is to whitewash the situation. The simpler and clearer the assessment, the easier it is to identify the problem.

Step 3: Set judgment criteria

With clear main indicators, you can find the judgment criteria. There are four basic ideas for finding standards:

1. Start from the overall results and look at the total amount. For example, if you need 100,000 new users this month, you must achieve 100,000.
2. Compare with no activity and look at the increase. For example, if there is no activity, the monthly salary is 50,000, but if there is activity, the salary must be 5+50,000, with the extra 50,000 as the standard.
3. Compare with past activities to see the efficiency. For example, the activity of attracting new members usually costs 100 yuan per new member, so it cannot be exceeded this time.
4. Compare with non-participating users and see the difference. For example, divide the group into non-participation group/participation group, and compare the number of new registrations/retention rate of the participating group.

From the company's perspective, the first method is definitely the most practical. But from the perspective of organizing activities, everyone likes to highlight their own contributions, so they tend to use methods 2, 3, or 4. Operations like to talk about: natural growth rate (what is the natural growth without any activities). And there is always a tendency to calculate the natural growth rate too low, or even make it a negative number, so that the activity appears to be extremely impressive.

It should be noted that the so-called natural growth rate can only be calculated when activities are not frequent. For many businesses (such as e-commerce and O2O), there are constant activities, with large activities embedded in small activities that are impossible to distinguish. This is not applicable at this time.

Similarly, the premise of setting up a reference group is that the reference group has no activity reminder and activity participation functions at all, and the quality of users in the reference group and the activity group is similar. If this restriction is not met, it is difficult to directly conclude that the activity is effective. These methods seem scientific, but in fact they are full of room for tricks . If you want to discuss the issue clearly, just set the standards simply and directly.

Step 4: Find the process that affects the results

The above steps 1, 2, and 3 are all to pave the way for reaching a judgment. There is a "good/bad" judgment. We can further analyze why it is good and why it is bad. At this time, it is necessary to refine and sort out the business processes and find the key points that can affect the results. For example, new customer acquisition activities, advertising channels, advertising copy, registration process, how to be prompted for benefits after registration, and the process of obtaining membership benefits will all have an impact (as shown in the figure below).

When analyzing these indicators, pay attention to the order of priority. For example, issues related to new user registration. We must first look at the delivery strength of each user source channel, whether the activity is launched in a timely manner, and when it is combined with the delivery. Only then will we conduct an in-depth analysis of the copy, event gifts, post-receipt behavior, etc.

IV. Summary

If you use this question in an interview, you can tell the difference between rookies, novices, and veterans at a glance:

Cainiao: I think we should consider four major aspects: user growth, product experience, brand influence, and commercial revenue, blablabla. Then you can seize the opportunity to ask him about the brand impact and how to measure it, and you can basically kill him with this question. The biggest gap between beginners and novices is that they don’t even know that data needs to be collected . Newbies love to come up with some hard-to-collect, but impressive-sounding metrics out of thin air.

Newbie: I think we should consider the number of new registrations, the retention of new users on the next day, 3 days, and 7 days, the membership collection rate of new users, the retention market of new users, the number of clicks of new users, the number of new users who purchase memberships, the frequency of new users using their rights and interests... (more than 100 various indicators are omitted here). Establish neural network model and conduct comprehensive evaluation.

Yes, newcomers like to pile up indicators first, and it seems that the more indicators they have, the better. Second, I like to talk about models. No matter whether the models can be used or not, or whether I have used them myself or not, I think it’s awesome if I talk about them . The only thing they ignore is the issue of standards . If you want to criticize the newcomers, just ask them “Why is a growth of 50,000 good, but 49,127 is bad?” “When you build a neural network evaluation model, who will label the good/bad activities? Should you label the entire activity or certain indicators? Why should different types of activities be labeled together?” These questions will basically confuse the newcomers.

After all, activity evaluation is a matter where clear thinking is more important than advanced technology, and maintaining integrity is more important than clear thinking . As long as you have a basic understanding of common operational activities: attracting new customers, promoting activation, retention, conversion, fission, etc. If you have an understanding of regular operating data and an understanding of past activities, you can basically give the correct answer. The fear is that you know too little about the business process and don’t even understand what others are doing.

Author: Down-to-earth Teacher Chen

Source: Down-to-earth Academy