User Churn Prediction Model, How to Evaluate Its Effectiveness?

The author of this article uses detailed examples to explain how to evaluate the effectiveness of a customer churn prediction model , as well as the purpose of a customer churn prediction model: to effectively retain and care for customers.

1. An important indicator: lift

An important indicator used to evaluate the prediction effect of the customer churn prediction model is the lift.

The so-called lift, in simple terms, is how much better the model is at predicting customer churn than not using the model.

As shown in the figure, customers are ranked from high to low according to the probability of churn. The points (10%, 50%) in the figure indicate that the top 10% of customers in terms of churn include 50% of the actual churned customers.

In other words, if a company has 3 million customers and the average churn rate is 1%, if it captures the top 10% of customers, it can actually capture 15,000 real churned customers (i.e. 3 million × 1% × 50%).

The blue line in the figure shows the results of randomly sampling customers without the guidance of a predictive model.

This line is actually very easy to understand. If 10% of the customers are sampled, 3 million × 10% × 1% lost customers can be captured, accounting for 3 million × 10% × 1% of the total actual lost customers ÷ (3 million × 1%) = 10%, so this line is actually a straight line with a slope of 45 degrees.

The red line in the figure represents the result predicted by the customer churn prediction model.

For example, the points on the line (10%, 50%) indicate that the top 10% of customers who churn include 50% of the actual churned customers. Therefore, for the top 10% of customers who churn, the effect of using the model prediction is 5 times that of not using the model prediction! This is called lift.

The red line in the above picture is the legendary ROC curve, the full name of which is Receiver Operating Characteristic Curve.

The blue line is the baseline. Generally speaking, the closer the ROC curve deviates from the baseline, the better the prediction effect of the model.

The degree of improvement is certainly important for judging the predictive performance of the model. However, people often only focus on the model prediction effect given by the lift, but ignore (or fail to evaluate) the "application effect" shown by the customer churn prediction model.

2. Focus on “application effect”

People are generally concerned about: With such a churn prediction model, or by carrying out retention care activities for high-potential churn customers under the guidance of this model, will the customer churn rate next month be significantly reduced?

This view is incorrect because the customer churn prediction model only reveals an objective law: "what kind of customers are more likely to churn."

The reality is that after using the customer churn prediction model, the customer churn rate often cannot be significantly reduced . The following is an example from the securities industry.

Suppose a brokerage firm A currently has 3 million customers and an average monthly churn rate of 1%.

In order to better establish a predictive model, only valid customers are modeled during the model development process. That is to say, before modeling, it is necessary to set certain conditions to eliminate non-valid customers, such as institutional customers, customers with extremely large or extremely small assets, customers with no trading behavior, etc.

In this way, the number of effective customers is 1.2 million, with an average monthly churn of 18,000 and a churn rate of 1.5%. Finally, Brokerage Company A developed a customer churn prediction model for effective customers. The effect of this model can be represented by the diagram above, which means that if the top 10% of customers with the greatest tendency to churn are selected as target active customers, 50% of all actual churn customers can be included.

Due to the shortage of resources in all aspects of Brokerage A, the number of customer service staff is limited. Therefore, Brokerage Company A decided to carry out one-on-one targeted retention and care activities for effective customers based on the list of customers with high churn tendency from the churn prediction model, while for ineffective customers, it hopes to retain them through ordinary marketing policies.

Brokerage firm A selected the target customers for this activity in descending order according to the churn tendency scores given by the churn prediction model. That is, it selected the top 5% of customers with high churn tendency from 1.2 million valid customers as target customers, i.e. 60,000.

Next, customer service staff will provide one-on-one retention and care work for these 60,000 customers during the "Retention Month". Brokerage A hopes to have a satisfactory result in the churn rate statistics at the end of the month.

Among these 60,000 customers, the number of actual lost customers is 120×5%×1.5%×5=4,500. It would be great if all of these 4,500 customers could be retained, but it is very difficult to do so in actual retention and care work.

We need to pay attention to several places where customer churn prediction models can cause dissipation in practical applications:

1. Among all customers, only effective customers are targeted for retention care. Assuming the ratio is a, here a=120/300=40%
2. When selecting target activity customers, only retain customers with a high tendency to churn, assuming that the selection ratio b=5%
3. In the process of customer retention, there is a contact success rate of target activity customers, assuming c
4. Among the customers who have successfully contacted customers, there is also the problem of success rate of retention. Assuming d

Based on Brokerage Firm A's previous experience in customer service, a, b, c, and d can all be estimated. Here we may assume that the successful contact rate c is 50%, and the retention success rate d of customers with a tendency to churn among the successfully contacted customers is 30%.

Assuming that the lost customers are evenly distributed among the customers who can be reached and those who cannot be reached, we can calculate the number of customers that broker A can successfully retain by taking retention care activities based on the churn prediction model:

Number of customers successfully retained = total number of customers × proportion of effective customers a × proportion of customers with high churn tendency b × average churn rate of effective customers × model improvement × contact success rate c × successful retention rate of contacted customers

d = 3,000,000 × 40% × 5% × 1.5% × 5 × 50% × 30% = 675 people

In this case, the overall churn rate = (30,000-675)/3,000,000 = 0.9775%, which is almost the same as the 1% without any activity!

From this we can see that the customer churn prediction model does not bring much change to the company regarding customer churn rate. Seeing such results, some people can't help but ask, do we still need to develop a churn prediction model? This is a very realistic problem.

3. Clear Purpose: Retain and Care for Customers

Is the purpose of establishing a churn prediction model to reduce customer churn rate or to improve the effectiveness of care and retention efforts?

If the goal is simply to significantly reduce customer churn rate, the churn prediction model will have relatively little effect. The reason is simple. The churn prediction model is actually a methodology, which cannot directly lead to a reduction in customer churn rate.

To give an example, it is like treating a patient. No matter how advanced the medical equipment is, it can only help the patient diagnose the problem, but cannot help the patient recover.

The churn prediction model in the securities industry can only help securities firms find customer groups with a higher tendency to churn in customer retention, but it cannot directly lead to a decrease in the churn rate . This point must be made clear.

Looking at the securities industry as a whole, brokerages usually define truly lost customers as those who have closed their capital accounts, transferred custody, and revoked their designations. However, the definition of churn in customer churn prediction models is usually based on whether the customer's assets have shrunk significantly.

In this way, the prediction model not only includes the above three types of customers, but also includes some customers who are likely to become dormant customers because their assets have shrunk severely and their expected losses exceed what they can bear, and they are deeply trapped.

Although these customers still appear to be clients of the brokerage firm, they have gradually turned from active customers to inactive customers and no longer contribute profit value to the brokerage firm.

Judging from the actual situation in the securities industry, customer loss due to actions such as closing capital accounts, transferring custody and revoking designation is inevitable and accounts for a certain proportion. However, the latter type of customers can be retained and cared for to remain active and continue to contribute profit value to the brokerage firm.

Therefore, the purpose of the customer churn prediction model should be to improve the effectiveness of retention care work and to maximize the customer's active status, rather than the so-called drastic reduction of customer churn rate .

The best time to keep a customer is before they leave. Prevention is better than cure.

Faced with increasingly fierce market competition, most companies are paying more and more attention to customer retention, and are doing their best to retain customers by continuously investing in customer care and retention.

But they usually face this problem: How to improve the efficiency of customer care and retention when corporate resources are scarce, and how to care for more customers who are actually about to churn with less customer contact cost?

This requires the use of a customer churn prediction model based on data mining.

Continuing with the above example, assume that brokerage firm A can reach 60,000 customers per month and focuses its contacts on high-value customers.

If we carry out care and retention work based on the list of the top 5% of customers with high churn tendency given by the customer churn prediction model, it is exactly 120×5%=60,000 people. At this time, the number of customers who can be successfully retained each month is 675.

Without model guidance, the number of customers that can be successfully retained each month is

Total number of customers × ratio of high-value customers a × ratio of customers with high churn tendency b × average churn rate of high-value customers × contact success rate c × successful retention rate of contacted customers

d = 3,000,000 × 40% × 5% × 1.5% × 50% × 30% = 135 people

Through a simple comparison, we can find that based on exactly the same personnel input, exactly the same contact success rate, and exactly the same retention success rate, the retention with model guidance successfully retained 675-135=540 more customers per month than the retention without model guidance.

Assuming that these successfully retained customers can continue to remain active for half a year (a conservative estimate), and that the average commission contributed by effective customers is 100 yuan per month, the additional income that can be obtained each month due to the improvement in retention efficiency will be 540×100×6=324,000 yuan.

At the end of the year, the total annual income will increase by 324,000×12=3,888,000 yuan.

This is already the most conservative estimate, because it is understood that the average monthly commission contribution of most customers is as high as several hundred yuan or even several thousand yuan.

Making a more conservative estimate, if the top 5% of customers with a high tendency to churn are selected as target customers under the guidance of the model, the improvement of the model is 3. In this case, the annual income can still increase by 1,944,000 yuan, and the return on investment is still very large!

In fact, our estimates ignore the costs of implementing retention activities. The reason for ignoring it is that the cost of conducting retention activities with or without model guidance is the same. We only need to compare how much more revenue is gained by conducting retention activities with model guidance than without model guidance.

These estimates are only to illustrate one point: the customer churn prediction model is not developed, deployed, and then thrown there. It cannot be run on time every month just to see whether the model prediction results are accurate or not. The key is to apply it to the actual customer retention and care work , so as to see real results.

The author of this article @jerryhuang_00bf compiled and published by (Qinggua Media). Please indicate the author information and source when reprinting!

Product promotion services: APP promotion services, advertising platform, Longyou Games