How to evaluate marketing activities?

01 Introduction

In actual efficiency evaluation work, not all marketing activities have conducted AB experiments, and not all companies have productized PSM models. In the absence of AB experiments and PSM modeling, are there other methods for evaluation? Today I will introduce to you a relatively common and easy-to-use analysis method, called the double difference method.

02 Introduction to DID

2.1 DID Overview

The English name of the double difference method DID is Differences-in-Differences, also known as "double difference method" and "difference in difference". The essence of double difference estimation is the fixed effect estimation of panel data. Double difference can be simply understood as differencing twice. As a powerful tool in policy effect evaluation, the double difference method is favored by more and more people. In summary, there are several reasons:

• It avoids the problem of endogeneity to a large extent: policies are generally exogenous to microeconomic entities, so there is no reverse causality problem;
• It alleviates the problem of omitted variable bias;
• The traditional method of evaluating policy effects is mainly to set a dummy variable to indicate whether the policy has occurred or not and then perform regression. In comparison, the double difference method has a more scientific model setting and can estimate the policy effects more accurately.
• The principle and model setting of the double difference method are simple, easy to understand and apply;

2.2 DID Model

The baseline DID model is set up as follows:

Among them, du is a grouping dummy variable. If individual i is affected by the implementation of the strategy, then individual i belongs to the treatment group and the corresponding du value is 1. If individual i is not affected by the implementation of the strategy, then individual i belongs to the control group and the corresponding du value is 0. dt is a dummy variable for strategy implementation. Before the strategy is implemented, dt takes the value of 0, and after the strategy is implemented, dt takes the value of 1. du·dt is the interaction term between the grouping dummy variable and the strategy implementation dummy variable. Its coefficient a3 represents the net effect after the strategy is implemented, which can be used to evaluate the effectiveness of the strategy.

Why can a3 reflect the net effect of the strategy? This can be reflected in the following table (the following table also reflects the true meaning of the five words of the double difference method):

The basic idea of ​​the double difference method is to construct a double difference statistic that reflects the effect of the strategy by comparing the differences between the control group and the experimental group before and after the implementation of the strategy. This idea and the content of the above table are transformed into a simple model (1). At this time, we only need to pay attention to the coefficient a3 of the interaction term in model (1) to obtain the desired net effect of the strategy under DID. The idea of ​​DID can be reflected in the following figure:

Model (1) is a simple model with only the grouping dummy variable du and the strategy implementation dummy variable dt. The polynomial regression equation it constitutes is also relatively simple. When actually applied to one's own strategy evaluation, there will not be only two influencing elements. Therefore, there will be more than two variables in the model, and specific problems need to be analyzed specifically.

One of the most important estimates of the double difference method is that there is a common trend between groups, also called a common effect, which is to assume that the two groups have the same trend of change when no strategy is imposed. If the two groups do not have a common trend without applying a strategy, the DID method will not work because the coefficient a3 obtained at this time is affected by other elements and cannot reflect the effect of the strategy implementation. Therefore, before using DID for strategy evaluation, it is necessary to conduct a robustness test of DID, mainly testing common trends to ensure that the application of DID is feasible.

03 DID Practice

Suppose the marketing department has launched an advertisement and then asks you to evaluate whether the advertisement is effective. The criterion for evaluating its effectiveness is whether it has brought about an increase in GMV. The most common method used by operations staff in daily evaluation is the before-and-after comparison: as shown below, among the users reached by the advertisement, the GMV before the advertisement was 1 million, and the GMV during the advertisement was 1.5 million, so the advertisement brought an increase of 500,000 in GMV. So are these 500,000 really all incremental gains brought about by this advertising campaign?

Obviously, it is unlikely, because there is a natural time difference between before and during the advertising. For example, before the advertising is April, and the advertising period is May. May naturally has holidays such as May Day. For tourism-related companies, transactions will naturally increase during holidays. Therefore, at this time, the double difference method mentioned above should be used to subtract a natural increment from the increment.

Which users should be selected as natural increments? The following are common ones in work:

• The advantage of directly using the market's non-advertising approach to reach users is that it is simple and crude, and you only need to refer to the market's natural trend. The disadvantage is that users reached by advertisements may themselves be users who are more likely to bring in GMV, while users not reached by advertisements may be relatively less active, and GMV growth is just like that of users reached by advertisements.
• Utilize a third-party PANEL database of Internet user samples to match the target user group that has visited the company from the PANEL database. Those who are reached by advertisements are the experimental group, and those who are not reached by advertisements are the control group. The advantage is that the PANEL database can basically represent the entire Internet user base, and the experimental group and the control group can be screened out based on whether they are reached by advertisements. The disadvantage is that if the target users of the company's advertisements are a small part of the PANEL database, such as a certain city group, then the final result may have a large error.
• The PSM method is used to screen out non-advertising-reached users. This is actually the PSM+DID mentioned in the previous article sharing the PSM method. It mainly uses PSM modeling to build a control group, and then uses DID to correct errors. Since this article mainly introduces how to perform evaluation when AB experiments and PSM modeling are not available, PSM+DID will not be discussed in detail here.

OK, let’s assume that we have built the control group, as shown in the table below. Then the net GMV growth is 300,000, which means that although the transaction volume of users reached by the advertisement increased by 500,000, if there was no advertisement, these users would naturally have a transaction volume growth of 200,000, so the final net GMV growth is 300,000.

04 Postscript

Double difference DID is indeed the fastest and easiest evaluation method in daily operations, but its accuracy is often not as good as AB experiments. If you really want to decide whether a project is worth investing more money in, it is recommended to conduct multiple DID evaluations on the basis of unchanged strategy to see whether the effect value is stable.

Author: A data person’s private land

Source: A data person’s private land (shujuren_qishu)