Application of indicators and quantification of operational activities

"Data-driven operation ", more and more people are talking about this concept. In my personal understanding, "data-driven operation" means abstracting operational strategies from the understanding of business data, tracking the implementation process of operational strategies with data, and ultimately evaluating the implementation effect of operational strategies with data .
The "data" we refer to in operations is data in a narrow sense, which is actually a variety of indicators.

• What are indicators?

  I think it is more accurate to equate the word "indicator" with the English word indicator. It is an indicator, a string of numbers that quantifies something.
  In fact, everyone uses indicators consciously or unconsciously in their work. As long as it works, why bother about its definition? What matters is how to use it well.
  As far as operations are concerned, I think when using indicators, we must first distinguish between process indicators and result indicators .

  Example 1: We are an e-commerce company and we want to run a promotion to increase the sales of product A. Among the traffic, clicks , favorites, purchases, average order value, and sales volume of product A’s page, which ones are process-oriented indicators and which ones are result-oriented indicators? Traffic, clicks, and collections are obviously process indicators, and sales are obviously result indicators, so what about purchase numbers and average order value? They are actually indicators in the customer purchasing process and belong to the entire conversion funnel. So first of all, they are process indicators; and "number of purchases × average order value = sales" are actually result indicators.

Therefore, process indicators and result indicators are not separated. The key lies in your operational purpose. Process indicators generally require high-frequency tracking and timely adjustments based on changes in the indicators; while result indicators are generally abstracted from the purpose of the activity and are therefore suitable for the evaluation of operational activities .

• How to design high-quality indicators

  To determine the quality of an indicator, we need to measure it from two aspects: reliability and validity .
  Reliability refers to the accuracy, precision and stability of an indicator in measuring something. The so-called validity means that the information measured by a certain indicator is indeed the information that the researcher expects to obtain .

  Example 2: Compare the indicator to a ruler; if the scale is accurate (1 mm is really 1 mm, not 1.2 mm), the precision is high (it can measure to the millimeter, not to the decimeter or centimeter), and the results are stable (the results of repeated measurements are consistent, rather than the ruler being longer or shorter at different times), then if these three points are excellent, then this is a good ruler, that is, an indicator with high reliability.
  Example 3: A ruler is used to measure length; no matter how good a ruler is (it has high reliability), if our purpose is to measure weight, then no matter how good the ruler is (no matter how high its reliability is), it cannot meet our needs (insufficient validity); to measure weight, we have to use a scale (it has high validity).

After the above two examples, you should be able to understand how to construct a good indicator. In addition to reliability and validity, good indicators also need to have the following characteristics: simple and fast calculation, easy to understand, generalizable (applicable to different operational activities and can be compared horizontally) , and sustainable (stable caliber and long-term availability).

• Index combination application - comprehensive index

  In many business scenarios, we need to keep an eye on more than one indicator. Taking our current business scenario as an example, to measure the comprehensive strength of a city operation team, we need to start from multiple angles such as capacity scale, order-taking ability, customer evaluation, team stability, and delivery costs. If each angle derives 1-2 indicators, you will fall into confusion.
  Therefore, rather than comprehensive measurement of multiple dimensions, we recommend using a comprehensive index. The most commonly used is the weighted composite index:
  Comprehensive team strength = transportation capacity × p1 + order-taking ability × p2 + customer evaluation × p3 + team stability × p4 + delivery cost × p5
  It should be noted that when calculating the comprehensive index, each indicator must first be standardized and converted into numbers in a unified unit before it can be added up. p1-p5 are weights. The more important an aspect is, the higher the weight will be. The sum of p1 to p5 must always be 1 .

• What core indicators should we focus on?

  For our current business, I think the core indicators that everyone needs to pay attention to are:
  Order acceptance rate, average cost per order, rate of positive reviews and evaluation rate <br />These are also result-based indicators. That's all, it's really not that complicated. From these three core indicators, combined with the combination of two dimensions:
  1. City → Grid → People/Stores
  2. Month → Week → Day → Peak → Hour <br />is enough to help us find and locate problems. At present, we are particularly concerned about the core indicators of the lunch rush hour. Data accumulation can only help you so far. Don’t expect the database to tell you everything. It’s impossible.
  What to do after discovering and locating the problem? You can use various methods such as surveys, visits, questionnaires, etc. to find the root cause of the problem, and then formulate operational actions to solve the problem.
  Of course, the data support group will continue to produce research results, making it easier and more accurate for everyone to formulate operational strategies and better implement them.

• Quantification of operational activities

  In the previous section, we sorted out the relevant knowledge of indicators. With this foundation, we can start discussing the quantification of operational activities.

  • What elements should an excellent operation have?
    In my opinion, an excellent operation activity should have at least:
    1. Clear campaign goals and budget
    2. Clear response relationship and execution process
    3. Reliable process indicators
    4. Accurately measure results
    5. Scientific and reasonable effect evaluation method
    6. Accumulation of knowledge
    7. A smart person in charge
  • What process and outcome indicators do we need?
    Taking the operation of our current O2O delivery business as an example, if we are doing a short-term activity, the process indicators we need to track include at least: budget, number of people covered by the activity, number of people who responded to the activity, number of people who met the activity rules, and expenditure percentage (the proportion of the amount spent to the budget) .
    For a long-term campaign or strategy, I recommend only tracking changes in outcome indicators, which are the core indicators we should focus on.
  • How to evaluate operational activities?
    This may be the most core part of this article. From my perspective, the evaluation of operational activities only answers two questions: Have the clear activity goals been achieved? Is the budget overspent?
    If the activity goals are achieved and the budget is controlled, then it is a well-operated activity!
  • How to determine whether the activity objectives have been achieved?
    Activity objectives must be formulated around core outcome indicators. To judge the effectiveness of an activity, we need to measure whether there is a qualitative change in the core result indicators. How to start an evaluation?
    First, break down the core business indicators into 1-2 layers. Why do this? Because core business indicators are often affected by multiple factors, we try to break them down into a layer that our operational actions can directly affect and that has a close relationship with the core indicators.

    Example 4: For example, if we are doing an activity with the goal of increasing the order acceptance rate during the lunch rush hour, then the order acceptance rate is our core result indicator. When evaluating, we split the order acceptance rate into the number of pushed orders and the number of active capacity based on the response relationship of the operational actions. What we directly affect is the number of active capacity. If the target group of our activity is the active capacity registered in the past month, then the active capacity indicator needs to be broken down by one layer to count the active capacity registered in the past month and the active capacity not registered in the past month.

    Secondly, we need to compare the result indicators from the time dimension, that is, the comparison before and after the activity. It should be noted that the comparison of data before and after the activity must span a full business cycle . For example, if our business has an obvious 7-day cycle fluctuation, then the operation activity must span a full seven days. At the same time, the mean method is used to compare the changes in the average level . Everyone should pay special attention to this point!

    Example 5: Continuing from Example 4, after the activity starts, we need to collect data for at least 7 days before we can start making evaluations. When evaluating, we calculated the average order acceptance rate 7 days after the event, and then compared it with the average order acceptance rate 7 days before the event. If conditions permit, I suggest taking the average data before the event across 2 periods, that is, 14 days, but make sure that no similar activities are taking place during these 14 days and there are no major changes in policies.

    So, how big does the difference have to be between the numbers before and after the event to be considered a difference rather than a random fluctuation? A rough estimate is 5%, that is, the difference in data before and after the activity reaches more than 5%, which can be said to be a significant effect of the activity . The sophisticated method requires extending the time and performing variance analysis on the two periods of data before and after the activity, which will not be elaborated here.
    In addition, for result-based indicators, it is not necessary to track them every day during the event. What is really important is that the span of the activity must exceed one business cycle. After more than one business cycle, the result indicators should be summarized and compared with those before the activity.
    Finally, I would like to add that, if conditions permit, it is recommended to find a reference system when implementing the activity to compare the effects of the activity horizontally. This will eliminate systemic influences. For example, the order acceptance rate is increasing all over the country. An event is held in a certain place and it is found that the order acceptance rate has increased. It is far-fetched to say that this is the effect of the event. We have grouped the cities, and among the cities in the same group, we can look for cities of similar size as a reference .

  • We must pay attention to budget control and keep a close eye on the input-output ratio of the activities .<br />Any operation cannot be separated from budget considerations, which is also our weakest link at present. It is very necessary to estimate the total budget before the event and keep an eye on the budget consumption (daily statistics) during the implementation of the event.
    In addition to paying attention to whether the total budget is overspent, we need to calculate ROI, that is, the return on investment. The calculation method of input-output ratio must be different for different activities.

    Example 6: For example, if we have an activity to mobilize transportation capacity, how much cost is spent, how many transportation riders are mobilized, and how many orders are delivered by this part of transportation capacity, we need to calculate cost/rider + average subsidy per transportation capacity per day, cost/order + average subsidy per order.

    In addition to calculating ROI, we also need to pay attention to the cost situation of the entire city when executing the activity and whether it meets our macro budget constraints. The best case scenario is that overall costs do not increase, or decrease, while the activity is being executed. That is, we need to move the money to where it should be used.

  • Measurement of subsequent impact after the event <br />After a short-term event is completed, it is also necessary to track how the result indicators change after the event. The ideal situation is that after an activity ends, the result-based indicators still maintain the excellent level at the time of the activity, which means that a short-term activity brings long-term positive benefits.
    We cannot ignore the lag effects and chain effects brought about by some operational activities.
  • Compare the effects of similar activities and summarize the experience <br />For similar activities or even the same activities conducted in different periods, we need to archive the activity data to observe the comparison between the activities. For many exploratory activities, it is necessary to plan them into a series of activities in order to achieve our experimental goals . For example, for price bottoming tests, we should conduct a series of tests in stages and by regional types, and summarize the rules from the data archives of each time.
    As operations become more sophisticated, we need to have the ability to plan and execute a series of activities !

• summary

  I didn’t want to write so many words. But there are too many things to express, so I can simply summarize them into the following points:

  1. The design of indicators needs to focus on reliability and validity
  2. Focus on 3-5 core indicators, combining the two dimensions of region and time
  3. The quantification of operational activities needs to distinguish between process indicators and result indicators
  4. The evaluation of operational effectiveness requires calculating the average across business cycles, and the difference is worth mentioning only if it is more than 5%.
  5. Pay attention to budget and input-output ratio
  6. Activities should archive data and experience and compare the effects between activities.

Hopefully, the information mentioned in this article will help you think more clearly, execute more efficiently, and achieve more significant results in your future operations! ~