Product operation: Application of data system under the growth model!

Just as people need to see the road ahead when walking, products and operations also need to open their "eyes" before making decisions. The left eye is data; the right eye is research. (Hey, don’t ask me why it’s not the left eye research, the right eye data…)

Through online data feedback, we can accurately identify problems, find patterns, verify conjectures, quell subjective disputes, and provide clear direction for the formulation and implementation of product improvements and operational optimization.

1. Data Function Setting of Internet Companies

Internet companies generally attach great importance to data, but the functional settings of data departments vary. Most companies will set up independent BI departments (such as Ctrip and JD.com), and some (such as Amazon) will also disperse data personnel across various teams.

There are three main roles common to data functions:

a. Data engineers are responsible for building the underlying data architecture, defining data tracking specifications, writing tracking codes (sometimes developers will also implant tracking codes), and establishing and managing database reports.

b. BI is responsible for capturing corresponding data items in the database according to business needs, writing SQL codes, and generating various reports. (Note: The functions of a traditional database administrator (DBA) are more like data engineer + BI – point of reference)

c. BA is responsible for analyzing the reports generated by BI, thoroughly interpreting the data based on business knowledge, and outputting observations and suggestions with clear guiding significance. BA personnel usually need to have strong business background knowledge, be able to accurately understand the business conditions and reasons for fluctuations behind the data, and output analysis conclusions in business "language".

My experience in practice is that the two organizational structures each have obvious advantages and disadvantages, and the advantages and disadvantages are completely opposite.

When data personnel are concentrated in one department, database management and report customization are very professional and efficient. However, because they are far away from the business departments, their understanding of the business is affected and they are relatively weak in data definition and interpretation.

The opposite is true when data functions are dispersed across business lines. There is also a serious problem of duplicate data extraction, which not only wastes manpower, but also causes the same data to be different in different departments due to differences in caliber definitions. For example, conversion rate = number of orders / number of visitors. Some departments remove the "suspected robot" part from the number of visitors, while some departments unify the number of visitors as "two-hop visitors", resulting in obvious differences in conversion rate data.

A better approach is to concentrate data engineers and BI in the data department, and set up BA personnel in each business line to connect the two sides.

2. Data Usage

The areas that require data observation on the Internet are very broad, and each sub-field has different core KPIs. Based on the influencing factors behind the core goal split, we should put forward data needs in a targeted manner and formulate data reports.

The data is usually used in the following ways:

1. General data reports

Regular data reports are mainly used for core data that require long-term continuous observation. For example:

• Traffic funnel monitoring can be divided into core funnel data such as home page bounce rate, business details page arrival rate (divided into two major branches: browsing-business details and searching-business details), car addition rate, settlement rate, and settlement completion rate.
• User channel source information, such as the number of users from each channel, the number of new customers, the proportion of orders, conversion status, etc.
• Category conversion rate fluctuations, such as traffic, orders, and SKU sales volume of each category.
• Traffic distribution efficiency, such as CTR of each channel/column, reach of product detail pages, conversion, and repeat visit rate.

When core data items of routine monitoring experience super-threshold fluctuations or trend fluctuations, a special analysis is usually triggered, and corresponding countermeasures are taken based on the analysis results to push the data back to the normal range.

For routine data reports, it is recommended to customize online reports through the company's BI system and conduct observation and analysis according to the monitoring frequency.

1. Special Analysis

Thematic data analysis usually determines data items according to the main influencing factors of the topic, splits the observation dimensions, captures multi-dimensional data, analyzes a certain thematic target, finds the data dimensions where the influencing factors are located, draws conclusions, and guides subsequent actions. For example:

• Analyze the status or effects of a major event, such as the data summary after the Double 11 promotion.
• There are significant fluctuations in core data, such as analysis of the reasons for the continued increase in conversion rates on the Web platform.
• A trend may emerge, such as the number of users from a certain payment channel continuing to decline.
• A specific research topic, such as analysis of shopping guide and consumption characteristics of those born after 1995.

1. AB Testing

A common confusion for product managers is that after a function or channel is launched, some changes occur in the target data. However, there are many factors that influence the changes, such as differences caused by time factors (such as higher conversion rates on weekdays than on weekends), competitor actions, seasonal factors, etc. Fluctuations in core data are often the result of the combined effect of these influencing factors, and it is difficult to accurately define how much direct impact the function itself has brought.

Operations often have similar demands. For example, when the homepage icon turns red or the guidance copy is adjusted, the data fluctuates, but it is difficult to determine to what extent it is the effect of this specific operational action.

In the above situation, the best method is to do AB testing: take two data sets, evenly and randomly distribute various influencing factors (such as region, user group characteristics) in the selection of data set samples, and implement specific product functions or operational actions on one of the data sets; in the same period of time, observe the differences in the target data on the two test sets, so as to accurately determine the exact effect of the function/action to be observed.

There are two points to note here:

1. In order to ensure the accuracy of the statistical results, a larger sample size and statistical time are required (number of results = number of users * statistical time, either the number of users is large enough and the statistical period can be slightly shorter; if the number of users is small, a longer statistical period is required).

2. If there are a few samples in a certain sample that have a huge impact on the mean (such as an order with a huge amount), they need to be excluded to reduce the deviation caused by chance.

3. Personalization

This is an era of big data. User groups with huge differences are faced with a huge amount of products and choices. The poor experience brought by "one size fits all" is no longer applicable.

Every user will leave his or her own clues and footprints in the system, reflecting his or her phased needs in terms of product categories, price ranges, brand preferences, etc. The system can effectively discover the current needs of current users through data and make effective recommendations, and users will also feel that the system "understands me" and have a good shopping experience.

Amazon’s early “Everything Store” concept has gradually transformed into “Everyone Store” in the current era, which is what we often call “one thousand faces for one thousand people”.

Data is the basis for thousands of faces for thousands of people. Through machine learning and algorithm design, the system can make intelligent recommendations in each module and automatically assemble scenarios that match the current user. This is one of the most important ways to use data. I will focus on this part in subsequent articles with actual cases.

3. Customization of routine data reports and data monitoring

In order to make the best use of BI resources and highlight your own focus, when customizing conventional data reports, do not be too comprehensive. There are two main points that need to be fully considered: the North Star indicator and the frequency of indicator monitoring.

1. North Star Indicator

For any business to be continuously optimized and improved and to achieve better results, it is necessary to correctly establish core indicators, continuously monitor, and analyze the gap between actual data and expected progress in stages, triggering corresponding adjustments to ensure that business development and plans remain consistent.

This set of ideas is summarized as PDCA in project management theory, namely Plan, Do, Check, Act. It is also called the Deming cycle in project management and continuous quality improvement. This system is the core method of business goal management. Students who are interested can refer to project management theory. This article will not go into details.

From the PDCA concept, we can see that the setting of goals, the judgment of execution effectiveness, and the effectiveness of corrective actions all require good data indicators to measure and serve as the basis for judging whether the final goal has been achieved. This measurable indicator, which has a direct positive correlation with the goal, is called the North Star Indicator.

The North Star indicator system is usually divided into multiple levels. The establishment and selection of indicators at each level are to better support the achievement of indicators at the previous level, so as to ultimately achieve the company's top-level strategy (company-level North Star indicators).

Here is a practical example. The operating scale of an e-commerce company is often measured by the company's annual sales volume (GMV), which means that GMV is one of the North Star indicators of the entire company. There are many ways to split and calculate turnover. Here is a common simplified calculation method:

GMV = AC * Freq * Conversion * AOS

• AC: Number of active customers
• Freq: average visit frequency of customers
• Conversion: Conversion rate
• AOS: Average unit price

The above four core indicators are second-level core indicators, which are usually assigned to each department for their respective responsibility.

For example, the marketing department is responsible for traffic, number of users and their activity, the product and operation department is responsible for conversion rate indicators, and the category line is responsible for the average unit price indicator. These indicators then become the North Star indicators for each department. If the core influencing factors of an indicator are scattered across multiple departments, the same department will take the lead.

In order to achieve the above-mentioned secondary indicators, further decomposition can be carried out. Take the number of active customers as an example:

AC = RC + NC – EC

• RC: Number of retained customers
• NC: Number of new customers
• EC: Number of lost customers

These indicators can then be further allocated to functional teams responsible for attracting new customers and retaining new customers, becoming the North Star indicators for these teams, which will be led by these teams.

The team responsible for attracting new customers can further split the new customer acquisition indicators into channels, such as paid channels, free channels, etc., and define core indicators and set goals for the next level.

Similarly, the functional team or personnel at the next level responsible for paid channels can be further divided into specific channels, such as ad networks, SEM, app stores, etc., and further set specific goals for each channel. This goes down layer by layer until you reach the bottom layer that is directly controllable.

By analogy, the conversion rate indicators that products and operations are responsible for can be split along the conversion funnel into home page to product details, search to product details, product details addition rate, shopping cart settlement rate, payment success rate, etc., and through progressive splitting, they can be broken down into each team and become their respective North Star indicators.

For each functional department/team, the indicators at the level they are responsible for and the indicators at the next level should become the monitoring content of regular data reports, and the report format should be formulated accordingly and data requirements should be put forward to the BI department.

From a macro perspective, the achievement of the third-level indicators can ensure the achievement of the second-level indicators, and the achievement of the second-level indicators can ensure the achievement of the top-level indicators, thereby providing protection for business goals. Therefore, the reasonable division of the indicator system and strict monitoring and correction are crucial to the realization of the company's goals.

1. Frequency of indicator monitoring

The cycle of regular data reports is usually daily, weekly, monthly and quarterly. Real-time data monitoring is usually required for emergency response (such as downtime and emergency incident handling), while semi-annual and annual reports are mostly statistics of business results. The cycle is too long, and the problems discovered and responses are too slow, which are usually not within the scope of regular data reports.

Each business unit has different characteristics and requires targeted data statistics frequency settings. The following is an example of monitoring conversion rates at the product and operation levels:

Real-time monitoring

When observing the effectiveness of activities during the big promotion, traffic changes rapidly, peaks occur one after another, and hot-selling products are sometimes sold out. At this time, data observation should be accurate to the smallest granularity and even monitor the data curve in real time. Respond quickly to problems reflected in the data (such as sold out, downtime, technical failures, unsalable items in prime resource positions, page display errors, fluctuations caused by price setting errors, etc.), optimize promotional products and resource positions, and use the horse racing mechanism to adjust the venue traffic distribution to maximize the effect of the big promotion.

Daily Report

For daily promotional activities, you can observe the performance of promotional categories and methods in the overall conversion funnel on a daily basis, locate problem points and quickly carry out targeted optimization; such as changing products, changing promotional rules, updating activity pages/activity columns, configuring promotional tags, etc., to achieve the best activity effect.

Weekly Report

In terms of operations, such as homepage or channel operations, you can observe the column users' interest index on a weekly or monthly basis through dimensions such as CTR of each section, dwell time, business detail arrival rate, car addition rate, conversion rate, and revisit frequency. For weak links, conduct in-depth analysis through data (such as user flow tracking, regional click heat analysis, bounce analysis, etc.), and appropriately combine qualitative and quantitative in-depth interviews with user research to optimize the channel entrance interaction design, page information architecture design, channel sub-column layout, information display, marketing copy, etc., to achieve the best results.

Monthly/quarterly reports

In the mobile era, the frequency of mobile packet sending is limited (mostly one packet is sent every two weeks to one month), and core indicators that are highly dependent on technical functions are often counted on a monthly or quarterly basis.

For example, for the functional iteration of the core conversion funnel module and the efficiency effect of the new product module, you can use months or quarters as units (related to the technology release cycle and the new column user education cycle), combine seasonal factors, and vertically compare the fluctuations of the corresponding data year-on-year and month-on-month to find the links that can be optimized and improved.

Operational actions generally bring faster data responses, focusing on daily and weekly reports to guide operations; while product actions are generally affected by technical releases, have a moderate data response cycle, and are more focused on monthly or quarterly reports, but all seek to respond quickly after discovering a problem.

In general, annual reports may be more suitable for financial considerations of corporate strategies and business lines. Apart from summaries of achievements and gains and losses, they are relatively less used on the product and operational sides.

The above is an example of conversion rate.

If it is user operation and growth, you can also develop and monitor corresponding data reports based on the frequency of user channel sources and activation, communication effect (short cycle, such as day or week), activity, category penetration, transaction status, per capita value (medium cycle, such as month), retention rate, churn return rate, and life cycle status (long cycle, such as quarter or half a year/year), and trigger corresponding adjustment actions.

Finally, when developing reports, it is recommended not to put too many levels of data on the same report, which will create an ocean of data, make the table overly complex, and lose focus. It is usually best for a report to contain two levels of indicators.

For example, the report of the first-level indicator only contains the data of the first- and second-level indicators. The fluctuation of the first-level indicator can be observed from the second-level indicator to find the cause of the fluctuation. If you need to go deeper, it is recommended to customize a secondary indicator report containing the second and third level indicator data. And so on.

4. Special Topic Analysis

We often encounter some unexpected abnormal situations at work, such as a sharp fluctuation in user conversion rate at a certain stage, a surge or sharp drop in transaction amount, a sharp drop in CTR of a certain column, etc., or we observe certain trend changes (such as evolution of consumer shopping preferences and changes in brand consumption trends). At this time, a thematic analysis is usually conducted to clarify the next steps in solving the problem.

1. Special analysis triggering reasons

Thematic analysis is mainly triggered by the following situations:

a. In data reports, we often see fluctuations in some core data indicators. When the fluctuation range exceeds a predefined warning threshold, analysis should be triggered (regardless of whether it is a positive or negative fluctuation) to understand the reasons behind the fluctuation and take corresponding countermeasures.

The magnitude of fluctuation that is worthy of triggering analysis depends on the business sensitivity corresponding to the characteristics of the indicator itself. There is no fixed rule for setting the threshold, you can set it according to your tolerance for the impact. A common mistake here is to be too sensitive to normal small fluctuations, which triggers frequent analysis, but ultimately no valuable discoveries are made. This is a natural fluctuation and a waste of manpower.

What is a normal fluctuation can be judged by analyzing the year-on-year and month-on-month statistics of the same indicator over a long period of time.

For example, the figure above shows the distribution of traffic by time period during a five-week period. Look carefully, is there anything unusual?

You guessed it right, there will be a large amount of traffic at 0 o'clock! The traffic peaks at 9:00, 14:00, and 19:00 are consistent with the access patterns of mobile users during the morning commute, when returning to their seats in the afternoon, and during the evening commute. However, the appearance of such a large amount of traffic at 0 o'clock is very abnormal and should trigger a special analysis.

b. In the data report, the data reflects a continuous change in the same trend, for example, 7 consecutive positive or negative growths. At this point, even if the preset abnormal warning threshold has not been reached, analysis should be performed to understand the reasons behind the trend.

Some students may ask, why 7 times?

In fact, this is not absolute. When a continuous trend appears, the more data points in the same direction, the greater the possibility that there is some non-accidental factor behind it. From a statistical point of view, if it is accidental factors that cause seven consecutive points to develop in the same direction, the probability is only 1/128, or about 8%. Therefore, the confidence level that there are non-accidental factors behind the 7-point trend change is high enough. If it is a particularly critical indicator, it may be time to conduct an analysis if five consecutive points develop in the same direction (97% certainty). Students who want to learn more can search for "7-point principles" and look up theoretical knowledge related to PMP or statistics.

Of course, we should eliminate the influencing factors that we have already understood. For example, the continuous decline in traffic as we get closer to the Spring Festival, or the continuous increase in sales of the new season's clothing as we approach the change of seasons, are all normal phenomena. Unless the fluctuations are too large and seriously deviate from the year-on-year situation, such trends are not worth wasting manpower to analyze.

c. Be interested in the reasons behind certain data and need to analyze and understand the information behind the data. This has nothing to do with the fluctuation of the data itself, but is just about gaining a deeper understanding of the reasons or factors behind the data.

For example, we can analyze why third-party merchants account for 48% of the traffic on the platform, so as to formulate a more balanced traffic distribution strategy to support self-operated or third-party businesses. We can also analyze why the proportion of users from paid channels is low or the cost per customer is too high, so as to make more accurate and cost-effective traffic purchases and placements.

1. Common methods of thematic analysis

To summarize simply, the main approach to thematic analysis is to comprehensively split the underlying structure of the volatility data indicators according to multiple dimensions, observe and compare the underlying data, find out in which sub-dimension the abnormal fluctuation occurs, and lock in that dimension, proceed step by step, and decompose in depth until the answer is finally found.

In the process of splitting into lower dimensions, it is necessary to consider multiple angles and make repeated comparisons. For example, if you find abnormal fluctuations in conversion rate in a certain week, you can split and observe it according to the following dimensions:

Dimension 1: Product Category

Split it into each category and observe whether a sharp fluctuation in the conversion rate of a certain category drives fluctuations in the overall conversion rate.

Case 1:

In one week, we found that the conversion rate of the entire site soared by nearly 2%. After observing the conversion rate of each category through the secondary report, we found that the conversion rate fluctuations mainly occurred in the beauty category. Further observation of the sales of various SKUs in the beauty category revealed that three products, namely, facial cleansing devices, water flossers, and a certain facial mask, had huge sales in a short period of time. The launch prices of these three items were much lower than those on JD.com and Tmall, and they were confirmed with the marketing department, which then placed them on the “What’s Worth Buying” website, resulting in a large influx of users and a surge in sales. The sales of these three hot-selling items drove fluctuations in the conversion rate of the entire site.

Case 2:

Once, the purchasing and sales department of a clothing line forgot to set mutual exclusion when setting up promotional coupons for a certain brand of clothing, resulting in users being able to collect coupons repeatedly and use them in combination. This technical loophole was disclosed by someone on the Wuyun platform, causing a large number of users and scalpers to rush in and buy the product for free, selling thousands of units in a very short period of time, causing the conversion rate to soar instantly. Because it is difficult to absolutely avoid errors when manually setting prices and promoting promotions, such problems occur from time to time on various e-commerce platforms.

Dimension 2: User Group

Split it into various user groups to observe whether the fluctuation of conversion rate is caused by changes in the purchasing behavior of a certain user group. Note that users themselves can be split according to many dimensions:

• gender
• Region: Province, Region
• Consumer price segment: high, medium and low price segment
• Consumption style type: such as fashion people, mothers and babies, digital addicts, reading enthusiasts, housewives...

Case 3:

During a week of data observation, we found a surge in the conversion rate of the entire site. Through regional and category analysis, we found that it was due to the high temperatures in East China, which led to a surge in sales of products such as air conditioners and fans in East China, pushing up the conversion rate of the entire site. The severe smog in Beijing also led to a surge in sales of products such as air purifiers and masks in the area.

Dimension 3: Channel Source

Split it into various user source channels and observe the sales situation corresponding to each channel.

For example, sometimes the conversion rate increases significantly. Analysis shows that this is because the marketing department has launched popular products in the prime resource positions of certain shopping guide websites, which has generated huge traffic and sales from this channel and thus pushed up the overall conversion rate. Of course, the phenomenon of fake orders in some channels will often cause fluctuations in the overall conversion rate.

Dimension 4: Conversion Funnel

Observe the changes in the conversion rates of the segmented links in the conversion funnel, such as from home page to product details, from product details to shopping cart, from shopping cart to checkout, and from checkout to payment.

Case 4:

One week, the conversion rate was lower than the warning value, and funnel analysis revealed that the success rate of the payment process had dropped significantly. After breaking down the payment channels, it was found that the payment success rate of a certain bank channel dropped to zero. After communicating with the bank, we confirmed that the bank had upgraded its payment interface and there was a problem with the upgraded version, which caused payment failure through this payment channel and caused fluctuations in the overall conversion rate.

Case 5:

Once, after the technical team launched a new version, they found that the conversion rate had dropped. Through funnel analysis, they found that there was a significant drop in the registration success rate of new users. Further analysis of the registration process revealed that a mandatory real-name authentication step was added to the product functionality, causing some users to give up registration at this step due to various considerations. After communicating with the product manager, the real-name authentication was changed to be skippable, and the authentication was guided in the subsequent stage. This change restored the registration success rate and solved the problem.

Dimension 5: Device Platform

Observe the conversion rates of various platforms such as iOS, Android, PC, Web, and various app versions. For example, we sometimes found that newly released Android packages had technical failures, resulting in large-scale login failures for users, which in turn affected the overall conversion rate.

Dimension 6: Sales Channel

Many platforms will connect to the next-level distribution channels, and changes in sales in each channel will also cause fluctuations in the overall conversion rate. Sometimes, when a channel carries out extremely effective advertising, it will significantly promote sales in that channel and thus affect the overall conversion rate.

Dimension 7: Traffic or sales period distribution

Split it into various user source channels and observe the sales situation corresponding to each channel.

For example, sometimes the conversion rate increases significantly. Analysis shows that this is because the marketing department launched a hot-selling product in the prime resource position of "What's Worth Buying", which generated huge traffic and sales from this channel and pushed up the overall conversion rate. Of course, the phenomenon of fake orders in some channels will often cause fluctuations in the overall conversion rate.

Case 6:

There was one time when an alarm was raised about a drop in conversion rate, and data analysis showed that sales were evenly distributed in terms of users, channels, categories, etc. Finally, the product manager and the BA jointly investigated and found that there was a large amount of traffic between 0:00 and 7:00, and the traffic was concentrated at the time when the hour just arrived. From this, it can be basically inferred that these traffics are not real customers, but are caused by some program scripts triggered at the hour. Finally, we followed up with the technical team for analysis and confirmed that a certain search engine crawler had begun to centrally crawl the platform's products and price information.

Dimension 8: User Account or Merchant

Sometimes a merchant or certain users receive abnormally large orders, resulting in huge fluctuations in the overall conversion rate, average order price, etc. (this phenomenon is often caused by fake orders). Such issues can be discovered by breaking down sales by merchant or user account.

In the actual analysis that the data team and I have done, problems are often found in the above eight dimensions. It does not rule out the possibility of more dimensions, and everyone can make analogies based on their own business characteristics.

The above is just one example of how to break down your conversion rate. Any indicator can usually be broken down until the problem is finally found, and the eight dimensions listed above are applicable to most online situation analyses.

The specific approach is: after splitting the indicators to the next level according to each dimension, observe whether the indicators of each dimension at the next level evenly reflect the fluctuation. If so, we can basically rule out the possibility that it is caused by factors in this dimension. By breaking down and observing each dimension of the same level one by one, you will usually find that a sub-indicator under a certain dimension fluctuates dramatically. Lock on that indicator, and break down and observe its lower-level indicators again, layer by layer, and you will eventually find a conclusion.

Related reading:

1. Product operation: 2 major ways to get started to accurately control private domain traffic!

2. Product operation: How can products improve user stickiness?

3. Practical Tips + Case Studies | How to improve user stickiness through habits?

4. How to conduct user behavior analysis and improve user stickiness?

5. How to establish a user incentive system to enhance user stickiness?

6. How can products improve user stickiness? Here are 3 tips

Author: Xu Xiaopeng

Source: Product meets operation