Changle SEO Training: Analysis of the reasons for the sudden increase in the frequency of Baidu spider crawling and the non-inclusion of website pages

The sudden increase in the number of websites crawled by Spider often brings great trouble to the sites. Many of them turn to the platforms for BaiduSpiderIP whitelist, but in fact, BaiduSpiderIP may change at any time, so they dare not announce it, for fear that the webmaster’s failure to set it in time will affect the crawling effect. How does Baidu calculate and allocate the number of crawling frequencies? What are the reasons for the sudden increase in site crawling frequency?

Changle SEO Training: Analysis of the reasons for the sudden increase in the frequency of Baidu spider crawling and the non-inclusion of website pages

In general, Baiduspider will comprehensively calculate the crawling frequency based on the site size, the number of new links generated by the website every day in history, the comprehensive quality score of the crawled web pages, etc., while taking into account the maximum crawling value that the website can bear set by the webmaster in the crawling frequency tool.

From the cases of sudden increase in crawling frequency that we have investigated so far, the reasons can be divided into the following categories:

1. Baiduspider found that there were a lot of JS codes in the site, and called a large number of resources to parse and crawl the JS codes.

2. The spiders of other departments of Baidu (such as business, pictures, etc.) are crawling, but the frequency and quantity are not well controlled. Sorry.

3. The crawled links are not scored well enough and there are too many spam links, which causes the spider to crawl again.

4. The site was attacked, someone imitated Baidu crawler

If the webmaster has ruled out problems with himself or herself and counterfeit issues and confirms that the frequency of BaiduSpider crawling is too large, he or she can submit feedback through the Feedback Center. Remember to provide detailed screenshots of the crawling logs.

Analysis of the reasons why Baidu does not include pages

Currently, there are two ways for Baidu spider to crawl new links. One is to actively discover and crawl, and the other is to obtain data from the link submission tool of Baidu webmaster platform. Among them, the data "collected" through the active push function is the most popular with Baidu spider. For webmasters, if the link has not been included for a long time, it is recommended to try using the active push function, especially for new websites. Actively pushing the homepage data will facilitate the capture of internal page data.

So students may ask, why can’t I see the data displayed online even after I submitted it? There are many factors involved. In the process of spider crawling, the factors that affect online display are:

1. Website blocking: Don’t laugh, there are really students who blocked Baidu Spider while submitting data to Baidu. Of course, the result is that their websites cannot be included.

2. Quality screening: Baidu spider 3.0 has taken its recognition of low-quality content to a new level, especially time-sensitive content. It starts quality assessment and screening from the crawling stage, filtering out a large number of over-optimized pages. According to internal regular data evaluation, the number of low-quality web pages has dropped by 62% compared to before.

3. Crawl failure: There are many reasons for crawl failure. Sometimes you have no problem accessing the website in the office, but Baidu spider encounters trouble. The site should always pay attention to ensure the stability of the website at different times and places.

4. Quota limit: Although we are gradually relaxing the quota for actively pushed crawling, if the number of site pages suddenly explodes, it will still affect the crawling and collection of high-quality links. Therefore, in addition to ensuring stable access, the site must also pay attention to website security to prevent hacker injections.