Crawling is routine of accessing your website by poke bot from poke engines similar to Google, Bing, live search, yahoo etc. Crawling rate is magnitude during that a poke bot visites your website. From a begin of poke engine epoch there has been no carry out over crawling rateby webmasters though newly most webmaster has felt a need of control crawling rate.

The categorical reason is with enlarge crawling magnitude your server bucket might increase specifically for vast websites. As most of a poke engines have been attempting to have genuine time poke formula as well as softened infrastructure has since a intensity of enlarge crawling for each website by most folds.

Google Webmaster does provides a choice to carry out Crawling rate as well as right away Bing has proposed charity behind crawling around directives of a Robots Exclusion Protocol (REP) as listed in a site’s robots.txt file.

How to Set Crawl Delay Parameter for Bing as well as MSNbot Crawler

In a robots.txt file, inside of a general user representative section, supplement a crawl-delay gauge as shown in a e.g. below:

User-agent: *

Crawl-delay: 1

The crawl-delay gauge accepts usually positive, total numbers as values. Consider a worth listed after a colon as a relations volume of throttling down we wish to request to MSNBot from a default yield rate. The aloft a value, a some-more throttled down a yield rate will be.

Here is illustration of effect on crawling if you set following crawl-delay parameter.

Source: Bing Webmaster Blog