NickStallman.net

Google Crawl Rate – Wow!

by on Oct.16, 2010, under google, internet

Well usually Google’s maximum crawl rate in Google Webmaster Tools is limited to 0.5 requests/second as the fastest rate possible with the slider.

The growth of PropertyNow however has caused that speed to increase – first to 0.8 requests/second then to 1.5 requests/second then even faster still to 2.5 requests/second.

That is the fastest I’ve ever heard of. There has been a increase in quality content, and also the server can handle the speed, so naturally Google wants to crawl it all as soon as possible and refresh it as often as possible. The logs show that they do actually push that limit but in bursts rather than constantly.

This change has also coincided with alterations of the search results. All keywords dropped temporarily during the speedy crawl period.

Very interesting stuff indeed.


1 Comment for this entry

Leave a Reply

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Blogroll

A few highly recommended websites...