When crawling the web, there’s always a speed limit. A spider can't fetch faster than the host willing to send the pages.

When crawling the web, there’s always a speed limit. A spider can't fetch faster than the host willing to send the pages.