An AI crawler is overloading my server. How do I slow it down without hurting my Google rankings?

What should I check, step by step?

1. Measure before you act: group the last hour or day of access logs by user-agent token, source IP and path, and find the one or two clients behind the spike. Google says sharp crawl increases are often caused by the site itself (faceted navigation, calendars with endless date URLs), so look at which URLs are being hit too.

2. Verify the claimed bot: compare the IP with the operator's published list (OpenAI publishes gptbot.json and searchbot.json, Perplexity perplexitybot.json, Anthropic a bots.json linked from its crawler article), or for Googlebot and Bingbot use reverse DNS plus a forward lookup. Unverified traffic using a bot's name is a scraper; block or rate-limit it without touching the real bot.
3. Slow bots that honor Crawl-delay with a named group: User-agent: ClaudeBot then Crawl-delay: 1 (Anthropic's own example), or Crawl-delay for Bingbot. Bing's help page describes the value as a time window in which it crawls at most one URL: 5 means about 17,280 URLs a day, 10 about 8,640.
4. Copy your rules into every named group you add. RFC 9309 says a crawler obeys the group that names it and only falls back to User-agent: * when none does, and Anthropic says rules that name ClaudeBot take priority over the general ones. A Crawl-delay-only ClaudeBot group leaves ClaudeBot with no Disallow lines at all.
5. Don't add Crawl-delay for Googlebot. Google ignores it. If Googlebot is the cause and it's an emergency, return 503 or 429 (with Retry-After in seconds or as an HTTP date) to some crawl requests for hours, at most 1 to 2 days, and stop once the load drops. Google lowers the crawl rate for the whole hostname and raises it again on its own.
6. Never use 401 or 403 to slow Google. Google says 4xx codes other than 429 have no effect on crawl rate, and a robots.txt that returns 4xx is treated as if there were no robots.txt at all.
7. For bots that don't document Crawl-delay (GPTBot, OAI-SearchBot, PerplexityBot), cut the URL space with Disallow on heavy or low-value paths (search results, filters, endless calendars, raw data dumps), or put a per-bot rate limit at your CDN that returns 429 with Retry-After for that user agent and its verified IPs only.
8. Keep robots.txt itself fast and returning 200. Google stops crawling for the first 12 hours if robots.txt returns 5xx and then uses its last good copy for up to 30 days, so an overloaded server that also fails on robots.txt pauses your Google crawling.
9. Still overloaded? Contact the operator with log samples: Anthropic lists claudebot@anthropic.com, Google has a form for reporting an unusually high crawl rate (it can take several days and you can't request more crawling), and Bing Webmaster Tools offers Crawl Control by hour of day plus a 'Bing is crawling too much' support option.

Which crawlers honor Crawl-delay, and what slows Googlebot, Bingbot, ClaudeBot, GPTBot and PerplexityBot?

Googlebot: no Crawl-delay. Google lists the robots.txt fields it supports and says other fields such as crawl-delay aren't supported. Its levers are fixing crawl traps on your site, short-term 500/503/429 responses, and the special crawl-rate request form.

• Bingbot: honors Crawl-delay whether it's in a Bingbot group or the * group, an exception to its usual one-group rule. Crawl Control in Bing Webmaster Tools sets speed per hour of the day, but a Crawl-delay line in robots.txt takes precedence over it.
• ClaudeBot: Anthropic says it supports the non-standard Crawl-delay extension and aims to be thoughtful about how quickly it crawls the same domains. It also warns that blocking its IPs instead of using robots.txt may not work reliably, because the bot then can't read your robots.txt.
• GPTBot, OAI-SearchBot and PerplexityBot: as of November 9, 2026, OpenAI's and Perplexity's crawler pages don't mention Crawl-delay. Use robots.txt Disallow for the paths you don't want crawled and an edge rate limit for volume. Blocking OAI-SearchBot or PerplexityBot outright removes you from those AI search products, and that's a separate decision from managing load.

How long can I return 503 or 429 to Googlebot before it hurts my rankings?

Google's crawl-rate page says many 500, 503 or 429 responses make it slow down across the whole hostname, including URLs that still return content, and it speeds up again on its own when the errors stop.

• The same page warns against doing this for longer than 1 to 2 days: if Googlebot sees these codes on the same URL for multiple days, the URL may be dropped from Google's index. Lowering the crawl rate also means new pages are found more slowly and changes such as prices take longer to show in Search.
• Retry-After comes from RFC 9110 HTTP Semantics. It can be a number of seconds (Retry-After: 120) or an HTTP date, and Google says its crawlers read it on 503 and 429 responses.

How do I add Crawl-delay for one bot in robots.txt without dropping my other rules for it?

Say your * group has Disallow: /search/ and Disallow: /api/. If you add 'User-agent: ClaudeBot' with only 'Crawl-delay: 1', ClaudeBot now matches that group and nothing else, so /search/ and /api/ are open to it. Put both Disallow lines under the ClaudeBot group as well.

• Bing has the same rule: if a Bingbot group exists, Bingbot discards everything else in the file, except that it still reads Crawl-delay from the * group. Its help pages recommend repeating the generic directives in any bot-specific group.