Amazonbot is one of the busiest crawlers in our logs. Should we block it, does blocking it keep our site out of Alexa answers, and what are Amzn-SearchBot and Amzn-User? Is there anything we lose by blocking Amazon's crawler?

What should I check, step by step?

1. Count each Amazon token separately in your logs (Amazonbot, Amzn-SearchBot, Amzn-User) before you decide; they do different jobs and are controlled separately.

2. To keep Amazon's training-capable crawler out, add a group naming it: "User-agent: Amazonbot" then "Disallow: /" (or Disallow lines for the sections you want kept out).
3. Leave Amzn-SearchBot allowed if you want to be eligible for Alexa and other Amazon search experiences. Give it its own group if your other search-bot rules don't fit it: Amazon says that without a group naming it, it follows the directives you give other search bots.
4. Don't treat a Disallow as a wall for Amzn-User. Amazon says it may not follow all robots.txt directives because a customer started the fetch. Anything that must stay private belongs behind a login.
5. For page-by-page control while still allowing the crawl, add <meta name="robots" content="noarchive"> to pages you don't want used for model training. Amazon says it also honors noindex and none, and rel=nofollow on links.
6. Don't use Crawl-delay to slow Amazon down: Amazon says these agents don't support it. Use rate limits at your server or CDN instead.
7. Verify that a request claiming to be Amazon comes from Amazon: each agent has its own published IP list, so check the source address against the right one before you trust or block on the user-agent string.
8. If you're an Amazon Associate, weigh the Content Partners terms before blocking Amazonbot, and read them in full: the benefits are tied to allowing continued access, and Amazon says you can leave at any time.
9. Allow about 24 hours for a change to apply, and remember a cached copy of robots.txt may be up to 30 days old. Keep a dated copy of each version.

What is the difference between Amazonbot, Amzn-SearchBot and Amzn-User?

Amazon's crawler page (last updated October 26, 2026) describes three user agents and says each setting is independent of the others. Amazonbot is used to improve Amazon's products and services, to provide more accurate information to customers, and may be used to train Amazon AI models. Its user-agent string contains "Amazonbot/0.1".

• Amzn-SearchBot is used to improve search experiences in Amazon products and services; allowing it makes your content eligible to appear in search experiences such as Alexa. Amazon says it doesn't crawl content for generative AI model training, and that if robots.txt doesn't mention it but allows other search bots, it crawls according to the directives given to other search bots.
• Amzn-User supports user actions, such as fetching live information to answer an Alexa question on a customer's behalf. Amazon says it doesn't crawl for generative AI model training, and that because its actions can be initiated by a user, it may not follow all robots.txt directives. Each of the three has its own published IP address list.

Does Amazonbot obey robots.txt, Crawl-delay and the noarchive meta tag?

Amazon says its automated crawling respects the Robots Exclusion Protocol (the user-agent line plus allow and disallow rules). The crawlers fetch robots.txt for each host separately, so example.com/robots.txt doesn't cover shop.example.com, and they may use a cached copy from the last 30 days. If the file can't be fetched, they behave as if it doesn't exist, so an outage on robots.txt reads as permission.

• On the page itself, Amazon says the agents respect rel=nofollow on links and the robots meta values noarchive, which Amazon reads as "do not use the page for model training", noindex and none. That gives you a middle path the robots.txt line doesn't: let Amazonbot crawl, but mark pages you don't want used for training.
• They don't support Crawl-delay. If Amazonbot's volume is the problem, a CDN or server rate limit is the tool that works, and it doesn't depend on the crawler's cooperation.

What do you give up by blocking Amazonbot, and what is Amazon Content Partners?

Amazon's crawler page now says that if you allow Amazonbot in robots.txt, you may be eligible for benefits with Amazon Content Partners. Amazon's announcement (published June 15, 2026, updated in October) describes the program as voluntary, with no fees and no minimum traffic, open to eligible U.S.-based independent content creators who enroll their content and allow Amazon continued access to it.

• The listed benefits are AI traffic management through AWS WAF (see which AI crawlers visit and block, rate-limit, allow or monetize them), an extra 1% commission on eligible Amazon Associates sales, access to Amazon Publisher Services advertising, and $100 a month in AWS hosting credits with CloudFront included. Amazon says participants can leave at any time.
• So the choice is a trade, and only you can price it: if your site earns little from Amazon Associates, blocking Amazonbot costs you little; if it earns a lot, the extra 1% may be worth more than keeping Amazon out of training. Either way, blocking Amazonbot alone doesn't affect Amzn-SearchBot, so Alexa search eligibility is a separate decision.