{
  "question": "Amazonbot is one of the busiest crawlers in our logs. Should we block it, does blocking it keep our site out of Alexa answers, and what are Amzn-SearchBot and Amzn-User? Is there anything we lose by blocking Amazon's crawler?",
  "slug": "should-you-block-amazonbot-and-what-is-the-difference-between-amazonbot-amzn-searchbot-and-amzn-user",
  "url": "https://www.rightshold.com/answers/should-you-block-amazonbot-and-what-is-the-difference-between-amazonbot-amzn-searchbot-and-amzn-user",
  "short_answer": "It depends on what you want to keep out, because Amazon now runs three separately controlled user agents. Amazonbot is the general crawler: Amazon says it's used to improve its products and services and may be used to train Amazon AI models. Amzn-SearchBot powers search experiences such as Alexa, and Amazon says it doesn't crawl for generative AI model training. Amzn-User fetches live pages when a customer asks Alexa something, doesn't crawl for training either, and may not follow all robots.txt directives because it acts on a user's request. Each one reads only the robots.txt group that names it, so \"User-agent: Amazonbot\" with \"Disallow: /\" keeps the training-capable crawler out without touching Alexa search eligibility. If you'd rather let Amazonbot crawl but keep specific pages out of training, Amazon says all three honor the page-level noarchive robots meta tag as \"do not use the page for model training\". Two cautions: changes can take about 24 hours, and the crawlers may use a cached robots.txt up to 30 days old. And blocking Amazonbot now has a stated cost for some sites: Amazon's crawler page says sites that allow Amazonbot may be eligible for its Content Partners program, which offers U.S.-based independent creators a +1% commission boost on eligible Amazon Associates sales, AWS hosting credits and AI traffic tools in return for continued access.",
  "checklist": [
    "Count each Amazon token separately in your logs (Amazonbot, Amzn-SearchBot, Amzn-User) before you decide; they do different jobs and are controlled separately.",
    "To keep Amazon's training-capable crawler out, add a group naming it: \"User-agent: Amazonbot\" then \"Disallow: /\" (or Disallow lines for the sections you want kept out).",
    "Leave Amzn-SearchBot allowed if you want to be eligible for Alexa and other Amazon search experiences. Give it its own group if your other search-bot rules don't fit it: Amazon says that without a group naming it, it follows the directives you give other search bots.",
    "Don't treat a Disallow as a wall for Amzn-User. Amazon says it may not follow all robots.txt directives because a customer started the fetch. Anything that must stay private belongs behind a login.",
    "For page-by-page control while still allowing the crawl, add <meta name=\"robots\" content=\"noarchive\"> to pages you don't want used for model training. Amazon says it also honors noindex and none, and rel=nofollow on links.",
    "Don't use Crawl-delay to slow Amazon down: Amazon says these agents don't support it. Use rate limits at your server or CDN instead.",
    "Verify that a request claiming to be Amazon comes from Amazon: each agent has its own published IP list, so check the source address against the right one before you trust or block on the user-agent string.",
    "If you're an Amazon Associate, weigh the Content Partners terms before blocking Amazonbot, and read them in full: the benefits are tied to allowing continued access, and Amazon says you can leave at any time.",
    "Allow about 24 hours for a change to apply, and remember a cached copy of robots.txt may be up to 30 days old. Keep a dated copy of each version."
  ],
  "sections": [
    {
      "heading": "What each Amazon crawler does",
      "points": [
        "Amazon's crawler page (last updated October 26, 2026) describes three user agents and says each setting is independent of the others. Amazonbot is used to improve Amazon's products and services, to provide more accurate information to customers, and may be used to train Amazon AI models. Its user-agent string contains \"Amazonbot/0.1\".",
        "Amzn-SearchBot is used to improve search experiences in Amazon products and services; allowing it makes your content eligible to appear in search experiences such as Alexa. Amazon says it doesn't crawl content for generative AI model training, and that if robots.txt doesn't mention it but allows other search bots, it crawls according to the directives given to other search bots.",
        "Amzn-User supports user actions, such as fetching live information to answer an Alexa question on a customer's behalf. Amazon says it doesn't crawl for generative AI model training, and that because its actions can be initiated by a user, it may not follow all robots.txt directives. Each of the three has its own published IP address list."
      ]
    },
    {
      "heading": "How Amazon reads robots.txt and meta tags",
      "points": [
        "Amazon says its automated crawling respects the Robots Exclusion Protocol (the user-agent line plus allow and disallow rules). The crawlers fetch robots.txt for each host separately, so example.com/robots.txt doesn't cover shop.example.com, and they may use a cached copy from the last 30 days. If the file can't be fetched, they behave as if it doesn't exist, so an outage on robots.txt reads as permission.",
        "On the page itself, Amazon says the agents respect rel=nofollow on links and the robots meta values noarchive, which Amazon reads as \"do not use the page for model training\", noindex and none. That gives you a middle path the robots.txt line doesn't: let Amazonbot crawl, but mark pages you don't want used for training.",
        "They don't support Crawl-delay. If Amazonbot's volume is the problem, a CDN or server rate limit is the tool that works, and it doesn't depend on the crawler's cooperation."
      ]
    },
    {
      "heading": "What blocking Amazonbot can cost you",
      "points": [
        "Amazon's crawler page now says that if you allow Amazonbot in robots.txt, you may be eligible for benefits with Amazon Content Partners. Amazon's announcement (published June 15, 2026, updated in October) describes the program as voluntary, with no fees and no minimum traffic, open to eligible U.S.-based independent content creators who enroll their content and allow Amazon continued access to it.",
        "The listed benefits are AI traffic management through AWS WAF (see which AI crawlers visit and block, rate-limit, allow or monetize them), an extra 1% commission on eligible Amazon Associates sales, access to Amazon Publisher Services advertising, and $100 a month in AWS hosting credits with CloudFront included. Amazon says participants can leave at any time.",
        "So the choice is a trade, and only you can price it: if your site earns little from Amazon Associates, blocking Amazonbot costs you little; if it earns a lot, the extra 1% may be worth more than keeping Amazon out of training. Either way, blocking Amazonbot alone doesn't affect Amzn-SearchBot, so Alexa search eligibility is a separate decision."
      ]
    }
  ],
  "checks": [],
  "ask_rightshold": {
    "start": "https://www.rightshold.com/start?ref=seo",
    "mcp_endpoint": "https://www.rightshold.com/mcp",
    "ask": "What does example.com's robots.txt tell Amazonbot and the other AI crawlers today, which group did each one match, and is our site counted as opted out of AI training?",
    "mcp_call": {
      "tool": "read_ai_policy",
      "arguments": {
        "domain": "example.com"
      }
    },
    "also": [
      "Free, no key, cached about 6 hours. For 16 AI crawler, search and user-fetch tokens, including Amazonbot, it reports whether the path is allowed and which group matched, plus Content-Signal lines, ai.txt, W3C TDMRep and whether llms.txt exists. Pass a path to check one section.",
      "It lists Amazonbot as mixed-purpose, so blocking it doesn't count toward the training opt-out summary. It doesn't check Amzn-SearchBot or Amzn-User, and it doesn't read page-level noarchive tags; check those in your own files."
    ],
    "scope": "Rightshold reads a site's public crawl and AI-use signals. It isn't affiliated with Amazon or the Content Partners program and can't make any crawler honor a preference. Crawler behavior is as Amazon describes it on its crawler page (last updated October 26, 2026) and the program as described in Amazon's announcement, both checked on November 10, 2026; read the program terms yourself before enrolling. Not legal or financial advice."
  },
  "sources": [
    {
      "title": "Amazon: Amazon Web Search crawlers (Amazonbot, Amzn-SearchBot, Amzn-User; last updated October 26, 2026)",
      "url": "https://developer.amazon.com/amazonbot"
    },
    {
      "title": "Amazon: Amazonbot published IP addresses",
      "url": "https://developer.amazon.com/amazonbot/ip-addresses/"
    },
    {
      "title": "Amazon: Amzn-SearchBot published IP addresses",
      "url": "https://developer.amazon.com/amazonbot/searchbot-ip-addresses/"
    },
    {
      "title": "Amazon: Amzn-User published IP addresses",
      "url": "https://developer.amazon.com/amazonbot/live-ip-addresses/"
    },
    {
      "title": "About Amazon: Amazon Content Partners is now available for independent content creators (published June 15, 2026, updated October 2026)",
      "url": "https://www.aboutamazon.com/news/innovation-at-amazon/amazon-content-partners-preview"
    },
    {
      "title": "IETF: RFC 9309, Robots Exclusion Protocol",
      "url": "https://www.rfc-editor.org/rfc/rfc9309"
    }
  ],
  "related": [
    {
      "title": "How to block AI training crawlers in robots.txt without leaving AI search",
      "url": "https://www.rightshold.com/answers/how-to-block-ai-training-crawlers-in-robots-txt-without-leaving-ai-search"
    },
    {
      "title": "How to slow down AI crawlers overloading your server without hurting Google rankings",
      "url": "https://www.rightshold.com/answers/how-to-slow-down-ai-crawlers-overloading-your-server-without-hurting-google-rankings"
    },
    {
      "title": "What is the __usage_ai-training robots.txt line?",
      "url": "https://www.rightshold.com/answers/what-is-the-usage-ai-training-robots-txt-line-and-which-ai-companies-will-honor-it"
    },
    {
      "title": "Why an AI training opt-out or robots.txt is not access control",
      "url": "https://www.rightshold.com/answers/why-an-ai-training-opt-out-or-robots-txt-is-not-access-control"
    }
  ],
  "computed_from": "Curated guidance; license verdicts computed from Rightshold's license profiles at render time. Not legal advice.",
  "updated": "2026-11-10"
}