Answers · AI crawlers
Requests with Claude-Agent in the user agent are showing up in my logs and look like a normal Chrome browser. What is it, how do I check it's really Anthropic, and do my robots.txt rules for ClaudeBot and Claude-User cover it?
Last updated
TL;DR: Claude-Agent is the token Anthropic adds to the user agent of a real Chrome browser it hosts, which Claude products such as Claude Duet use to complete tasks at a user's direction. Anthropic's help article says this agentic browser use is separate from its robots (ClaudeBot, Claude-User and Claude-SearchBot) and that your preferences for those are independent, so ClaudeBot and Claude-User groups in robots.txt don't decide whether Claude-Agent gets in; the article doesn't mention robots.txt for it at all. Match the Claude-Agent token, not the full string, and confirm the source IP is listed under Claude-Agent at claude.com/.well-known/sail. On November 14, 2026 that file held nine IPv4 addresses, no IPv6, and marked Claude-Agent exhaustive, which under the SAIL draft it follows means a request claiming Claude-Agent from any other address isn't Anthropic's. It's a different list from claude.com/crawling/bots.json, which covers the crawlers. Refresh it automatically: the file sets expires_after to 3600 seconds.
What should I check, step by step?
1. Match on the Claude-Agent token anywhere in the User-Agent header. Anthropic says other parts of the string, such as the Chrome version, can change.
- Check the source IP against the Claude-Agent entry at https://claude.com/.well-known/sail before you treat a request as Anthropic's. Anyone can send the token.
- Treat a Claude-Agent claim from an address outside that list as not Anthropic's: the file marks the service exhaustive, which in the SAIL draft means it sends from nowhere else.
- Don't verify it against claude.com/crawling/bots.json. That list (14 prefixes, creationTime October 20, 2026, on November 14) is for Anthropic's crawlers and fetchers; Claude-Agent has its own.
- Refresh the list automatically, not by hand. The file sets expires_after = 3600 seconds, and the draft says not to refetch an unexpired copy more often than every 1,800 seconds.
- Decide Claude-Agent separately from ClaudeBot, Claude-User and Claude-SearchBot. Anthropic says those preferences are independent, so a robots.txt group for ClaudeBot doesn't settle it.
- If you allow it, exempt only requests where token and IP both match from bot challenges or bot rate limits, and keep sign-in, payment and account changes behind whatever you require of people.
- Keep a dated copy of the SAIL file next to your rule. If a visit is questioned later, that shows what you checked against at the time.
What is Claude-Agent in a user agent string?
Anthropic's help article "How do Anthropic-hosted browsers identify themselves when they browse the web on a user's behalf?" (read November 14, 2026) says some Claude products, including Claude Duet, can use a browser hosted by Anthropic to complete tasks at a user's direction, which it calls agentic browser use. The requests come from a real browser on Anthropic's infrastructure and carry that browser's own User-Agent with the token Claude-Agent added.
- Anthropic's example is "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-Agent/1.0) Chrome/154.0.0.0 Safari/537.36". Because it's a full browser, a visit can load the page's scripts and assets like a person's Chrome would, so it won't look like a one-page fetch in your logs.
- Anthropic keeps it apart from its robots. Its crawler article lists three, ClaudeBot (collects content that could contribute to training), Claude-User (fetches pages when a person asks Claude something) and Claude-SearchBot (indexes for search quality), all controlled through robots.txt. The browser article says allowing or disallowing Claude-Agent traffic controls whether your users can ask Claude services that use agentic browsing to complete tasks on your site, and that your preferences for the robots are independent and managed separately.
How do you verify that a Claude-Agent request really comes from Anthropic?
Anthropic says the traffic comes from the IP addresses published at https://claude.com/.well-known/sail under Claude-Agent, that the addresses may change, and that you should update rules from the file automatically rather than hardcode them. It also warns that anyone can send the same User-Agent header.
- Fetched on November 14, 2026, the file was served as application/sail+toml with schema draft-kelley-webbotauth-sail-00, modified 2026-10-22T18:36:00Z and expires_after = 3600. It had one prefix set, agentic-browser-use, holding nine single IPv4 addresses and no IPv6, marked exclusive = true, and one service, Claude-Agent, marked exhaustive = true and pointing at that set.
- The article describes no request signature for Claude-Agent, so the IP check is the verification. A Web Bot Auth check finds nothing to verify on these requests.
What do exhaustive and exclusive mean in a SAIL file like Anthropic's?
SAIL (Source Address Identity List) is an individual Internet-Draft from Anthropic, draft-kelley-webbotauth-sail-00, posted October 22, 2026 and expiring April 25, 2027; Anthropic notes it isn't yet a standard. It defines the /.well-known/sail location and a TOML format in which an operator lists the addresses its own services send from.
- Section 7 of the draft allows exactly these conclusions. A request claiming a listed service from one of that service's addresses is confirmed. If the service is marked exhaustive, a claim from any other address is refuted, and the consumer must not give it the treatment reserved for that service. If a prefix set is exclusive, a request from it claiming any other of the publisher's services is refuted. Anthropic's file sets both, so a Claude-Agent claim from outside the nine addresses, or a ClaudeBot claim from inside them, isn't Anthropic's.
- The draft also says what the list can't tell you: traffic from a listed address that makes no claim, or claims to be someone else, isn't shown to be the publisher's, and listing an address doesn't prove the publisher holds it. Your copy lapses at the earliest of its own expiry or 30 days after you fetched it, and a document you can't parse in full must be ignored entirely.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “We decided how our site treats Claude-Agent: requests whose user agent carries the token and whose IP is in Anthropic's SAIL list skip our bot challenge, other requests claiming Claude-Agent are refused, and checkout still needs the customer to sign in. Record the rule and the SAIL file we checked against.”
{
"tool": "record_action",
"arguments": {
"action_type": "security.config_change",
"outcome": "success",
"occurred_at": "2026-11-14T13:00:00Z",
"client_event_id": "edge-rule:claude-agent-sail-2026-11-14",
"summary": "Edge rule: Claude-Agent token plus source IP in claude.com/.well-known/sail (service Claude-Agent, exhaustive) skips the bot challenge; Claude-Agent claims from other IPs refused; checkout and account changes still require customer sign-in. SAIL file modified 2026-10-22T18:36:00Z, refreshed hourly.",
"action_sha256": "<sha256 of the SAIL file you fetched>"
}
}- Needs your API key (or accept_terms:true once to open a free account; the key comes back once). It returns a ledger-committed receipt of what you submitted and when. It proves the entry was recorded, not that your edge enforces the rule.
- It doesn't fetch the SAIL file, check IP addresses, read your firewall or see any Claude-Agent traffic. Run the check in your own stack; the hash ties the receipt to the exact file you used.
Rightshold isn't affiliated with Anthropic and can't see Anthropic's traffic, Claude Duet sessions or your logs. The user-agent example, IP lists and file contents are as Anthropic published them, and the SAIL rules as written in draft-kelley-webbotauth-sail-00, all read November 14, 2026; the addresses can change at any refresh and the draft may change. Not legal advice.
What are the sources?
- Anthropic: How do Anthropic-hosted browsers identify themselves when they browse the web on a user's behalf?
- Anthropic: Claude-Agent SAIL file
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Anthropic: crawler IP list (bots.json)
- IETF: draft-kelley-webbotauth-sail-00, SAIL: Source Address Identity List (October 22, 2026)
Related
- How to block spoofed ChatGPT-User and Perplexity-User requests at the edge without blocking the real ones
- How to verify which AI agent is hitting your site with Web Bot Auth
- What is Trusted Agent Delegation, and how would it tie an AI agent to the user behind it?
- How to let a headless browser AI agent complete checkout without turning off bot protection
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'