Answers · AI crawlers
How do I verify which AI agent is hitting my site with Web Bot Auth signatures?
Last updated
TL;DR: Check the request's signature, not its user-agent. With Web Bot Auth, an agent signs each request with HTTP Message Signatures (RFC 9421) and sends three headers: Signature, Signature-Input (tag="web-bot-auth", a keyid that is the key's JWK thumbprint, created and expires) and Signature-Agent, the HTTPS URL where its public keys are published. You fetch that key set, pick the key the keyid names, and verify the signature. If it checks out, the request was signed by a key that URL publishes, so you can allowlist, rate limit or block by that URL. ChatGPT's cloud browser signs as "https://chatgpt.com", with keys at chatgpt.com/.well-known/http-message-signatures-directory, and Google signs some Google-Agent requests as https://agent.bot.goog. A valid signature is identity, not permission, and it isn't everywhere yet: the protocol is still an IETF working-group draft and Google says it doesn't sign every request, so keep IP-list or reverse-DNS checks as the fallback.
What should I check, step by step?
1. Make sure the headers reach you. OpenAI's troubleshooting advice is to confirm that proxies in front of your origin keep Signature, Signature-Input and Signature-Agent.
- If your CDN or bot manager already verifies Web Bot Auth, use its verified result. OpenAI says Cloudflare, Akamai, HUMAN and Vercel recognize ChatGPT's signed traffic and verify the signatures themselves.
- To verify it yourself, parse Signature-Input and require tag="web-bot-auth", a keyid, created and expires that haven't passed, @authority or @target-uri among the covered components, and the signature-agent header covered too.
- Resolve Signature-Agent over HTTPS only, accept only a 200 response and don't follow redirects. ChatGPT and Google send their origin and publish keys at /.well-known/http-message-signatures-directory on it, served as application/http-message-signatures-directory+json.
- Select the key whose JWK SHA-256 thumbprint equals the keyid and verify per RFC 9421. Cache the key set according to its Cache-Control header, and drop cached keys that disappear from it; Google says those may be expired or revoked.
- Attach policy only to the signer URL you resolved yourself, such as chatgpt.com or agent.bot.goog. Never attach it to the user-agent string or to an unverified Signature-Agent value.
- Keep the old checks as a fallback for unsigned requests: the operator's published IP lists or a reverse-then-forward DNS lookup. Google says not all of Google-Agent's requests are signed.
- After verifying, still decide what the agent may do. Apply rate limits per signer, log the signer URL with each request, and keep logins, payments and forms behind your normal rules.
What a valid signature proves, and what it doesn't
The working-group draft (draft-ietf-webbotauth-httpsig-protocol-01, October 29, 2026, by authors at Cloudflare and Google) says the Signature-Agent URL is the agent's identifier only once the verifier has fetched it and found a key that verifies the request. Until then it's a claim, and verifiers must not attach policy to it.
- Even then, the draft says a valid signature says nothing about who operates the agent, whether it is benign or whether the request is authorized. Those decisions stay with you. An agent can also abandon one URL and start over with another, so a block by URL works against honest agents, not determined ones.
Replay and scope
A signature covering only @authority verifies for any method, path or body sent to that host until it expires, so anyone who sees one request can reuse it until then. The draft recommends expires no more than 24 hours after created and suggests covering @method and @path to narrow it.
- Cloudflare's docs recommend a short expires (a minute is often enough) and say Cloudflare doesn't currently validate nonces or keep a database of seen nonces. If replay matters to you, track nonces yourself for the signature's lifetime.
- Google notes that for latency-sensitive requests you can respond first and verify within the signature's expiry window, then apply the result to the caller's later requests. A signature's expiry is separate from how long you cache the key set.
Where implementations differ
The current draft makes Signature-Agent a dictionary keyed by the signature label, such as g="https://agent.bot.goog", which is how Google sends it. Earlier drafts used a bare quoted string, such as "https://chatgpt.com". The draft requires signers to send the dictionary form and lets verifiers accept both.
- Cloudflare's Web Bot Auth docs (updated October 8, 2026) say its verification of registered bots expects the bare-string form and fails the dictionary form. If you run your own verifier, accept both forms.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “This request says it comes from ChatGPT's agent. Does its Web Bot Auth signature verify, and does my policy (only chatgpt.com and agent.bot.goog) allow it?”
{
"tool": "gate_check",
"arguments": {
"policy": {
"allowed_signature_agents": [
"chatgpt.com",
"agent.bot.goog"
]
},
"request": {
"method": "GET",
"url": "<the requested URL>",
"headers": {
"signature-agent": "<as received>",
"signature-input": "<as received>",
"signature": "<as received>"
}
}
}
}- Free, no key. It resolves the signer's HTTPS key directory (the well-known directory, a JWKS URL or a Signature Agent Card) and checks the signature against your policy. It returns allow or deny with reasons such as web_bot_auth_expired or web_bot_auth_unknown_keyid, plus the signer it resolved. Allow answers can be cached for at most 5 minutes, and for less when the signature expires sooner.
- allowed_signature_agents takes host names such as chatgpt.com, or the exact https URL of one signer's JWKS or Signature Agent Card. require_web_bot_auth: true accepts any signer whose directory resolves.
- POST /v1/http-signatures/verify checks only the signature. The Gate page (/gate) has free CC0 middleware snippets that run the same check on incoming requests.
Rightshold checks the signatures and requests you pass it. It doesn't sit in front of your site, see your traffic or block anything unless you add the check to your stack, it doesn't track nonces for replay, and it can't identify requests that carry no signature. A verified signature identifies the key's publisher, not that the agent is safe or allowed. Not legal advice.
What are the sources?
- IETF: draft-ietf-webbotauth-httpsig-protocol-01, HTTP Message Signatures for automated traffic (October 29, 2026)
- IETF Web Bot Auth working group documents
- RFC 9421: HTTP Message Signatures
- OpenAI Help: ChatGPT cloud browser allowlisting (Web Bot Auth)
- Google: Authenticate requests with Web Bot Auth (experimental)
- Cloudflare docs: Web Bot Auth (updated October 8, 2026)
Related
- How to tell ChatGPT-User and other AI assistant fetches from AI training crawlers in server logs
- How to block AI training crawlers in robots.txt without leaving AI search
- Can you fingerprint an AI agent?
- How to accept x402 payments on an API
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'