Answers · agent controls
How do I set per-agent rate limits and quotas that do not throttle human browsers?
Last updated
TL;DR: Split traffic into two quota buckets before you count anything. Verified agents (a short-lived credential id, a Web Bot Auth signer URL, or another cryptographic identity you checked) get identity-bound limits: requests per minute, per day and a parallel-call ceiling keyed to that agent, not its IP. Human browsers stay on the ordinary IP, cookie or session bucket you already use for people, so a bursty shopping agent cannot starve shoppers sharing a NAT, and a shared office IP cannot lock out every agent behind it. Cloudflare's rate-limiting docs warn that applying rate limits to verified bots can hurt SEO, and Advanced Rate Limiting can count by headers or cookies while lower plans mostly count by IP — so put agent quotas at your origin or API gateway once identity is known, and keep search crawlers on a separate allow path. Return 429 with Retry-After to agents; never fold them into the human challenge path that would block real users.
What should I check, step by step?
1. Decide the identity for agents before the counter runs: a verified agent credential id (crd_...), a Web Bot Auth Signature-Agent URL after signature check, or your own API key issued per agent. Do not key agent quotas only on User-Agent or client IP.
- Keep humans on a separate bucket: IP (with NAT-aware options where you have them), session cookie or logged-in account id. Do not put browser traffic into the agent counter.
- At the edge, classify first: verified search crawlers (Googlebot and peers) stay on their allowlist; unverified automation can be challenged or blocked; only traffic that presents a checked agent identity enters the agent quota path.
- Set agent windows that match abuse, not human browsing: requests per minute for loops, requests per hour or day for scrape volume, and a max concurrency so one agent cannot open hundreds of parallel connections.
- When an agent is over quota, return HTTP 429 with Retry-After and a stable machine-readable body. Do not send that agent through a CAPTCHA or browser challenge meant for people — that burns human challenge budget and still fails headless clients.
- Exempt or soft-limit honest search crawlers separately from marketplace agents. Cloudflare documents that rate-limiting verified bots can affect SEO; prefer allowlists and crawl-delay style controls for them.
- Log which bucket each request used (human_ip, human_session, agent_credential, bot_signer, unverified) so a throttled homepage is diagnosable without guessing.
- Test with a shared-NAT scenario: many humans behind one IP plus one looping agent. The agent should hit 429 while browsers on the same IP keep loading.
Why one IP bucket throttles the wrong clients
Agents and people often share egress: cloud NATs, corporate proxies and mobile CGNAT mean many humans look like one IP, while one agent farm can rotate many IPs. Counting only by IP either locks out a whole office when an agent misbehaves, or lets a rotating agent ignore the limit.
- Cloudflare rate-limiting rules on Free/Pro mainly count by IP; counting by headers, cookies or custom characteristics is listed for Advanced Rate Limiting on Enterprise. So identity-bound agent quotas usually belong in your API gateway or origin after you have verified the credential or signature — not only in a cheap edge IP rule.
- RFC 9309 (robots.txt) is not an access-control or quota system: it is a voluntary crawl preference. Quotas for agents that call your marketplace API need authenticated identity and a counter you enforce.
Identity-bound agent quotas vs browser buckets
Agent path: require a short-lived scoped credential or a valid Web Bot Auth signature (HTTP Message Signatures, RFC 9421, with tag web-bot-auth). Key the counter on credential id or signer URL. Apply rpm/rph and concurrency. Revoke the credential or drop the signer from the allowlist when abuse continues.
- Human path: keep session or IP limits sized for browsing and forms. Do not decrement the human bucket when a verified agent calls /api/*. Do not decrement the agent bucket when a logged-in shopper loads HTML.
- Search crawlers: treat as a third class. Verify by published IP ranges, reverse DNS or Web Bot Auth where available, allowlist them for indexable URLs, and avoid folding them into either the marketplace agent quota or the human challenge flow.
What to return so agents slow down without harming humans
RFC 6585 defines 429 Too Many Requests; include Retry-After in seconds. Agents can back off; humans hitting a soft IP limit can get a gentler page or a challenge — those are different UX paths.
- Publish RateLimit-Policy / RateLimit style headers when you can (the IETF RateLimit header draft), including a partition that names the agent id for agent responses, so operators see which quota fired.
- Do not put the quota in the system prompt. A model told to "go slow" can ignore it; the gateway that refuses the next call is the control that holds.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “Issue a short-lived credential for my marketplace buyer agent so I can put per-agent rate limits on that id without touching human browser quotas.”
{
"tool": "issue_agent_credential",
"arguments": {
"agent": "marketplace-buyer",
"public_key_pem": "<Ed25519 PEM>",
"config": {
"model_id": "provider/model@version",
"system_prompt_sha256": "<hex>",
"tools": [
"catalog.read",
"order.create"
],
"code_version": "v1.0.0"
},
"scope": [
"marketplace:read",
"marketplace:order"
],
"ttl_hours": 24,
"signature": "<agent key signs the CredentialRequest>"
}
}- Needs your API key. verify_agent_credential is free and keyless: your edge or API gateway checks the credential before counting it toward an agent quota.
- Send the credential id as agent in POST /v1/guard/check under a guard policy with rate_per_minute and rate_per_hour so each action by that agent shares one window. Over-limit checks get a plain deny; you see rate_limit_minute or rate_limit_hour at GET /v1/guard/decisions.
- precall_check can refuse a tool call before it runs when the agent's policy says it is over budget or rate. revoke_agent_credential drops that one agent without changing human session limits.
Rightshold verifies and records agent credentials and guard decisions. It does not sit as your CDN or count raw HTTP requests to your origin; put high-volume request counters in your gateway once identity is known. Not legal advice.
What are the sources?
- Cloudflare: Rate limiting rules (characteristics, verified bots and SEO warning)
- Cloudflare: Verified bots
- Cloudflare: Web Bot Auth
- RFC 9421: HTTP Message Signatures
- RFC 9309: Robots Exclusion Protocol
- RFC 6585: 429 Too Many Requests
- IETF draft: RateLimit header fields for HTTP
- Agent credentials on Rightshold
Related
- How to rate limit an AI agent's API calls per key
- How to verify which AI agent is hitting your site with Web Bot Auth
- How to give an AI agent a scoped, expiring credential instead of a long-lived API key
- Why an AI training opt-out or robots.txt is not access control
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'