Rightshold™ Sign up

Answers · agent controls

How do I rate limit an AI agent's API calls per key?

Last updated

TL;DR: Give every agent its own key, then count and limit requests by that key in a layer the agent can't change: an API gateway or LLM proxy in front of the provider, or a check in your tool code before each call. Provider limits won't do it for you: OpenAI says its rate limits are defined at the organization and project level, not per user, and Anthropic enforces limits per organization with optional per-workspace limits. Set limits in requests per minute and per hour (and tokens per minute for LLM calls), answer over-limit calls with HTTP 429 and a Retry-After header, and make the agent's client honor it with capped, jittered backoff instead of retrying in a tight loop. Pair the rate limit with a spend cap, because a slow agent can still run up a bill.

What should I check, step by step?

1. Issue one key per agent (and per environment), never one shared key. A per-key limit is only as good as the identity behind it.

  1. Keep the real provider keys in the gateway or proxy; give the agent a key that only that layer accepts, so it can't call the provider directly and skip the limit.
  2. Pick the windows: requests per minute for bursts, requests per hour or day for runaway loops, tokens per minute for LLM traffic, and a cap on parallel requests if the agent fans out.
  3. Count by the key's id, not the client IP: agents behind one NAT or cloud egress share IPs, and one agent can rotate through many.
  4. When a key is over its limit, return 429 Too Many Requests with Retry-After in seconds, and say which limit was hit so the developer (not the model) can fix it.
  5. Make the agent's HTTP client respect Retry-After and back off with jitter and a retry ceiling; log every 429 per key so a looping agent shows up on a dashboard.
  6. Add a hard spend or budget limit per key next to the rate limit, and a way to revoke the key at once when the logs show a loop.
  7. If you run several gateway instances, keep the counters in a shared store (or accept that each instance counts separately) and test the limit with a burst before you rely on it.

Why the provider's limits aren't per key

OpenAI's rate-limit guide says limits are defined at the organization level and the project level, not the user level; separate hard spend limits for an organization or project return 429 once reached.

Where to enforce it

An LLM proxy with virtual keys: LiteLLM's proxy, for example, can set rpm_limit, tpm_limit and max_parallel_requests per key or team, and max_budget with budget_duration. Its docs warn that budgets need a database; without one the global budget check is skipped and requests keep flowing.

Telling the agent it's limited

RFC 6585 defines 429 Too Many Requests; the response SHOULD explain the condition and MAY include Retry-After.

How do I ask Rightshold about this (free, no key)?

Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “Give my support agent its own short-lived credential, so I can rate limit and revoke it on its own instead of sharing one key.”

One-click setup

{
  "tool": "issue_agent_credential",
  "arguments": {
    "agent": "support-agent",
    "public_key_pem": "<Ed25519 PEM>",
    "config": {
      "model_id": "provider/model@version",
      "system_prompt_sha256": "<hex>",
      "tools": [
        "crm.read",
        "email.send"
      ],
      "code_version": "v1.0.0"
    },
    "scope": [
      "support:*"
    ],
    "ttl_hours": 24,
    "signature": "<agent key signs the CredentialRequest>"
  }
}

Rightshold doesn't sit in front of your provider or count raw API traffic. The guard answers allow or deny for actions your code asks about, one network call and one ledger record each, so use it for agent actions and tool calls, and a gateway or proxy for high-volume request limiting.

What are the sources?

Related

Not legal advice. JSON · Markdown · All answers

Get a free API key in one call

No card and no email needed. 100 free credits; the license and policy checks stay free without a key.

curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_...   # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
  -H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'
Get a keyQuickstart