Answers · agent controls
How do I return tiered API responses for verified AI agents vs anonymous bots, without blocking human browsers?
Last updated
TL;DR: Serve capability tiers, not a single allow/deny wall. Classify every request into a response class before you build the body: human browser (session or ordinary HTML), verified agent (short-lived credential or checked Web Bot Auth signature with scope), anonymous automation (no usable identity), and verified search crawler (allowlisted). Humans keep the full public page. Verified agents get a machine-shaped payload sized to their scope — for example catalog summaries at scope marketplace:browse, contracted prices and hold ids only at marketplace:paid_read. Anonymous bots get a stable public stub or 401 with WWW-Authenticate telling them how to present a credential (RFC 7235), not a CAPTCHA that burns your human challenge budget. This is graceful degradation by identity: the same URL can return different fields by Accept and by verified tier, without cloaking the human HTML search engines need. It is not a rate-limit page (that is about counting) and not a hard gate that refuses every unpaid field (that is covered separately).
What should I check, step by step?
1. Define three or four response classes up front: public_summary, human_html, agent_scoped (keyed by credential scope), and deny_or_challenge for unverified automation on sensitive routes.
- Classify before serialization: verify the agent credential or Web Bot Auth signature first, then pick the response builder. Do not build the paid body and strip fields afterwards — that leaks timing and accidental fields.
- Keep human HTML honest and indexable on the marketing or product URL. Put dense machine feeds on /api/... or negotiate with Accept: application/json so crawlers and shoppers are not served a hollow page.
- For verified agents, map scope → fields. A browse-tier credential returns ids, titles and public prices; a paid-read credential adds contract prices, inventory holds and PII your agreement allows. Missing scope on an otherwise valid credential is 403 Forbidden (RFC 7235 / RFC 7231), not a silent empty list.
- For anonymous automation on agent-oriented routes, return 401 Unauthorized with a WWW-Authenticate challenge that names your agent credential scheme and a link to registration docs. Do not send that client through a browser CAPTCHA.
- Use one stable JSON error shape (code, message, how_to_upgrade) so agents can branch. Prefer not to reveal whether a hidden SKU exists when the caller lacks the tier that may see it.
- Log class, credential id (if any), scope and HTTP status so you can see whether humans were accidentally folded into the bot path.
- Test three clients on the same path: a logged-out browser, a credentialed agent, and curl with no auth. The browser must still get usable HTML; only the agent path should change with scope.
Why should API responses for AI agents use capability tiers instead of a single allow or deny?
A hard gate answers yes or no. A tiered response answers how much: public stub, scoped agent JSON, or full human page. Marketplaces and SaaS APIs need the middle option so cooperative agents can work without you opening every field to scrapers.
- RFC 7235 says a 401 must carry WWW-Authenticate with at least one challenge when the request lacks valid credentials; a server that has valid credentials that still are not enough for the resource should use 403. Use that split: unknown caller → 401 + how to get a credential; known agent lacking scope → 403 with a machine-readable upgrade hint.
- Cloudflare's bot products separate verified bots, likely automated traffic and humans. Mirror that idea at your origin for response content: verification changes which serializer runs, not only whether a WAF rule fires.
How do I classify verified agents, humans and anonymous bots before building an API response?
Order of checks: (1) is this a verified search crawler you allowlist for HTML? (2) is there a valid, unrevoked agent credential (or Web Bot Auth signature) with scope for this route? (3) is there a human session? (4) otherwise treat as anonymous automation.
- verify_agent_credential (free) answers validity, revocation and optional scope in one call. Cache positive results briefly; fail closed on paid tiers if verification is down.
- Never key tiers only on User-Agent strings. Spoofable labels belong in logs, not in the branch that unlocks contract prices.
How do I keep human browsers on full HTML when AI agents get tiered API responses?
Graceful degradation fails when a bot fight breaks the shop. Do not put ordinary browsers into the anonymous-automation 401 path. Session or first-party cookie traffic stays on human_html even when an agent from the same NAT is limited.
- If you negotiate content types, document it: text/html for people and crawlers, application/json for agents with credentials. Same resource family, different representations — not a secret URL that only agents know while Google sees a blank page.
- Pair tiers with per-agent quotas on a separate control (covered on the rate-limit page). Tiers decide the body; quotas decide how often an agent may ask.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “Before I build this API response, verify the caller's agent credential and tell me whether it still has marketplace:browse so I can return the browse-tier JSON instead of a 401.”
{
"tool": "verify_agent_credential",
"arguments": {
"id": "crd_<id from the Authorization header>",
"scope": "marketplace:browse"
}
}- Free and keyless over MCP or GET /v1/credentials/verify. On success, serialize the browse-tier body; on failure, return 401 with WWW-Authenticate and a how_to_upgrade link, or 403 if the credential is valid but lacks scope.
- issue_agent_credential (needs your API key) mints short-lived scoped credentials for each tier you sell (browse, paid_read, settle).
- declare_intent / precall_check can record the intended fetch and purpose before your handler runs so the audit trail shows why a tier was granted.
- revoke_agent_credential drops one abusive agent without changing human HTML or other agents' tiers.
Rightshold verifies credentials and records intents you send. It does not host your API or choose your field lists; your handler must pick the response class after verification. Not legal advice.
What are the sources?
- RFC 7235: HTTP Authentication (401, WWW-Authenticate; 403 when credentials are inadequate)
- RFC 7231: 403 Forbidden
- MDN: HTTP authentication
- Cloudflare: Verified bots
- Cloudflare: Web Bot Auth
- RFC 9421: HTTP Message Signatures
- Agent credentials on Rightshold
- Agent gate on Rightshold
Related
- How to require a verified agent credential before an API returns paid or sensitive data
- How to set per-agent rate limits and quotas that do not throttle human browsers
- How to verify which AI agent is hitting your site with Web Bot Auth
- How to give an AI agent a scoped, expiring credential instead of a long-lived API key
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'