Answers · AI crawlers
Does Anubis (the proof-of-work challenge) actually stop AI scrapers, and what does it cost real visitors and good bots?
Last updated
TL;DR: It stops the cheap ones, not the determined ones. Anubis is an MIT-licensed reverse proxy by Xe Iaso (Techaro) that makes a browser solve a SHA-256 proof-of-work puzzle in JavaScript, then gives it a signed cookie. Most scrapers that pretend to be browsers don't run JavaScript, so they never get past it: GNOME's sysadmin reported that only 3% of about 81,000 requests in two and a half hours passed, and a Duke University Libraries pilot found it blocked more than 4 million unwanted requests a day. But the puzzle itself is cheap for anyone who bothers. Security researcher Tavis Ormandy calculated that a free-tier Google Cloud VM could mine a week's tokens for 11,508 sites (the project's GitHub star count, which he used as a stand-in for deployments) in about 6 minutes, and in August 2025 Codeberg said AI crawlers had learned to solve its Anubis challenges. The costs land on real visitors: a few seconds' wait at the default difficulty, JavaScript and cookies required (12 Duke users reported problems in one week, mostly with cookies disabled), and, in the default aggressive AI preset, blocks on AI agents that fetch a page for a person. Treat it as one cheap layer, not access control.
What should I check, step by step?
1. Decide what you're protecting. Anubis earns its keep on expensive dynamic pages (git forges, search, archives) hit by bots that don't run JavaScript. It is much less useful against scrapers that drive real headless browsers.
- Keep the default allow rules for search crawlers. Anubis's default policy allows Googlebot, Bingbot, Applebot, DuckDuckGo, the Internet Archive and others by user agent plus published IP ranges, so they aren't challenged. Check them after every upgrade.
- Pick an AI preset on purpose. The default aggressive preset blocks all AI and LLM user agents regardless of purpose, including assistants fetching a page because a person asked. The moderate preset blocks training and undocumented bots but allows documented user-fetch and search agents such as ChatGPT-User, OAI-SearchBot and Perplexity-User.
- Keep robots.txt too. Anubis's own preset notes that to fully opt out of some AI training, such as Google's, you must set the flags in robots.txt, because those controls aren't a separate crawler you can block.
- Leave keep-the-internet-working paths unchallenged: robots.txt, /.well-known/, favicons, feeds, and any API, git or webhook endpoints that non-browser clients need.
- Plan for visitors without JavaScript or cookies. Since version 1.20.0 there is a non-JavaScript meta-refresh challenge (off by default when LWN reviewed it); turn it on if your audience needs it, and publish a contact for people who get stuck.
- Don't raise the difficulty to fight determined scrapers. LWN reports difficulty 4 (the default) takes a few seconds and 6 can take minutes, and, as Ormandy argues, the cost falls on visitors' own devices far more than on a scraper's datacenter.
- Watch for the arms race: track the pass rate and origin load after rollout. When scrapers start passing, layer rate limits, verified-bot allowlists and per-path rules rather than relying on the puzzle.
How does Anubis decide which visitors get a proof-of-work challenge?
Anubis sits between your reverse proxy (Caddy, NGINX, Apache) and your application. Each request is matched against policy rules with three actions: allow, challenge or deny. By default it challenges anything that looks like a browser, usually any user agent containing Mozilla, because most scrapers claim to be browsers. Passing the challenge gets a signed JWT cookie, and Ormandy notes the default cookie lifetime in the code is 7 days.
- The idea comes from Hashcash, Adam Back's 1997 anti-spam scheme: a small compute cost that's nothing for one visitor but adds up for a bulk sender. Anubis's default policy also denies known pathological bots, allows listed search crawlers on verified IP ranges, and imports an aggressive AI-blocking preset.
Can AI scrapers get past Anubis?
It works against high-volume bots that don't execute JavaScript. GNOME sysadmin Bart Piotrowski's numbers, reported by LibreNews and OSnews, showed about 81,000 requests in two and a half hours with only 3% passing. Duke's pilot blocked more than 4 million unwanted requests a day while letting real users and well-behaved bots through. LWN lists the Linux kernel mailing list archive, sourcehut and FFmpeg among early users.
- It doesn't hold up against an operator willing to spend a little. Ormandy measured about 2^21 SHA-256 operations a second on a free-tier e2-micro VM and worked out that minting a week's tokens for 11,508 sites (the GitHub star count he used as a proxy for deployments) takes about 6 minutes, roughly no money, adding that the scheme has the problem backwards because it mostly limits people without resources. On August 15, 2025 Codeberg said, 'It seems like the AI crawlers learned how to solve the Anubis challenges', after a period of extreme slowness. Its staff said Anubis remained useful and that they were looking at other tools such as Iocaine. Drew DeVault, who had deployed Anubis on SourceHut, later called it the nuclear option and switched to go-away.
Does Anubis block real users, search engines or AI assistants?
People wait for the puzzle once per cookie period, a few seconds at the default difficulty, and need JavaScript and cookies, or the experimental meta-refresh challenge. In Duke's pilot, 12 people reported problems in one week, usually because cookies were disabled. Non-browser clients and unusual setups can be caught too: Ormandy opens his analysis by asking why Anubis was blocking his access to the Linux kernel.
- AI agents acting for a person are blocked under the aggressive preset whether or not they're well behaved, because it matches AI user agents regardless of purpose. If you want those through, use the moderate preset or a rule that lets verified agents bypass the challenge. Base that rule on a verified signature or published IP range, not on a header being present, since anything a client sends can be copied.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “This request says it's an AI agent fetching a page for a user and carries Web Bot Auth Signature headers. Does the signature verify, and which signer is it, so my middleware can let it skip the proof-of-work challenge?”
{
"tool": "gate_check",
"arguments": {
"policy": {
"require_web_bot_auth": true
},
"request": {
"method": "GET",
"url": "<the requested URL>",
"headers": {
"user-agent": "<as received>",
"signature-agent": "<as received>",
"signature-input": "<as received>",
"signature": "<as received>"
}
}
}
}- Free, no key. It resolves the signer's HTTPS key directory and returns allow or deny with reasons, plus the signer it resolved. Allow answers can be cached for at most 5 minutes, and for less when the signature expires sooner. allowed_signature_agents narrows it to signers you name, such as chatgpt.com.
- Anubis doesn't call Rightshold. You'd run this check in your own middleware or edge worker in front of Anubis and route only verified requests around the challenge. The Gate page (/gate) has free CC0 middleware snippets that run the same check.
- It verifies signatures, not addresses: it doesn't check IP lists or reverse DNS, so keep Anubis's IP-based allow rules for unsigned crawlers such as Googlebot.
Rightshold checks the Web Bot Auth signatures and gate policies you send it. It isn't affiliated with Anubis or Techaro, doesn't run challenges, sit in front of your site or block scrapers, and can't identify requests that carry no signature. A verified signature identifies the key's publisher, not that the agent is well behaved. Anubis details are from its docs, policy files and the reports cited, and may have changed in later releases.
What are the sources?
- LWN.net: Anubis sends AI scraperbots to a well-deserved fate (July 10, 2025)
- Anubis: default bot policy (botPolicies.yaml, TecharoHQ/anubis)
- Anubis: aggressive and moderate AI-blocking presets (TecharoHQ/anubis data/meta)
- Tavis Ormandy: Anubis (proof-of-work cost analysis)
- The Register: Codeberg beset by AI bots that now bypass Anubis defense (August 16, 2025)
- The Register: Anubis guards gates against hordes of LLM bot crawlers (July 9, 2025)
- OSnews: FOSS infrastructure is under attack by AI companies (GNOME Anubis numbers via LibreNews)
Related
- How to detect AI agents that pose as a regular Chrome browser
- How to sign your AI agent's HTTP requests with Web Bot Auth so websites can verify it
- How to catch crawlers that ignore robots.txt with a disallowed trap URL
- The Gate
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'