Answers · AI crawlers
Which Mistral AI crawlers visit my site, what's the difference between MistralAI-User, MistralAI-Index and MistralAI-Training, and how do I block Mistral from training on my content without disappearing from answers in Mistral's Vibe assistant?
Last updated
TL;DR: Mistral documents three robots.txt tokens, one per job. MistralAI-Training crawls the web to build datasets for training Mistral's generative models and, Mistral says, isn't used for search or live answers. MistralAI-Index crawls automatically to build the index behind Mistral search, which helps answer questions in Vibe, Mistral's assistant, and Mistral says what it collects isn't used for generative AI training of any kind. MistralAI-User fetches a page when a Vibe user's question needs it and links the source in the answer; it doesn't crawl on its own or collect training data. To refuse training but stay in Vibe's answers, add a group for MistralAI-Training with Disallow: / and leave the other two alone. Name MistralAI-Training explicitly: in robots.txt a crawler only follows the group that names it, or the * group if none does, so blocking MistralAI-User and MistralAI-Index leaves training under your * rules. Mistral publishes IP lists for the User and Index crawlers but none for the training crawler, so a request claiming to be MistralAI-Training can't be checked by address. Mistral is a signatory of the EU's General-Purpose AI Code of Practice, whose copyright chapter commits signatories to follow robots.txt when crawling.
What should I check, step by step?
1. Decide per job: training (MistralAI-Training), Mistral search indexing (MistralAI-Index) and live fetches for Vibe users (MistralAI-User). Each token is separate; a rule for one doesn't touch the others.
- To refuse training only, add "User-agent: MistralAI-Training" followed by "Disallow: /". Search indexing and user fetches keep working, so your pages can still be cited with a link in Vibe.
- To refuse all three, list the three User-agent lines in one group above "Disallow: /". Blocking MistralAI-User also means Vibe can't open your page when a user asks about it.
- Always name MistralAI-Training if training is what you object to. A group for MistralAI-User and MistralAI-Index, or for an undocumented name like MistralBot, doesn't apply to it; it falls back to your * group, which on most sites allows almost everything.
- Don't rely on the shared __usage_ai-training line for Mistral. As of November 11, 2026, the companies that had announced they'd honor it were OpenAI, Anthropic, Google and Microsoft; Mistral wasn't among them.
- Use path rules if you want Vibe to reach only some sections: a MistralAI-User group with Allow: lines for those paths and Disallow: / for the rest works, because the longest matching rule wins.
- Verify MistralAI-User and MistralAI-Index requests against Mistral's published IP lists (mistral.ai/mistralai-user-ips.json and mistral.ai/mistralai-index-ips.json) before giving them any special treatment, and refresh the lists regularly.
- Treat MistralAI-Training as an unverifiable claim: Mistral publishes no IP list for it and doesn't document request signing, so enforce a refusal with robots.txt plus an edge rule on the user agent rather than an allowlist.
- Keep dated copies of your robots.txt. If you later need to show what you told Mistral's crawlers and when, that's the record.
What is the difference between MistralAI-User, MistralAI-Index and MistralAI-Training?
Mistral's crawler page (docs.mistral.ai/robots, checked November 11, 2026) lists three agents. MistralAI-User is for user actions in Vibe: when a user asks a question, Vibe may visit a web page to help answer and include a link to the source in its response. Mistral says it isn't used for crawling the web automatically or for collecting generative AI training content. Its user-agent string is "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)".
- MistralAI-Index is for automated crawling for indexing only. It indexes content for Mistral search, which helps answer user questions in Vibe, and Mistral says content it crawls isn't used for generative AI training of any kind. Its user agent ends in "MistralAI-Index/1.0; +https://docs.mistral.ai/robots)".
- MistralAI-Training crawls web content to help build datasets for training Mistral's generative AI models, and Mistral says webmasters can disallow it in robots.txt. Mistral says it isn't used for search indexing or to answer live user queries in Vibe. Its user agent ends in "MistralAI-Training/1.0; +https://docs.mistral.ai/robots)".
- Mistral appears on the European Commission's list of signatories of the General-Purpose AI Code of Practice. The code's copyright chapter commits signatories to crawl only lawfully accessible content, follow robots.txt instructions and other machine-readable opt-outs, and publish information about their crawlers.
Does blocking MistralAI-User or MistralAI-Index also stop Mistral training on my site?
Under RFC 9309, a crawler obeys the group whose User-agent line matches its product token (case-insensitively), and only if no group names it does it fall back to the * group. Groups don't inherit from each other. So a file that blocks MistralAI-User and MistralAI-Index says nothing to MistralAI-Training, which then follows whatever your * group allows.
- This happens on real sites. On November 11, 2026, the robots.txt of one large French news site disallowed MistralAI-User, MistralAI-Index and a token called MistralBot, which Mistral doesn't document, but had no group for MistralAI-Training, leaving the training crawler under a * group that allows most of the site. Another large French daily's file allowed MistralAI-User only on a few sections and disallowed the rest, a pattern that keeps chosen pages available to Vibe users.
- A single shared training opt-out wouldn't fix this for Mistral yet. The "User-Agent: __usage_ai-training" group proposed in an IETF individual draft on October 30, 2026 had been backed by OpenAI, Anthropic, Google and Microsoft as of early November 2026, with Meta described as supportive but uncommitted; Mistral hadn't announced support. Until it does, the MistralAI-Training token is the documented way to say no.
How do you verify a request really comes from a Mistral AI crawler?
Mistral links two IP lists from its crawler page. On November 11, 2026, mistral.ai/mistralai-user-ips.json held 4 IPv4 addresses (file creation time February 19, 2025) and mistral.ai/mistralai-index-ips.json held 2 (creation time April 19, 2026). A request that says MistralAI-User or MistralAI-Index from an address outside those lists isn't from Mistral.
- There is no published list for MistralAI-Training, and Mistral's page doesn't mention Web Bot Auth or another signature scheme for any of the three. Anyone can type the training crawler's user agent, so you can't verify it, only refuse it. That's acceptable when you're refusing: a fake that claims to be MistralAI-Training and is blocked by your user-agent rule loses nothing you wanted to give it.
- robots.txt states a preference; it doesn't stop a crawler that ignores it. If you need to enforce a refusal, add a rule at your CDN or server that returns 403 for the MistralAI-Training user agent, and check your logs for requests from Mistral's published addresses that don't carry the matching token.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “Before we add Mistral rules, what does our robots.txt tell the AI training, AI search and user-fetch crawlers today, and which of them fall through to our * group?”
{
"tool": "read_ai_policy",
"arguments": {
"domain": "example.com"
}
}- Free, no key, cached about 6 hours. For 16 AI crawler, search and user-fetch tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot and others) it reports whether the path is allowed and which group matched; matched_group "*" is the fall-through this page warns about. It also reports Content-Signal lines, ai.txt, W3C TDMRep and whether llms.txt exists.
- It doesn't check the three MistralAI tokens yet, so read your Mistral groups yourself, and it reports what your files say, not what Mistral has crawled or used.
Rightshold reads a site's public crawl and AI-use signals. It isn't affiliated with Mistral AI, can't see what Mistral has crawled or trained on and can't make any company honor a preference. Crawler descriptions, user-agent strings and IP lists are as Mistral published them, and the robots.txt files described were fetched, on November 11, 2026; they may change. Not legal advice.
What are the sources?
- Mistral Docs: Mistral crawlers (MistralAI-User, MistralAI-Index, MistralAI-Training)
- Mistral AI: MistralAI-User IP addresses (JSON)
- Mistral AI: MistralAI-Index IP addresses (JSON)
- European Commission: The General-Purpose AI Code of Practice (signatories)
- IETF: RFC 9309, Robots Exclusion Protocol
- OpenAI: Making it easier for websites to express AI training preferences (October 30, 2026)
- TechRadar: Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard (November 8, 2026)
Related
- How to block AI training crawlers in robots.txt without leaving AI search
- What is the __usage_ai-training robots.txt line, and which AI companies will honor it?
- Do AI companies have to follow robots.txt in the EU under the AI Act code of practice?
- Why an AI training opt-out or robots.txt is not access control
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'