Answers · AI crawlers
Siri AI now answers questions from the web. How do I keep my site's content out of Siri and Apple's AI-generated answers without dropping out of Spotlight, Siri Suggestions and Safari search, and does blocking Applebot-Extended do that?
Last updated
TL;DR: Not with Applebot-Extended alone. Apple's Applebot page (published September 4, 2026) separates three uses of what Applebot crawls, each with its own control. Search in Spotlight, Siri and Safari follows your robots.txt rules for Applebot, so blocking Applebot takes you out of Apple search. Training of Apple's foundation models follows Applebot-Extended: disallowing it opts your content out of training, but Applebot-Extended doesn't crawl, and Apple says it doesn't affect search inclusion or ranking. The third use is the one Siri AI's web answers rely on: Apple says Applebot data may provide additional context and up-to-date content when AI models generate output in Apple products, such as answers to broad world knowledge questions in Siri and Search. The control for that is the nosnippet robots meta tag (or an X-Robots-Tag: applebot: nosnippet header), applied page by page. The cost is that suggestions for those pages show only the title, with no description. For paywalled pages, page-level schema.org isAccessibleForFree: false keeps the page eligible for search while Apple says it won't use the content as context for AI output. Siri AI started rolling out in beta in English on September 14, 2026.
What should I check, step by step?
1. Decide the three questions separately: Apple search (Applebot), training (Applebot-Extended) and AI-generated answers (nosnippet or isAccessibleForFree). One switch doesn't cover the others.
- To opt out of training, add a group "User-agent: Applebot-Extended" with "Disallow: /" (or Disallow lines for the sections you want excluded). Leave Applebot itself allowed if you want to stay in Spotlight, Siri and Safari search.
- To keep a page out of Siri and Search AI answers, add <meta name="robots" content="nosnippet"> to it, or send X-Robots-Tag: applebot: nosnippet as a header (that also works for PDFs and other non-HTML files). Apple says it won't use content tagged nosnippet as context for AI-generated output.
- Count the cost of nosnippet before applying it site-wide: Apple says it won't generate a description or web answer for that page, and suggestions to visit it show only the title. A standard nosnippet meta tag (not one aimed at applebot) applies to other search engines too, Google included.
- For paywalled or metered pages, mark the page's JSON-LD with "isAccessibleForFree": false. Apple says the page stays eligible for search but the content isn't used as context for AI output. It works per page; section-level hasPart markup isn't supported.
- Check what Applebot actually matches in your robots.txt. Apple says that if your rules don't mention Applebot but do mention Googlebot, Applebot follows the Googlebot rules, so a Googlebot group written for Google can quietly govern Apple too.
- Don't block the JavaScript, CSS or XHR your pages need: Apple says Applebot may render pages in a browser and can't index the content properly if those resources are blocked.
- Verify that a request claiming to be Applebot is from Apple before trusting or blocking it: Apple says genuine traffic reverse-resolves to *.applebot.apple.com (check that the name resolves back to the same IP), or matches a CIDR prefix in its published applebot.json file.
- Don't use Crawl-delay for Apple: Apple says Applebot doesn't follow it and adjusts its crawl rate itself when a site slows down or returns errors. Use a server or CDN rate limit if you need one.
- Keep a dated record of each change. Apple doesn't say how long a robots.txt or meta-tag change takes to apply, and it offers no publisher report of how often your pages appear in Siri answers.
What is the difference between Applebot, Applebot-Extended and nosnippet for Apple's AI features?
Applebot is Apple's crawler. Apple says the data it crawls powers search across Apple's ecosystem, including Spotlight, Siri and Safari, and that allowing Applebot in robots.txt lets your content appear in search results for Apple users worldwide. Applebot follows robots.txt rules aimed at it, and Apple says noindex keeps a page out of Spotlight and Siri Suggestions.
- Applebot-Extended is a control token, not a crawler. Apple says it doesn't crawl webpages and is only used to decide how data crawled by Applebot may be used: disallowing it opts your content out of training Apple's general-purpose foundation models powering generative AI features in Apple Intelligence, Services and Developer Tools. Pages that disallow it can still be in search results, and Apple says site rules for Applebot-Extended aren't considered in ranking for Search.
- AI-generated answers are a separate use. Apple says Applebot-crawled data may provide additional context and up-to-date content when AI models generate output for display in Apple products and services, for example answering broad world knowledge questions in Siri and Search with answers that may include links to sources. Publishers opt out of that with nosnippet on specific content; isAccessibleForFree: false does the same for paywalled pages. Apple also notes that even with Applebot-Extended disallowed and nosnippet applied, Applebot may still crawl your pages and they stay discoverable in Spotlight, Siri and Safari.
Does Siri AI use content from websites to answer questions?
Apple released Siri AI on September 14, 2026 as a beta in English, with French, Japanese, Korean, Portuguese and Spanish announced for the following month. Apple's newsroom says it can use broad world knowledge to get up-to-date information on virtually any topic, drawing on information from across the web. On iPad and Mac it's built into Spotlight; Apple said it wouldn't be available initially in the EU on iOS, iPadOS and watchOS.
- That makes the nosnippet decision a live one. Before, a site deciding about Apple mostly weighed search (Applebot) against training (Applebot-Extended). Now a third outcome, your content used as context in an answer that may or may not link to you, has its own switch, and blocking training doesn't flip it.
- Apple's page says answers may include links to sources, but it doesn't say when links appear, for which questions or how many. There's no Apple equivalent of a search console report for these answers, so your own logs and referrer data are the only measure of what allowing them brings in.
What do you lose by blocking Applebot or using nosnippet, and does Applebot follow Googlebot rules?
Blocking Applebot: you leave Apple search in Spotlight, Siri and Safari altogether, which is more than most sites mean to give up. Disallowing Applebot-Extended: Apple says no effect on search or ranking; the cost is only that your content isn't used to improve Apple's models. nosnippet: no description or web answer for that page, title only; a generic nosnippet also shortens your results on other search engines, so the applebot-specific X-Robots-Tag form is the narrower tool.
- The Googlebot fallback is easy to miss. If your file has a Googlebot group and no Applebot group, Apple says Applebot follows the Googlebot instructions, not your "User-agent: *" rules. Give Applebot its own group if you want different rules for Apple.
- Two Apple user agents behave differently. Applebot's search user-agent string looks like Safari with "(Applebot/0.1; +http://www.apple.com/go/applebot)" appended, and Apple says the advertised browser version changes over time, so match on the Applebot token. The iTMS user agent, which may also come from applebot.apple.com hosts, doesn't follow robots.txt; Apple says it only crawls URLs tied to content registered on Apple Podcasts.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “What does example.com's robots.txt tell Applebot-Extended and the other AI training crawlers today, which group did each one match, and is our site counted as opted out of AI training?”
{
"tool": "read_ai_policy",
"arguments": {
"domain": "example.com"
}
}- Free, no key, cached about 6 hours. For 16 AI crawler, search and user-fetch tokens, including Applebot-Extended (counted as a training token), it reports whether the path is allowed and which group matched, plus Content-Signal lines, ai.txt, W3C TDMRep and whether llms.txt exists. Pass a path to check one section.
- It doesn't check Applebot itself, apply Apple's Googlebot fallback (a token with no group of its own is matched against your "User-agent: *" group), or read nosnippet, X-Robots-Tag or isAccessibleForFree on your pages; check those in your own files and headers.
Rightshold reads a site's public crawl and AI-use signals. It isn't affiliated with Apple, can't see how Apple uses your content and can't make any crawler honor a preference. Controls are as described on Apple's Applebot page (published September 4, 2026) and the Siri AI release in Apple's newsroom (September 14, 2026), both checked on November 10, 2026; Apple may change them. Not legal advice.
What are the sources?
- Apple Support: About Applebot (Applebot-Extended, nosnippet, X-Robots-Tag, isAccessibleForFree; published September 4, 2026)
- Apple: Applebot IP CIDR prefixes (applebot.json)
- Apple Newsroom: Siri AI, a profoundly more capable and personal assistant, is here (September 14, 2026)
- Apple Newsroom: Major updates for Apple's software platforms are now available (September 14, 2026)
- IETF: RFC 9309, Robots Exclusion Protocol
Related
- Keeping your site out of Google AI Overviews without leaving Google Search
- How to block AI training crawlers in robots.txt without leaving AI search
- Should you block Amazonbot? Amazonbot vs Amzn-SearchBot vs Amzn-User
- Why an AI training opt-out or robots.txt is not access control
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'
