Answers · AI evidence
How do I prove an AI-generated report or summary cited sources that actually said what it claims?
Last updated
TL;DR: Capture the evidence at the moment the report is written, because the sources won't wait. For every claim, keep the exact bytes of the source the model was given (not just its URL), a SHA-256 of those bytes, the quoted passage and where it sits in the source, and a support verdict for the claim: supported, partly supported or not supported, and who or what judged it. Seal that grounding manifest together with the report's hash, so anyone holding the report and the saved sources can later recompute the hashes and re-read each passage next to its claim. This matters because citations from generative systems often don't support what they're attached to: in Stanford's 2023 audit of four generative search engines, only 51.5% of generated sentences were fully supported by their citations and only 74.5% of citations supported their sentence. And a link checked later may show something else: Pew Research found 38% of webpages that existed in 2013 were no longer accessible a decade later. On Rightshold, record_ai_run signs the report's hash with your grounding manifest's hash (as params) onto a public ledger, and verify_ai_run lets a reader check the report against the receipt for free.
What should I check, step by step?
1. Save each source exactly as the model received it: the retrieved text, PDF or HTML bytes, with the URL, the retrieval time and the HTTP status. For web pages, a WARC capture (the web archive format standardized as ISO 28500) keeps the response and its headers together.
- Hash each saved source with SHA-256 and keep the bytes somewhere durable. A hash with no copy of the source proves nothing to a reader who can't get the original.
- Split the report into claims (a sentence is the usual unit) and map each claim to the source and passage that backs it: source hash, quoted text, and character offsets or page numbers.
- Check each quote mechanically: the quoted passage must appear in the saved source byte for byte. If you use Anthropic's citations feature, its cited_text is extracted from your documents, so the pointer is valid by construction. That still only shows the passage exists, not that it supports the claim.
- Judge support per claim, separately from existence: supported, partly supported or not supported. Record who judged it (a named reviewer role, or the checker model and its version) and send unsupported claims back before the report goes out.
- Put the claim map, source hashes, verdicts and judge into one grounding manifest, serialize it canonically (sorted keys) and hash it.
- Seal the report hash and the manifest hash together at release time with an independent timestamp, so a manifest written after a complaint can't pass for one written before.
- Write down what the record can't show: that the sources themselves are true, that a claim with no citation was checked, or that the judge was right.
Why isn't a URL enough to prove what an AI-generated report's source said?
A URL points at whatever the server returns today. Pew Research Center's May 2024 study of Common Crawl samples found that a quarter of pages that existed between 2013 and 2023 were gone by October 2023, and 38% of pages from 2013. Pew counted only pages that no longer exist; pages whose content changed at the same address weren't measured, and those are the harder case, because the link still works and shows a different text.
- Memento (RFC 7089) is the web's standard way to ask an archive for a page as it was at a given time, and it helps when a public archive happened to capture the page. It can't help with sources it never saw, such as search results, paywalled pages, internal documents or the chunks your retrieval step pulled from a vector store. For those, your own saved copy is the only record of what the model was shown.
What should I check for each claim in an AI-generated report, and what does each check prove?
Existence: the cited source is one the model actually received in this run. Prove it by matching the citation to a saved source hash, not by re-fetching the URL.
- Quote accuracy: the passage quoted or pointed to is really in that source. This is a string match against the saved bytes and can be fully automated.
- Support: the passage backs the claim it's attached to. This is the part that fails most. Liu, Zhang and Liang define citation recall (share of statements fully supported by their citations) and citation precision (share of citations that support their statement); their human audit of Bing Chat, NeevaAI, perplexity.ai and YouChat found averages of 51.5% and 74.5%, and responses rated more useful tended to have less accurate citations. Support needs a reader, human or model, and the record should say which.
How do I seal an AI report's citation and grounding check together with its run record?
The report hash ties the record to one exact text: change a word and it no longer matches. The manifest hash ties that text to the specific sources, passages and verdicts you had at release. Sealing both in one signed, timestamped receipt shows the check existed before anyone disputed the report.
- With record_ai_run, put the manifest's SHA-256 and summary counts in params. Rightshold stores params as params_sha256 over sorted-key canonical JSON, so keep the exact params object; a reader recomputes its hash and compares it with the receipt.
- Send the report as output and add a contains check listing every citation marker you expect (for example [1] to [14]). Rightshold signs whether each marker appeared in the text. It doesn't open the sources or judge support; those results are yours, sealed as you sent them.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “Seal this AI-written market summary with its grounding check: 14 citations, 13 supported and 1 partly supported, with the hash of my grounding manifest (claim map, source hashes, verdicts), so a client can check later what was cited and checked before it went out.”
{
"tool": "record_ai_run",
"arguments": {
"model": "claude-sonnet-4-5-20250929",
"provider": "anthropic",
"prompt_sha256": "<sha256 of the exact request, including the retrieved source text>",
"output": "<the final summary text with its [1]..[14] markers>",
"params": {
"grounding_manifest_sha256": "<sha256 of canonical(grounding manifest)>",
"sources": 9,
"citations": 14,
"supported": 13,
"partly_supported": 1,
"unsupported": 0,
"judge": "analyst review + quote string match"
},
"checks": [
{
"type": "contains",
"all": [
"[1]",
"[2]",
"[3]",
"[4]",
"[5]",
"[6]",
"[7]",
"[8]",
"[9]",
"[10]",
"[11]",
"[12]",
"[13]",
"[14]"
]
}
],
"ran_at": "2026-11-07T22:00:00Z"
}
}- Needs an API key. Output text is hashed on arrival and never stored; only its SHA-256 and the check results go into the signed receipt, which is written to Rightshold's append-only ledger and anchored to Bitcoin through OpenTimestamps.
- verify_ai_run is free and needs no key: a client sends the receipt id and the summary text to confirm it's the version that was recorded, then hashes the canonical params object you share to match params_sha256, and hashes the manifest you share to match grounding_manifest_sha256 inside it.
- In recorded mode the receipt is marked reported_by caller: it proves what you recorded and when, not that your verdicts are right. Keep source copies and the manifest yourself; Rightshold doesn't hold them.
Rightshold doesn't fetch, store or read your sources, and doesn't judge whether a passage supports a claim. It signs and timestamps the hashes, parameters and deterministic text checks you send, and lets anyone verify them later. Not legal advice.
What are the sources?
- Liu, Zhang and Liang: Evaluating Verifiability in Generative Search Engines (arXiv 2304.09848)
- Pew Research Center: When Online Content Disappears (May 17, 2024)
- RFC 7089: HTTP Framework for Time-Based Access to Resource States (Memento)
- Library of Congress: WARC, Web ARChive file format (ISO 28500)
- Anthropic docs: Citations (cited_text and citation locations)
- RFC 8785: JSON Canonicalization Scheme (JCS)
- Verified AI Runs on Rightshold
Related
- How to check AI-generated citations for hallucinations
- How to prove which model version, prompt and settings produced an AI output
- How to check that an AI agent's delivered file matches the output hash recorded for its run
- How to prove what an AI model said at a specific time
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'