# Does a passing AI eval score prove what the AI did in one specific incident?

**Short answer:** No. An eval measures a rate: how often a model, prompt and settings passed the cases you chose, on the day you ran it. An incident is one event: one input, one retrieved context, one reported model version, one output, at one time. A 98% pass rate is fully consistent with this case being in the 2%, and if the incident's input wasn't in the test set the score says nothing direct about it. What answers "what happened" is a per-run record kept when it happened: the exact request and retrieved context, the model the provider reports, the settings, the exact output and the provider's request id, hashed and timestamped. Evals answer different questions: was this kind of failure foreseeable, did you test for it, and did a later change make it more or less likely. Keep both kinds of evidence, label which is which, and preserve the incident record before you change the prompt, model or retrieval.

## Checklist
1. Freeze the incident first. Pull the run record (request body with system prompt version, retrieved documents and tool results, the model value the provider returned, settings, exact output, request id) before anyone edits the prompt, swaps the model or re-indexes documents. Provider copies are short-lived: OpenAI's data controls page lists abuse-monitoring logs kept up to 30 days by default, and Anthropic says it deletes API inputs and outputs within 30 days by default.
2. Write the two questions down separately. What happened in this run is answered by the run record. How often does this happen is answered by evals. Mixing them is how a pass rate ends up quoted as if it described one conversation.
3. Check whether the eval set covers the incident at all: the same kind of input, the same retrieval source, the same tools. If it doesn't, the old score is about other cases. Add the incident input and close variants as new cases.
4. Compare configurations. If the model value, system_fingerprint (OpenAI) or modelVersion (Gemini) in the incident record differs from the eval run's, or the prompt template commit differs, the eval measured a different setup.
5. Re-run the incident input many times on the pinned model and settings and report the failure rate with its sample size. One clean re-run doesn't show the original output didn't happen: OpenAI calls seed best effort and says model outputs are variable by nature.
6. Quote eval results with n. Zero failures in n independent trials still leaves a 95% upper bound of about 3/n on the failure rate (the statistical rule of three), so 0 failures in 100 runs is consistent with a failure rate of up to about 3%.
7. Keep the eval run itself checkable: hashes of the dataset, the scorer version, the settings and the result log, with a signed timestamp. "We tested this" is evidence only if someone else can see what was tested and when.
8. If the EU AI Act's high-risk rules apply to your system, they expect automatic logs (Article 12), kept at least six months by providers (Article 19) and deployers (Article 26(6)). Regulation (EU) 2026/1744 moved these Chapter III requirements to 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems.

## What does an AI eval pass rate actually measure, and what doesn't it tell you about one conversation?
- An eval runs a fixed set of cases through a configuration and counts passes. Its result is a property of that set, that configuration and that day. It is the right tool for comparing versions, catching regressions and estimating how often a kind of failure happens.
- It isn't a record of any production conversation. Production inputs differ from test cases, retrieved context changes as documents are edited, and providers change backends: OpenAI documents system_fingerprint as the backend configuration fingerprint and tells you to compare it with seed when outputs differ.
- A small n hides rare failures. A failure that happens once in 500 conversations will usually not show up in a 100-case eval, and at production volume it still happens every day.

## What evidence do I need to show what an AI system did in one specific incident?
- The exact inputs the model saw: system prompt, conversation so far, retrieved passages and tool results, as sent. Logging only the user's message loses most of what decided the answer.
- The provider's own identifiers: the model value in the response (not only the alias you requested), system_fingerprint or modelVersion where returned, and the request id. OpenAI recommends logging x-request-id in production; Anthropic returns a request-id header on every response.
- The exact output and what happened next: whether it was shown to a user unchanged, edited by post-processing, or acted on by an agent. Then a hash of each part and a timestamp from someone other than you, so a record rebuilt after the complaint can be told apart from one kept at the time.
- NIST's Generative AI Profile (NIST AI 600-1) lists incident disclosure among its primary considerations and says logging, recording and analyzing incidents, plus change-management records, version history and metadata, help the people responding to them.

## How should eval results and per-run records be used together in an AI incident dispute?
- Use the run record to establish the facts of the incident: what was asked, what context was supplied, which model answered and what it said. Use evals to answer whether the failure was a known risk, whether you tested for it and how often it happens.
- Don't fix first. Changing the prompt or model before saving the incident's configuration destroys the comparison you'll need. For serious incidents with high-risk systems, EU AI Act Article 73(6) says the provider must not alter the system in a way that may affect later evaluation of the causes before informing the authorities.
- After the fix, run the old and new configurations on the incident input and its variants and keep both results. That shows the change addressed this failure, which neither the old score nor the new one shows alone.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "Record a signed receipt for the incident run as I captured it: the reported model, hashes of the exact request (system prompt, retrieved context, tool results) and of the output the user saw, the provider request id and my incident reference."

```json
{
  "tool": "record_ai_run",
  "arguments": {
    "model": "claude-sonnet-4-5-20250929",
    "provider": "anthropic",
    "prompt_sha256": "<sha256 of the exact request JSON, including system prompt and retrieved context>",
    "output_sha256": "<sha256 of the exact output the user saw>",
    "params": {
      "request_id": "<request-id header value>",
      "template_commit": "<git sha of the prompt template>",
      "retrieval_snapshot": "<index or document version id>",
      "incident_ref": "<your ticket id>"
    },
    "ran_at": "2026-11-09T15:12:00Z"
  }
}
```
- Needs an API key. Prompt and output can be sent as hashes only; text sent in plain is hashed on arrival and never stored. params is stored as params_sha256 over sorted-key canonical JSON, so keep the exact params object.
- ran_at is the time you report; the receipt's issued_at is Rightshold's clock. A receipt recorded after the complaint shows the record hasn't changed since then, not that it existed at ran_at. Recording each run as it returns is what makes the timing evidence.
- Record the eval run the same way (dataset, scorer and result-log hashes in params) so the eval evidence is checkable too. verify_ai_run is free and needs no key: anyone can compare a prompt or output text with the recorded hashes.
- In recorded mode the receipt is marked reported_by caller: it shows what you recorded, not that the provider served that model or that the output was right.

_Rightshold records the hashes and fields you send, signs them and lets anyone check them later. It doesn't run your evals, decide what caused an incident, or judge whether an output was correct. Whether the EU AI Act applies to your system, and what a court or regulator will accept, are questions for a lawyer. Not legal advice._

## Sources
- [EUR-Lex: Regulation (EU) 2024/1689 (AI Act), Articles 12, 19, 26 and 73](https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng)
- [EUR-Lex: Regulation (EU) 2026/1744 (Digital Omnibus on AI), new dates for Chapter III](https://eur-lex.europa.eu/eli/reg/2026/1744/oj/eng)
- [NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative AI Profile (incident disclosure)](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf)
- [OpenAI API reference: Overview (x-request-id, model outputs are variable, pinned versions and evals)](https://platform.openai.com/docs/api-reference/debugging-requests)
- [OpenAI API reference: The chat completion object (model, system_fingerprint)](https://platform.openai.com/docs/api-reference/chat/object)
- [OpenAI: Data controls in the OpenAI platform (abuse monitoring retention)](https://platform.openai.com/docs/guides/your-data)
- [Gemini API reference: GenerateContentResponse (modelVersion, responseId)](https://ai.google.dev/api/generate-content)
- [Anthropic: Errors (request-id header)](https://platform.claude.com/docs/en/api/errors)
- [Anthropic Privacy Center: How long do you store my organization's data?](https://privacy.claude.com/en/articles/7996866-how-long-do-you-store-my-organization-s-data)
- [Verified AI Runs on Rightshold](https://www.rightshold.com/verified-ai-runs)

## Related
- [How to prove which model version, prompt and settings produced an AI output](https://www.rightshold.com/answers/how-to-prove-which-model-version-prompt-and-settings-produced-an-ai-output)
- [How to prove a reported eval score wasn't cherry-picked from many runs](https://www.rightshold.com/answers/how-to-prove-a-reported-eval-score-wasnt-cherry-picked-from-many-runs)
- [How to investigate a complaint that your AI chatbot made something up](https://www.rightshold.com/answers/how-to-investigate-a-complaint-that-your-ai-chatbot-made-something-up)
- [Verified AI Runs](https://www.rightshold.com/verified-ai-runs)

_Not legal advice. Updated 2026-11-09. JSON: https://www.rightshold.com/answers/does-a-passing-ai-eval-prove-what-happened-in-one-specific-incident.json_