Does a passing AI eval score prove what the AI did in one specific incident?

What should I check, step by step?

1. Freeze the incident first. Pull the run record (request body with system prompt version, retrieved documents and tool results, the model value the provider returned, settings, exact output, request id) before anyone edits the prompt, swaps the model or re-indexes documents. Provider copies are short-lived: OpenAI's data controls page lists abuse-monitoring logs kept up to 30 days by default, and Anthropic says it deletes API inputs and outputs within 30 days by default.

2. Write the two questions down separately. What happened in this run is answered by the run record. How often does this happen is answered by evals. Mixing them is how a pass rate ends up quoted as if it described one conversation.
3. Check whether the eval set covers the incident at all: the same kind of input, the same retrieval source, the same tools. If it doesn't, the old score is about other cases. Add the incident input and close variants as new cases.
4. Compare configurations. If the model value, system_fingerprint (OpenAI) or modelVersion (Gemini) in the incident record differs from the eval run's, or the prompt template commit differs, the eval measured a different setup.
5. Re-run the incident input many times on the pinned model and settings and report the failure rate with its sample size. One clean re-run doesn't show the original output didn't happen: OpenAI calls seed best effort and says model outputs are variable by nature.
6. Quote eval results with n. Zero failures in n independent trials still leaves a 95% upper bound of about 3/n on the failure rate (the statistical rule of three), so 0 failures in 100 runs is consistent with a failure rate of up to about 3%.
7. Keep the eval run itself checkable: hashes of the dataset, the scorer version, the settings and the result log, with a signed timestamp. "We tested this" is evidence only if someone else can see what was tested and when.
8. If the EU AI Act's high-risk rules apply to your system, they expect automatic logs (Article 12), kept at least six months by providers (Article 19) and deployers (Article 26(6)). Regulation (EU) 2026/1744 moved these Chapter III requirements to 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems.

What does an AI eval pass rate actually measure, and what doesn't it tell you about one conversation?

An eval runs a fixed set of cases through a configuration and counts passes. Its result is a property of that set, that configuration and that day. It is the right tool for comparing versions, catching regressions and estimating how often a kind of failure happens.

• It isn't a record of any production conversation. Production inputs differ from test cases, retrieved context changes as documents are edited, and providers change backends: OpenAI documents system_fingerprint as the backend configuration fingerprint and tells you to compare it with seed when outputs differ.
• A small n hides rare failures. A failure that happens once in 500 conversations will usually not show up in a 100-case eval, and at production volume it still happens every day.

What evidence do I need to show what an AI system did in one specific incident?

The exact inputs the model saw: system prompt, conversation so far, retrieved passages and tool results, as sent. Logging only the user's message loses most of what decided the answer.

• The provider's own identifiers: the model value in the response (not only the alias you requested), system_fingerprint or modelVersion where returned, and the request id. OpenAI recommends logging x-request-id in production; Anthropic returns a request-id header on every response.
• The exact output and what happened next: whether it was shown to a user unchanged, edited by post-processing, or acted on by an agent. Then a hash of each part and a timestamp from someone other than you, so a record rebuilt after the complaint can be told apart from one kept at the time.
• NIST's Generative AI Profile (NIST AI 600-1) lists incident disclosure among its primary considerations and says logging, recording and analyzing incidents, plus change-management records, version history and metadata, help the people responding to them.

How should eval results and per-run records be used together in an AI incident dispute?

Use the run record to establish the facts of the incident: what was asked, what context was supplied, which model answered and what it said. Use evals to answer whether the failure was a known risk, whether you tested for it and how often it happens.

• Don't fix first. Changing the prompt or model before saving the incident's configuration destroys the comparison you'll need. For serious incidents with high-risk systems, EU AI Act Article 73(6) says the provider must not alter the system in a way that may affect later evaluation of the causes before informing the authorities.
• After the fix, run the old and new configurations on the incident input and its variants and keep both results. That shows the change addressed this failure, which neither the old score nor the new one shows alone.