How do I make LLM eval results reproducible and tamper-evident?

What should I check, step by step?

1. Use a dated or pinned model ID (for example gpt-4o-2024-08-06), never a moving alias, and save the model value the API returns, since that's what actually answered.

2. Record the generation settings: temperature, top_p, max tokens, seed, tools and system prompt. With OpenAI, also save system_fingerprint from each response, since it changes when the backend configuration changes.
3. Hash the inputs: the dataset file, the prompt templates and the scorer or grader code (a git commit is fine). Put the hashes in the log.
4. Run more than one epoch and report the spread, not just the best run. Sampling makes single runs noisy, and OpenAI calls seed best effort.
5. Keep provider request and response ids. Inspect, for example, logs ModelEvent.request_ids and ModelOutput.response_id for OpenAI, Anthropic and several other providers.
6. When the run is final, compute a SHA-256 of the log file and get that hash timestamped independently, so anyone can see if a score was edited later.
7. If you edit scores after review, use a tool that keeps the history (Inspect's edit_score keeps the original value and provenance) and hash the edited log as a new version.

Why exact reproduction usually fails

OpenAI's API reference says seed makes a best effort to sample deterministically and determinism is not guaranteed. A changed system_fingerprint means the backend changed between runs.

• Models are retired. OpenAI's deprecations page lists shutdown dates and says GET /v1/models returns a shutdown_date for models that have one. After shutdown, an old eval can't be re-run on that model at all, so the saved log is the only record.
• Anthropic's models overview says every Claude model ID is a pinned snapshot, including the dateless IDs used from the 4.6 generation on, while for earlier models the short alias is a pointer that resolves to a dated ID. If your config used an alias, record the dated ID it resolved to.

What a good eval log contains

Inspect's EvalLog is a useful reference. It has the eval spec (task, model, creation time), the plan (solvers and generation config), aggregate results, token usage, and every sample with its input, output, target and score. inspect log export-config writes the run config back out so the same setup can be run again.

• The log is a file, and Inspect's docs show reading it, editing scores or tags, and writing it back. That's useful for corrections, but it means the file alone doesn't show the published numbers are the ones the run produced. A hash recorded and timestamped at the end of the run does.
• Keep the dataset hash next to the score. A result is meaningless if the test set changed between two runs being compared.

Deterministic checks versus model graders

Rules like 'is valid JSON matching this schema', 'contains no API keys' or 'matches this regex' give the same answer every time and can be re-checked by anyone with the output.

• Model-graded scores change with the grader model. Record the grader's model snapshot and prompt like any other model call, and treat its scores as less stable.