How do I prove a reported eval score wasn't cherry-picked as the best of many runs?

What should I check, step by step?

1. Before the first run, write a run plan: the model snapshots or checkpoints, the number of runs, the seeds, any prompt variants, how failed runs are handled, and the statistic you will report (mean over runs, pass@1, or pass@k with k and samples per problem stated). Hash the plan file.

2. Give every run a run_index from 1 to N and the seed the plan assigns it. If you need more runs, write a new plan version first; don't keep running until a good one turns up.
3. Record a receipt as each run finishes: the results-file hash, the plan hash, run_index, the seed and the score. Record failed, crashed and timed-out runs too, with their outcome.
4. Report N, the mean, the spread (standard deviation or a confidence interval), and the minimum and maximum. If you also report the best run, label it best-of-N and give N.
5. Choose a checkpoint or prompt for release on a separate dev split, then run the chosen one on the held-out split. Selecting on the test split is the cherry-pick, even if every run is recorded.
6. If a run is excluded (an infrastructure failure, say), keep its receipt and give the reason. The exclusion rule should already be in the plan.
7. Give the reviewer the plan file and the full list of receipt ids. They check each receipt and signature with verify_ai_run, confirm run_index covers 1 to N with no gaps and the same plan hash, and confirm every issued_at is later than the plan's own receipt.

Why does reporting the best of N eval runs inflate the score?

Every eval score is a noisy estimate. If you run N variants or seeds and keep the highest, you are reporting the top of a distribution, not its centre. The Leaderboard Illusion (Singh et al., 2025) states it formally: with N of at least 2 and non-zero variance, the expected value of the best-performing variant is strictly greater than that of a single submission, so a best-of-N rating is biased upward.

• Their audit of Chatbot Arena found an undisclosed practice that let some providers test many private variants and publish or retract scores as they chose, with 27 private Meta variants observed in the lead-up to Llama 4. In simulation, testing 20 variants raised the best score found by about 50 points over no private testing.
• They also ran it for real. Two identical Aya-Vision-8B checkpoints scored 1052 and 1069, with four models ranked between them; two different Aya-Vision-32B candidates scored 1097 and 1059. Their recommendations include prohibiting score retraction after submission and disclosing how many private variants were tested before launch.

Should I report pass@1, pass@k or the mean over seeds for an AI eval?

Sampling many times and counting a problem as solved if any sample passes is a real, useful metric — as long as it is labeled. The Codex paper reports that Codex solved 28.8% of HumanEval problems with a single sample and 70.2% with 100 samples per problem. A reader who sees only "70.2%" with no k would badly overestimate single-shot performance.

• Say which one you are reporting, the k, the number of samples drawn per problem, the temperature, and how many runs the figure averages over. If runs vary, the spread is part of the result.
• The American Statistical Association's 2016 statement on p-values puts it as a principle: proper inference requires full reporting and transparency. For evals, that means reporting every run you planned, not just the one you liked.

Can signed run receipts prove I reported every eval run?

A record_ai_run receipt is signed by Rightshold, gets an issued_at time set by Rightshold when it is recorded, and sits on the public ledger. It shows that this results file, with this plan hash and run_index, was recorded at that time and hasn't changed since.

• It can't show that you recorded every run. Someone can still run experiments off the books and record only the good ones. What receipts add is that an honest full disclosure becomes checkable: the plan fixes N before any results exist, each run is pinned to a hash and a time, and a gap in run_index, a run recorded before its plan, or a score that doesn't match its results file is visible to anyone you give the ids to.
• If the selection happened earlier — a provider testing many private variants, or a team trying prompts on the test split before writing a plan — no receipt recorded afterwards can undo it. Write the plan first.