# How do I prove a reported eval score wasn't cherry-picked as the best of many runs?

**Short answer:** Write down how many runs you will do and how the reported number is computed before you start, record a receipt for every run as it finishes (including the bad and the crashed ones), and report the count, the mean and the spread across all of them. Picking the best of N runs, seeds, prompts or checkpoints raises the number even when nothing got better. The Leaderboard Illusion (Singh et al., 2025) found that some providers tested many private variants on Chatbot Arena and published only the best; it identified 27 private variants tested by Meta in the lead-up to the Llama 4 release, showed that the expected best of N noisy estimates is higher than any single one, and submitted two identical Aya-Vision-8B checkpoints that scored 1052 and 1069. Labeled honestly, the gap can be huge and legitimate: the Codex paper (Chen et al., 2021) reports 28.8% of HumanEval problems solved with one sample and 70.2% with 100 samples per problem. Both are fair when named; presenting the second as the first is not. The American Statistical Association's statement on p-values lists "proper inference requires full reporting and transparency" as a principle. On Rightshold, record_ai_run issues a signed receipt with a server-set issued_at for each run; put the plan hash, run_index and planned run count in params. Rightshold can't show that you didn't skip recording a run. What makes an omission visible is a plan hashed first plus the full list of receipt ids with run_index running 1 to N. This page is about selective reporting across runs. Pre-registering a pass/fail threshold, judging a negligible change with an equivalence margin, and making a single run reproducible are covered on separate pages.

## Checklist
1. Before the first run, write a run plan: the model snapshots or checkpoints, the number of runs, the seeds, any prompt variants, how failed runs are handled, and the statistic you will report (mean over runs, pass@1, or pass@k with k and samples per problem stated). Hash the plan file.
2. Give every run a run_index from 1 to N and the seed the plan assigns it. If you need more runs, write a new plan version first; don't keep running until a good one turns up.
3. Record a receipt as each run finishes: the results-file hash, the plan hash, run_index, the seed and the score. Record failed, crashed and timed-out runs too, with their outcome.
4. Report N, the mean, the spread (standard deviation or a confidence interval), and the minimum and maximum. If you also report the best run, label it best-of-N and give N.
5. Choose a checkpoint or prompt for release on a separate dev split, then run the chosen one on the held-out split. Selecting on the test split is the cherry-pick, even if every run is recorded.
6. If a run is excluded (an infrastructure failure, say), keep its receipt and give the reason. The exclusion rule should already be in the plan.
7. Give the reviewer the plan file and the full list of receipt ids. They check each receipt and signature with verify_ai_run, confirm run_index covers 1 to N with no gaps and the same plan hash, and confirm every issued_at is later than the plan's own receipt.

## Why does reporting the best of N eval runs inflate the score?
- Every eval score is a noisy estimate. If you run N variants or seeds and keep the highest, you are reporting the top of a distribution, not its centre. The Leaderboard Illusion (Singh et al., 2025) states it formally: with N of at least 2 and non-zero variance, the expected value of the best-performing variant is strictly greater than that of a single submission, so a best-of-N rating is biased upward.
- Their audit of Chatbot Arena found an undisclosed practice that let some providers test many private variants and publish or retract scores as they chose, with 27 private Meta variants observed in the lead-up to Llama 4. In simulation, testing 20 variants raised the best score found by about 50 points over no private testing.
- They also ran it for real. Two identical Aya-Vision-8B checkpoints scored 1052 and 1069, with four models ranked between them; two different Aya-Vision-32B candidates scored 1097 and 1059. Their recommendations include prohibiting score retraction after submission and disclosing how many private variants were tested before launch.

## Should I report pass@1, pass@k or the mean over seeds for an AI eval?
- Sampling many times and counting a problem as solved if any sample passes is a real, useful metric — as long as it is labeled. The Codex paper reports that Codex solved 28.8% of HumanEval problems with a single sample and 70.2% with 100 samples per problem. A reader who sees only "70.2%" with no k would badly overestimate single-shot performance.
- Say which one you are reporting, the k, the number of samples drawn per problem, the temperature, and how many runs the figure averages over. If runs vary, the spread is part of the result.
- The American Statistical Association's 2016 statement on p-values puts it as a principle: proper inference requires full reporting and transparency. For evals, that means reporting every run you planned, not just the one you liked.

## Can signed run receipts prove I reported every eval run?
- A record_ai_run receipt is signed by Rightshold, gets an issued_at time set by Rightshold when it is recorded, and sits on the public ledger. It shows that this results file, with this plan hash and run_index, was recorded at that time and hasn't changed since.
- It can't show that you recorded every run. Someone can still run experiments off the books and record only the good ones. What receipts add is that an honest full disclosure becomes checkable: the plan fixes N before any results exist, each run is pinned to a hash and a time, and a gap in run_index, a run recorded before its plan, or a score that doesn't match its results file is visible to anyone you give the ids to.
- If the selection happened earlier — a provider testing many private variants, or a team trying prompts on the test split before writing a plan — no receipt recorded afterwards can undo it. Write the plan first.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "Record a signed receipt for run 3 of the 5 planned in my hashed run plan: the plan hash, run index, seed, the results-file hash and its score, so a reviewer can check every planned run was reported."

```json
{
  "tool": "record_ai_run",
  "arguments": {
    "model": "<model under test, dated snapshot>",
    "provider": "<provider or eval-harness>",
    "prompt_sha256": "<sha256 of the frozen eval set file>",
    "output_sha256": "<sha256 of this run's per-item results>",
    "params": {
      "run": "multi-run-eval",
      "plan_sha256": "<sha256 of the run plan>",
      "run_index": 3,
      "runs_planned": 5,
      "seed": 1003,
      "temperature": 0.7,
      "statistic": "mean_over_runs",
      "score": 0.812,
      "outcome": "completed"
    },
    "ran_at": "2026-11-08T07:25:00Z"
  }
}
```
- Needs an API key. Only the hashes and fields you send are stored; record one receipt per run, including failed runs (outcome: failed).
- Record the plan itself first (record_action with action_type eval_plan.recorded and action_sha256 set to the plan hash) so every run receipt can be shown to come after it.
- params is kept as params_sha256 over sorted-key canonical JSON (up to 4,000 bytes), so keep the exact params object for each run.
- verify_ai_run is free and keyless: send each receipt and signature exactly as returned, plus that run's results text, to confirm the signature, that it is on record, and that the text matches output_sha256.
- ledger_verify is free and keyless and checks the public ledger the receipts sit on.

_Rightshold doesn't run your evals, compute your statistics or know how many runs you did. It signs and timestamps each run record you send. Receipts make a full disclosure checkable; they can't prove that no unrecorded runs exist. Not legal advice._

## Sources
- [Singh et al.: The Leaderboard Illusion (2025)](https://arxiv.org/abs/2504.20879)
- [Chen et al.: Evaluating Large Language Models Trained on Code (Codex, HumanEval, 2021)](https://arxiv.org/abs/2107.03374)
- [American Statistical Association: statement on statistical significance and p-values (March 7, 2016; principle 4, full reporting and transparency)](https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf)
- [Verified AI Runs on Rightshold](https://www.rightshold.com/verified-ai-runs)

## Related
- [How to pre-register an eval's pass/fail threshold and analysis plan before looking at the scores](https://www.rightshold.com/answers/how-to-pre-register-an-evals-pass-fail-threshold-and-analysis-plan-before-looking-at-the-scores)
- [How to decide whether an eval score change is negligible with a pre-set equivalence margin](https://www.rightshold.com/answers/how-to-decide-whether-an-eval-score-change-is-negligible-with-a-pre-set-equivalence-margin)
- [How to make LLM eval results reproducible and tamper-evident](https://www.rightshold.com/answers/how-to-make-llm-eval-results-reproducible-and-tamper-evident)
- [How to keep an eval set out of model training data and detect contamination](https://www.rightshold.com/answers/how-to-keep-an-eval-set-out-of-model-training-data-and-detect-contamination)

_Not legal advice. Updated 2026-11-08. JSON: https://www.rightshold.com/answers/how-to-prove-a-reported-eval-score-wasnt-cherry-picked-from-many-runs.json_