How do I pre-register an eval's pass/fail threshold and analysis plan before I look at the scores?

What should I check, step by step?

1. Before scoring — or before unblinding scores you already collected — write a plan: the primary metric and how it is computed, the pass/fail threshold (and whether the claim is superiority, non-inferiority or equivalence), secondary metrics, inclusion and exclusion rules, pairing, the sample size or stop rule, and how you will handle multiple comparisons.

2. If this is a comparison, include the equivalence or superiority margin and α in the same plan. "We'll see if it improved" is not a gate. How to decide whether an eval score change is negligible with a pre-set equivalence margin covers the TOST side.
3. Pin the instrument in the plan: the eval-set content hash and the judge id (judge model snapshot, rubric, prompt, scale, settings and parser). A threshold without a named set and judge can be moved by swapping either.
4. Hash the canonical plan file (SHA-256). Keep the file where it cannot be quietly overwritten: a version-control tag, or a registry such as OSF Registries or an AsPredicted PDF, which is time-stamped and has a unique URL for verification.
5. Run the eval against that plan. Do not change the threshold, drop items after seeing them, switch the primary metric, or keep adding items until the number crosses the line.
6. When you record the run, put plan_sha256 (and the eval-set and judge hashes) in params and the results-file hash as output_sha256. record_ai_run keeps params as params_sha256 over sorted-key canonical JSON.
7. Report confirmatory results against the plan, and label any extra cuts as exploratory. Planned and unplanned work are both useful; they are not the same claim.
8. If the threshold has to change (a new risk tolerance, a bug in the metric, a new set version), write a new plan with a new hash. Keep the old plan and say what changed. Do not edit the original file in place.

Why does looking at eval scores before setting the pass/fail threshold move the goalposts?

Nosek, Ebersole, DeHaven and Mellor (2018) write that progress depends on generating hypotheses from existing observations and testing them with new ones, and that mistaking the generation of postdictions for the testing of predictions reduces the credibility of findings. Ordinary biases such as hindsight — seeing an outcome as more predictable after it has occurred — make this easy to do by accident. Their remedy is to define the questions and the analysis plan before observing the outcomes.

• They describe hypothesizing after the results are known as circular: generating a hypothesis from the data and then evaluating it on the same data. Researchers also have many degrees of freedom to run different tests and report the attractive ones. In an LLM eval that looks like raising the pass bar after a strong run, lowering it after a miss so the model still "ships," dropping categories that failed, switching from exact match to a looser judge rubric once exact match looks bad, trying many prompt variants and presenting the best as the planned one, or watching a live dashboard and stopping when the number looks good.
• The Center for Open Science distinguishes planned (usually confirmatory) work, where results are held to the highest standards and p-values keep their diagnostic value, from unplanned (usually exploratory) work, which is for discovery and whose p-values lose that value. Both belong in a report if you label them. What does not belong is presenting an after-the-fact threshold as if it had been the gate all along.

What should I write in an eval analysis plan before I look at the scores?

Primary metric and its exact definition: the denominator, how items are aggregated, and any pairwise swap for an LLM judge.

• The pass/fail threshold and the claim type: for example "at least 92% on the frozen set," "no more than 1.5 points worse than the baseline" (non-inferiority), or "the difference sits inside ±δ" (equivalence).
• Inclusion and exclusion: which items count, how invalid or refused outputs are scored, and any rule for removing leaked or contaminated items. That removal rule has to be fixed in advance, or "we dropped the contaminated ones" becomes another way to move the gate after seeing scores.
• Multiple comparisons: if you will look at a dozen categories, say whether the ship gate is the primary metric only, a pre-specified hierarchy, or an adjustment such as Bonferroni or Holm.
• A stop rule: a fixed n, or a sequential rule written down beforehand. Do not keep adding items until the run passes.
• AsPredicted produces a one-page, time-stamped PDF with a unique URL; public pre-registrations cannot be modified and are backed up in the Wayback Machine. OSF Registries, described by the Center for Open Science, let you specify the plan in advance of the study (including before you analyse an existing data set). Hash the plan yourself even if you also register it, so the run receipt can point at bytes and not only at a URL.

How do I update an eval pass/fail threshold without silently editing the original plan?

The Center for Open Science notes that when you write up preregistered research you should transparently disclose any changes from the plan. A new risk tolerance, a bug in the scorer or a new eval-set version is a reason to write a new plan, not a reason to rewrite the old file.

• Give the new plan its own version and hash. Old results stay bound to the old plan_sha256. If someone silently edits the original file, the old receipt's hash will no longer match — which is the point of recording it.
• Exploratory follow-ups are fine when labeled. If you want to test a new threshold as a confirmatory claim, bind it to a new plan and, where you can, to items that were not already used to pick it.