# How do I pre-register an eval's pass/fail threshold and analysis plan before I look at the scores?

**Short answer:** Write the metric, the pass/fail threshold, inclusion and exclusion rules, the multiple-comparison plan and the stop rule down before you look at the scores, then hash that plan and store the hash with the later run record. Nosek, Ebersole, DeHaven and Mellor (2018) call this preregistration: define the research questions and analysis plan before observing the outcomes, so predictions stay distinct from postdictions. The Center for Open Science puts it plainly: the same data cannot be used to generate and test a hypothesis. In LLM evals, peeking and then picking a friendlier threshold, dropping hard items, or promoting a secondary metric that "also looks good" is hypothesizing after the results are known. You may update a threshold by publishing a new versioned plan, not by editing the old one in place. record_ai_run stores the plan hash in params (kept as params_sha256) alongside the later output hash; verify_ai_run checks the receipt.

## Checklist
1. Before scoring — or before unblinding scores you already collected — write a plan: the primary metric and how it is computed, the pass/fail threshold (and whether the claim is superiority, non-inferiority or equivalence), secondary metrics, inclusion and exclusion rules, pairing, the sample size or stop rule, and how you will handle multiple comparisons.
2. If this is a comparison, include the equivalence or superiority margin and α in the same plan. "We'll see if it improved" is not a gate. How to decide whether an eval score change is negligible with a pre-set equivalence margin covers the TOST side.
3. Pin the instrument in the plan: the eval-set content hash and the judge id (judge model snapshot, rubric, prompt, scale, settings and parser). A threshold without a named set and judge can be moved by swapping either.
4. Hash the canonical plan file (SHA-256). Keep the file where it cannot be quietly overwritten: a version-control tag, or a registry such as OSF Registries or an AsPredicted PDF, which is time-stamped and has a unique URL for verification.
5. Run the eval against that plan. Do not change the threshold, drop items after seeing them, switch the primary metric, or keep adding items until the number crosses the line.
6. When you record the run, put plan_sha256 (and the eval-set and judge hashes) in params and the results-file hash as output_sha256. record_ai_run keeps params as params_sha256 over sorted-key canonical JSON.
7. Report confirmatory results against the plan, and label any extra cuts as exploratory. Planned and unplanned work are both useful; they are not the same claim.
8. If the threshold has to change (a new risk tolerance, a bug in the metric, a new set version), write a new plan with a new hash. Keep the old plan and say what changed. Do not edit the original file in place.

## Why does looking at eval scores before setting the pass/fail threshold move the goalposts?
- Nosek, Ebersole, DeHaven and Mellor (2018) write that progress depends on generating hypotheses from existing observations and testing them with new ones, and that mistaking the generation of postdictions for the testing of predictions reduces the credibility of findings. Ordinary biases such as hindsight — seeing an outcome as more predictable after it has occurred — make this easy to do by accident. Their remedy is to define the questions and the analysis plan before observing the outcomes.
- They describe hypothesizing after the results are known as circular: generating a hypothesis from the data and then evaluating it on the same data. Researchers also have many degrees of freedom to run different tests and report the attractive ones. In an LLM eval that looks like raising the pass bar after a strong run, lowering it after a miss so the model still "ships," dropping categories that failed, switching from exact match to a looser judge rubric once exact match looks bad, trying many prompt variants and presenting the best as the planned one, or watching a live dashboard and stopping when the number looks good.
- The Center for Open Science distinguishes planned (usually confirmatory) work, where results are held to the highest standards and p-values keep their diagnostic value, from unplanned (usually exploratory) work, which is for discovery and whose p-values lose that value. Both belong in a report if you label them. What does not belong is presenting an after-the-fact threshold as if it had been the gate all along.

## What should I write in an eval analysis plan before I look at the scores?
- Primary metric and its exact definition: the denominator, how items are aggregated, and any pairwise swap for an LLM judge.
- The pass/fail threshold and the claim type: for example "at least 92% on the frozen set," "no more than 1.5 points worse than the baseline" (non-inferiority), or "the difference sits inside ±δ" (equivalence).
- Inclusion and exclusion: which items count, how invalid or refused outputs are scored, and any rule for removing leaked or contaminated items. That removal rule has to be fixed in advance, or "we dropped the contaminated ones" becomes another way to move the gate after seeing scores.
- Multiple comparisons: if you will look at a dozen categories, say whether the ship gate is the primary metric only, a pre-specified hierarchy, or an adjustment such as Bonferroni or Holm.
- A stop rule: a fixed n, or a sequential rule written down beforehand. Do not keep adding items until the run passes.
- AsPredicted produces a one-page, time-stamped PDF with a unique URL; public pre-registrations cannot be modified and are backed up in the Wayback Machine. OSF Registries, described by the Center for Open Science, let you specify the plan in advance of the study (including before you analyse an existing data set). Hash the plan yourself even if you also register it, so the run receipt can point at bytes and not only at a URL.

## How do I update an eval pass/fail threshold without silently editing the original plan?
- The Center for Open Science notes that when you write up preregistered research you should transparently disclose any changes from the plan. A new risk tolerance, a bug in the scorer or a new eval-set version is a reason to write a new plan, not a reason to rewrite the old file.
- Give the new plan its own version and hash. Old results stay bound to the old plan_sha256. If someone silently edits the original file, the old receipt's hash will no longer match — which is the point of recording it.
- Exploratory follow-ups are fine when labeled. If you want to test a new threshold as a confirmatory claim, bind it to a new plan and, where you can, to items that were not already used to pick it.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "Record a signed receipt for this eval run against a plan I hashed before looking at the scores: the plan hash, the pass/fail threshold, hashes of the frozen eval set and judge, and the results-file hash."

```json
{
  "tool": "record_ai_run",
  "arguments": {
    "model": "<model or system under test, dated snapshot>",
    "provider": "<provider or eval-harness>",
    "prompt_sha256": "<sha256 of the frozen eval set file>",
    "output_sha256": "<sha256 of the per-item scores and results table>",
    "params": {
      "run": "preregistered-eval",
      "plan_sha256": "<sha256 of the pre-registered analysis plan>",
      "plan_version": "v1",
      "judge_id": "<sha256 of rubric + judge prompt + scale + settings>",
      "eval_set_sha256": "<sha256>",
      "metric": "holdout_accuracy",
      "threshold": 0.92,
      "claim": "pass_if_primary_ge_threshold",
      "alpha": 0.05,
      "n_items": 400,
      "primary_score": 0.934,
      "passed": true,
      "exploratory_notes": "none"
    },
    "ran_at": "2026-11-08T07:05:00Z"
  }
}
```
- Needs an API key. Only hashes and the fields you send are stored. params is kept as params_sha256 over sorted-key canonical JSON (up to 4,000 bytes), so keep the exact params object.
- Hash the plan and record plan_sha256 before you look at the scores. A threshold written after seeing the result is not pre-registered, even if you later put it in params.
- verify_ai_run is free and needs no key: send the receipt id plus the results file to confirm it matches output_sha256, then recompute params_sha256 from the params object you share, and recompute plan_sha256 from the dated plan file.
- ledger_verify is free and keyless and checks the public ledger the receipt sits on.

_Rightshold does not write your analysis plan, score your eval or decide whether a run passed. It signs and timestamps the hashes and fields you send so a later reader can see which plan a result was claimed against. A receipt does not prove the plan was written before the scores unless the plan file's own dated record (or registry timestamp) shows that. Not legal advice._

## Sources
- [Nosek, Ebersole, DeHaven and Mellor: The preregistration revolution (Proc Natl Acad Sci USA, 2018)](https://pmc.ncbi.nlm.nih.gov/articles/PMC5856500/)
- [Center for Open Science: Preregistration](https://www.cos.io/initiatives/prereg)
- [AsPredicted: pre-registration platform (time-stamped one-page PDFs)](https://aspredicted.org/)
- [Verified AI Runs on Rightshold](https://www.rightshold.com/verified-ai-runs)

## Related
- [How to decide whether an eval score change is negligible with a pre-set equivalence margin](https://www.rightshold.com/answers/how-to-decide-whether-an-eval-score-change-is-negligible-with-a-pre-set-equivalence-margin)
- [How to version an LLM judge and keep a frozen eval set so scores stay comparable](https://www.rightshold.com/answers/how-to-version-an-llm-as-a-judge-and-keep-a-frozen-eval-set-so-scores-stay-comparable)
- [How to report false-positive rates with confidence intervals on a frozen holdout](https://www.rightshold.com/answers/how-to-report-false-positive-rates-with-confidence-intervals-on-a-frozen-holdout)
- [How to make LLM eval results reproducible and tamper-evident](https://www.rightshold.com/answers/how-to-make-llm-eval-results-reproducible-and-tamper-evident)

_Not legal advice. Updated 2026-11-08. JSON: https://www.rightshold.com/answers/how-to-pre-register-an-evals-pass-fail-threshold-and-analysis-plan-before-looking-at-the-scores.json_