How do I decide whether an eval score change is negligible using a pre-set equivalence margin (TOST)?

What should I check, step by step?

1. Before you score the new system — or at least before you look at the new scores — write down the metric, whether items are paired, the smallest important difference δ, α, and the interval method.

2. Choose δ as the smallest change that would actually alter the decision (a ship gate, a cost/quality tradeoff, a risk budget). Do not pick δ after seeing the interval so that the result just fits.
3. Prefer a paired comparison when both systems score the same frozen items: take the per-item difference, then a confidence interval for the mean difference. Use an unpaired interval only when the item sets differ, and say so.
4. Compute one confidence interval for the difference (new minus old). Do not decide equivalence by whether two separate score intervals overlap.
5. For TOST at level α, check that the (1−2α) interval for the difference lies entirely inside (−δ, δ). Equivalently, reject both one-sided nulls H0: Δ ≤ −δ and H0: Δ ≥ +δ at level α. At α = 0.05 that is the 90% interval inside the margin.
6. Report the interval, δ, α, n, pairing and the method. Distinguish four outcomes: equivalent; different from zero but not equivalent; a real but practically negligible difference (the interval excludes zero and still sits inside ±δ); and undecided (the interval is too wide to fit inside the margin).
7. Do not claim equivalence from p > 0.05 on a difference test, or from an estimated-power check after a non-significant result (the older "power approach" Schuirmann compared TOST against).
8. Hash the plan (δ, α, pairing, metric, inclusion rules) together with the eval-set hash and judge id. Put those values in params when you record the run, and the results-file hash as output_sha256.

Why isn't a non-significant eval score change the same as a negligible one?

A usual significance test asks whether you can reject a difference of exactly zero. Failing to reject that null means the data are not surprising under "no difference." It does not mean the difference is small. Lakens (2017) notes that researchers often conclude an effect is absent from a non-significant result, and that when p > α the only formally correct conclusion is that the data are not surprising assuming the null; the test may simply have lacked power.

• An underpowered eval — a small holdout, a noisy LLM judge, high item-to-item variance — will often fail to detect a change even when the true change is large. That is the opposite of evidence that the change is negligible.
• Equivalence tests flip the burden of proof. Walker and Nowacki (2011) explain that in a traditional comparison the burden rests on showing a difference, so if the evidence is weak, equality cannot be ruled out; in an equivalence study the goal is to demonstrate equivalency, so that is where the burden rests. The type I error you control is wrongly declaring two systems equivalent.
• Superiority (the new system is better by at least some amount), difference (not exactly zero) and equivalence (the difference sits inside ±δ) are three different claims. A result can be statistically different from zero and still equivalent if the whole interval sits inside the margin — a real but unimportant change — or neither significant nor equivalent, which means you do not yet know.

How do I pick an equivalence margin and test the difference between two eval scores?

δ is the smallest effect size of interest (SESOI; Lakens). For an eval, that is the smallest accuracy, F1 or win-rate change that would change your decision. Set it from product cost, risk or a historical meaningful gap — not from this run's interval.

• FDA's May 2026 guidance on statistical approaches to establishing bioequivalence says BE comparisons normally rely on (1) a criterion, (2) a confidence interval for the criterion and (3) a predetermined BE limit. The recommended method is the two one-sided tests procedure (Schuirmann 1987): two one-sided tests at α (usually 0.05), which is the same as checking that the 100(1−2α)% confidence interval lies entirely inside [T1, T2]. For average bioequivalence of pharmacokinetic measures the usual limit is 80.00 percent to 125.00 percent for the ratio of the averages. That 80–125 range is a clinical convention for drugs, not a default for LLM scores; your eval needs its own predetermined numeric margin (for example ±1.5 percentage points of accuracy).
• Schuirmann (1987) compared TOST to the then-common power approach (test no-difference at 0.05 and require an estimated power of 0.80) and found that for α = 0.05 TOST has uniformly superior properties in most cases. Do not treat "we were powered, and the difference was not significant" as a substitute for an equivalence test.
• If both systems answer the same frozen items, use paired differences. Pairing removes item difficulty as noise and usually needs fewer items. If the sets differ, use an unpaired interval and do not mix the two.
• Use the confidence interval for the difference, not "the two 95% intervals overlap, so the scores are the same." Overlap of two separate intervals is the wrong object: Walker and Nowacki, Lakens and the FDA guidance all put the interval for the difference (or ratio) inside the pre-set margin. Two overlapping score intervals can hide a difference that a direct interval for Δ would show, and they cannot tell you whether Δ sits inside ±δ.

How do I record a pre-registered equivalence margin with an eval run?

A later reader should be able to see that δ and α were fixed before the new scores, and which set and judge produced them. Hash a canonical plan file: metric, pairing, δ, α, interval method, inclusion rules, eval-set hash and judge id.

• record_ai_run stores the hashes and the params you send. Params are kept as params_sha256 over sorted-key canonical JSON, so keep the exact object. Put plan_sha256, delta, alpha, pairing, eval_set_sha256 and judge_id in params, and the results-file hash as output_sha256. verify_ai_run checks the receipt; a reader recomputes params_sha256 from the params object you share.
• Rightshold does not run TOST, pick δ or score your items. The receipt proves what was recorded and when, not that the two systems are equivalent.