How do I report false-positive rates with confidence intervals on a frozen holdout?

What should I check, step by step?

1. Freeze the holdout: write-protect or content-address the item ids, inputs, and gold labels before any scoring run. Record holdout_set_sha256 and labels_sha256.

2. Freeze the harness: pin eval code commit, scorer version, model snapshot (or classifier build), decoding/threshold settings, and random seeds. Record harness_version (git SHA or release tag).
3. Blind scorers: humans or LLM judges must not see gold labels (and ideally not the system under test's identity) while assigning predictions or error tags.
4. Score once on the frozen set. Count TP/FP/FN/TN per category (or per task type). Do not drop hard items after seeing scores.
5. For each category rate (e.g. FP / (FP+TN) or FP / scored_negatives — state the denominator), report the point estimate and a 95% Wilson or bootstrap CI. Avoid bare Wald intervals when n is small or the rate is near 0/1.
6. Publish n, FP, FN, the CI method, and the pinned harness_version + holdout_set_sha256 beside the rates. If you bootstrap, publish the seed and number of resamples.
7. Never retrain, calibrate thresholds, or select prompts on the holdout. If you must iterate, carve a separate tuning split and keep the holdout sealed.
8. Hash the final metrics table and eval log; call record_ai_run with those hashes and params. Optionally commit a recorder head and later verify_recorder on a disclosed chain.

Freeze and blind before you score

NIST's AI Technology Evaluation (AITE) program scores models on sequestered, blind data specifically to mitigate train/test contamination. Your internal holdout should follow the same idea: inaccessible for training and tuning until the report is locked.

• NIST CAISI guidance on evaluation cheating notes that sharing a representative subset can help third parties while still preserving a held-out test set, and that publishing evaluation configurations (and review methodology) builds confidence even when full transcripts stay private.
• Inspect-style eval logs keep the spec, plan, aggregate results, and per-sample scores together. Export or hash that log so the CI numbers you quote map to a single frozen run, not a moving spreadsheet.

Rates with intervals, not bare percentages

A false-positive rate without n and a CI looks precise and is not. NIST Dataplot's proportion confidence-limits documentation describes Wilson, adjusted Wald, Jeffreys, and exact binomial methods, defaults to Wilson, and cites Agresti–Coull and Brown–Cai–DasGupta on coverage — including recommending Wilson or Jeffreys for n ≤ 40.

• Report FP and FN per category (toxicity subtype, tool-error class, intent label, etc.). Aggregate-only FP hides categories where the detector is unsafe.
• Bootstrap CIs are fine when you resample the frozen holdout with a fixed seed; still publish method, seed, and B. Do not bootstrap a set you altered after peeking.
• Pin harness_version and holdout_set_sha256 in the same table as the rates. A later "same" FP% on a different target hash is a different claim.

What can go wrong?

Holdout items were used to pick the decision threshold last week; today's FP% is an in-sample fit dressed as a test.

• Only a point estimate is published ("FP = 1.2%") with no n and no CI, so a reviewer cannot tell 1/80 from 12/1000.
• Scorers saw labels or the vendor name, inflating agreement on easy negatives and under-counting hard false positives.
• Harness commit is "latest main" with no SHA; the metrics CSV was edited after the run without a new receipt.
• Retraining on holdout failures "just to check" — then reporting the new model on the same holdout as if it were still unseen.