# How do I report false-positive rates with confidence intervals on a frozen holdout?

**Short answer:** Publish per-category false-positive (and false-negative) rates with confidence intervals on a holdout set that was frozen and labeled before you scored it — not on a slice you reshuffled after seeing results. Blind scorers to labels (and preferably to which system produced each output). Prefer Wilson score intervals or bootstrap intervals for binomial rates; NIST's Dataplot guidance defaults to Wilson for proportion confidence limits and notes better coverage than the plain normal (Wald) interval, especially for small n or rates near 0 or 1. Pin the harness version and a hash of the target set (and label file) in the report so a reviewer can tell whether the numbers still match the bytes. Never retrain, retune thresholds, or pick prompts on the holdout. Seal the eval package with record_ai_run; use verify_recorder when you disclose a hash-chained log that should end at a committed head.

## Checklist
1. Freeze the holdout: write-protect or content-address the item ids, inputs, and gold labels before any scoring run. Record holdout_set_sha256 and labels_sha256.
2. Freeze the harness: pin eval code commit, scorer version, model snapshot (or classifier build), decoding/threshold settings, and random seeds. Record harness_version (git SHA or release tag).
3. Blind scorers: humans or LLM judges must not see gold labels (and ideally not the system under test's identity) while assigning predictions or error tags.
4. Score once on the frozen set. Count TP/FP/FN/TN per category (or per task type). Do not drop hard items after seeing scores.
5. For each category rate (e.g. FP / (FP+TN) or FP / scored_negatives — state the denominator), report the point estimate and a 95% Wilson or bootstrap CI. Avoid bare Wald intervals when n is small or the rate is near 0/1.
6. Publish n, FP, FN, the CI method, and the pinned harness_version + holdout_set_sha256 beside the rates. If you bootstrap, publish the seed and number of resamples.
7. Never retrain, calibrate thresholds, or select prompts on the holdout. If you must iterate, carve a separate tuning split and keep the holdout sealed.
8. Hash the final metrics table and eval log; call record_ai_run with those hashes and params. Optionally commit a recorder head and later verify_recorder on a disclosed chain.

## Freeze and blind before you score
- NIST's AI Technology Evaluation (AITE) program scores models on sequestered, blind data specifically to mitigate train/test contamination. Your internal holdout should follow the same idea: inaccessible for training and tuning until the report is locked.
- NIST CAISI guidance on evaluation cheating notes that sharing a representative subset can help third parties while still preserving a held-out test set, and that publishing evaluation configurations (and review methodology) builds confidence even when full transcripts stay private.
- Inspect-style eval logs keep the spec, plan, aggregate results, and per-sample scores together. Export or hash that log so the CI numbers you quote map to a single frozen run, not a moving spreadsheet.

## Rates with intervals, not bare percentages
- A false-positive rate without n and a CI looks precise and is not. NIST Dataplot's proportion confidence-limits documentation describes Wilson, adjusted Wald, Jeffreys, and exact binomial methods, defaults to Wilson, and cites Agresti–Coull and Brown–Cai–DasGupta on coverage — including recommending Wilson or Jeffreys for n ≤ 40.
- Report FP and FN per category (toxicity subtype, tool-error class, intent label, etc.). Aggregate-only FP hides categories where the detector is unsafe.
- Bootstrap CIs are fine when you resample the frozen holdout with a fixed seed; still publish method, seed, and B. Do not bootstrap a set you altered after peeking.
- Pin harness_version and holdout_set_sha256 in the same table as the rates. A later "same" FP% on a different target hash is a different claim.

## What can go wrong?
- Holdout items were used to pick the decision threshold last week; today's FP% is an in-sample fit dressed as a test.
- Only a point estimate is published ("FP = 1.2%") with no n and no CI, so a reviewer cannot tell 1/80 from 12/1000.
- Scorers saw labels or the vendor name, inflating agreement on easy negatives and under-counting hard false positives.
- Harness commit is "latest main" with no SHA; the metrics CSV was edited after the run without a new receipt.
- Retraining on holdout failures "just to check" — then reporting the new model on the same holdout as if it were still unseen.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "Record a signed receipt for this frozen-holdout eval: harness version, holdout-set hash, per-category FP/FN counts, and the CI method — without uploading the raw items."

```json
{
  "tool": "record_ai_run",
  "arguments": {
    "model": "detector/policy@harness-sha",
    "provider": "eval-harness",
    "prompt_sha256": "<sha256 of holdout ids + labels file>",
    "output_sha256": "<sha256 of metrics table + eval log>",
    "params": {
      "harness_version": "<git sha>",
      "holdout_set_sha256": "<sha256>",
      "labels_sha256": "<sha256>",
      "categories": [
        {
          "name": "pii_leak",
          "n": 200,
          "fp": 3,
          "fn": 5,
          "fp_rate": 0.015,
          "fp_ci_wilson_95": [
            0.0051,
            0.0433
          ]
        }
      ],
      "ci_method": "wilson",
      "confidence": 0.95
    },
    "ran_at": "2026-11-06T22:00:00-05:00"
  }
}
```
- Needs your API key. record_ai_run stores the hashes and params you send, not the holdout items, and returns a signed ledger receipt; verify_ai_run is free.
- verify_recorder (free) checks a disclosed hash-chained run of log entries against a head you committed earlier, and says when that head was committed.
- How to make LLM eval results reproducible and tamper-evident covers pinning model snapshots, seeds, and hashing the final eval log.
- Verified AI Runs: https://www.rightshold.com/verified-ai-runs

_Rightshold records signed AI-run receipts and can verify disclosed recorder chains for agents that call it. It does not run your eval, compute Wilson or bootstrap intervals, blind your scorers, or stop a team that retrains on the holdout. Enforce freeze/blind/CI reporting in your harness and review process. Not legal advice._

## Sources
- [NIST Dataplot: Proportion Confidence Interval (Wilson default; Agresti–Coull; Brown–Cai–DasGupta)](https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/propconf.htm)
- [NIST AITE: sequestered / blind evaluation to mitigate train–test contamination](https://pages.nist.gov/ai-technology-evaluation/)
- [NIST CAISI: Practices for detecting and preventing evaluation cheating (held-out test set; publish configurations)](https://www.nist.gov/caissi/4-practices-detecting-and-preventing-evaluation-cheating)
- [Inspect AI: Log Files (EvalLog fields, export-config)](https://inspect.aisi.org.uk/eval-logs.html)
- [Verified AI Runs on Rightshold](https://www.rightshold.com/verified-ai-runs)

## Related
- [How to make LLM eval results reproducible and tamper-evident](https://www.rightshold.com/answers/how-to-make-llm-eval-results-reproducible-and-tamper-evident)
- [How to prove what an AI model said at a specific time](https://www.rightshold.com/answers/how-to-prove-what-an-ai-model-said-at-a-specific-time)
- [How to keep an agent risk/approval record outside the agent's editable session](https://www.rightshold.com/answers/how-to-keep-an-agent-risk-approval-record-outside-the-agents-editable-session)
- [Verified AI Runs](https://www.rightshold.com/verified-ai-runs)

_Not legal advice. Updated 2026-11-06. JSON: https://www.rightshold.com/answers/how-to-report-false-positive-rates-with-confidence-intervals-on-a-frozen-holdout.json_