Answers · AI evidence
How do I version an LLM-as-a-judge (judge model and rubric) and keep a frozen eval set, so AI eval scores stay comparable over time?
Last updated
TL;DR: Treat the judge as part of the measuring instrument and give it a version, the same way you version the model under test. A judge version is the exact judge model snapshot (never an alias like -latest), the rubric text, the judge prompt template, the score scale, the decoding settings and the parsing code, hashed together into one judge id. Freeze the eval set too: a fixed list of item ids, inputs and reference answers with a content hash, kept apart from anything you tune on. Then only compare scores that share both the judge id and the eval-set hash. When either has to change (a rubric fix, or the judge model being retired), don't splice the new numbers onto the old chart: run a bridge, scoring the same frozen outputs with the old and new judge, report how far they agree, and start a new series. Record each run's judge id, eval-set hash and scores somewhere that can't be quietly edited, so a later reader can tell which numbers are comparable.
What should I check, step by step?
1. Pin the judge model to a dated snapshot, not an alias. Aliases move to new snapshots, and a different judge is a different ruler even if the model under test hasn't changed.
- Put the rubric, the judge prompt template, the score scale and any few-shot examples in version control, and hash them. A one-word change to the rubric is a new judge version.
- Record the judge's decoding settings (temperature, max tokens, reasoning effort, seed where offered) and the code that parses its output into a score, and include them in the judge id.
- Freeze the eval set: item ids, inputs, reference answers and labels in one file with a content hash. Keep a separate dev split for tuning prompts and rubrics, and never edit the frozen set in place; a fixed or added item makes a new set version.
- Keep a small human-labeled calibration set and measure the judge's agreement with it (for example percent agreement or Cohen's kappa) every time the judge version changes.
- For pairwise judging, run each pair in both orders and only count a win that survives the swap, so position bias doesn't move scores between versions.
- Store every run with its judge id, eval-set hash, model-under-test snapshot, per-item scores and the aggregate, and only plot or compare runs with matching judge id and eval-set hash.
- Watch the judge model's retirement date and plan the bridge run before it's switched off, while you can still call the old judge.
Why does an LLM judge need its own version?
Zheng et al. (Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023) found strong LLM judges matched human preferences at over 80% agreement, but also documented position, verbosity and self-enhancement biases, and noted that absolute single-answer scores are likely to fluctuate more than pairwise results if the judge model changes.
- So a score of 7.8 means 'judge J, rubric R, on set S gave 7.8'. Change J or R and the same outputs can score differently, which looks like a regression or an improvement in the model under test when nothing about it changed.
- The paper's own case studies were made with a dated GPT-4 snapshot and it warns they may not be fully reproducible with future versions. That is the normal state of hosted judges, which is why the snapshot has to be part of the record.
What goes into an LLM judge's version id?
Judge model snapshot and provider, the rubric text, the judge prompt template, the score scale and labels, few-shot examples, decoding settings, and the version of the parsing code.
- Hash a canonical bundle of those (for example JSON with a fixed key order, or RFC 8785) and use the hash as the judge id. Two runs with the same id used the same instrument; any difference shows up as a different id.
- Keep the human-readable parts next to the hash: a reviewer should be able to read the rubric a score came from, not just see that it changed.
How do I keep an eval set frozen?
Freeze means the file a score was computed on can be identified exactly: a list of item ids and their content, hashed, with the hash stored next to every result.
- Fix mistakes by publishing a new set version, not by editing the old one. Report on the new version from then on, and keep the old version so old results can still be checked.
- Keep the frozen set out of anything you tune against. If rubric or prompt edits are tested on it, it slowly becomes a training set and later scores stop meaning what earlier ones did.
- If the set has to stay private, publish only its hash and item count. That still lets you show later that results came from the same set.
What do I do when I have to change the LLM judge?
Model providers retire snapshots. OpenAI's deprecations page says generally available models get at least six months' notice, and its List models endpoint can return a shutdown_date for each model, so you can see a judge's end date coming.
- Before the old judge goes away, run a bridge: score the same stored outputs on the frozen set with the old and new judge versions, and report agreement and the score shift per category.
- Then start a new series under the new judge id. Show old and new series side by side with the bridge result, rather than joining them into one line.
- Re-check agreement with the human calibration set under the new judge. A judge that agrees with the old judge but not with people hasn't solved anything.
What are the common failure modes?
The judge is called through an alias, the provider moves the alias, and a step change in scores gets blamed on the model under test.
- Someone tidies the rubric wording between runs and nobody records it, so two numbers on the same chart came from different instruments.
- Items are added to or fixed in the 'frozen' set in place, and the old results can no longer be reproduced or checked.
- The judge sees which system produced each answer, or always sees candidate A first, and its biases drift with the version.
- Scores are kept in a spreadsheet that anyone can edit, so months later there's no way to show which judge and set a published number came from.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “Record a signed receipt for this judged eval run: the judge id (judge model snapshot + rubric + prompt + settings hash), the frozen eval-set hash, the model under test, and the scores.”
{
"tool": "record_ai_run",
"arguments": {
"model": "<model under test, dated snapshot>",
"provider": "<provider>",
"prompt_sha256": "<sha256 of the frozen eval set file>",
"output_sha256": "<sha256 of per-item judge scores + eval log>",
"params": {
"judge_model": "<judge model, dated snapshot>",
"judge_id": "<sha256 of rubric + judge prompt + scale + settings + parser version>",
"rubric_version": "v3",
"eval_set_version": "v2",
"eval_set_items": 400,
"judge_temperature": 0,
"pairwise_swap": true,
"mean_score": 7.4,
"human_agreement": 0.82
},
"ran_at": "2026-11-07T08:00:00Z"
}
}- Needs an API key. Only hashes and the fields you send are recorded; the receipt is Ed25519-signed and written to Rightshold's append-only ledger.
- verify_ai_run (free, no key) checks a receipt's signature and, given the original files, that they match the recorded hashes. Use it before comparing a new score with an old one: matching judge_id and eval-set hash means the two runs used the same judge and set.
- For a judge change, record the bridge run the same way, with both judge ids in params, so the switch-over is on the record too.
- How to make LLM eval results reproducible and tamper-evident covers pinning the model under test, seeds and hashing the final eval log.
Rightshold doesn't run your eval, call the judge or score anything. It signs and timestamps the hashes and fields you send, so you can later show which judge version and eval set a score came from. A receipt doesn't show the rubric is good or the judge is right. Not legal advice.
What are the sources?
- Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685)
- Wang et al.: Large Language Models are not Fair Evaluators (arXiv 2305.17926)
- OpenAI API: Deprecations (notice periods; shutdown_date on List models)
- RFC 8785: JSON Canonicalization Scheme
Related
- How to make LLM eval results reproducible and tamper-evident
- How to report false-positive rates with confidence intervals on a frozen holdout
- How to replay an AI agent run deterministically from a recorded call order
- Verified AI Runs
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'