# New agent benchmarks like Google DeepMind's Multimodal Knowledge Agent (MKA), launched November 10, 2026 with Artificial Analysis, and GitHub's ReviewBench are graded by an AI model rather than by people or test suites. How much should we rely on those leaderboard scores when choosing a model for our AI app, and what should we check?

**Short answer:** Use it to shortlist, not to decide. Model-graded agent benchmarks are getting more careful: MKA's 200 tasks are kept private by Artificial Analysis so they can't leak into training data, every model runs in the same open-source harness (Stirrup) with the same tools, sandbox and turn limit, a GPT-6 Luna (medium) agentic judge marks each weighted rubric criterion, and Google reports that domain experts agreed with the autorater 96% of the time. ReviewBench publishes its rubric, judge and data, and reports 96.6% agreement between independent senior engineers and its labels. But an agreement rate is an average across many checks, so a few points of disagreement can be as large as the gap between the models you're comparing; the judge is itself a model from one lab; results depend on the harness, turn limit and scoring rule; published costs exclude judge calls; and no model led all 10 MKA verticals. Before you choose, rerun a small set of your own tasks in your own setup, have a person grade a sample, and keep a record of each run's model version, prompt, judge and dataset version so your own numbers can be checked later.

## Checklist
1. Find out who or what grades the benchmark: a test suite, people, or a model. If it's a model, note which one and at what setting, and whether it's from the same lab as any model being ranked.
2. Look for a published agreement rate between the grader and human experts, how it was measured (how many items, which graders) and whether it was checked on the same kinds of tasks you care about.
3. Compare the score gap between the models you're choosing with the grader's disagreement rate. A two-point lead on a judge that disagrees with experts about one time in twenty isn't a clear win.
4. Check the setup the score depends on: harness, tools, turn limit, internet access, number of attempts per task and whether failed or abandoned runs score zero.
5. Use the breakdown closest to your work (vertical, file type, severity) instead of the headline. On MKA, no model led every vertical.
6. Read cost per task with its footnotes; Artificial Analysis excludes judge calls from MKA costs.
7. Rerun 10 to 30 of your own tasks with the shortlisted models in your own app, have a person grade a sample blind, and record which version, prompt and grader produced each result.
8. Re-check after model updates; a leaderboard score belongs to one dated model version and benchmark configuration.

## How are the MKA and ReviewBench AI agent benchmarks graded?
- MKA, built by Google DeepMind with domain experts across 10 verticals and 83 occupations, has 200 tasks that each pair a prompt and a bundle of visual input files (drawings, schematics, scans) with an expert rubric of roughly 70 yes/no checks, weighted 75% on factual accuracy and 25% on visual design. Artificial Analysis runs every task in its Stirrup harness with code execution, web search and fetch, image viewing and a 250-turn limit. GPT-6 Luna (medium) acts as an agentic judge in its own sandbox, without the input files, and marks each criterion satisfied or not; criteria carry weights from 1 to 5. The headline score, Climb WSR, is the mean weighted share of the harder "Climb" criteria satisfied per task. Google's post says experts agreed with the autorater 96% of the time, and 10 showcase tasks with rubrics are public on Hugging Face under Apache-2.0.
- ReviewBench, which GitHub launched on October 5, 2026, has 219 pull requests from 187 public repositories in 19 languages. Its golden set of findings comes from human reviewers, follow-up commits, static analysis and several frontier models, validated against one rubric by Claude Sonnet 5 as the judge. Grounded metrics count only known findings; augmented metrics let the judge credit valid findings nobody had labeled. GitHub says senior engineers who hadn't built the dataset re-labeled every golden finding and agreed 96.6% of the time, and that it versions the dataset, judge and matcher used in every evaluation.

## What does a grader's 96% agreement with experts tell you about a benchmark score?
- A 96% agreement rate means the grader and the experts disagreed on about 4 in 100 judgments in the validation sample. That's good, but neither post says how the disagreements were spread between easy baseline checks and the harder criteria that separate strong models, and the harder ones are where leaderboard gaps come from. On MKA's first leaderboard the top three Climb WSR scores were 57%, 54% and 53%, and Artificial Analysis lists one run per task.
- The grader is a model too. A judge can favor answers that look like its own, reward confident formatting, or miss errors it can't see; MKA's judge doesn't see the input files, so it can't catch a deliverable that misreads a drawing in a plausible way unless the rubric spells out the right answer. Artificial Analysis uses multi-judge panels from different labs on some other benchmarks to reduce self-preference; MKA, as described on its page, uses one judge.
- Setup matters as much as the model. Artificial Analysis notes on other benchmarks that differences in judge, document handling or harness make its scores not directly comparable with the original authors' numbers, and the same applies when you compare a leaderboard score with what you see in your own app.

## How do you turn a public AI benchmark into evidence for your own model choice?
- Shortlist from the public breakdown closest to your work, then run your own tasks in your own harness, prompts and tools. Keep the task set fixed and private, grade with the same rubric each time, and have a person grade a random sample so you know your own grader's agreement rate.
- Record each run as it happens: model and dated version, provider, your prompt or system-prompt version, a hash of the input and of the exact output, the judge model and its version, the rubric and dataset version, and the score. That lets you show later which model produced which result, re-check a disputed score against the original output, and notice when a provider's model changes under the same name.
- If you publish or sell on the strength of an eval, say how it was graded and what the grader's measured agreement was, the way MKA and ReviewBench do. Claims about AI accuracy are claims like any other, and a dated record of the runs behind them is what you'll be asked for.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "Record a signed receipt for each run in our model-selection eval: the candidate model and dated version, hashes of the task input and of the exact deliverable, and in params our task id, judge model, rubric version, dataset version and the judge's score, so we can later show which model produced which graded result."

```json
{
  "tool": "record_ai_run",
  "arguments": {
    "model": "<candidate model id and dated version>",
    "provider": "<model provider>",
    "prompt_sha256": "<sha256 of the task prompt and inputs as sent>",
    "output_sha256": "<sha256 of the exact deliverable or answer>",
    "params": {
      "task_id": "<your private task id>",
      "harness": "<harness name and version>",
      "judge_model": "<judge model and setting>",
      "rubric_version": "<version>",
      "dataset_version": "<version>",
      "judge_score": 0.71,
      "human_graded": false
    },
    "ran_at": "2026-11-11T17:00:00Z"
  }
}
```
- Needs an API key. Send hashes, not your task content, and keep the inputs and outputs in your own store; params are stored as a hash, so keep the exact params object too.
- verify_ai_run is free and needs no key: anyone you share a result with can check that the output you show them matches the hash recorded at run time.
- Record the human-graded sample the same way with human_graded true and a reviewer reference instead of a name, so the agreement rate you quote can be traced to specific runs.
- A receipt shows what was recorded and when. It doesn't show the grade was right or that the benchmark measures what you need.

_Rightshold records the hashes and run details you send and lets anyone check them later. It doesn't run benchmarks, grade outputs or rank models, and it isn't affiliated with Google DeepMind, Artificial Analysis, GitHub or any model provider. Benchmark details are from the MKA announcement and Artificial Analysis pages published November 10, 2026, the MKA showcase dataset card, and GitHub's ReviewBench post of October 5, 2026, read on November 11, 2026; leaderboards change as models are added. Not procurement advice._

## Sources
- [Google: Introducing the Multimodal Knowledge Agent benchmark (November 10, 2026)](https://blog.google/innovation-and-ai/models-and-research/gemini-models/multimodal-knowledge-agent-benchmark/)
- [Artificial Analysis: Multimodal Knowledge Agent (MKA) benchmark (harness, judge, Climb WSR, cost scope)](https://artificialanalysis.ai/evaluations/mka)
- [Artificial Analysis: MKA launch article (November 10, 2026)](https://artificialanalysis.ai/articles/google-deepmind-multimodal-knowledge-agent)
- [Artificial Analysis: Intelligence benchmarking methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking)
- [Hugging Face: google/multimodal-knowledge-agent (10 showcase tasks)](https://huggingface.co/datasets/google/multimodal-knowledge-agent)
- [GitHub Blog: ReviewBench, an open benchmark for AI code review (October 5, 2026)](https://github.blog/ai-and-ml/github-copilot/reviewbench-an-open-benchmark-for-ai-code-review/)

## Related
- [How to version an LLM-as-a-judge and keep a frozen eval set so scores stay comparable](https://www.rightshold.com/answers/how-to-version-an-llm-as-a-judge-and-keep-a-frozen-eval-set-so-scores-stay-comparable)
- [How to prove a reported eval score wasn't cherry-picked from many runs](https://www.rightshold.com/answers/how-to-prove-a-reported-eval-score-wasnt-cherry-picked-from-many-runs)
- [How to keep an eval set out of model training data and detect contamination](https://www.rightshold.com/answers/how-to-keep-an-eval-set-out-of-model-training-data-and-detect-contamination)
- [How to make LLM eval results reproducible and tamper-evident](https://www.rightshold.com/answers/how-to-make-llm-eval-results-reproducible-and-tamper-evident)
- [Does a passing AI eval prove what happened in one specific incident?](https://www.rightshold.com/answers/does-a-passing-ai-eval-prove-what-happened-in-one-specific-incident)

_Not legal advice. Updated 2026-11-11. JSON: https://www.rightshold.com/answers/how-much-should-you-trust-an-ai-agent-benchmark-score-when-another-ai-model-grades-it.json_