New agent benchmarks like Google DeepMind's Multimodal Knowledge Agent (MKA), launched November 10, 2026 with Artificial Analysis, and GitHub's ReviewBench are graded by an AI model rather than by people or test suites. How much should we rely on those leaderboard scores when choosing a model for our AI app, and what should we check?

What should I check, step by step?

1. Find out who or what grades the benchmark: a test suite, people, or a model. If it's a model, note which one and at what setting, and whether it's from the same lab as any model being ranked.

2. Look for a published agreement rate between the grader and human experts, how it was measured (how many items, which graders) and whether it was checked on the same kinds of tasks you care about.
3. Compare the score gap between the models you're choosing with the grader's disagreement rate. A two-point lead on a judge that disagrees with experts about one time in twenty isn't a clear win.
4. Check the setup the score depends on: harness, tools, turn limit, internet access, number of attempts per task and whether failed or abandoned runs score zero.
5. Use the breakdown closest to your work (vertical, file type, severity) instead of the headline. On MKA, no model led every vertical.
6. Read cost per task with its footnotes; Artificial Analysis excludes judge calls from MKA costs.
7. Rerun 10 to 30 of your own tasks with the shortlisted models in your own app, have a person grade a sample blind, and record which version, prompt and grader produced each result.
8. Re-check after model updates; a leaderboard score belongs to one dated model version and benchmark configuration.

How are the MKA and ReviewBench AI agent benchmarks graded?

MKA, built by Google DeepMind with domain experts across 10 verticals and 83 occupations, has 200 tasks that each pair a prompt and a bundle of visual input files (drawings, schematics, scans) with an expert rubric of roughly 70 yes/no checks, weighted 75% on factual accuracy and 25% on visual design. Artificial Analysis runs every task in its Stirrup harness with code execution, web search and fetch, image viewing and a 250-turn limit. GPT-6 Luna (medium) acts as an agentic judge in its own sandbox, without the input files, and marks each criterion satisfied or not; criteria carry weights from 1 to 5. The headline score, Climb WSR, is the mean weighted share of the harder "Climb" criteria satisfied per task. Google's post says experts agreed with the autorater 96% of the time, and 10 showcase tasks with rubrics are public on Hugging Face under Apache-2.0.

• ReviewBench, which GitHub launched on October 5, 2026, has 219 pull requests from 187 public repositories in 19 languages. Its golden set of findings comes from human reviewers, follow-up commits, static analysis and several frontier models, validated against one rubric by Claude Sonnet 5 as the judge. Grounded metrics count only known findings; augmented metrics let the judge credit valid findings nobody had labeled. GitHub says senior engineers who hadn't built the dataset re-labeled every golden finding and agreed 96.6% of the time, and that it versions the dataset, judge and matcher used in every evaluation.

What does a grader's 96% agreement with experts tell you about a benchmark score?

A 96% agreement rate means the grader and the experts disagreed on about 4 in 100 judgments in the validation sample. That's good, but neither post says how the disagreements were spread between easy baseline checks and the harder criteria that separate strong models, and the harder ones are where leaderboard gaps come from. On MKA's first leaderboard the top three Climb WSR scores were 57%, 54% and 53%, and Artificial Analysis lists one run per task.

• The grader is a model too. A judge can favor answers that look like its own, reward confident formatting, or miss errors it can't see; MKA's judge doesn't see the input files, so it can't catch a deliverable that misreads a drawing in a plausible way unless the rubric spells out the right answer. Artificial Analysis uses multi-judge panels from different labs on some other benchmarks to reduce self-preference; MKA, as described on its page, uses one judge.
• Setup matters as much as the model. Artificial Analysis notes on other benchmarks that differences in judge, document handling or harness make its scores not directly comparable with the original authors' numbers, and the same applies when you compare a leaderboard score with what you see in your own app.

How do you turn a public AI benchmark into evidence for your own model choice?

Shortlist from the public breakdown closest to your work, then run your own tasks in your own harness, prompts and tools. Keep the task set fixed and private, grade with the same rubric each time, and have a person grade a random sample so you know your own grader's agreement rate.

• Record each run as it happens: model and dated version, provider, your prompt or system-prompt version, a hash of the input and of the exact output, the judge model and its version, the rubric and dataset version, and the score. That lets you show later which model produced which result, re-check a disputed score against the original output, and notice when a provider's model changes under the same name.
• If you publish or sell on the strength of an eval, say how it was graded and what the grader's measured agreement was, the way MKA and ReviewBench do. Claims about AI accuracy are claims like any other, and a dated record of the runs behind them is what you'll be asked for.