Answers · AI evidence
How do I check whether an API provider is serving a degraded or quantized version of a model?
Last updated
TL;DR: Don't argue from vibes. Pin the provider, run a fixed probe set that exercises the things degradation breaks first — tool-call formatting, long outputs, non-English text — run the same probes against a reference endpoint (usually the model maker's own API) at the same settings, and compare pass rates with confidence intervals. Record every probe run before you compare, so the result can't be rearranged afterwards. Degradation is real and often intermittent. Anthropic's September 17, 2025 postmortem describes three infrastructure bugs that intermittently degraded Claude between August and early September 2025: a routing error that sent some Sonnet 4 requests to servers configured for the upcoming 1M-token context window (0.8% of requests at first, 16% at the worst hour on August 31, and "sticky", so affected users kept hitting it), a TPU misconfiguration that occasionally produced Thai or Chinese characters in English answers, and an XLA:TPU compiler bug in approximate top-k. Anthropic says its evaluations didn't capture what users were reporting. Across third-party hosts of open-weight models the gaps can be large: Moonshot AI's K2 Vendor Verifier, testing kimi-k2-thinking on 2025-11-15, measured 100% tool-call schema accuracy on Moonshot's own API against about 87% for some third-party endpoints. record_ai_run gives each probe run a signed, timestamped receipt naming the provider and quantization; verify_ai_run checks it. This page is about comparing providers or endpoints of the same model. Watching one endpoint for drift over time is covered on a separate page.
What should I check, step by step?
1. Pin the endpoint. Through a router, name the provider and turn fallbacks off; on OpenRouter that's provider.order or provider.only with allow_fallbacks false, and quantized variants have their own slugs (for example deepinfra/fp8). Otherwise you don't know which backend answered.
- Record what the provider declares: quantization (OpenRouter can filter on it with provider.quantizations), context length, and any parameters it ignores. OpenRouter's require_parameters option routes only to providers that support every parameter you send.
- Build a fixed probe set aimed at known failure modes: tool calls checked against their JSON Schema, a few long-output tasks (Kimi Vendor Verifier uses AIME 2025 as a long-output stress test that catches KV cache bugs and quantization degradation), and prompts that catch stray characters or language switches.
- Run the same probes at the same settings on a reference endpoint and on the provider under test, in the same time window, with enough samples per probe to put a confidence interval on each pass rate.
- Check the settings were actually applied before you compare scores. Moonshot found that a significant portion of anomalous benchmark reports came from misused decoding parameters, and its verifier now validates parameter enforcement before running any benchmark.
- Record each probe run as it finishes: provider, endpoint slug, quantization, settings, the probe-set hash and the results-file hash. Then compare.
- Re-run on a schedule. Anthropic's routing bug grew after a load-balancing change on August 29, 2025, so a provider that passed once can fail later.
What does a degraded or quantized model from an API provider look like?
Quantization reduces precision to save memory and compute. OpenRouter's docs say quantized models may show degraded performance on certain prompts, depending on the method, and list levels from int4 and fp4 up to bf16 and fp32.
- It isn't only quantization. In Anthropic's postmortem the causes were routing, a runtime misconfiguration and a compiler bug, each producing different symptoms on different platforms at different rates. Anthropic states it never reduces model quality because of demand, time of day or server load; the problems were bugs, and they still looked to users like a quietly worse model.
- Moonshot built its vendor verifier because users kept reporting precision problems with Kimi K2 tool calls across providers. Its leaderboard reports tool-call trigger similarity against the official API and JSON Schema accuracy per provider; on the 2025-11-15 kimi-k2-thinking run the official API scored 100.00% schema accuracy and some third-party endpoints, including a self-hosted vLLM baseline, came in around 87%.
How do I design probes that detect a degraded model endpoint?
Short general-knowledge prompts rarely separate a healthy endpoint from a damaged one; Anthropic notes Claude often recovers well from isolated mistakes. Use checks with a clear pass or fail: does the tool call parse and validate against its schema, does a long answer stay coherent to the end, is the output in the requested language.
- Fix the probe set file and hash it, keep settings identical across endpoints, and sample enough times. Because hosted outputs vary even at temperature 0, compare rates with intervals, not single answers.
- Keep the evidence minimal and exact: the request bodies, the raw responses and the scoring script version. Someone who disputes your result should be able to rerun the scorer on your saved responses.
What does a signed receipt add when comparing AI model providers?
A record_ai_run receipt per probe run fixes which endpoint, quantization and settings you tested, which probe set you used and what results file came out, with a time set by Rightshold when it was recorded.
- That turns "provider X seemed worse last month" into a dated comparison you can hand to the provider or your own team, and it stops the comparison from being redrawn after the fact. It doesn't prove the provider's internal configuration; it proves what you observed and when.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “Record a signed receipt for this probe run against a pinned provider: the endpoint slug, declared quantization, settings, the probe-set hash, the results-file hash and the tool-call schema pass rate, so I can compare it with the reference endpoint.”
{
"tool": "record_ai_run",
"arguments": {
"model": "<model id as the provider names it>",
"provider": "<provider/endpoint slug, e.g. deepinfra/fp8>",
"prompt_sha256": "<sha256 of the frozen probe set file>",
"output_sha256": "<sha256 of this run's raw responses + scores>",
"params": {
"run": "provider-probe",
"reference": false,
"quantization_declared": "fp8",
"fallbacks": "off",
"temperature": 0.6,
"top_p": 0.95,
"n_probes": 200,
"samples_per_probe": 3,
"tool_schema_pass_rate": 0.94,
"long_output_pass_rate": 0.81
},
"ran_at": "2026-11-09T12:00:00Z"
}
}- Needs an API key. Record the reference endpoint's run the same way with reference: true, using the same probe-set hash.
- params is kept as params_sha256 over sorted-key canonical JSON (up to 4,000 bytes), so keep the exact params object for each run.
- verify_ai_run is free and keyless: send each receipt and signature exactly as returned, plus that run's results text, to confirm the signature, that it is on record, and that the text matches output_sha256.
- ledger_verify is free and keyless and checks the public ledger the receipts sit on.
Rightshold doesn't run probes, inspect a provider's hardware or weights, or rate providers on this page. It signs and timestamps the run records you send. A receipt shows what you observed from an endpoint and when, not why it behaved that way. Not legal advice.
What are the sources?
- Anthropic: A postmortem of three recent issues (September 17, 2025)
- Moonshot AI: K2 Vendor Verifier (tool-call trigger similarity and schema accuracy by provider)
- Kimi: Kimi Vendor Verifier (pre-verification of decoding parameters; AIME 2025 long-output stress test)
- OpenRouter: Provider routing (order, only, allow_fallbacks, require_parameters, quantizations, endpoint variants)
- Verified AI Runs on Rightshold
Related
- How to detect silent model drift between AI runs with recorded baselines
- Why an LLM gives different answers at temperature 0, and how to prove what it said
- How to prove which model version, prompt and settings produced an AI output
- How to make LLM eval results reproducible and tamper-evident
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'