Why does an LLM give different answers at temperature 0, and how do I prove what it actually said?

What should I check, step by step?

1. Don't promise anyone that re-running the same prompt at temperature 0 will reproduce an answer. Treat every hosted response as a one-off observation.

2. Save the exact request body (model, messages, tools, temperature, top_p, seed, max tokens) and the full raw response, not a copy-paste of the text.
3. Hash both files (SHA-256) right after the call and record the hashes, the model value the API returned, any system_fingerprint or equivalent, and the time — before anyone disputes the answer.
4. When an answer is questioned, compare the saved response with the recorded hash. Use a re-run only as a fresh sample, and say so.
5. If you need repeatability for tests or evals, sample several times and report the spread, or replay recorded model outputs instead of calling the model again.
6. If you self-host and need bit-identical outputs, look at batch-invariant inference (the Thinking Machines post demonstrates it on vLLM) and expect a speed cost: their unoptimized deterministic run took 55 s against 26 s for default vLLM, 42 s with an improved attention kernel.
7. Record the serving stack you controlled (engine, version, GPU type, batch-invariant mode on or off) next to the output hash, because changing any of them can change outputs.

Why does an LLM give different answers at temperature 0?

The usual explanation is "floating point plus concurrency": GPU threads finish in random order, so sums come out slightly differently. The Thinking Machines post says that's not the full picture. Floating-point addition isn't associative, so adding the same numbers in a different order can give a different result, but the kernels in an LLM's forward pass don't use the order-dependent atomic adds that would make a single run random. Run the same batch twice and you get the same bits.

• What changes is the batch. Matrix multiplication, RMSNorm and attention kernels pick different reduction strategies for different batch sizes, so the result for your request depends on the batch size it was computed in. You don't control the batch size; server load does. From your side the system is nondeterministic even though the server is deterministic for any fixed set of requests.
• It isn't only GPUs. The post notes the same issue applies to endpoints served from CPUs or TPUs, and Anthropic's September 17, 2025 postmortem describes a separate case: in December 2024 its TPU implementation would occasionally drop the most probable token when temperature was zero, which it worked around at the time.

Can re-running a prompt prove what an AI model said earlier?

A re-run tells you what the model says now, on today's backend and today's load. It can't confirm or refute what it said last week. Small early differences also compound: in the Feynman test the completions agreed for 102 tokens and then split.

• Seeds don't fix this on hosted APIs. OpenAI documents seed as best effort and tells you to compare system_fingerprint to see whether the backend configuration changed between requests.
• So the evidence for "the model said X" has to be created when it said X: the saved request and response, hashed and recorded at the time, with the model value the API returned.

How does Rightshold record what an AI model said so it can be checked later?

record_ai_run takes the model, provider, the SHA-256 of the request and of the response, your settings and the time you ran it, and returns a receipt signed by Rightshold with an issued_at set when it was recorded. The prompt and output stay with you; only the hashes and fields you send are stored.

• Later, verify_ai_run takes the receipt, its signature and the saved output text, and confirms the signature, that the receipt is on record, and that the text matches the recorded hash. It does not call the model again.