Answers · AI evidence
How do I make an AI agent run deterministically replayable from a recorded call order, with the seed kept as provenance and a versioned schedule format?
Last updated
TL;DR: Don't try to make the model deterministic; make the run replayable. Record every non-deterministic input in the order it happened: each model response, each tool result, clock reads, random draws and the order in which concurrent calls completed. That ordered record (the schedule) is what replay consumes: on replay, the harness returns recorded results instead of calling the model or tools, so the same code takes the same path. Keep the seed, model snapshot, system_fingerprint and sampling settings in the record as provenance, because provider seeds are best effort and a re-run with the same seed can still differ. Give the schedule format a version field and record the generator (harness name, version and commit) that wrote it, so a replayer can refuse a schedule it doesn't understand instead of misreading it. Then hash-chain the schedule entries and commit the head somewhere you can't edit, so a replay can be shown to use the run that actually happened.
What should I check, step by step?
1. Route every source of non-determinism through one recorder: model calls, tool calls, HTTP fetches, time.now(), random(), UUID generation, environment reads, and the completion order of parallel calls.
- Write one schedule entry per event in the order it was observed: seq, kind (model, tool, clock, random, join), a call id, a hash of the request, the response (or its hash plus a pointer to stored content), and the previous entry's hash.
- Put the schedule format version and the generator (harness name, version, git commit) in a header, and refuse to replay a version the replayer doesn't support.
- Record provenance alongside, not instead of, the results: model snapshot id, the model value the API returned, system_fingerprint where offered, temperature, top_p, seed, tool definitions hash and prompt hashes.
- In replay mode, match each call by seq and request hash and return the recorded response. A mismatch (the code asked for something different) is a divergence: stop and report the first differing seq rather than calling the live model.
- For parallel tool calls, record the order results were consumed and replay in that order, even if the original calls completed out of order.
- Hash-chain entries (prev = hash of the previous entry) and commit the head when the run ends, so a later edit to any entry breaks the chain.
- Keep schedules alongside the code version that produced them. A replay against changed agent code is a test of the new code, not a reproduction of the old run.
Why is the seed provenance and not a guarantee of reproducibility?
Providers describe seeds as best effort. OpenAI documents seed together with system_fingerprint, which changes when the backend configuration changes; the same seed on a different backend may give different output.
- Batching, floating-point order on GPUs, and model snapshot retirement all change outputs between runs. Models also get retired, so an exact re-run may be impossible a few months later.
- Record the seed so a reader knows what was asked for, and record the outputs so the run can be replayed without the model at all.
How do I record the call order and replay an agent run from it?
This is how durable-execution engines work. Temporal records an event history for each workflow and replays the workflow code against it after a crash; its docs require workflow code to be deterministic and to get time, randomness and I/O through the SDK or activities so replay sees the recorded values.
- LangGraph's durable execution docs give the same rule: wrap side effects and non-deterministic operations in tasks so their results are saved and reused when a run resumes instead of being re-executed.
- Record-and-replay for HTTP is older still. vcrpy records requests and responses to a cassette and plays them back in tests. An agent schedule is a cassette that also covers model calls, clock reads, random draws and ordering.
- Ordering is part of the record. Two tool calls that finish in a different order can change what the model sees next, so the schedule stores the order results were delivered to the agent.
How do I version the schedule format and its generator?
Give the file a format id and version (for example schedule/1) and bump it when entry fields change meaning. A replayer that silently accepts a newer format can misread entries and report a false reproduction.
- Record the generator: harness name, version and commit. If a bug in the recorder dropped clock reads, you can find every schedule it wrote.
- Canonicalize before hashing (RFC 8785 JSON Canonicalization Scheme or a fixed key order), so the same entry always hashes the same way across languages.
- Keep a small replay test in CI: replay a stored schedule against the current harness and fail if it diverges, so format or harness changes don't break old records unnoticed.
How do I make the run record tamper-evident?
Chain each entry to the previous one by hash and sign entries with the agent's key, so a disclosed stretch can be checked for gaps and edits.
- Commit the chain head to a log you don't control at the end of the run (or periodically during long runs), so you can show the schedule existed in this form at that time.
- Disclose only what's needed: a stretch of entries that chains to a committed head proves those steps without publishing the whole run.
Where do agent replays go wrong?
A direct time.now() or random() call bypasses the recorder; the replay takes a different branch on the first run after midnight.
- Parallel tool results are recorded by completion time but consumed in a different order; replay diverges at the first join.
- The schedule has no format version, and a recorder upgrade changes a field's meaning; old schedules replay wrongly without an error.
- On a missing recording the replayer falls back to the live model, and the 'reproduction' is really a new run.
- The seed is reported as proof the run is reproducible, but the model snapshot was retired and nothing else was recorded.
How do I ask Rightshold about this (free, no key)?
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code, then ask: “Record a signed receipt for this agent run: the model snapshot, the seed and sampling settings as provenance, the schedule format version and generator, and the hashes of the prompt and of the final schedule file.”
{
"tool": "record_ai_run",
"arguments": {
"model": "gpt-4o-2024-08-06",
"provider": "openai",
"prompt_sha256": "<sha256 of the system prompt + task input>",
"output_sha256": "<sha256 of the final schedule file>",
"params": {
"seed": 42,
"temperature": 0,
"system_fingerprint": "<from the response>",
"schedule_format": "schedule/1",
"generator": "my-harness 0.4.2 (<git sha>)",
"entries": 128
},
"ran_at": "2026-11-07T06:00:00Z"
}
}- Needs an API key. Only hashes and the fields you send are recorded; the receipt is Ed25519-signed and written to Rightshold's append-only ledger. verify_ai_run (free) later checks a receipt and, given the text, that it matches the recorded hashes.
- For the per-step chain, the flight recorder fits: your agent signs each schedule entry ({seq, at, kind, summary, prev}), recorder_checkpoint commits the head, and verify_recorder (free) checks that a disclosed stretch of entries is signed, consecutive and ends at a committed head, and says when it was committed.
- Neither proves the replay is correct or that the named model produced the outputs. They prove which schedule you recorded and that it hasn't changed since.
Rightshold doesn't record, run or replay your agent. It signs and timestamps the hashes and settings you send and checks disclosed log stretches against heads you committed. Not legal advice.
What are the sources?
- Temporal docs: Workflow Definition (deterministic constraints, replay)
- Temporal docs: Events and Event History
- LangGraph: Durable execution (tasks for side effects and non-determinism)
- vcrpy: record and replay HTTP interactions
- OpenAI API reference: Create chat completion (seed, system_fingerprint)
- RFC 8785: JSON Canonicalization Scheme
- Python docs: random (Notes on Reproducibility)
Related
- How to make LLM eval results reproducible and tamper-evident
- How to prove what an AI model said at a specific time
- How to log every AI agent tool call for audit
- Verified AI Runs
Not legal advice. JSON · Markdown · All answers
Get a free API key in one call
No card and no email needed. 100 free credits; the license and policy checks stay free without a key.
curl -s -X POST https://www.rightshold.com/v1/accounts/signup -H "Content-Type: application/json" -d '{"accept_terms":true}'
export RIGHTSHOLD_API_KEY=rs_... # the api_key from the reply (shown once)
curl -s https://www.rightshold.com/v1/licenses/check -H "Authorization: Bearer $RIGHTSHOLD_API_KEY" \
-H "Content-Type: application/json" -d '{"license":"MIT","intended_use":{"commercial_use":true}}'