We already trace every LLM call in LangSmith or Langfuse. Is that an audit trail we can rely on if an AI answer is disputed?

What should I check, step by step?

1. Check sampling first. In Langfuse, LANGFUSE_SAMPLE_RATE or sample_rate below 1 (or an OpenTelemetry sampler) means some answers were never traced. An audit record needs every answer that reached a customer.

2. Check retention. LangSmith cloud: base 14 days, extended up to 180 days. Langfuse: no automatic deletion unless a project retention period is set, but owners and admins can set one as short as 3 days. Compare that with how long complaints, chargebacks and claims can take to arrive.
3. Check who can delete. In Langfuse anyone with delete rights can remove a trace in the UI or by API, and Langfuse says data removed by retention can't be recovered. In LangSmith traces can be deleted before they expire. List who holds those rights, including API keys used by apps and agents.
4. Check what the trace holds: the exact request with system prompt, retrieved context and tool results, the raw output, the model value the provider returned, settings, and the provider's request id. Debug traces often truncate, redact or summarize, which is fine for debugging and not for evidence.
5. Add a record that sits outside the tracing tool: at send time, hash the request and response and record the hash somewhere append-only that your own admins can't edit, with the trace id alongside so the two can be matched later.
6. Export full traces to write-once storage on a schedule (Langfuse offers blob storage export to S3, GCS or Azure; S3 Object Lock in compliance mode or a locked Cloud Storage bucket stops early deletion).
7. Keep evals and audit records apart. An eval dataset built from traces measures quality across chosen cases; it doesn't show what one customer was told. Saving a trace to a dataset copies selected fields, not the whole trace.

What are LLM tracing tools like LangSmith and Langfuse designed for?

Debugging, monitoring and evaluation: seeing why a chain failed, what it cost, how long it took and how answers score. For those jobs, sampling, short retention and easy deletion are features. They cut cost and make it simple to remove test traffic or a user's data on request.

• LangSmith describes base traces as short-lived and suited to quick debugging, with extended retention for traces that carry valuable feedback. After retention ends, traces are no longer accessible in the UI or API and their inputs and outputs are deleted.
• Langfuse documents deleting single traces, batches of traces and all traces matching a filter. Its project retention policy deletes traces, observations and scores nightly once they age out, and those can't be recovered, while its own audit logs and dataset items are unaffected.

What does an AI audit trail need that an observability trace doesn't provide?

Complete: one record for every answer that reached a person, written at the time, not a sample.

• Protected: the people whose work it may be used to judge, including admins and the app's own API keys, can't edit or delete it without that showing. NIST SP 800-53 AU-9 calls for protecting audit information from unauthorized modification and deletion, and AU-11 for keeping audit records for a defined period.
• Kept long enough, and checkable by someone outside your team: a customer, auditor or court shouldn't have to trust your admin console that nothing changed.

How do I use LLM tracing and an audit trail together?

Keep tracing as it is. At the point the answer goes out, compute a SHA-256 of the exact request and response the trace captured and record it, with the trace id, in an append-only record outside the tracing tool.

• Rightshold's record_ai_run does that step: it takes the model, provider, the hashes and params such as your trace id, and returns a signed receipt with Rightshold's own timestamp on its append-only ledger. Later, verify_ai_run (free) checks whether a trace or a quoted answer still matches. If someone edited or deleted the trace, the receipt still shows what existed at send time, as long as you kept the original or an export of it.