{
  "question": "We already trace every LLM call in LangSmith or Langfuse. Is that an audit trail we can rely on if an AI answer is disputed?",
  "slug": "is-an-llm-observability-trace-the-same-as-an-audit-trail-of-ai-answers",
  "url": "https://www.rightshold.com/answers/is-an-llm-observability-trace-the-same-as-an-audit-trail-of-ai-answers",
  "short_answer": "Not on its own. Tracing tools such as LangSmith and Langfuse are built for debugging and evaluation, and their defaults show it. LangSmith keeps base traces for 14 days and extended traces for up to 180 days on its cloud service, then deletes them, and traces can be deleted before then. Langfuse lets users delete a trace from the UI or with DELETE /api/public/traces/{traceId}, lets project owners and admins set a retention period as short as 3 days, and its SDKs can sample, so with a sample rate of 0.2 only 20% of traces are kept at all. An audit trail has to be complete (every answer, not a sample), protected from change or deletion by the people it may be used against, and kept for as long as a dispute can arrive. NIST SP 800-53 puts the second point plainly in AU-9: protect audit information from unauthorized access, modification and deletion. Keep the tracing tool for what it's good at, and add an append-only record at the point each answer goes out: a hash of the request and response, recorded outside the tracing tool, with the full trace exported to storage your team can't quietly edit.",
  "checklist": [
    "Check sampling first. In Langfuse, LANGFUSE_SAMPLE_RATE or sample_rate below 1 (or an OpenTelemetry sampler) means some answers were never traced. An audit record needs every answer that reached a customer.",
    "Check retention. LangSmith cloud: base 14 days, extended up to 180 days. Langfuse: no automatic deletion unless a project retention period is set, but owners and admins can set one as short as 3 days. Compare that with how long complaints, chargebacks and claims can take to arrive.",
    "Check who can delete. In Langfuse anyone with delete rights can remove a trace in the UI or by API, and Langfuse says data removed by retention can't be recovered. In LangSmith traces can be deleted before they expire. List who holds those rights, including API keys used by apps and agents.",
    "Check what the trace holds: the exact request with system prompt, retrieved context and tool results, the raw output, the model value the provider returned, settings, and the provider's request id. Debug traces often truncate, redact or summarize, which is fine for debugging and not for evidence.",
    "Add a record that sits outside the tracing tool: at send time, hash the request and response and record the hash somewhere append-only that your own admins can't edit, with the trace id alongside so the two can be matched later.",
    "Export full traces to write-once storage on a schedule (Langfuse offers blob storage export to S3, GCS or Azure; S3 Object Lock in compliance mode or a locked Cloud Storage bucket stops early deletion).",
    "Keep evals and audit records apart. An eval dataset built from traces measures quality across chosen cases; it doesn't show what one customer was told. Saving a trace to a dataset copies selected fields, not the whole trace."
  ],
  "sections": [
    {
      "heading": "What tracing tools are designed for",
      "points": [
        "Debugging, monitoring and evaluation: seeing why a chain failed, what it cost, how long it took and how answers score. For those jobs, sampling, short retention and easy deletion are features. They cut cost and make it simple to remove test traffic or a user's data on request.",
        "LangSmith describes base traces as short-lived and suited to quick debugging, with extended retention for traces that carry valuable feedback. After retention ends, traces are no longer accessible in the UI or API and their inputs and outputs are deleted.",
        "Langfuse documents deleting single traces, batches of traces and all traces matching a filter. Its project retention policy deletes traces, observations and scores nightly once they age out, and those can't be recovered, while its own audit logs and dataset items are unaffected."
      ]
    },
    {
      "heading": "What an audit trail needs that a trace doesn't promise",
      "points": [
        "Complete: one record for every answer that reached a person, written at the time, not a sample.",
        "Protected: the people whose work it may be used to judge, including admins and the app's own API keys, can't edit or delete it without that showing. NIST SP 800-53 AU-9 calls for protecting audit information from unauthorized modification and deletion, and AU-11 for keeping audit records for a defined period.",
        "Kept long enough, and checkable by someone outside your team: a customer, auditor or court shouldn't have to trust your admin console that nothing changed."
      ]
    },
    {
      "heading": "Using both without doubling the work",
      "points": [
        "Keep tracing as it is. At the point the answer goes out, compute a SHA-256 of the exact request and response the trace captured and record it, with the trace id, in an append-only record outside the tracing tool.",
        "Rightshold's record_ai_run does that step: it takes the model, provider, the hashes and params such as your trace id, and returns a signed receipt with Rightshold's own timestamp on its append-only ledger. Later, verify_ai_run (free) checks whether a trace or a quoted answer still matches. If someone edited or deleted the trace, the receipt still shows what existed at send time, as long as you kept the original or an export of it."
      ]
    }
  ],
  "checks": [],
  "ask_rightshold": {
    "start": "https://www.rightshold.com/start?ref=seo",
    "mcp_endpoint": "https://www.rightshold.com/mcp",
    "ask": "When this traced answer goes out, record a signed receipt with the model, hashes of the request and response the trace captured, and the trace id, so we can check later that the trace wasn't changed.",
    "mcp_call": {
      "tool": "record_ai_run",
      "arguments": {
        "model": "<model value from the response>",
        "provider": "openai",
        "prompt_sha256": "<sha256 of the request JSON in the trace>",
        "output_sha256": "<sha256 of the response JSON in the trace>",
        "params": {
          "trace_id": "<LangSmith or Langfuse trace id>",
          "request_id": "<x-request-id from the provider>",
          "sample_rate": 1
        },
        "ran_at": "2026-11-09T22:00:00Z"
      }
    },
    "also": [
      "Needs an API key. Send hashes only, not the trace contents; if you send text, Rightshold stores only its SHA-256.",
      "verify_ai_run is free and needs no key. Send the receipt id, or the receipt and signature plus the request or output text from the trace (or from your export), to check it matches what was recorded.",
      "Hash the same bytes every time: decide whether you hash the raw provider JSON or the text shown to the customer, and keep to it.",
      "A receipt shows what was recorded and when Rightshold recorded it. It doesn't show the trace is complete, that the provider served that model, or that the answer was true."
    ],
    "scope": "Rightshold doesn't connect to LangSmith, Langfuse or any tracing tool, and it doesn't store your traces. It signs and timestamps the hashes you send and lets anyone check them later. Retention and deletion details above are from each vendor's docs as of November 2026 and vary by plan and self-hosting; check your own settings. Not legal advice."
  },
  "sources": [
    {
      "title": "LangSmith: Usage and billing (base 14 days, extended 180 days, deletion after retention)",
      "url": "https://docs.langchain.com/langsmith/usage-and-billing"
    },
    {
      "title": "LangSmith: Data purging and compliance (retention settings, 180-day maximum from September 14, 2026)",
      "url": "https://docs.langchain.com/langsmith/data-purging-compliance"
    },
    {
      "title": "Langfuse: Data Deletion (single, batch and filtered trace deletion)",
      "url": "https://langfuse.com/docs/administration/data-deletion"
    },
    {
      "title": "Langfuse: Data Retention (project retention, minimum 3 days, audit logs and datasets unaffected)",
      "url": "https://langfuse.com/docs/administration/data-retention"
    },
    {
      "title": "Langfuse: Sampling (client-side sample rate)",
      "url": "https://langfuse.com/docs/observability/features/sampling"
    },
    {
      "title": "NIST SP 800-53 Rev. 5: AU-9 (protection of audit information) and AU-11 (audit record retention)",
      "url": "https://csrc.nist.gov/publications/detail/sp/800-53/rev-5/final"
    },
    {
      "title": "Verified AI Runs on Rightshold",
      "url": "https://www.rightshold.com/verified-ai-runs"
    }
  ],
  "related": [
    {
      "title": "Can you get an old AI API response back from the provider to settle a dispute?",
      "url": "https://www.rightshold.com/answers/can-you-get-an-old-ai-api-response-back-from-the-provider-to-settle-a-dispute"
    },
    {
      "title": "How to make an AI agent's audit log append-only so even an admin can't silently delete entries",
      "url": "https://www.rightshold.com/answers/how-to-make-an-ai-agents-audit-log-append-only-so-even-an-admin-cant-silently-delete-entries"
    },
    {
      "title": "Does a passing eval prove what happened in one AI incident?",
      "url": "https://www.rightshold.com/answers/does-a-passing-ai-eval-prove-what-happened-in-one-specific-incident"
    },
    {
      "title": "Verified AI Runs",
      "url": "https://www.rightshold.com/verified-ai-runs"
    }
  ],
  "computed_from": "Curated guidance; license verdicts computed from Rightshold's license profiles at render time. Not legal advice.",
  "updated": "2026-11-09"
}