{
  "question": "Our AI agent passes about 80% of our test tasks when we run each task once. Is that how often it will get a given customer's request right in production, and what is pass^k (pass hat k), which some agent benchmarks report instead?",
  "slug": "what-is-pass-hat-k-and-why-one-run-per-task-overstates-how-consistent-an-ai-agent-is",
  "url": "https://www.rightshold.com/answers/what-is-pass-hat-k-and-why-one-run-per-task-overstates-how-consistent-an-ai-agent-is",
  "short_answer": "No. A one-run score is the agent's average success rate, not how often it succeeds every time on the same task. pass^k (read \"pass hat k\") is the chance that all k independent tries at the same task succeed, averaged over tasks. Sierra's τ-bench paper (2024) proposed it for agents that must be consistent, such as customer service, because pass@k, the older code-generation metric from OpenAI's Codex paper (2021), counts a task as solved if any one of k tries passes. With n trials per task and c successes, the paper's unbiased estimates are pass^k = C(c,k)/C(n,k) and pass@k = 1 − C(n−c,k)/C(n,k), averaged across tasks; for k = 1 both equal the plain success rate. The gap gets large fast. In τ-bench's retail tasks, the best function-calling agent tested (gpt-4o) succeeded on about 61% of single tries, but its pass^8 was under 25%. A task the agent solves 6 times in 8 has pass@4 of 100% and pass^4 of about 21%. So run each task several times, report pass^k with k and n stated next to pass^1, and keep a record of every trial so the figure can be checked.",
  "checklist": [
    "Decide k and the number of trials per task (n, at least k) before running. Write them down with the task set version and the scoring rule.",
    "Run every task n times with the same prompt, tools and settings. Change nothing between trials except the randomness you want to measure, such as sampling or a simulated user.",
    "Score each trial pass or fail with the same rule. Count c, the passes, per task.",
    "Compute pass^k per task as C(c,k)/C(n,k), which is 0 whenever c is less than k, then average across tasks. Report pass^1 alongside it.",
    "Don't estimate pass^k as (pass^1)^k from one run per task. That assumes every task is equally hard and every trial independent, and it's wrong when some tasks always pass and others usually fail.",
    "Label pass@k and pass^k clearly. pass@k rises with k and pass^k falls; a chart that doesn't say which can mislead in either direction.",
    "Look at the tasks with the lowest per-task pass rate. Those are the requests where a customer may get a different outcome each time they ask.",
    "Record every trial, failures included, so the c and n behind each task can be checked later."
  ],
  "sections": [
    {
      "heading": "pass@k and pass^k answer different questions",
      "points": [
        "OpenAI's Codex paper (2021) defined pass@k for code generation: generate n samples per problem (it used n = 200 and k up to 100), count the c that pass the unit tests, and estimate pass@k as 1 − C(n−c,k)/C(n,k), averaged over problems. It notes that the shortcut 1 − (1 − p̂)^k, built from the single-sample pass rate, is biased, and that pass@k amounts to choosing the best of k samples with an oracle that already knows the tests. Its Codex model solved 28.8% of HumanEval problems with one sample and 70.2% with 100 samples per problem.",
        "That is the right question when you can check every answer before using one, as with code and unit tests. An agent talking to a customer gets one go. Sierra's τ-bench paper (2024) defined pass^k as the chance that all k independent trials of a task succeed, averaged across tasks, with the unbiased estimate C(c,k)/C(n,k) per task. It says pass^1 equals pass@1, the average reward, and reports that as its main comparison metric.",
        "Worked example: a task run n = 8 times passes c = 6 times. pass@4 is 1, because any 4 tries include at least one pass. pass^4 is C(6,4)/C(8,4) = 15/70, about 0.21. Both describe the same 8 runs."
      ]
    },
    {
      "heading": "What τ-bench measured",
      "points": [
        "τ-bench simulates conversations between an agent and a user played by a language model, in two customer-service domains: retail (115 tasks) and airline (50 tasks). The agent gets API tools and a written policy. A trial passes only if the final database matches the one allowed outcome and the agent's replies include the required information; the paper notes that this check is necessary but not sufficient, since an agent could, for example, make a change without the explicit confirmation the policy requires.",
        "The agent ran at temperature 0 and the simulated user at temperature 1, with at least 3 trials per task in the main results. gpt-4o with function calling scored 61.2% pass^1 in retail and 35.2% in airline, the best of the models tested. In retail its pass^8 fell below 25%.",
        "The authors' conclusion: in real-world use it isn't enough to build agents with high average success; the pass^k trend matters too. Even with the agent at temperature 0, results varied from trial to trial, because the simulated user's replies varied."
      ]
    },
    {
      "heading": "Making a reliability number checkable",
      "points": [
        "A pass^k figure depends on n, k, the task set and the scoring rule, and on every failed trial being counted. If failures quietly drop out, pass^k goes up, so keep a record of each trial as it finishes.",
        "For each trial, keep the task id, the trial number out of n, the model and dated version, a hash of the transcript or final state, the pass or fail result and the time. Anyone reviewing the figure can then recount c for each task and recompute pass^k.",
        "pass^k measures consistency on your task set only. It says nothing about requests unlike your tasks, and a model update can change it, so rerun it when the model or prompt changes."
      ]
    }
  ],
  "checks": [],
  "ask_rightshold": {
    "start": "https://www.rightshold.com/start?ref=seo",
    "mcp_endpoint": "https://www.rightshold.com/mcp",
    "ask": "Record a signed receipt for trial 5 of 8 on task R-017 of our agent reliability eval: the model version, hashes of the task prompt and the full transcript, and in params the task id, trial index, trials planned, k, task-set version and whether the trial passed, so a reviewer can recount the passes behind our pass^k figure.",
    "mcp_call": {
      "tool": "record_ai_run",
      "arguments": {
        "model": "<agent model id and dated version>",
        "provider": "<model provider>",
        "prompt_sha256": "<sha256 of the task prompt, policy and tool list as sent>",
        "output_sha256": "<sha256 of this trial's full transcript and final state>",
        "params": {
          "run": "pass-hat-k-eval",
          "task_id": "R-017",
          "trial_index": 5,
          "trials_planned": 8,
          "k": 4,
          "task_set_version": "<version>",
          "scoring_rule": "final-state-match-and-required-info",
          "passed": false
        },
        "ran_at": "2026-11-13T12:30:00Z"
      }
    },
    "also": [
      "Needs an API key. Send hashes and the fields shown; keep the transcripts in your own store. params are stored as a hash, so keep the exact params object for each trial.",
      "Record every trial, passed or failed. A reviewer can then check that trial_index runs 1 to trials_planned for every task.",
      "verify_ai_run is free and needs no key: a reviewer sends a receipt, its signature and the transcript text to confirm it matches what was recorded.",
      "A receipt shows a trial was recorded and when. It can't show that no trials were run and left out; recording the run plan first with record_action makes gaps visible."
    ],
    "scope": "Rightshold records the trial details you send and lets anyone check them later. It doesn't run agents, simulate users, score trials or compute pass^k, and it isn't affiliated with Sierra, OpenAI or any benchmark. Definitions and figures are from the τ-bench paper (arXiv 2406.12045) and the Codex paper (arXiv 2107.03374), read on November 13, 2026. Not procurement advice."
  },
  "sources": [
    {
      "title": "Sierra: τ-bench, a benchmark for tool-agent-user interaction in real-world domains (2024; defines pass^k)",
      "url": "https://arxiv.org/abs/2406.12045"
    },
    {
      "title": "τ-bench code and data (GitHub)",
      "url": "https://github.com/sierra-research/tau-bench"
    },
    {
      "title": "OpenAI: Evaluating Large Language Models Trained on Code (Codex, HumanEval, unbiased pass@k estimator, 2021)",
      "url": "https://arxiv.org/abs/2107.03374"
    },
    {
      "title": "Verified AI Runs on Rightshold",
      "url": "https://www.rightshold.com/verified-ai-runs"
    }
  ],
  "related": [
    {
      "title": "How to prove a reported eval score wasn't cherry-picked from many runs",
      "url": "https://www.rightshold.com/answers/how-to-prove-a-reported-eval-score-wasnt-cherry-picked-from-many-runs"
    },
    {
      "title": "Why an LLM gives different answers at temperature 0, and how to prove what it said",
      "url": "https://www.rightshold.com/answers/why-does-an-llm-give-different-answers-at-temperature-0"
    },
    {
      "title": "How to report false-positive rates with confidence intervals on a frozen holdout",
      "url": "https://www.rightshold.com/answers/how-to-report-false-positive-rates-with-confidence-intervals-on-a-frozen-holdout"
    },
    {
      "title": "How much should you trust an AI agent benchmark score when another AI model grades it?",
      "url": "https://www.rightshold.com/answers/how-much-should-you-trust-an-ai-agent-benchmark-score-when-another-ai-model-grades-it"
    }
  ],
  "computed_from": "Curated guidance; license verdicts computed from Rightshold's license profiles at render time. Not legal advice.",
  "updated": "2026-11-13"
}