# Our AI agent passes about 80% of our test tasks when we run each task once. Is that how often it will get a given customer's request right in production, and what is pass^k (pass hat k), which some agent benchmarks report instead?

**Short answer:** No. A one-run score is the agent's average success rate, not how often it succeeds every time on the same task. pass^k (read "pass hat k") is the chance that all k independent tries at the same task succeed, averaged over tasks. Sierra's τ-bench paper (2024) proposed it for agents that must be consistent, such as customer service, because pass@k, the older code-generation metric from OpenAI's Codex paper (2021), counts a task as solved if any one of k tries passes. With n trials per task and c successes, the paper's unbiased estimates are pass^k = C(c,k)/C(n,k) and pass@k = 1 − C(n−c,k)/C(n,k), averaged across tasks; for k = 1 both equal the plain success rate. The gap gets large fast. In τ-bench's retail tasks, the best function-calling agent tested (gpt-4o) succeeded on about 61% of single tries, but its pass^8 was under 25%. A task the agent solves 6 times in 8 has pass@4 of 100% and pass^4 of about 21%. So run each task several times, report pass^k with k and n stated next to pass^1, and keep a record of every trial so the figure can be checked.

## Checklist
1. Decide k and the number of trials per task (n, at least k) before running. Write them down with the task set version and the scoring rule.
2. Run every task n times with the same prompt, tools and settings. Change nothing between trials except the randomness you want to measure, such as sampling or a simulated user.
3. Score each trial pass or fail with the same rule. Count c, the passes, per task.
4. Compute pass^k per task as C(c,k)/C(n,k), which is 0 whenever c is less than k, then average across tasks. Report pass^1 alongside it.
5. Don't estimate pass^k as (pass^1)^k from one run per task. That assumes every task is equally hard and every trial independent, and it's wrong when some tasks always pass and others usually fail.
6. Label pass@k and pass^k clearly. pass@k rises with k and pass^k falls; a chart that doesn't say which can mislead in either direction.
7. Look at the tasks with the lowest per-task pass rate. Those are the requests where a customer may get a different outcome each time they ask.
8. Record every trial, failures included, so the c and n behind each task can be checked later.

## What is the difference between pass@k and pass^k for AI agent evals?
- OpenAI's Codex paper (2021) defined pass@k for code generation: generate n samples per problem (it used n = 200 and k up to 100), count the c that pass the unit tests, and estimate pass@k as 1 − C(n−c,k)/C(n,k), averaged over problems. It notes that the shortcut 1 − (1 − p̂)^k, built from the single-sample pass rate, is biased, and that pass@k amounts to choosing the best of k samples with an oracle that already knows the tests. Its Codex model solved 28.8% of HumanEval problems with one sample and 70.2% with 100 samples per problem.
- That is the right question when you can check every answer before using one, as with code and unit tests. An agent talking to a customer gets one go. Sierra's τ-bench paper (2024) defined pass^k as the chance that all k independent trials of a task succeed, averaged across tasks, with the unbiased estimate C(c,k)/C(n,k) per task. It says pass^1 equals pass@1, the average reward, and reports that as its main comparison metric.
- Worked example: a task run n = 8 times passes c = 6 times. pass@4 is 1, because any 4 tries include at least one pass. pass^4 is C(6,4)/C(8,4) = 15/70, about 0.21. Both describe the same 8 runs.

## What did the τ-bench benchmark find about how consistent AI agents are across repeated tries?
- τ-bench simulates conversations between an agent and a user played by a language model, in two customer-service domains: retail (115 tasks) and airline (50 tasks). The agent gets API tools and a written policy. A trial passes only if the final database matches the one allowed outcome and the agent's replies include the required information; the paper notes that this check is necessary but not sufficient, since an agent could, for example, make a change without the explicit confirmation the policy requires.
- The agent ran at temperature 0 and the simulated user at temperature 1, with at least 3 trials per task in the main results. gpt-4o with function calling scored 61.2% pass^1 in retail and 35.2% in airline, the best of the models tested. In retail its pass^8 fell below 25%.
- The authors' conclusion: in real-world use it isn't enough to build agents with high average success; the pass^k trend matters too. Even with the agent at temperature 0, results varied from trial to trial, because the simulated user's replies varied.

## How do you make an AI agent's pass^k reliability figure checkable?
- A pass^k figure depends on n, k, the task set and the scoring rule, and on every failed trial being counted. If failures quietly drop out, pass^k goes up, so keep a record of each trial as it finishes.
- For each trial, keep the task id, the trial number out of n, the model and dated version, a hash of the transcript or final state, the pass or fail result and the time. Anyone reviewing the figure can then recount c for each task and recompute pass^k.
- pass^k measures consistency on your task set only. It says nothing about requests unlike your tasks, and a model update can change it, so rerun it when the model or prompt changes.

## Ask Rightshold (free, no key)
Add the MCP server https://www.rightshold.com/mcp to Claude, Cursor, ChatGPT, Grok or VS Code (one-click setup: https://www.rightshold.com/start?ref=seo), then ask: "Record a signed receipt for trial 5 of 8 on task R-017 of our agent reliability eval: the model version, hashes of the task prompt and the full transcript, and in params the task id, trial index, trials planned, k, task-set version and whether the trial passed, so a reviewer can recount the passes behind our pass^k figure."

```json
{
  "tool": "record_ai_run",
  "arguments": {
    "model": "<agent model id and dated version>",
    "provider": "<model provider>",
    "prompt_sha256": "<sha256 of the task prompt, policy and tool list as sent>",
    "output_sha256": "<sha256 of this trial's full transcript and final state>",
    "params": {
      "run": "pass-hat-k-eval",
      "task_id": "R-017",
      "trial_index": 5,
      "trials_planned": 8,
      "k": 4,
      "task_set_version": "<version>",
      "scoring_rule": "final-state-match-and-required-info",
      "passed": false
    },
    "ran_at": "2026-11-13T12:30:00Z"
  }
}
```
- Needs an API key. Send hashes and the fields shown; keep the transcripts in your own store. params are stored as a hash, so keep the exact params object for each trial.
- Record every trial, passed or failed. A reviewer can then check that trial_index runs 1 to trials_planned for every task.
- verify_ai_run is free and needs no key: a reviewer sends a receipt, its signature and the transcript text to confirm it matches what was recorded.
- A receipt shows a trial was recorded and when. It can't show that no trials were run and left out; recording the run plan first with record_action makes gaps visible.

_Rightshold records the trial details you send and lets anyone check them later. It doesn't run agents, simulate users, score trials or compute pass^k, and it isn't affiliated with Sierra, OpenAI or any benchmark. Definitions and figures are from the τ-bench paper (arXiv 2406.12045) and the Codex paper (arXiv 2107.03374), read on November 13, 2026. Not procurement advice._

## Sources
- [Sierra: τ-bench, a benchmark for tool-agent-user interaction in real-world domains (2024; defines pass^k)](https://arxiv.org/abs/2406.12045)
- [τ-bench code and data (GitHub)](https://github.com/sierra-research/tau-bench)
- [OpenAI: Evaluating Large Language Models Trained on Code (Codex, HumanEval, unbiased pass@k estimator, 2021)](https://arxiv.org/abs/2107.03374)
- [Verified AI Runs on Rightshold](https://www.rightshold.com/verified-ai-runs)

## Related
- [How to prove a reported eval score wasn't cherry-picked from many runs](https://www.rightshold.com/answers/how-to-prove-a-reported-eval-score-wasnt-cherry-picked-from-many-runs)
- [Why an LLM gives different answers at temperature 0, and how to prove what it said](https://www.rightshold.com/answers/why-does-an-llm-give-different-answers-at-temperature-0)
- [How to report false-positive rates with confidence intervals on a frozen holdout](https://www.rightshold.com/answers/how-to-report-false-positive-rates-with-confidence-intervals-on-a-frozen-holdout)
- [How much should you trust an AI agent benchmark score when another AI model grades it?](https://www.rightshold.com/answers/how-much-should-you-trust-an-ai-agent-benchmark-score-when-another-ai-model-grades-it)

_Not legal advice. Updated 2026-11-13. JSON: https://www.rightshold.com/answers/what-is-pass-hat-k-and-why-one-run-per-task-overstates-how-consistent-an-ai-agent-is.json_