Our AI agent passes about 80% of our test tasks when we run each task once. Is that how often it will get a given customer's request right in production, and what is pass^k (pass hat k), which some agent benchmarks report instead?

What should I check, step by step?

1. Decide k and the number of trials per task (n, at least k) before running. Write them down with the task set version and the scoring rule.

2. Run every task n times with the same prompt, tools and settings. Change nothing between trials except the randomness you want to measure, such as sampling or a simulated user.
3. Score each trial pass or fail with the same rule. Count c, the passes, per task.
4. Compute pass^k per task as C(c,k)/C(n,k), which is 0 whenever c is less than k, then average across tasks. Report pass^1 alongside it.
5. Don't estimate pass^k as (pass^1)^k from one run per task. That assumes every task is equally hard and every trial independent, and it's wrong when some tasks always pass and others usually fail.
6. Label pass@k and pass^k clearly. pass@k rises with k and pass^k falls; a chart that doesn't say which can mislead in either direction.
7. Look at the tasks with the lowest per-task pass rate. Those are the requests where a customer may get a different outcome each time they ask.
8. Record every trial, failures included, so the c and n behind each task can be checked later.

What is the difference between pass@k and pass^k for AI agent evals?

OpenAI's Codex paper (2021) defined pass@k for code generation: generate n samples per problem (it used n = 200 and k up to 100), count the c that pass the unit tests, and estimate pass@k as 1 − C(n−c,k)/C(n,k), averaged over problems. It notes that the shortcut 1 − (1 − p̂)^k, built from the single-sample pass rate, is biased, and that pass@k amounts to choosing the best of k samples with an oracle that already knows the tests. Its Codex model solved 28.8% of HumanEval problems with one sample and 70.2% with 100 samples per problem.

• That is the right question when you can check every answer before using one, as with code and unit tests. An agent talking to a customer gets one go. Sierra's τ-bench paper (2024) defined pass^k as the chance that all k independent trials of a task succeed, averaged across tasks, with the unbiased estimate C(c,k)/C(n,k) per task. It says pass^1 equals pass@1, the average reward, and reports that as its main comparison metric.
• Worked example: a task run n = 8 times passes c = 6 times. pass@4 is 1, because any 4 tries include at least one pass. pass^4 is C(6,4)/C(8,4) = 15/70, about 0.21. Both describe the same 8 runs.

What did the τ-bench benchmark find about how consistent AI agents are across repeated tries?

τ-bench simulates conversations between an agent and a user played by a language model, in two customer-service domains: retail (115 tasks) and airline (50 tasks). The agent gets API tools and a written policy. A trial passes only if the final database matches the one allowed outcome and the agent's replies include the required information; the paper notes that this check is necessary but not sufficient, since an agent could, for example, make a change without the explicit confirmation the policy requires.

• The agent ran at temperature 0 and the simulated user at temperature 1, with at least 3 trials per task in the main results. gpt-4o with function calling scored 61.2% pass^1 in retail and 35.2% in airline, the best of the models tested. In retail its pass^8 fell below 25%.
• The authors' conclusion: in real-world use it isn't enough to build agents with high average success; the pass^k trend matters too. Even with the agent at temperature 0, results varied from trial to trial, because the simulated user's replies varied.

How do you make an AI agent's pass^k reliability figure checkable?

A pass^k figure depends on n, k, the task set and the scoring rule, and on every failed trial being counted. If failures quietly drop out, pass^k goes up, so keep a record of each trial as it finishes.

• For each trial, keep the task id, the trial number out of n, the model and dated version, a hash of the transcript or final state, the pass or fail result and the time. Anyone reviewing the figure can then recount c for each task and recompute pass^k.
• pass^k measures consistency on your task set only. It says nothing about requests unlike your tasks, and a model update can change it, so rerun it when the model or prompt changes.