τ-bench Leaderboard

What this benchmark measures

• tool-using agent behaviour
• policy adherence
• multi-turn user interaction

Relevant to:

• customer-service agent screening
• agent reliability assessment

Failure modes it exercises:

• rule violations and inconsistent agent behaviour across repeated attempts

Method and its limits

Simulated multi-turn conversations in which an agent uses domain-specific API tools under written policy rules; success is measured by comparing the final database state to an annotated goal state, with pass^k measuring consistency across repeated trials.

• The user is simulated by a language model, so conversational difficulty depends on that simulator.
• Domain coverage is narrow; the paper names retail specifically.
• pass^1 overstates deployable reliability, which is why the authors introduced pass^k.

Dataset

Dataset

tau-bench

Type

simulated tool-agent-user conversations with annotated goal states

Freshness

aging

Benchmark release

2024-06-17

The primary paper registers the simulated-conversation design and the retail domain. ModelRefs does not register which domain or version each score was produced on.

How to read this score

pass^k is the figure that matters for deployment. The paper reports that state-of-the-art function-calling agents including gpt-4o succeed on under 50% of tasks, with pass^8 below 25% in retail, so a single-attempt success rate should not be read as reliability.

Similarity to real tasks: High — Established from the primary methodology: tau-bench simulates dynamic conversations between a language-model user and an agent equipped with domain API tools and written policy guidelines, then compares the database state at the end of the conversation with an annotated goal state. Verifying an outcome by final database state, rather than by text similarity, closely resembles how a deployed service agent would be judged. The simulated user remains a model rather than a real customer.

Data contamination risk: Medium — Tasks and policies are published in a public repository. No run-specific overlap audit is registered.

Benchmark gaming risk: Medium — Scaffolding, retry policy, and the choice of user simulator materially change results, and none is registered per score here.

What you still need to test yourself

• Rebuild the evaluation with your own policies, tools, and escalation paths.
• Measure pass^k on your own tasks; single-run success rates will overstate reliability.

This benchmark supports decisions about:

• Compare agent consistency rather than best-case success.
• Estimate how often an agent will break a written policy rule.

Limitations

• Not evidence that an agent is safe to deploy against real customers.
• Results depend heavily on agent scaffolding, which this page does not record.

Primary source reviewed 2026-10-11. tau-bench is treated as an aging 2024 benchmark.

What this evidence does not cover

• Four models are currently scored on this page, and neither the domain nor the pass^k setting behind each score is registered.

Sources

• tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains Yao, Shinn, Razavi, Narasimhan · accessed 2026-10-11Supports the simulated-conversation design, database-state grading, the pass^k metric, and the reported under-50% success and sub-25% pass^8 figures; it does not support any model score shown here.
• tau-bench repository Sierra Research · accessed 2026-10-11Official task and policy implementation; the exact revision and scaffolding used for any score here are not registered.

How this benchmark is scored

Primary source: https://github.com/sierra-research/tau-bench

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to τ-bench Leaderboard.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path