Last updated: 2026-10-29
CLOSEBENCH
The reference benchmark for AI sales agents.
Coding agents have SWE-bench. Tool-use agents have τ-bench. Selling — the part of agentic AI that touches money, makes promises and has to stay honest under pressure — had no reproducible benchmark. CloseBench is that benchmark.
Open source: github.com/AndreuwMetal/closebench · dataset on Hugging Face
What it does
It does not score a model on a prompt. It puts the whole sales agent — system prompt, tools, guardrails and state — into realistic WhatsApp conversations against an LLM-simulated buyer with a hidden budget and attitude, through the agent's own production entry point.
| What it measures | Meaning |
|---|---|
| Success rate | Goal reached (payment, demo, handoff, qualify-out or an ethical no-sale) with zero violations |
| pass^k | The scenario passed in all k runs: reliability, not a lucky sample |
| Violations | Lying, invented prices, ignoring an opt-out, faking being human, obeying injected instructions… one is enough to fail |
| Cost | Real token usage turned into dollars per conversation |
Why the compliance gate is absolute
The most persuasive strategy is often the deceptive one. A benchmark that scores persuasion without gating honesty quietly rewards lying. In CloseBench, one lie fails the scenario no matter how good the sell was.
Objective facts anchor the judge
The scorer reads the truth from mocks and SQLite — was a checkout created, for how much, did a guardrail fire — and gives those facts to the LLM judge. The judge rules on style and policy; it cannot be talked into believing a sale happened that didn't.
Who it is for
Teams deploying AI agents that sell, book or negotiate, and who need to know before launch whether the agent closes reliably and stays honest.