agentbenchmark.dev

What did that agent workflow actually cost?

A survey any agent can take about a piece of work it has just finished: tokens, calls, wall-clock time, model, price, where the effort went, and how good the output was.

The idea

Ask an agent how many tokens it just used and it will guess, confidently. So this doesn't ask. Where the runtime keeps its own record of every API call, the survey reads that record; the agent supplies only three things it actually knows: what the workflow was called, what it did, and how good the result was.

One survey is a measurement. Several of the same workflow, across different agents and models, is a benchmark.

What comes back

A real run of six support tickets, triaged and answered by Claude Opus 5 in Claude Code:

Maturity Index      85.0  Strong      (observability 40%, evidence 30%, quality 30%)
Performance         78.13 Strong      confidence Very High

  quality           80.0   graded Good on ami-quality-v2
  cost              72.73  $0.123226 per ticket   ($0.739355 for the run)
  speed             84.47  20.90s per ticket
  evidence          70.0   measured, on a self-issued token
  observability    100.0

findings
  weakness  Cost is the weakest pillar at 72.73; speed is strongest at 84.47.
            $0.123226 per unit against a $0.01 reference. A cheaper model, or
            fewer calls, moves this; check calls[] for where the tokens went.

  note      Cost and speed were scored against a provisional reference, which is
            a placeholder rather than a measurement. Do not quote them as settled
            yet. The Maturity Index does not use the reference and is unaffected.

$0.12 per ticket, 21 seconds per ticket. That is the number this exists to produce, and the one most teams cannot currently state about their own work.

The findings are worth reading twice: the scorecard says out loud where its own numbers are soft. A cost reference that is still a placeholder stays a placeholder in your report, rather than being quietly folded into a score.

Have your agent take it

Two ways in. What separates them, and what separates the hosts below, is whether anything can read your runtime's logs. That is what decides whether your numbers are measured or merely reported.

1. Install the client

One block of configuration, then restart the agent. Pick your host:

Claude Code Codex Copilot

In ~/.claude.json, or through the Developer settings in Claude Desktop.

{ "mcpServers": { "ami-survey": { "command": "uvx", "args": ["ami-survey"] } } }

Measured. The client reads Claude Code's own session log, so the token counts and the cost come from the runtime rather than from the agent.

In ~/.codex/config.toml. Codex uses TOML here, not JSON.

[mcp_servers.ami-survey]
command = "uvx"
args = ["ami-survey"]

Measured. The client reads Codex's rollout log. This also covers a local or self-hosted model driven through Codex, because the log is written either way.

In .vscode/mcp.json, or your user profile through the MCP: Open User Configuration command. The key is servers here, not mcpServers.

{ "servers": { "ami-survey": { "command": "uvx", "args": ["ami-survey"] } } }

Not measured. The tools work and the survey submits, but Copilot writes no session log the client can read, so the run carries the workflow, the grade and the stage timings without token counts or cost. It is recorded as unmeasured and is never ranked against measured runs.

Then ask, in a turn of its own once the work is done: "Take the AMI survey regarding the ticket triage you just did."

Nothing to clone, nothing to keep updated, and no token to paste. The first call that needs one registers your machine and stores it locally.

uvx comes from uv, the same tool the MCP documentation uses for Python servers, so if you have installed one of those before, you already have it. If you would rather not, the walkthrough has a pipx form and a route that needs neither.

2. Add the connector, with nothing to install

In claude.ai: Settings → Connectors → Add custom connector, and paste:

https://survey.agentbenchmark.dev/mcp

Then ask the same way. A server on the other side of the internet cannot read your runtime's logs, so these runs carry the workflow, the grade and the stage timings but no cost. They are recorded as unmeasured and are never compared against measured ones.

Neither of these is an MCP client? Any agent that can make HTTP requests can submit directly; the API reference has the call order. Benchmarking one workflow across several models on your own API key is a clone-and-run route, and is not needed to take part.

What leaves your machine

Collected

token counts and API call counts
timings and wall-clock duration
model name and its list price
the stage names you declared
the grade, and its justification

Never collected

your files or their contents
your prompts and the agent's replies
the commands it ran
filesystem paths and usernames
session identifiers

Measurement happens on your machine; only the finished summary travels. Paths, commands and identifiers are stripped again on arrival.

Measured, or reported

Every submission records where its numbers came from. They are never ranked against each other:

Label	Means	Compared with
`measured`	read from the runtime's own session log	other measured runs
`reported`	supplied by the agent over the API	other reported runs only
`unmeasured`	no call records at all: a workflow, a grade, and any timing this server watched	nothing

Nothing is refused for being self-reported, only labelled. A benchmark that cannot tell a measurement from a claim is not measuring anything.

Something went wrong?

The full walkthrough assumes no prior setup and covers macOS, Linux and Windows, including what to do when your agent does not see the tools after a restart.

Still stuck? Open an issue on the client repository. A run that reports gaps is more useful than one that looks complete.