The Token Value Benchmark

Price per million tokens cannot be compared across models, because a token is not the same thing in each:

Tokenisers differ. The same sentence is a different number of tokens in each family.
Verbosity differs. Two correct answers can differ several-fold in output tokens.
Reasoning differs. A thinking model may spend many hidden tokens, billed as output, before the first visible word.
Caching differs. A model whose cache holds a long instruction cheaply is worth more than its list price suggests.
Failure is the real cost. A model at half the price that fails one task in five is not cheaper once the retry is counted.

So the benchmark measures cost per successful task, with quality held to a bar. Price per token is an input to that, not the answer.

The standard agent in every install

The Token Value Benchmark is an agent, atlantic · platform · benchmark · token-value (atlantic-platform:benchmark-token-value), shipped with every engine. It is off by default and nothing schedules it: it is the one agent whose job is spending money on models, and an agent that started itself would be a bill nobody agreed.

It runs in your account, through your own gateway, so your allow list, residency rule, budgets and kill switch all apply to it. Its tasks, their material and the answers never leave your account. Only counts, timings, grades and costs reach LightHouse.

The suite

The standard suite (aal-token-value, version 1.0.0) ships with the engine: 21 tasks over the eight task classes the router routes by, on material Atlantic invented, so you can read every task. Three tasks per class where the answer can be checked mechanically, two for the rubric-graded classes. The long-context tasks read a 65,000-character document generated from the suite’s version, so everyone running 1.0.0 reads the same one.

Your own tasks go in a file in your account (var/benchmark/client-tasks.json by default), in the same shape:

{
  "tasks": [
    {
      "id": "claims-ext-01",
      "task_class": "extraction",
      "input_tokens_est": 1200,
      "max_output_tokens": 200,
      "instruction": "From the material, return JSON with exactly the keys policy_number, loss_date and amount_eur. Return JSON only.",
      "material": "…your own text…",
      "check": { "kind": "json_fields", "expect": ["policy_number", "loss_date", "amount_eur"] }
    }
  ]
}

Your tasks are reported beside the standard suite’s (tasks_source: client), never averaged into them, and never change the suite’s version. A change of suite version is a change of baseline, and is labelled as one.

21 tasks is enough to separate models on cost per successful task. It is not a quality leaderboard, and does not claim to be.

Grading

Deterministic checks wherever a task allows one: exact, label, contains-all, regex, JSON fields, JSON schema, citation, tool call and numeric.
A rubric judged by a named model for summarising and drafting. A run with rubric tasks and no judge is refused at the start, rather than leaving two classes ungraded.
A sample of grades is surfaced for a person to confirm, and the report says how many were confirmed.
JSON that does not parse is a fail. A citation to a document the task never gave is a fail. A call the gateway throttles into failure is a fail, with its reason. Nothing is skipped.

The dry-run estimate and the hard cap

Estimate. Before any model is called, the benchmark projects the whole run’s cost from each task’s declared size and your price card. The projection allows twice each task’s visible output limit, because a thinking model’s hidden tokens are billed as output: under-projecting is the dangerous direction.
Refuse a run with no cap, a run whose estimate exceeds its cap, and a run with rubric tasks and no judge.
Check the cap before every attempt, never after, with the same projection the estimate used. Checking afterwards would let every run pass its cap by one call.

In LightHouse the cap is between $1 and $1,000. Your deployment can set its own ceiling, and a cap above the agent’s own budget is refused, so the cap you approve is the one the run is stopped against.

Starting a run

A client_admin, signed in as a person: Agents → Routing, start a run, with its cap, the task classes, the judge and a reason. Or the API:

curl -X POST "https://$AAL_TENANT.atlanticagenticlabs.com/api/v1/benchmark/runs" \
  -H "Cookie: CF_Authorization=$CF_AUTHORIZATION" \
  -H "Content-Type: application/json" \
  -d '{"cap_usd": 10, "classes": ["retrieval_qa", "extraction"],
       "reason": "measure our own claims tasks before switching the router on"}'

import os

import httpx

r = httpx.post(
    f"https://{os.environ['AAL_TENANT']}.atlanticagenticlabs.com/api/v1/benchmark/runs",
    cookies={"CF_Authorization": os.environ["CF_AUTHORIZATION"]},
    json={"cap_usd": 10, "classes": ["retrieval_qa", "extraction"],
          "reason": "measure our own claims tasks before switching the router on"},
)
print(r.status_code, r.json())   # 202, with the command's id

const res = await fetch(`https://${process.env.AAL_TENANT}.atlanticagenticlabs.com/api/v1/benchmark/runs`, {
  method: "POST",
  headers: { Cookie: `CF_Authorization=${process.env.CF_AUTHORIZATION}`, "Content-Type": "application/json" },
  body: JSON.stringify({
    cap_usd: 10,
    classes: ["retrieval_qa", "extraction"],
    reason: "measure our own claims tasks before switching the router on",
  }),
});
console.log(res.status, await res.json()); // 202, with the command's id

Nothing runs in LightHouse, and no model is called from Atlantic’s side. The request reaches your account as a signed benchmark.start command, which the agent picks up on its next pass. A key cannot start a run (403 person_required). Add judge_model when the classes include summarising or drafting.

In your account, your engineers can run the same thing from the command line, through your gateway:

# What a run would cost. Calls no model.
python -m aal_engine.benchmark estimate --policy policy.yaml --cap 10

# The run itself: capped, graded, and written where the router reads it
python -m aal_engine.benchmark run --policy policy.yaml --cap 10 \
    --judge eu.anthropic.claude-opus-4-6 \
    --client-tasks var/benchmark/client-tasks.json \
    --out var/benchmark/report.json \
    --evidence-out var/benchmark/tenant-evidence.json

The league table and the plain sentence

GET /api/v1/benchmark (any viewer; format=csv for a spreadsheet), the MCP tool benchmark, or Agents → Routing in LightHouse. Per task class and model:

Measure	What it says
Cost per successful task	The headline: the run’s cost divided by the tasks that met the bar. A model that passed nothing has none, never a small number
Quality at the bar	The share that met it, and the failures by reason
Tokens per task	Input, output, cached and reasoning, separately
Value per unit cost, relative token value	For comparing models of different grades on the same work
Cache effect	Warm against cold on the same tasks; it can be negative, which is a finding
Latency at the bar	Time to first token and to finish, over the attempts that passed
Escalation cost	For a cheap model, its cost including the escalations its failures would buy

And one sentence per class that you can act on. From Atlantic’s own published run, on retrieval question-answering:

For retrieval question-answering on the standard suite, Claude Haiku 4.5 costs an estimated $0.00035 per successful answer against Claude Opus 4.6’s $0.0021, at the same quality bar.

Atlantic’s published figures

The engine ships with Atlantic’s own evidence (version 2.0.0), produced by this benchmark on suite 1.0.0 on 31 October 2026: through a gateway in Atlantic’s own AWS account in eu-west-1, against Amazon Bedrock’s EU cross-region profiles for Claude Haiku 4.5, Sonnet 4.5, Sonnet 4.6, Opus 4.5 and Opus 4.6, with Opus 4.6 as the named judge, priced on AWS’s published Bedrock rates. Two classes, from that run:

Task class (bar)	Model	Quality	Cost per successful task, estimated	p95 latency
Retrieval Q&A (0.90)	Haiku 4.5	1.00	$0.00035	667 ms
	Sonnet 4.6	1.00	$0.00112	1,012 ms
	Sonnet 4.5	1.00	$0.00130	2,211 ms
	Opus 4.5	1.00	$0.00196	2,899 ms
	Opus 4.6	1.00	$0.00211	1,991 ms
Long context (0.90)	Haiku 4.5	0.50	$0.0345	1,385 ms
	Sonnet 4.5	1.00	$0.0529	3,905 ms
	Opus 4.6	1.00	$0.0885	6,769 ms
	Sonnet 4.6	0.50	$0.106	1,396 ms
	Opus 4.5	0.50	$0.172	2,780 ms

The second class is why the bar matters: on long context, the cheapest model per successful task (Haiku 4.5) does not meet the bar, so the router would choose Sonnet 4.5.

Read these with their resolution. With two or three tasks per class, quality resolves only to halves or thirds, and one task moves a model across a bar. Over three runs on the same day, extraction, classification, retrieval question-answering and tool use were 1.00 for every model every time; coding, long context, summarising and drafting moved by one task between runs. Treat the four stable classes as measurements and the other four as a floor until the suite is larger. Cached and reasoning tokens were zero on that run, because the Bedrock backend did not then mark a cache point or set a thinking budget; the gateway now marks one.

Your own results replace Atlantic’s, for the same model and class, however much older Atlantic’s are.

What it feeds

The router: your results are written where the router reads them, so its next decision is taken on your numbers. See How a model earns its scores.
The routing lever: the router’s projected costs, and so the saving, are built from these measurements.

Coming

Running on a schedule, or automatically when something changes (a new model, a price change, a model version moving): Coming. Today a person starts every run.
A trend in Insights that shows a model getting slower or worse over time: Coming