How a model earns its scores

The router chooses on a measurement or it does not choose. This page is about where those measurements come from, how long they count for, and how to make them yours.

A score, and what it carries

One score is one model on one task class, measured by the Token Value Benchmark:

Field	What it is
quality	The share of the class’s tasks the model passed, graded by deterministic checks or a named judge
tasks	How many tasks the share is over
tokens	Mean input, output, cached and reasoning tokens per task, which the router uses to project each call’s cost
latency_ms_p95	The 95th-percentile latency over the attempts that passed
cost_per_success_usd	Cost per successful task, the benchmark’s headline
suite_version, measured_on	Which suite, and when
Source	aal (Atlantic’s published figures) or yours

Whose scores count

1. Yours, written by the benchmark in your account, where the router reads them (var/benchmark/tenant-evidence.json by default, or the path in your policy’s routing.evidence). For the same model and class, yours always win, however much older Atlantic’s are. When yours go stale, it is your figure that is reported as stale, rather than the router quietly falling back to Atlantic’s.
2. Atlantic’s published figures, shipped with the engine and produced by the same benchmark on the same suite, so you can reproduce them. Version 2.0.0 was measured on 31 October 2026 on Amazon Bedrock’s EU profiles; the benchmark page has the method and a sample of the results. Set use_published: false in your policy to route on your own figures alone.

Model IDs in the evidence are families, so eu.anthropic.claude-haiku-4-5 and the same family’s other regional profiles read the same score.

How long a score counts

A score is fresh for 90 days by default (max_evidence_age_days), and only if it was measured on a suite version at or above your floor (min_suite_version). A change of suite version is a change of baseline: scores measured on an older suite go stale, rather than being compared with newer ones.

A stale or missing score drops the candidate (stale_evidence, no_evidence). If every candidate drops, the call keeps its agent’s model. A router that guessed at quality would be choosing on price alone.

How often scores are re-measured

When you run the benchmark. A client_admin starts a run in Agents → Routing, with a cap, or your engineers run it from the command line in your account; see Starting a run.

• Re-measuring automatically on a schedule, or when something changes (a new model, a price change, a model version moving): Coming
• A trend that shows a model getting slower or worse over time: Coming

Until then, the freshness window is the safeguard: a score you have not renewed in 90 days stops routing calls.

Running the evaluations on your own data

1. Write your own tasks into the client tasks file in your account (var/benchmark/client-tasks.json), in the suite’s shape: each task’s class, its instruction and material, its size, and the check that grades it. The benchmark page has an example. Your material never leaves your account.
2. Start a run with a cap. It estimates the whole run first and refuses to start if the estimate exceeds the cap.
3. Confirm the sample of grades it surfaces for a person, for the rubric-graded classes.
4. Read the league table in Agents → Routing. Your results are now the router’s evidence for those models and classes.

Your tasks are reported beside the standard suite’s, never averaged into it, so you can see where your work and the standard suite disagree.

What the figures cannot yet carry

The standard suite has two or three tasks per class, so a quality figure resolves only to halves or thirds, and one task moves a model across a bar. On Atlantic’s published run, four classes (extraction, classification, retrieval question-answering, tool use) were stable across three runs; four (coding, long context, summarising, drafting) moved by one task. Read the stable classes as measurements and the others as a floor. Your own tasks, in larger numbers, are the way past this.

Coming: replay evaluation

The benchmark measures a model on a suite: the kind of work, not your agent’s own days. Replay evaluation would take an agent’s recorded work, re-run it on a candidate model with the tool results replayed as they were recorded, and score it with the agent’s own checks, its golden set and a calibrated blind judge. It would answer a different question: whether this agent can move to a cheaper model without getting worse. Coming

Until it exists, moving an agent to a different model tier still needs a written reason at release.