# How a model earns its scores

> How scores are measured, and how to run the evaluations on your own data.

The router chooses on a measurement or it does not choose. This page is about where those measurements come from, how long they count for, and how to make them yours.

## A score, and what it carries

One score is one model on one task class, measured by [the Token Value Benchmark](/cost/token-value-benchmark/):

| Field                          | What it is                                                                                                  |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------- |
| `quality`                      | The share of the class’s tasks the model passed, graded by deterministic checks or a named judge            |
| `tasks`                        | How many tasks the share is over                                                                            |
| `tokens`                       | Mean input, output, cached and reasoning tokens per task, which the router uses to project each call’s cost |
| `latency_ms_p95`               | The 95th-percentile latency over the attempts that passed                                                   |
| `cost_per_success_usd`         | Cost per successful task, the benchmark’s headline                                                          |
| `suite_version`, `measured_on` | Which suite, and when                                                                                       |
| Source                         | `aal` (Atlantic’s published figures) or yours                                                               |

## Whose scores count

1. **Yours**, written by the benchmark in your account, where the router reads them (`var/benchmark/tenant-evidence.json` by default, or the path in your policy’s `routing.evidence`). For the same model and class, yours always win, however much older Atlantic’s are. When yours go stale, it is your figure that is reported as stale, rather than the router quietly falling back to Atlantic’s.
2. **Atlantic’s published figures**, shipped with the engine and produced by the same benchmark on the same suite, so you can reproduce them. Version 2.0.0 was measured on 31 October 2026 on Amazon Bedrock’s EU profiles; [the benchmark page](/cost/token-value-benchmark/#atlantics-published-figures) has the method and a sample of the results. Set `use_published: false` in your policy to route on your own figures alone.

Model IDs in the evidence are families, so `eu.anthropic.claude-haiku-4-5` and the same family’s other regional profiles read the same score.

## How long a score counts

A score is **fresh** for 90 days by default (`max_evidence_age_days`), and only if it was measured on a suite version at or above your floor (`min_suite_version`). A change of suite version is a change of baseline: scores measured on an older suite go stale, rather than being compared with newer ones.

A stale or missing score drops the candidate (`stale_evidence`, `no_evidence`). If every candidate drops, the call keeps its agent’s model. A router that guessed at quality would be choosing on price alone.

## How often scores are re-measured

When you run the benchmark. A `client_admin` starts a run in **Agents → Routing**, with a cap, or your engineers run it from the command line in your account; see [Starting a run](/cost/token-value-benchmark/#starting-a-run).

- Re-measuring automatically on a schedule, or when something changes (a new model, a price change, a model version moving): \[Coming]
- A trend that shows a model getting slower or worse over time: \[Coming]

Until then, the freshness window is the safeguard: a score you have not renewed in 90 days stops routing calls.

## Running the evaluations on your own data

1. Write your own tasks into the client tasks file in your account (`var/benchmark/client-tasks.json`), in the suite’s shape: each task’s class, its instruction and material, its size, and the check that grades it. The [benchmark page](/cost/token-value-benchmark/#the-suite) has an example. Your material never leaves your account.
2. Start a run with a cap. It estimates the whole run first and refuses to start if the estimate exceeds the cap.
3. Confirm the sample of grades it surfaces for a person, for the rubric-graded classes.
4. Read the league table in **Agents → Routing**. Your results are now the router’s evidence for those models and classes.

Your tasks are reported beside the standard suite’s, never averaged into it, so you can see where your work and the standard suite disagree.

## What the figures cannot yet carry

The standard suite has two or three tasks per class, so a quality figure resolves only to halves or thirds, and one task moves a model across a bar. On Atlantic’s published run, four classes (extraction, classification, retrieval question-answering, tool use) were stable across three runs; four (coding, long context, summarising, drafting) moved by one task. Read the stable classes as measurements and the others as a floor. Your own tasks, in larger numbers, are the way past this.

## Coming: replay evaluation

The benchmark measures a model on a **suite**: the kind of work, not your agent’s own days. Replay evaluation would take an agent’s recorded work, re-run it on a candidate model with the tool results replayed as they were recorded, and score it with the agent’s own checks, its golden set and a calibrated blind judge. It would answer a different question: whether **this agent** can move to a cheaper model without getting worse. \[Coming]

Until it exists, moving an agent to a different model tier still needs a written reason at release.

Source: https://docs.atlanticagenticlabs.com/router/evidence/