Evaluations

Every agent is evaluated before it is released and again while it runs. An evaluation scores what the agent retrieves and what it writes against a golden set: questions or tasks with the answers or documents a good agent must find.

Golden sets

A golden set belongs to the pack. Each entry names what a good answer must find or mention, and what it must never claim. The engine scores:

• retrieval: recall, precision, mean reciprocal rank and nDCG over the documents retrieved, and whether the right one came first;
• answers: grounded (no invented citation), complete (every required point present), weak support, citation rate, and forbidden claims.

Scoring is mechanical; no model judges another model’s answer. The engine’s own retrieval is measured the same way, on golden sets of its own, on every engine release.

The quality bar: the release gate

The release gate turns an evaluation into one decision a release can be held to:

1. Two floors that never move. Any invented citation or forbidden claim blocks, whatever else improved.
2. The worst check, never the average. Each metric is compared with the last release on the same golden set; one that falls beyond the tolerance (0.05 by default) blocks.
3. No answer evaluation, no release. Retrieval scores alone say nothing about what a model wrote.
4. Every model family your policy allows needs its own passing answer evaluation, so moving to another family is a measured change.
5. The decision is recorded, with every check, in the release ledger.

python -m aal_engine.agent.gate --current retrieval.json --current answers.json \
  --baseline last-release.json --policy gateway-policy.yaml --record decision.json

After release

The online monitor samples live outputs (more often for higher autonomy and more sensitive data) and checks each in code: citations to passages never retrieved, and the pack’s stop rules. A scheduled re-run of the golden set against the released version feeds the same monitor. A breach rolls the agent back to the last version that passed. See Versioning and rollback.

Running evaluations on your own data

Your golden set is built from your own documents and questions, and the evaluation runs in your deployment, against your own index. Nothing of it leaves your account except the scores. You run it by hand or on a schedule you set. Running it from LightHouse or a command line (lighthouse evals run) is Coming.

Results in LightHouse

• Reinforcement → Evaluations and releases shows each agent’s results and its releases.
• GET /api/v1/evals (filter by pack and family) and GET /api/v1/releases: see the Read reference.
• MCP: evals.

Each result is an eval record: pack, agent, version, model family, suite, metrics and whether it passed. Never a question or an answer.