Weave’s agent evals measure whether a change to a built-in agent’s prompt, a case or a rubric helped. Each case gives one agent (Loom, Tapestry, Shuttle, Spindle, Pattern, Weft or Warp) a realistic task and scores what it does. You run the same cases before and after a change, with repeats, and weave eval compare tells you whether any difference is bigger than the noise. Nothing tunes prompts automatically.

Published results from the CI workflow are on the Agent Evals dashboard. This page covers running evals yourself. The flags are documented in the CLI reference.

Evals run from the root of a checkout of the Weave repository, where the cases, rubrics and model matrix live. In a checkout, bun packages/cli/src/main.ts stands in for weave:

git clone https://github.com/weave-io/weave.git && cd weave
bun install
export OPENROUTER_API_KEY=<your-key>

Add --dry-run to any eval run to check the suite, case and model names for free. It makes no model calls and needs no key. Check your OpenRouter credits before a large run.

A plain weave eval run runs every model in the default matrix, which is the setting for a baseline, not for a quick check. While you work, run the development subset: the (at most two) models marked dev in evals/model-matrix.json, currently deepseek/deepseek-v4-flash-0731 and openai/gpt-6-luna.

weave eval run --models dev --repeat 3

# Narrow it to the suite you are changing
weave eval run --agent pattern --models dev --repeat 3

--repeat 3 runs each case three times per model, and the report shows a pass rate (PASS 3/3, FLAKY 2/3, FAIL 0/3) instead of one verdict. Models do not answer the same way every time, so a single run can flip a case either way. Repeats cost proportionally more.

The text cases on the dev subset with --repeat 3 cost about $0.18 (276 attempts, 24 Sep 2026). Add --track text to skip the trajectory cases, which need Podman and a sandbox image.

To find out why one case fails on one model, run just that case on just that model, with raw transcripts on:

weave eval run --agent weft-review --case weft-review-traced-true-positive \
  --model deepseek/deepseek-v4-flash-0731 --raw-artifacts

It costs one model call plus the judge calls the case makes, and nothing is published. The report shows the verdict, every scoring dimension that applies with its score (✗ marks one below its bar), a short Why: line and the path to the raw transcript:

  FAIL  weft-review-traced-true-positive on deepseek/deepseek-v4-flash-0731  (weft-review, required)
        Weighted total 0.00 (pass mark 0.50)
        ✗ executionCompleteness  0.00  below 0.95
        ✗ rationaleQuality       0.00  below 0.70
        Why: required execution case failed; dimensions: executionCompleteness, rationaleQuality
        Raw transcript: …/eval-bundles/runs/b42a14b-2026-09-23-001/raw/case-weft-review-traced-true-positive-….json

Open the raw file to see why: it holds the composed prompt, the model’s answer, the judge’s rationale and the runner’s diagnostics, such as the required signals it did not find. Raw files stay on your machine. Never commit them.

A case reported as ERROR instead of FAIL was not scored at all: the model’s answer was empty or truncated on every attempt, the request failed, or the judge could not answer. It says nothing about the prompt, so it is left out of pass rates, and the run exits 1.

Run the same evals on the commit before your change and on the commit with it, with the same filters and the same --repeat, then compare the two runs:

# 1. Baseline, on the commit before the change
git switch main
weave eval run --models dev --repeat 5
#    → Eval run 86eb974-2026-09-23-001: …

# 2. Candidate, on the commit with the change
git switch my-prompt-change
weave eval run --models dev --repeat 5
#    → Eval run 1a2b3c4-2026-09-23-001: …

# 3. Compare the two runs (run IDs or run directories)
weave eval compare 86eb974-2026-09-23-001 1a2b3c4-2026-09-23-001

eval compare reads only the two local bundles, so it makes no model calls and needs no key. For each suite and model it prints both pass rates with 95% intervals and a verdict:

• IMPROVED or REGRESSED: the difference is bigger than chance explains (Fisher’s exact test, Holm-adjusted across the rows, p < 0.05).
• no detectable change: the data cannot tell the runs apart. That is not the same as “equal”: with 25 attempts a side, a move from 60% to 80% usually goes undetected. Raise --repeat, or include more cases, to detect smaller changes.

It refuses runs that used different models, cases, repeat counts or judges, and dry runs. Run the baseline and the candidate close together, because a provider can change a model between them.

Six cases in loom-routing, tapestry-execution and shuttle-execution run a real OpenCode session in a Podman sandbox and score what the agent did: who it delegated to, which commands ran after its last edit, and whether a hidden verifier passes. Build the sandbox image once, and again after pulling changes:

podman build -t weave-sandbox-opencode-default -f sandboxes/opencode/Containerfile sandboxes/opencode
TMPDIR=~/.cache/weave-trajectory-tmp weave eval run --track trajectory --models dev

Each session can take 5 to 9 minutes. Trajectory cases list their own allowed models, so on the dev subset five of them run, all on DeepSeek V4 Flash.

CI runs the trajectory cases in their own job and publishes them as a separate run. tryweave.io/evals shows that run under Runtime scenarios and keeps the text run as its main results.

Every case is scored from 0 to 1 on routing, delegation, execution and rationale, leaving out the dimensions that do not apply. A case passes when one of its routing, delegation or execution scores reaches 0.95. Deterministic rules score what can be read from the answer, such as the chosen route. The rest is judged by TypeSafe Jev (typesafe/jev-1.13), a judging model rather than a chat model, so it is never one of the models under test and never grades its own answers. Each run records the judge’s ID and version, and changing the judge means starting a new baseline.

• CLI reference: weave eval run and weave eval compare
• Agent Evals dashboard
• Evals overview and the detailed Agent Evals reference in the Weave repository
• Eval baseline, 24 Sep 2026: per-suite pass rates, noise bands, and time and cost of each run