Which models actually hold up.
Transparent evidence for how models perform inside Weave's multi-agent orchestration. Every run is reproducible, every case is inspectable, and every metric comes from real execution — no synthetic benchmarks, no cherry-picked demos.
75 model-case evaluations covering 25 distinct test cases across 3 models and 8 capability suites. Each case was evaluated with each model.
Model × capability matrix
The central artifact: which models excel at which capabilities. Each cell is the pass rate for that model on that suite's cases.
| Model | Loom Routing | Tapestry Execution | Shuttle Execution | Spindle Tools | Pattern Planning | Weft Review | Warp Security | Tapestry Category Routing | Overall |
|---|---|---|---|---|---|---|---|---|---|
GPT-5.5 OpenAI | 100% | 100% | 100% | 50% | 50% | 100% | 50% | 0% | 48% |
Claude Sonnet 4.5 Anthropic | 100% | 100% | 50% | 50% | 50% | 50% | 100% | 0% | 44% |
Claude Opus 4.5 Anthropic | 67% | 100% | 50% | 50% | 50% | 50% | 100% | 0% | 40% |
Capability suites
Each suite tests one orchestration capability. Aggregate pass rates show overall reliability; model-level detail reveals strengths and weaknesses.
Loom Routing
Tests delegation accuracy from Loom to specialist agents
Tapestry Execution
Verifies plan execution fidelity in Tapestry
Shuttle Execution
Tests domain specialist task completion
Spindle Tools
Evaluates tool selection and usage accuracy
Pattern Planning
Assesses plan quality and decomposition
Weft Review
Tests code review quality and issue detection
Warp Security
Validates security vulnerability detection
Tapestry Category Routing
Tests category-based task routing within Tapestry
The allSuitesGreen flag is the strict suite gate emitted by the eval runner. One non-green suite makes the run false, regardless of overall pass rate.
Current status: false (33/75 evaluations passed — 44% overall pass rate).
Case-level evidence
Expand any case to see execution details. Pass means the case met all criteria. Partial is a quality annotation indicating incomplete success, and does not by itself determine the aggregate result. Fail means the case did not meet criteria.
This is an early production run, so historical trend data is sparse. Once multiple production runs accumulate, this section will show pass-rate trends over time, regression detection, and model performance evolution.
Methodology and context
All evaluations are produced by weave-io/weave and published in the weave-agent-evals repository. Each case is a real-world task scenario executed through Weave's orchestration layer.
Models are evaluated on identical cases to ensure fair comparison. Pass/fail is determined by automated verification of task completion, code correctness, and adherence to constraints.
Sanitized public reports are available in the data repository. View this run's report ↗