A/B-style comparisons
Compare two runs side-by-side: same scenarios, different system prompts or models.
01
Set up
- Run the same scenario catalogue against two endpoints.
- Tag each run with a `comparison_group` label.
- Open the compare view at `/runs/compare?ids=...`.
02
Read the diff
- Per-category coverage delta is the headline metric.
- Brittleness delta matters more than aggregate coverage.
- Statistical significance ships once n≥30 per scenario.