Run comparison
Compare two runs side-by-side: aggregate scores, MITRE coverage heatmap delta, and per-finding lifecycle changes. The replay surface for change validation.
Key points
- Inputs are two run_ids; the response carries score deltas, technique-level coverage changes, and finding diffs.
- Findings classify as new (only in B), resolved (only in A), or regressed (still present + same severity).
- Brittleness deltas are surfaced separately so a literal-coverage improvement that loses paraphrase coverage is visible.
- Available from the UI at /runs/compare?ids=A,B and from the API at GET /v1/runs/{a}/compare/{b}.
01
Practical comparisons
- Pre-deploy vs. post-deploy of a prompt change — confirms a hardening landed and did not introduce new gaps.
- Last night vs. tonight — surfaces silent regression in upstream model behavior.
- Production vs. staging — quantifies the defense gap between environments.
02
What it does not do
- It is not a longitudinal trend view — compare two specific runs, not a window of runs.
- It does not merge or rerun — produce both inputs from real run executions before comparing.
- It compares within a tenant only; cross-tenant comparison is intentionally not exposed.