Brittleness
Brittleness is the drop in defense block-rate when adversaries paraphrase a probe instead of using literal wording. It is the single number that distinguishes a keyword filter (collapses on paraphrasing) from a control-level defense (does not).
Key points
- Reported in percentage points (pp), e.g. -50pp = block-rate falls from 67% to 17%.
- Always paired with an attribution: which defense family produced the drop.
- A flag, not a verdict — actionability depends on the category and your threat model.
- Drives the upsell conversation toward budget controls and structural defenses.
01
Brittleness classes
- keyword: defense depends on string-level matching; brittle by construction.
- control: defense is structural (rate limit, tool registry); wording-agnostic.
- unknown: classification not yet wired — surfaced to invite manual review.
02
How we measure it
- Run the same scenario family against originals (literal) and paraphrases.
- Compute block-rate for each set; brittleness = paraphrase rate − original rate.
- Baseline measurement: 10 replications, 85 paraphrases per scenario, n=17 probes.