Attack simulations
A simulation is one controlled run of probes against your LLM application — scoped to an allowed target, executed by our engine, and graded against OWASP LLM Top 10 and MITRE ATT&CK.
An attack simulation is a bounded exercise: the engine runs a curated probe set against one target you own, inside a scope you approved, and records everything needed to replay the result. This page explains what a single run yields and when to schedule one.
Simulations are the unit of evidence in DoesItDefend. Scores, coverage states, comparisons, and evidence packs all derive from completed runs, so understanding what a run promises — and what it deliberately does not — makes every other concept easier to read.
Key points
- Every simulation has an explicit target_scope; out-of-scope targets are rejected.
- Run creation is idempotent: reusing the same Idempotency-Key returns the existing run, while model responses may still vary across new executions.
- A completed run persists findings and evidence artifacts. Its numeric score is decision-grade only when canonical persisted evidence supports a conclusive assessment; otherwise public and comparison scores remain N/A.
- Simulation profiles range from quick (≈5 minutes) to deep (≈30 minutes).
What a simulation produces
- Per-category findings tagged with OWASP LLM IDs and ATT&CK techniques.
- Per-category states (open / remediated / tested / untested) derived from conclusive persisted evidence.
- Eligible Starter+ runs can produce a checksum-verified evidence pack. LLM runs with eligible findings can also include request/response transcripts and a remediation playbook.
Every artifact a run generates is written to persisted storage before any score is computed. That ordering is deliberate: a finding you cannot re-open with its inputs attached is an anecdote, not evidence. Treat the run record, not the dashboard, as the source of truth.
When to run one
- Before each release, against a staging endpoint.
- Weekly against production (smoke profile).
- After any change to system prompts, tools, or model versions.
Frequency follows risk, not ritual. A release that touches system prompts, tool definitions, or retrieval changes the attack surface and deserves a full run; unchanged surfaces are better served by a short smoke profile than by rerunning everything on a fixed schedule.
Related resources