LLM security testing: what to test and what the evidence proves
LLM security testing asks whether an AI application keeps its intended boundaries when inputs, context, tools, and model behavior become adversarial.
A useful LLM security test is a controlled, repeatable attempt to violate a defined security boundary. It records the input, expected defense, observed behavior, and evidence needed to reproduce the result. A prompt list alone is not a security program.
Who this is for: For AppSec, AI engineering, security engineering, and product teams preparing an LLM application for release.
Test the application, not only the model
An LLM application combines model instructions, retrieval, tools, identity, business logic, filters, and observability. A safe refusal from the base model does not prove that the assembled system protects data or limits actions.
Start by naming assets and trust boundaries: system instructions, retrieved documents, credentials, tool permissions, user data, output consumers, and cost controls. Then connect each test to a plausible abuse path.
- Direct and indirect prompt injection
- Sensitive information disclosure
- Unauthorized tool use and excessive agency
- Cross-user or cross-tenant context leakage
- Unbounded consumption and operational abuse
A repeatable security-testing loop
Keep the workflow deterministic around the model so a result can support an engineering decision.
- 01
Define the boundary
State what the system must never reveal, execute, retrieve, or spend without authorization.
- 02
Map representative attacks
Use public taxonomies such as OWASP and MITRE ATLAS, then adapt cases to the actual architecture.
- 03
Vary the behavior
Test paraphrases, encodings, multi-turn pressure, retrieved instructions, and tool-mediated paths instead of one magic phrase.
- 04
Observe the defense
Record which control blocked the action and distinguish a safe block from a transport failure or inconclusive response.
- 05
Replay after remediation
Freeze the case, rerun it, and preserve evidence that the original path is closed without breaking legitimate behavior.
Worked example: test one permission boundary
Consider a calendar assistant that may read the signed-in user’s events but must never retrieve a coworker’s private calendar. First confirm that the user can read one of their own events. Then run a sanitized cross-user request and record the application response, tool call, authorization decision, and audit signal.
If the tool is never called because authorization blocks the request, the recorded path is blocked by a control. If the calendar service is unavailable, the result is inconclusive—not secure. This small example shows why the verdict must describe both the observed effect and the control that produced it.
What a test can and cannot conclude
- Show that a specific attack path succeeded or was blocked under recorded conditions
- Compare guardrail behavior across controlled variants
- Produce reproducible evidence for remediation and release gates
- Reveal missing controls across model, application, and tool layers
- Prove the application is secure against every future attack
- Replace threat modeling, code review, or authorization testing
- Turn one model refusal into proof of systemic protection
- Estimate real-world prevalence without operational data
Minimum evidence before calling a result actionable
- 01Named asset and security boundary
- 02Attack taxonomy and scenario identifier
- 03Sanitized input fingerprint and expected result
- 04Observed outcome separated from infrastructure errors
- 05Defense control or detection signal that fired
- 06Reproduction steps and post-fix replay status
Need this applied to your LLM application?
Scope an authorized assessment with human-reviewed remediation, one replay-validated retest, and a checksum-verified Evidence Pack.
Request a scoped assessmentPrimary sources
- OWASP Top 10 for LLM Applications 2025
OWASP GenAI Security Project
Public risk taxonomy for LLM and GenAI applications.
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1)
National Institute of Standards and Technology
Risk-management guidance for governing, mapping, measuring, and managing GenAI risks.
- MITRE ATLAS — Adversarial Threat Landscape for Artificial-Intelligence Systems
MITRE
Living knowledge base of adversary tactics and techniques against AI-enabled systems.