How to test prompt injection without mistaking a prompt list for coverage
Prompt injection testing is a boundary test: can untrusted instructions change protected behavior, reveal data, or trigger actions?
Start from a prohibited outcome, create a clean control, vary the attack delivery path, and observe both model output and downstream effects. A test is inconclusive when the target never processed the input or when no expected defense was observable.
Who this is for: For AppSec engineers, AI application developers, red teams, and QA teams building pre-release checks.
A safe prompt-injection test plan
- 01
Choose one prohibited outcome
Examples include exposing a protected instruction, retrieving another user’s context, or invoking a tool outside the caller’s authorization.
- 02
Establish a benign control
Confirm the endpoint, response parser, identity, and tool path work normally before interpreting adversarial responses.
- 03
Exercise delivery classes
Cover direct input, retrieved content, multi-turn context, transformed wording, and tool output where those paths exist.
- 04
Measure effects, not keywords
Check protected data, actions, authorization, and detection signals. A response that omits a refusal phrase may still be safe.
- 05
Separate result states
Use pass, fail, blocked-by-control, inconclusive, and infrastructure-error states so outages never look like defenses.
- 06
Freeze and replay
Preserve a sanitized case identifier and rerun it after every relevant model, prompt, retrieval, or guardrail change.
Why variation belongs in the test
LLM behavior is sensitive to wording and conversational context. Testing only an obvious instruction override gives a brittle view of risk and encourages keyword filters that do not enforce authorization.
Variation should remain bounded and reproducible: change one delivery dimension at a time, retain the expected protected outcome, and record which control made the decision.
- Paraphrase and language variation
- Direct versus retrieved instructions
- Single-turn versus gradual context pressure
- Encoding or formatting variation
- Tool result and agent-memory injection
Worked example: separate a safe block from an unavailable target
A document assistant is allowed to summarize an approved folder and forbidden from following instructions embedded inside retrieved documents. Establish a benign control with an ordinary approved document. Then use a sanitized test document containing an untrusted instruction and observe whether protected data or a tool action is reached.
A policy block with an audit signal is a blocked result. A timeout before retrieval is an infrastructure error. This distinction matters because an outage must not be counted as a defense. A normal answer that ignores the embedded instruction is useful evidence for this recorded path, but it is not proof against every wording or delivery method.
What this playbook can and cannot conclude
- Validate recorded prompt-injection paths against an owned target
- Reveal regressions when the same cases are replayed
- Distinguish application controls from model refusals
- Authorize testing against third-party systems
- Publish operational bypass payloads or protected prompts
- Prove immunity to unseen prompt-injection techniques
- Treat an unavailable endpoint as a passed defense
Practice against an owned sandbox
Use a deliberately vulnerable target you control before touching a production application. DoesItDefend’s public demo uses owned sandbox infrastructure and exposes sanitized evidence rather than customer data.
Primary sources
- OWASP Top 10 for LLM Applications 2025
OWASP GenAI Security Project
Defines prompt injection as the leading LLM application risk in the 2025 list and describes direct and indirect forms.
- Ignore Previous Prompt: Attack Techniques For Language Models
Perez & Ribeiro, NeurIPS ML Safety Workshop
Early systematic research on goal hijacking and prompt leaking through adversarial instructions.
- MITRE ATLAS — Adversarial Threat Landscape for Artificial-Intelligence Systems
MITRE
Maps prompt injection, jailbreak, and related AI attack techniques.