LLM jailbreaking — techniques & defenses
AI Security · AI sec engineer · AppSec · Red team (AI)
How prompts bypass guardrails; how to defend RAG and agent apps.
Concept map
User/Doc → Model → Tools → Data Attack: direct/indirect injection → Defense: trust boundaries + policy
1. Common jailbreak patterns
Role-play (DAN-style), encoding (Base64/ROT), multi-turn gradual escalation, indirect injection via retrieved documents, tool abuse on agents.
2. Real-world impact
Data exfil from chatbots connected to internal docs; agents calling tools they should not (email send, shell).
- Threat model includes untrusted user + untrusted RAG content
3. Defenses
System prompt hardening (not sole fix), input/output filtering, tool allow-lists, human approval for sensitive actions, separate privilege for RAG vs user message, logging and alerting on policy violations.
4. Testing
Use Garak/Promptfoo on apps you own; red-team multi-turn scenarios; regression in CI.
Case studies
Samsung ChatGPT incident (2023)
Employees pasted sensitive code into public LLM — policy + DLP + enterprise LLM gateways as defense.
Indirect prompt injection
Malicious content in webpages/emails retrieved by RAG can instruct model to exfil — treat retrieval as untrusted input (OWASP LLM Top 10).
Commands
- See linked tools and /commands catalog.