GuidesAI reliability

Prompt injection: understanding the attack and defending against it

By the Banzai Labs teamUpdated on 2 min read

“Ignore your rules”Page or email readLeast privilegeConfirmationValidationText that is read is never an order
An instruction hidden in a document must not become an order: least privilege, confirmation and validation.

Two forms

Where the fake instruction comes from
Direct
  • The user types it in the conversation
  • “Forget your rules and…”
  • Visible in the logs
Indirect
  • Hidden in a page, an email, a PDF
  • The agent reads it while doing its job
  • Invisible to the user

Defenses

Defense Effect
Least privilege A support agent cannot issue refunds
Human confirmation Every irreversible action waits for a click
Outside content tagged The prompt states this text is data, not an instruction
Validated output Schema, allowlist of URLs and recipients
No secrets in the prompt Nothing to leak
Logging Trace what triggered an action

The golden rule

  1. 1ReadThe agent may read untrusted content…
  2. 2Act…or act with significant rights…
  3. 3Not both without a safeguardOtherwise a human confirmation steps in.

Background: AI agents on real data and tool use. Testing: test an AI chatbot.

Frequently asked questions

Is a good system prompt enough?
No. It reduces the risk without removing it. The real protection lies in the architecture: what the agent is allowed to do, and what requires confirmation.
Are recent models resistant?
More than before, thanks to their training. Injection still tops the OWASP Top 10 for LLM applications.
Which agents are most exposed?
Those that read outside content (web, emails, received documents) and can act (send, pay, delete).

Read next

All guides