---
kind: concept
slug: prompt-injection
title: Prompt injection
summary: >-
  Text planted in an LLM application's input can override the developer's instructions. A survey names attack families and defences without numbers. Two system cards report author-run results on instruction-priority and injection tests, on their own test sets. A review of agent systems names the threat, with no experiments.
tags: [security, llm, robustness]
papers: ["2402.00888", "2412.16720", "2506.04133", "2601.03267"]
related: [adversarial-robustness, data-poisoning, prompt-engineering, llm-agent, multi-agent-systems, tool-use, red-teaming, safety-filter]
status: draft
updated_at: 2026-10-31
---

## Overview

An application built on a language model usually combines a fixed instruction
from the developer with data from a user or a web page. Prompt injection hides
an instruction inside that data, and the model may follow it instead of the
developer's instruction [arxiv:2402.00888 §4.1.1] [arxiv:2402.00888 §6.1]. The
problem is the mixing of trusted and untrusted text in one prompt. Most defences
try either to clean or separate the untrusted text, or to detect that it looks
wrong. Model-side approaches train the model to prefer the developer's
instruction when messages conflict [arxiv:2412.16720 §4.2] [arxiv:2601.03267 §3.5],
and to ignore instructions found in web or connector content
[arxiv:2601.03267 §3.6].

Four cards list this concept. The first is a survey with no experiments of its
own [arxiv:2402.00888 §1] [arxiv:2402.00888 §9], so its numbers are figures it
quotes from other work, or "not stated". The second is a system card for
OpenAI o1 that reports author-measured results on instruction-conflict tests
[arxiv:2412.16720 §4.2]. Its card does not present those tests as
prompt-injection attacks. The third is the system card for the GPT-5 family.
It reports author-run instruction-hierarchy and prompt-injection evaluations
against o3 and GPT-4o [arxiv:2601.03267 §3.5] [arxiv:2601.03267 §3.6]. The
fourth is a review of multi-agent LLM systems. It treats prompt injection as a
threat that can spread from one agent to the others, and lists controls for it,
with no experiments of its own [arxiv:2506.04133 §3] [arxiv:2506.04133 §4].

## Variants

### Attacks

The survey lists several attack methods [arxiv:2402.00888 §4.1.1]. Each one
is shown below with the setting the survey gives for it.

- **Black-box injection (HOUYI).** Modelled on traditional web injection. It
  has three parts: a prompt built to fit the application, a context-partition
  step that prepares the model to accept the injection, and the malicious
  payload [arxiv:2402.00888 §4.1.1]. Card: [Security and Privacy Challenges of Large Language Models: A Survey](/papers/2402.00888).
- **Universal gradient-based injection.** Inspired by Greedy Coordinate
  Gradient. The survey says it needs few training instances and works even
  against basic safety training, but may fall short against perplexity-based
  defences [arxiv:2402.00888 §4.1.1]. Card: [Security and Privacy Challenges of Large Language Models: A Survey](/papers/2402.00888).
- **Injection into LLM judges (JudgeDeceiver).** An optimisation-based attack
  on a model that scores other model outputs [arxiv:2402.00888 §4.1.1]. Card:
  [Security and Privacy Challenges of Large Language Models: A Survey](/papers/2402.00888).

The survey also notes that several of these attacks were shown mainly on
application pipelines or smaller models, not on large LLMs themselves
[arxiv:2402.00888 §4.1.1].

### Agent systems

The review on agentic multi-agent systems lists prompt injection among the
threats to such systems [arxiv:2506.04133 §1]. It says these systems are open
to prompt injection and that its effect can cascade across agents
[arxiv:2506.04133 §3]. For cascading attacks it cites role-swapping on the
ChatDev framework, reported in prior work [arxiv:2506.04133 §3]. The review
adds that malicious content in one modality, such as text inside an image, can
change behaviour in another [arxiv:2506.04133 §4]. It reports none of these
attacks as its own experiments. Card: [TRiSM for Agentic AI](/papers/2506.04133).

The GPT-5 card names prompt injection as a risk because its models can browse
the web and use connectors, so instructions can be hidden in content they read
[arxiv:2601.03267 §3.6]. Its external red-team assessment of connectors ran for
two weeks, with two outside groups. It reports 47 findings, 10 of them notable,
and says the mitigations were deployed before release [arxiv:2601.03267 §4.2].
The card does not describe the attack methods in that assessment.
Card: [OpenAI GPT-5 System Card](/papers/2601.03267).

### Defences

The survey describes two groups of external defences. It says only limited
studies have looked at defences against prompt injection [arxiv:2402.00888 §6.1].

- **Prevention.** Pre-process the untrusted data to remove the injected
  instruction, and/or redesign the developer's instruction prompt. The survey
  names paraphrasing, re-tokenization, data-prompt isolation, instructional
  prevention and sandwich prompting [arxiv:2402.00888 §6.1]. It describes
  paraphrasing and re-tokenization in some detail, and names the others only.
  Card: [Security and Privacy Challenges of Large Language Models: A Survey](/papers/2402.00888).
- **Detection.** Check whether the data prompt is intact. The survey splits
  detection into response-based (examines the model's answer) and
  prompt-based (examines the data prompt). Perplexity-based detection is a
  prompt-based method: added instructions tend to raise the perplexity of the
  data [arxiv:2402.00888 §6.1]. Card: [Security and Privacy Challenges of Large Language Models: A Survey](/papers/2402.00888).
- **Agent controls.** The review's application-security table lists prompt
  hygiene, plan-then-execute, an action selector, tool isolation, output
  filters, cross-agent checks and human-in-the-loop holds, with least-privilege
  access control [arxiv:2506.04133 §4]. It also says to validate tool outputs
  before they run [arxiv:2506.04133 §6]. The review names these controls and
  gives no effect size for any of them. Card: [TRiSM for Agentic AI](/papers/2506.04133).
- **Connector session rule.** The GPT-5 card gives a rule for connector use:
  after a connector call, browsing in that session reads only cached pages,
  because sensitive connector content may already be in context. The card
  says this may change as safety training improves [arxiv:2601.03267 §3.6].
  It is the card's own rule, and no effect size is given for it. Card:
  [OpenAI GPT-5 System Card](/papers/2601.03267).

### Model-side instruction hierarchy

The o1 system card describes training the model to rank three message types:
system over developer, and developer over user. Training used examples of
conflicting messages and supervised the model to follow the higher-priority one
[arxiv:2412.16720 §4.2]. The card gives no training algorithm or data size
[arxiv:2412.16720 §2], so this variant cannot be rebuilt from the card. Card:
[OpenAI o1 System Card](/papers/2412.16720).

The card's results are author-reported on pre-release checkpoints, and the
production model may differ [arxiv:2412.16720 §3]. The table gives GPT-4o and
o1 scores as the card reports them. The card does not name the metric in this
table [arxiv:2412.16720 §4.2].

| Evaluation | GPT-4o | o1 |
|---|---|---|
| Developer vs user message conflict | 0.68 | 0.78 |
| System vs developer message conflict | 0.74 | 0.80 |
| System vs user message conflict | 0.68 | 0.78 |
| Tutor jailbreak, system message | 0.33 | 0.95 |
| Tutor jailbreak, developer message | 0.58 | 0.92 |
| Phrase protection, user message | 0.74 | 0.91 |
| Phrase protection, developer message | 0.82 | 0.70 |
| Password protection, user message | 0.85 | 1 |
| Password protection, developer message | 0.69 | 0.96 |

The developer-message phrase-protection row is lower for o1 than for GPT-4o,
while the other rows are higher [arxiv:2412.16720 §4.2].

The GPT-5 card uses the same three-level order: system over developer, and
developer over user [arxiv:2601.03267 §3.5]. It also trains the models to
ignore instructions in web or connector content [arxiv:2601.03267 §3.6]. The
card gives no training recipe for these, and the safe-completion method it
mentions is described in a separate paper [arxiv:2601.03267 §3.1]. Card:
[OpenAI GPT-5 System Card](/papers/2601.03267).

The card's instruction-hierarchy table is author-run and compares
gpt-5-thinking and gpt-5-main with OpenAI o3 and GPT-4o. The authors note
regressions for gpt-5-main and say they will follow up with a fix
[arxiv:2601.03267 §3.5]. Higher is better in the table below, as in the card.

| Evaluation | gpt-5-thinking | OpenAI o3 | gpt-5-main | GPT-4o |
|---|---|---|---|---|
| System prompt extraction, realistic user attacks on system message | 0.990 | 0.997 | 0.885 | 0.885 |
| System prompt extraction, academic user attacks on system message | 0.991 | 0.982 | 0.930 | 0.825 |
| System prompt extraction, academic developer attacks on system message | 0.991 | 0.982 | 0.789 | 0.561 |
| Phrase protection, malicious user message | 0.940 | 0.975 | 0.619 | 0.735 |
| Phrase protection, malicious developer message | 0.911 | 0.921 | 0.404 | 0.449 |

The card also reports prompt-injection evaluations, where only gpt-5-thinking
and o3 are compared [arxiv:2601.03267 §3.6]. Higher is better.

| Evaluation | gpt-5-thinking | OpenAI o3 |
|---|---|---|
| Browsing prompt injections | 0.99 | 0.89 |
| Tool-calling prompt injections | 0.99 | 0.80 |
| Coding prompt injections | 0.97 | 0.94 |

## When to use which

Neither the survey nor the review gives a comparison of its defences or any
numbers for them, so the first four rows only map constraints to the mechanism
each source describes. They do not rank the options. The fifth and sixth rows
rest on the o1 and GPT-5 cards alone, and the seventh on the benchmark table of
the review.

| Situation | Use | Why |
|---|---|---|
| You control the developer's instruction prompt | Redesign the instruction prompt; instructional prevention; sandwich prompting | These are named as prevention that works on the instruction side [arxiv:2402.00888 §6.1]. No effect size is given. |
| You can rewrite the untrusted data before the model sees it | Paraphrasing or re-tokenization | The survey says paraphrasing breaks injected instructions and special characters, and re-tokenization breaks injected instructions into more tokens [arxiv:2402.00888 §6.1]. No effect size is given. |
| You can afford an extra check on the data or the answer | Perplexity-based or other prompt-based detection; response-based detection | Detection flags suspicious input or output rather than rewriting it [arxiv:2402.00888 §6.1]. The gradient-based attack may evade perplexity checks [arxiv:2402.00888 §4.1.1]. |
| Your agent has tools, shared memory or other agents | Least-privilege access per agent; tool isolation; allowlisted actions; plan-then-execute; validating tool outputs | The review names these as controls for agent applications and says access control matters most when agents share memory or permissions [arxiv:2506.04133 §4] [arxiv:2506.04133 §6]. No effect size is given. |
| Your agent browses the web or calls connectors | Cached-only browsing after a connector call; a model trained to ignore instructions in web or connector content | The GPT-5 card describes both and says the cached-only rule may change [arxiv:2601.03267 §3.6]. Its injection results are against o3 only, and no figure is given for the rule alone [arxiv:2601.03267 §3.6]. |
| You train or fine-tune the model yourself | Model-side instruction hierarchy | The o1 card reports gains on its own conflict tests after this training [arxiv:2412.16720 §4.2]. The GPT-5 card reports mixed results against o3 on its own tests [arxiv:2601.03267 §3.5]. Neither gives a training recipe [arxiv:2412.16720 §2] [arxiv:2601.03267 §3.5], and neither tests attacks outside its own tests. |
| You run a large commercial or open LLM | Test each defence on your model before relying on it | Most attacks and defences the survey covers were tested on smaller models [arxiv:2402.00888 §8]. The o1 and GPT-5 results are for OpenAI models and their own tests [arxiv:2412.16720 §4.2] [arxiv:2601.03267 §3.5]. |
| You need a benchmark that covers injection | HarmBench, reporting attack success rate and robustness | The review lists HarmBench as covering prompt injection and jailbreaks, with ASR and robustness metrics. It notes that HarmBench is limited for multi-agent systems [arxiv:2506.04133 §6 tab. 8]. |

## Implementation sketch

A prevention pipeline with a perplexity check, as the survey describes its
parts [arxiv:2402.00888 §6.1]. The survey does not give the order of the
steps, a threshold, or the model used for scoring; those choices are open
and are marked below.

```pseudocode
# Guarded LLM call: rewrite, check, then separate  [arxiv:2402.00888 §6.1]
# Input:  instruction (string, trusted), data (string, untrusted)
# Output: answer (string), or None when the data is flagged

def guarded_answer(instruction, data, llm, paraphrase, perplexity_flags):
    # 1. Rewrite the untrusted data to break injected instructions  [§6.1]
    cleaned = paraphrase(data)
    # 2. Flag the rewritten data if it looks unlikely  [§6.1]
    #    the scoring model and the rule are not stated in the survey
    if perplexity_flags(cleaned):
        return None
    # 3. Keep the trusted instruction apart from the untrusted data  [§6.1]
    #    the layout of the two parts is a free choice
    prompt = instruction + "\n\n" + cleaned
    return llm(prompt)
```

## Contradictions

None known between the four cards. The survey and the review report no
numbers for the prompt-injection defences, so there is nothing to disagree
with. The survey says the universal gradient-based attack may fall short against
perplexity-based defences [arxiv:2402.00888 §4.1.1], and it says proactive
detection has been effective in a work it cites [arxiv:2402.00888 §6.1]. Those
statements are about different attacks and defences, not a conflict over one
result.

The o1 and GPT-5 cards both report a phrase-protection row for developer
messages. Their row names are similar, but the cards test different models and
do not show the same setup, so their numbers are not compared. Both report a
lower score on that row than a reference model: o1 against GPT-4o, 0.70 against
0.82 [arxiv:2412.16720 §4.2], and gpt-5-main against GPT-4o, 0.404 against 0.449
[arxiv:2601.03267 §3.5]. The GPT-5 card notes regressions for gpt-5-main and
says a fix will follow [arxiv:2601.03267 §3.5]. The o1 card's instruction-conflict tests
are not the attacks the survey or the review describes, so its scores are not
compared with theirs.

## Papers

- [Security and Privacy Challenges of Large Language Models: A Survey](/papers/2402.00888): a taxonomy of prompt injection and other LLM attacks, with the defences named for each, and no experiments of its own.
- [OpenAI o1 System Card](/papers/2412.16720): a safety report for o1 and o1-mini, with instruction-hierarchy evaluations and author-reported scores; not a method paper.
- [OpenAI GPT-5 System Card](/papers/2601.03267): a safety report for the GPT-5 family, with author-run instruction-hierarchy and prompt-injection evaluations against o3 and GPT-4o; not a method paper.
- [TRiSM for Agentic AI](/papers/2506.04133): a review of trust, risk and security for multi-agent LLM systems, with a threat taxonomy, control tables and no experiments.