Prompt injection

Overview

An application built on a language model usually combines a fixed instruction from the developer with data from a user or a web page. Prompt injection hides an instruction inside that data, and the model may follow it instead of the developer's instruction 2402.00888 §4.1.1 2402.00888 §6.1. The problem is the mixing of trusted and untrusted text in one prompt. Most defences try either to clean or separate the untrusted text, or to detect that it looks wrong. Model-side approaches train the model to prefer the developer's instruction when messages conflict 2412.16720 §4.2 2601.03267 §3.5, and to ignore instructions found in web or connector content 2601.03267 §3.6.

Four cards list this concept. The first is a survey with no experiments of its own 2402.00888 §1 2402.00888 §9, so its numbers are figures it quotes from other work, or "not stated". The second is a system card for OpenAI o1 that reports author-measured results on instruction-conflict tests 2412.16720 §4.2. Its card does not present those tests as prompt-injection attacks. The third is the system card for the GPT-5 family. It reports author-run instruction-hierarchy and prompt-injection evaluations against o3 and GPT-4o 2601.03267 §3.5 2601.03267 §3.6. The fourth is a review of multi-agent LLM systems. It treats prompt injection as a threat that can spread from one agent to the others, and lists controls for it, with no experiments of its own 2506.04133 §3 2506.04133 §4.

Variants

Attacks

The survey lists several attack methods 2402.00888 §4.1.1. Each one is shown below with the setting the survey gives for it.

• Black-box injection (HOUYI). Modelled on traditional web injection. It has three parts: a prompt built to fit the application, a context-partition step that prepares the model to accept the injection, and the malicious payload 2402.00888 §4.1.1. Card: Security and Privacy Challenges of Large Language Models: A Survey.
• Universal gradient-based injection. Inspired by Greedy Coordinate Gradient. The survey says it needs few training instances and works even against basic safety training, but may fall short against perplexity-based defences 2402.00888 §4.1.1. Card: Security and Privacy Challenges of Large Language Models: A Survey.
• Injection into LLM judges (JudgeDeceiver). An optimisation-based attack on a model that scores other model outputs 2402.00888 §4.1.1. Card: Security and Privacy Challenges of Large Language Models: A Survey.

The survey also notes that several of these attacks were shown mainly on application pipelines or smaller models, not on large LLMs themselves 2402.00888 §4.1.1.

Agent systems

The review on agentic multi-agent systems lists prompt injection among the threats to such systems 2506.04133 §1. It says these systems are open to prompt injection and that its effect can cascade across agents 2506.04133 §3. For cascading attacks it cites role-swapping on the ChatDev framework, reported in prior work 2506.04133 §3. The review adds that malicious content in one modality, such as text inside an image, can change behaviour in another 2506.04133 §4. It reports none of these attacks as its own experiments. Card: TRiSM for Agentic AI.

The GPT-5 card names prompt injection as a risk because its models can browse the web and use connectors, so instructions can be hidden in content they read 2601.03267 §3.6. Its external red-team assessment of connectors ran for two weeks, with two outside groups. It reports 47 findings, 10 of them notable, and says the mitigations were deployed before release 2601.03267 §4.2. The card does not describe the attack methods in that assessment. Card: OpenAI GPT-5 System Card.

Defences

The survey describes two groups of external defences. It says only limited studies have looked at defences against prompt injection 2402.00888 §6.1.

• Prevention. Pre-process the untrusted data to remove the injected instruction, and/or redesign the developer's instruction prompt. The survey names paraphrasing, re-tokenization, data-prompt isolation, instructional prevention and sandwich prompting 2402.00888 §6.1. It describes paraphrasing and re-tokenization in some detail, and names the others only. Card: Security and Privacy Challenges of Large Language Models: A Survey.
• Detection. Check whether the data prompt is intact. The survey splits detection into response-based (examines the model's answer) and prompt-based (examines the data prompt). Perplexity-based detection is a prompt-based method: added instructions tend to raise the perplexity of the data 2402.00888 §6.1. Card: Security and Privacy Challenges of Large Language Models: A Survey.
• Agent controls. The review's application-security table lists prompt hygiene, plan-then-execute, an action selector, tool isolation, output filters, cross-agent checks and human-in-the-loop holds, with least-privilege access control 2506.04133 §4. It also says to validate tool outputs before they run 2506.04133 §6. The review names these controls and gives no effect size for any of them. Card: TRiSM for Agentic AI.
• Connector session rule. The GPT-5 card gives a rule for connector use: after a connector call, browsing in that session reads only cached pages, because sensitive connector content may already be in context. The card says this may change as safety training improves 2601.03267 §3.6. It is the card's own rule, and no effect size is given for it. Card: OpenAI GPT-5 System Card.

Model-side instruction hierarchy

The o1 system card describes training the model to rank three message types: system over developer, and developer over user. Training used examples of conflicting messages and supervised the model to follow the higher-priority one 2412.16720 §4.2. The card gives no training algorithm or data size 2412.16720 §2, so this variant cannot be rebuilt from the card. Card: OpenAI o1 System Card.

The card's results are author-reported on pre-release checkpoints, and the production model may differ 2412.16720 §3. The table gives GPT-4o and o1 scores as the card reports them. The card does not name the metric in this table 2412.16720 §4.2.

The developer-message phrase-protection row is lower for o1 than for GPT-4o, while the other rows are higher 2412.16720 §4.2.

The GPT-5 card uses the same three-level order: system over developer, and developer over user 2601.03267 §3.5. It also trains the models to ignore instructions in web or connector content 2601.03267 §3.6. The card gives no training recipe for these, and the safe-completion method it mentions is described in a separate paper 2601.03267 §3.1. Card: OpenAI GPT-5 System Card.

The card's instruction-hierarchy table is author-run and compares gpt-5-thinking and gpt-5-main with OpenAI o3 and GPT-4o. The authors note regressions for gpt-5-main and say they will follow up with a fix 2601.03267 §3.5. Higher is better in the table below, as in the card.

The card also reports prompt-injection evaluations, where only gpt-5-thinking and o3 are compared 2601.03267 §3.6. Higher is better.

When to use which

Neither the survey nor the review gives a comparison of its defences or any numbers for them, so the first four rows only map constraints to the mechanism each source describes. They do not rank the options. The fifth and sixth rows rest on the o1 and GPT-5 cards alone, and the seventh on the benchmark table of the review.

Implementation sketch

A prevention pipeline with a perplexity check, as the survey describes its parts 2402.00888 §6.1. The survey does not give the order of the steps, a threshold, or the model used for scoring; those choices are open and are marked below.

# Guarded LLM call: rewrite, check, then separate  [arxiv:2402.00888 §6.1]
# Input:  instruction (string, trusted), data (string, untrusted)
# Output: answer (string), or None when the data is flagged

def guarded_answer(instruction, data, llm, paraphrase, perplexity_flags):
    # 1. Rewrite the untrusted data to break injected instructions  [§6.1]
    cleaned = paraphrase(data)
    # 2. Flag the rewritten data if it looks unlikely  [§6.1]
    #    the scoring model and the rule are not stated in the survey
    if perplexity_flags(cleaned):
        return None
    # 3. Keep the trusted instruction apart from the untrusted data  [§6.1]
    #    the layout of the two parts is a free choice
    prompt = instruction + "\n\n" + cleaned
    return llm(prompt)

Contradictions

None known between the four cards. The survey and the review report no numbers for the prompt-injection defences, so there is nothing to disagree with. The survey says the universal gradient-based attack may fall short against perplexity-based defences 2402.00888 §4.1.1, and it says proactive detection has been effective in a work it cites 2402.00888 §6.1. Those statements are about different attacks and defences, not a conflict over one result.

The o1 and GPT-5 cards both report a phrase-protection row for developer messages. Their row names are similar, but the cards test different models and do not show the same setup, so their numbers are not compared. Both report a lower score on that row than a reference model: o1 against GPT-4o, 0.70 against 0.82 2412.16720 §4.2, and gpt-5-main against GPT-4o, 0.404 against 0.449 2601.03267 §3.5. The GPT-5 card notes regressions for gpt-5-main and says a fix will follow 2601.03267 §3.5. The o1 card's instruction-conflict tests are not the attacks the survey or the review describes, so its scores are not compared with theirs.

Papers

• Security and Privacy Challenges of Large Language Models: A Survey: a taxonomy of prompt injection and other LLM attacks, with the defences named for each, and no experiments of its own.
• OpenAI o1 System Card: a safety report for o1 and o1-mini, with instruction-hierarchy evaluations and author-reported scores; not a method paper.
• OpenAI GPT-5 System Card: a safety report for the GPT-5 family, with author-run instruction-hierarchy and prompt-injection evaluations against o3 and GPT-4o; not a method paper.
• TRiSM for Agentic AI: a review of trust, risk and security for multi-agent LLM systems, with a threat taxonomy, control tables and no experiments.