wikillm

Paper · 2026-10-28 · cs.AI, cs.CR, cs.LG

Where You Inject Matters: The Trust Region Effect in LLM Agents

With injected text held fixed, LLM agents follow it at different rates by file or surface: for GPT-5.4, README or TODO instructions were followed 2.6x as often as code or data-file ones. The end position was most effective within a surface. A two-stage RL attacker that also learns placement beats content-only training with policy-chosen placement (author-reported).

Prompt injectionLLM agentAdversarial robustnessprompt-injectionindirect-prompt-injectionllm-agentsagent-securityreinforcement-learning

arXivPDFMarkdown9 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

An AI agent that reads files, emails or web pages can be tricked by text hidden in those sources. Earlier research mostly asked which wording of such a hidden instruction works best. This paper asks a different question: where the text is placed matters too. The same sentence was followed more often when it sat in a project's README or TODO file than when it sat in a code file or a data file.

The authors built three test setups with many small workplace-style tasks. They kept the hidden text the same and moved it between files and between positions in a file. They then measured how often each agent did the unwanted action. The agents differed from one another in their results.

The paper also trains a model to write such text and choose a placement. It reports that learning the placement along with the wording can raise success when the model picks the placement itself. A fixed rule of thumb about where to place text was also strong, and in several settings beat the learned placement. For defenders, the practical point is that no file type should be treated as trusted just because it looks like documentation.

TL;DR

The paper measures how the location of an injected instruction changes whether LLM agents follow it. It names this the "trust region effect" §1 §4.2. Three agentic testbeds cover coding, tool use and daily workflow, with 30 benign tasks, 24 injection tasks and 36 environment surfaces §4.1. For GPT-5.4, instructions in repository documentation (README or TODO) were followed 2.6 times as often on average as the same instruction in a code or data file §4.2 (author-reported). The End position gave the highest conditional success in all three domains §4.2 tab. 3. A two-stage RL attacker that learns the placement beats content-only training when the placement is chosen by the policy tab. 5; content-only training with a fixed end-of-surface prior was competitive (author-reported). The paper provides no code.

Problem

Indirect prompt injection puts instructions into content an agent reads while doing a user's task, so the user never sees them §1. Prior work mostly varied the injected content, or studied placement only inside one webpage or one file §1 §2. The authors ask whether an injection's surface (which kind of environment item carries it) and its position within that surface change attack success, with the content held fixed §1 §3.1.

Their threat model: the attacker can change the content of an environment surface but not the user prompt, the system prompt or model weights §3.1.

Method

The paper has two parts: a measurement study and a learned attacker.

Injection formalism. An injection is a pair of content and placement. A placement is a surface plus a position within it, with positions Start, Middle and End §3.1 eq. 2. For each benign task, the candidate placements are all surface and position pairs over the surfaces the agent reliably reads for that task §3.3 eq. 3.

Testbeds. All three are built by composing existing tasks so that each task touches several surfaces §3.2.

  • Coding: 10 longer tasks built from Terminal-Bench-2 by combining 2 to 5 subtasks in one Docker environment; 7 injection risks §3.2. Surfaces include the repository README and TODO files, source files and data files app. B.
  • Tool use: 10 longer tasks built from AgentDojo by composing 6 to 8 atomic tasks; 12 injection risks §3.2. Surfaces include web pages, reviews, calendar entries, email bodies, cloud documents, local files and transaction memos app. B.
  • Workflow: 10 daily-workflow tasks built on DecodingTrust-Agent; 5 injection risks §3.2. Surfaces are GitHub, Slack, Confluence and Jira app. B.

Utility and attack success are scored by deterministic, script-based verifiers, not by an LLM judge §3.2 §4.1. Utility is a score in [0, 1]; attack success is binary §3.2.

Conditional attack success. The main metric counts only injections the agent actually read. The paper checks whether the payload text appears in what the agent retrieved §4.2 and reports ASR(ℓ) = Pr(attack success | injection read, ℓ) §4.1.

Placement-aware attacker. The attacker is a language model that takes the benign task, a malicious goal and the candidate placement set, and outputs content, a surface and a position §3.3 eq. 4. Training has two stages.

  • Stage I, content optimisation. Supervised warm-up on template-based examples §3.3.1 app. D. Then reinforcement learning with GRPO §3.3.1. During this stage the placement is overwritten by a fixed prior: rank surfaces by their average success on the training grid, pick the best-ranked surface available for the task, and use the End position §3.3.1 eq. 5 eq. 6. The ranking for tool use is in app. D.
  • Stage II, joint optimisation. Start from the Stage I checkpoint and use the placement the policy generates, without the override, under the same reward §3.3.2.

Reward. The reward multiplies a format gate by a term that combines verifier attack success, benign utility and an LLM-judge score for the content §3.3.1 eq. 7. The judge score is an average of three sub-scores, multiplied by a separate intent score eq. 7. The weights of the three terms are described as adjustable; their values are not stated §3.3.1.

Pseudocode
# Two-stage placement-aware attacker  [§3.3] [eq. 7]
# Input:  benign tasks, malicious goals, candidate placements per task
# Output: trained attacker policy

def train_attacker(tasks, goals, placements):
    # Warm-start the policy on structured examples  [§3.3.1] [app. D]
    policy = supervised_warmup(tasks, goals, placements)

    # Stage I: learn content; the placement is forced to the fixed prior
    # [§3.3.1] [eq. 6]
    policy = grpo_train(policy, tasks, goals, placement_mode="prior_end")

    # Stage II: learn content and placement jointly  [§3.3.2]
    policy = grpo_train(policy, tasks, goals, placement_mode="policy")

    return policy

Surface and position effects. The authors also test a template-based attack, which is a fixed framing with the goal inserted, and an LLM-based placement selector, which is prompted to choose a placement §4.4 app. D.

Hyperparameters

NameValueWhereNotes
benign tasks / injection tasks / surfaces / positions30 / 24 / 36 / 3§4.1Start, Middle, End per surface
evaluated agents8 (GPT-5.6, GPT-5.4, GPT-5-mini, GPT-4o, Gemini-3.1-Pro, Grok-4.6, Opus5, Sonnet5)§4.1GPT-4o only in tool use
agent frameworksClaude Agent SDK (Claude models, coding and workflow); OpenAI Agents SDK (other models, coding and workflow); AgentDojo tool calling (tool use)§4.1
victim sampling, temperature, step limitnot stated§4.1
trials per cellnot stated in the sections read§4.2 tab. 2Small cells are flagged as unstable §4.3
attacker policy modelQwen2.5-7B-Instruct§4.5
attacker training victimGPT-4o§4.5
leave-task split (train / val / test contexts)84 / 12 / 24§4.5Unseen benign tasks
leave-risk split (train / val / test contexts)60 / 30 / 30§4.5Unseen injection risks
GRPO generations per prompt24§4.5
GRPO rollouts per optimiser step96§4.5
learning rate1e-5§4.5Optimiser not stated
training epochs3§4.5
judge weight jsweep over 0, 0.3, 0.5, 1§4.5 tab. 5j = 0 uses only the verifier signal
reward weights λ_succ, λ_benign, λ_judgenot stated§3.3.1 eq. 7Described as adjustable
SFT warm-up settingsnot stated§3.3.1
LLM judge modelnot stated§3.3.1
fixed placement priorEnd position, surfaces ranked by training success§3.3.1 eq. 6 app. D
benign utility without injection56% (leave-risk), 88.9% for GPT-5.4§4.5 tab. 6

Results

Author-reported. Setup: three testbeds §4.1. Victim models are listed in the hyperparameter table above. Full tables are in the cited sources.

SettingMetricBaselineThis paperSource
GPT-4o, tool use, delivered injectionsignored / refused / attemptednot applicable81.3% / 0.7% / 18.0%tab. 1
GPT-5.4, delivered injectionsignored / refusednot applicable66.8% / 33.2%tab. 1
GPT-5.4, coding: README and TODO vs code and data filesASR on averagecode and data files2.6x more often§4.2
GPT-5.4, coding, README row (Recon / SQLi)conditional ASRcode file row (19 / 11)86 / 79tab. 2
Position effect, coding (start / mid / end)conditional ASRstart 12.6mid 11.1, end 13.0tab. 3
Position effect, workflow (start / mid / end)conditional ASRstart 2.3mid 3.3, end 5.7tab. 3
Position effect, tool use, GPT-4o (start / mid / end)conditional ASRstart 39.5mid 40.1, end 51.2tab. 3
Attacker, leave-risk, j = 0: Content-only with prior placementASRSelf-selected placement 37.062.7 (utility 48)tab. 5
Attacker, leave-risk, j = 0: Two-stage vs Self-selectedASR gainSelf-selected 37.0Two-stage 59.1 (+22.1)tab. 5
Attacker, leave-risk, judge weights 0.3 to 1ASR gain, Two-stage vs Self-selectedSelf-selected 26.8, 36.6, 7.7+20.7 to +24.8 pptab. 5
Attacker, leave-task, j = 0: Two-stage vs Self-selectedASR gainSelf-selected 75.7Two-stage 80.3 (+4.6)tab. 5
Attacker, leave-task, j = 0.5 and 1ASR gainSelf-selected 12.9, 77.9Two-stage 23.4, 83.3tab. 5
Transfer to GPT-5.4, learned attacker (matched placement)ASR / utilitytemplate attacks 0.033.3 / 88.9tab. 6
Tool use, template attack, random vs prior-ranked surface, leave-riskconditional ASRrandom 25.6prior 34.7app. D
Tool use, template attack, random vs prior-ranked surface, leave-taskconditional ASRrandom 38.8prior 77.1app. D
Learned content with prior placement, leave-riskconditional ASRtemplate at prior 34.762.7app. D
Overall ASR, Gemini-3.1-Pro (code / workflow / tool use)ASRnot applicable67.0 / 65.3 / 19.3app. D
Overall ASR, GPT-5.6 (code / workflow / tool use)ASRnot applicable0.6 / 0.0 / 0.0app. D
Tool use, placement selection at budget B = 1successLLM selector 0.42fixed prior 0.53§4.4

The attacker's training and test sets are small: 30 test contexts for the transfer test §4.5. Attack results on the leave-task split vary widely across judge weights (for example, Self-selected placement ranges from 12.9 to 77.9) tab. 5.

Paper vs code

No official code. The LaTeX source has no repository link, project page or code-availability statement [raw/src]. The paper's ethics statement says that artifacts are shared through responsible disclosure, which we could not verify §9. Searches done: the LaTeX sources were searched for links; GitHub repository searches for "trust region injection" and "injection placement agents"; and searches restricted to the Microsoft organisation (several authors are at Microsoft Research) for "injection placement" and "prompt injection trust region". The repository hits were unrelated third-party projects, and the organisation search returned none. WebSearch was not permitted in this session, so no general web search was run.

Pitfalls

  • Conditional and unconditional ASR differ. The main measurements use only injections the agent read §4.1. Comparing against unconditional numbers from other papers will mislead.
  • The surface effect is confounded with read order. A test over 24 read orders in one workflow task found that read depth changes ASR less than surface identity does app. D. The check covers one task, so it is not general.
  • Documentation surfaces are not uniformly stronger. Gemini-3.1-Pro opens README and TODO files in under 5% of runs, so its documentation numbers are based on very few delivered cells and are unstable §4.3.
  • The payload is fixed per condition. Non-environmental channels, such as the prompt or a tool description, were stronger than environmental surfaces in the coding testbed app. D. Placement results should not be read as a ranking of threats.
  • Training is narrow. The attacker was trained on the tool-use testbed only, with one victim (GPT-4o), and transfer was tested on one more victim (GPT-5.4) with 30 test contexts §4.5 tab. 6.
  • The fixed End prior is strong. The prior-placement variant matches or beats the learned placement policy in several settings §4.5 tab. 5. The learned placement is not shown to be the main source of gains.
  • The reward weights, SFT settings, judge model, victim sampling settings and number of trials per cell are not stated. Re-implementation will need choices the paper does not make.
  • Model names are author-reported. Several are unreleased or preview names (for example GPT-5.6, Opus5, Sonnet5, Grok-4.6) that could not be verified here.
  • Limitations stated by the authors: the trust effect is behavioural, not a measured internal state; three testbeds do not cover real environments; the study is about indirect prompt injection only app. E.
  • Dual-use. This card describes an attack method at the level of its structure, placement findings and training stages. It leaves out the injection templates and example payloads in the appendix app. D. For defence, the actionable finding is that no file type or position should be treated as safer by default; the paper suggests defences should account for where untrusted content appears §5.

Lineage

  • Builds on: AgentDojo (tool-use benchmark) §3.2; Terminal-Bench-2 (coding tasks) §3.2; DecodingTrust-Agent (workflow tasks) §3.2; GRPO §3.3.1.
  • Related prior work on placement: EIA and POISE (position inside one webpage or one file), MUZZLE (LLM-selected web-agent vessels) §2.
  • Related prior work on content: AutoInject and RL-Hammer, both RL-trained injection content with placement fixed §2.
  • Superseded by: not applicable.

Source and license

Adapted from “Where You Inject Matters: The Trust Region Effect in LLM Agents” by Chengquan Guo, Xiaodong Liu, Weiwei Yang, Mingqian Feng, Jialin Song, Qiyao Ma, Carolyn Buractaon, Christopher White, Bo Li, Jianfeng Gao (arXiv:2610.38867), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.