Organizations: Fullive-AI · Nanyang Technological University · Supply Chain Tech Team Y, JD.com · Peking University · University of Chinese Academy of Sciences · Soochow University · Beijing University of Posts and Telecommunications · University of Leeds · Beihang University
LLM agents that read untrusted content can be hijacked by prompt injections. Runtime monitors that try to catch a hijack before the agent acts, and benchmarks that score each task with a single run, share an assumption: whether an agent follows an injection is settled when the injection is read. We test this assumption with branching rollouts that replay an episode exactly up to the step at which the agent reads an injection and then resample the rest of the episode many times, splitting the outcome variance into a part fixed at read time and a part decided afterwards. Across four AgentDojo suites, a Qwen3 agent without thinking is largely decided at read time (0.91 of the outcome variance lies between states); with thinking on, this share drops to 0.62 (95% CI of the drop [-0.38, -0.22]), in the same direction for three attack templates and two success labels. How much is fixed at read time is a property of the model that the same measurement reveals, and thinking lowers the attack success rate of Qwen3 but raises it for two other models. Resampling from reasoning prefixes shows that the decision forms gradually as the agent reasons. Zero-shot chain-of-thought monitors built on cheaper models miss much of what is already decided, whereas stronger monitors, and a lookahead monitor that judges simulated next actions, come close to what the prefix determines. Finally, single-run labels of thinking agents disagree with a rerun up to 35% of the time, so benchmarks need repeated runs per task.
Figures & tables
Figure 1. Overview. (1) We replay an AgentDojo episode up to its exposure state s . (2) We rerun the agent K times from each state; the ring shows V , the share of outcome variance fixed at read time (Qwen3-8B, four suites). (3) Resampling from cuts at 30–70% of one run’s reasoning gives the share of the outcome already decided (banking). (4) Dots: each monitor’s mean gap to the decision ceiling over six cells. Example text is abridged and drawn runs are illustrative; all numbers are measured.
Claim and criterion
Outcome
C1 The early window in the context is produced by the layout: moving the tool block after the newest observation creates a knee near the injection that the native layout lacks.
C2 Without thinking, outcomes are largely fixed at read time; thinking lowers V and raises the share of mixed states on the same states, in the same direction for at least two model families.
Holds. V 0.91 → 0.62, [ − 0.38, − 0.22]; Qwen3 and Nemotron. Premise model-dependent.
C3 Weak CoT monitors reach a within-state AUC of 0.70 by 30% of the reasoning.
Fails. 0.52–0.65 at 30%.
C4 With thinking, on mixed states, de-meaned probes beat text monitors in the first 20% of the reasoning.
Fails in all 5 settings: − 0.07 to +0.03 vs. TF-IDF.
Table 1. Claims and decision criteria fixed before the experiments, and outcomes (Qwen3-8B unless noted).
banking
slack
native
late
native
late
Injection end e (median)
0.96
0.26
1.00
0.25
Knee of the probe curve
1.00
0.30
0.95
0.60
Direct-fire share
29%
7%
66%
53%
ASR, comply label
48%
29%
68%
54%
ASR, official label
34%
19%
60%
35%
Table 2. Layout control (Qwen3-8B, no thinking, same tasks): the tool block sits in the system prompt (native) or after the newest observation (late). Direct-fire: complying episodes that act in the step in which they read the injection.
Figure 2. Read-time share V : the share of outcome variance already fixed when the injection is read, with thinking off and on, on the same states (comply label; other attacks: Qwen3-8B). Right: 95% CI of the paired drop. Table 3 adds the official label and the controls.
Setting
States
ASR
Mixed states
V off
V on
ΔV , comply [95% CI]
ΔV , official [95% CI]
Qwen3-8B
banking
117
0.60 → 0.38
7% → 26%
0.91
0.66
− 0.25 [ − 0.49, − 0.07]
− 0.37 [ − 0.56, − 0.21]
slack
95
0.78 → 0.67
13% → 40%
0.79
0.47
− 0.32 [ − 0.45, − 0.21]
− 0.39 [ − 0.54, − 0.26]
workspace
202
0.08 → 0.06
2% → 10%
0.88
0.46
− 0.42 [ − 0.64, − 0.25]
− 0.33 [ − 0.53, − 0.16]
travel
107
0.49 → 0.44
9% → 59%
0.90
0.35
− 0.55 [ − 0.69, − 0.43]
− 0.50 [ − 0.64, − 0.37]
all four suites
518
0.42 → 0.32
7% → 28%
0.91
0.62
− 0.29 [ − 0.38, − 0.22]
− 0.32 [ − 0.40, − 0.26]
Table 3. Read-time share V with thinking off and on, on the same states. ASR and mixed states: off → on (mixed states with K matched across modes). ΔV : on minus off with paired 95% bootstrap CI; official: the same comparison with AgentDojo’s security() label (its state counts include the states the comply label cannot decide). Sampling only: the non-thinking prompt with the default vs. the thinking-mode sampling; matched sampling: the latter vs. thinking. Nemotron and SmolLM3 count branches whose reasoning hits the token budget as not complying.
Figure 3. How the decision forms while the agent reasons, and who reads it, on the same branches. Top: determined share D(f) , the share of the within-state outcome variance fixed by the reasoning prefix (band: 95% CI). Bottom: within-state AUC; the decision ceiling (grey band: 95% CI) bounds every monitor that reads the context and the prefix. Weak CoT monitor: DeepSeek-v4-flash without thinking; strong: DeepSeek-v4-flash with thinking; lookahead: GPT-5 judging the calls of four continuations without being told the injection. Hollow points: values fixed by definition at read time ( D=0 ; within-state AUC 0.5 ). All monitors: Appendix Table 5 .
Reads
Monitor
Gap
Below
Reasoning
weak CoT (GPT-4o, DeepSeek)
+0.09 to +0.28
15/15
strong CoT (3 monitors)
+0.03 to +0.17
3/18
DeepSeek, predict call first (banking)
+0.08 to +0.11
3/3
Simulated
lookahead, GPT-5
+0.01 to +0.04
0/9
calls
lookahead, Claude Opus 4.8
− 0.03 to +0.06
1/9
lookahead, DeepSeek
+0.03 to +0.23
6/9
Table 4. Monitors against the decision ceiling on the same branches: 3 settings × 3 reasoning fractions (banking: 3 cells). Gap: ceiling minus the monitor’s within-state AUC. Below: cells in which the paired 95% CI of the gap excludes zero.
Figure 4. Probability that two runs from the same state get the same label, thinking off vs. on. Right: runs per task needed for a majority label that matches the more likely outcome with 90% probability.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Reads the reasoning
Reads simulated calls
Supervised
Setting
f
D(f) [95% CI]
Ceiling
GPT-4o
DeepSeek
GPT-5
Claude
DS-think
GPT-5
Claude
GPT-5, 1 turn
Probe
TF-IDF
Qwen3-8B banking
0.3
0.36 [0.25, 0.47]
0.78
0.50 ∗
0.53 ∗
0.67
0.69
0.74
0.76
0.76
0.70
0.77
0.64
33 states, 127 branches
0.5
0.55 [0.44, 0.65]
0.88
0.60 ∗
0.66 ∗
0.79
0.72 ∗
0.77 ∗
0.86
0.88
0.81 ∗
0.72 ∗
0.65 ∗
0.7
0.62 [0.51, 0.72]
0.90
0.68 ∗
0.64 ∗
0.84
0.83
0.85
0.89
0.90
0.89
0.76 ∗
0.69 ∗
Qwen3-8B slack
0.3
0.37 [0.26, 0.49]
0.82
0.66 ∗
0.64 ∗
–
0.74
0.77
0.78
0.79
0.74
0.82
0.83
39 states, 152 branches
0.5
0.48 [0.35, 0.60]
0.91
0.74 ∗
0.70 ∗
–
0.81
0.88
0.88
0.88
0.81 ∗
0.83
0.78 ∗
Appendix
Table 5. Nested resampling: determined share D(f) , decision ceiling, and the within-state AUC of every monitor on the same branches at reasoning fraction f . CoT monitors read the reasoning prefix; lookahead judges read the calls of four continuations and are not told the injection (single turn: calls of the current turn only); supervised monitors are cross-validated across user tasks. ∗ / † : the paired 95% CI of the ceiling minus the monitor lies above/below zero. –: not run.
Setting
Agree
Noise
k90
Δ ASR
Flip
Qwen3-8B banking
0.95/0.83
3%/10%
1/3
− 0.229
0%
Qwen3-8B slack
0.93/0.77
5%/14%
1/7
− 0.125
0%
Qwen3-8B workspace
0.99/0.94
1%/4%
1/1
− 0.026
2%
Qwen3-8B travel
0.95/0.68
4%/19%
1/11
− 0.046
14%
Qwen3-14B banking
0.98/0.85
2%/9%
1/1
− 0.246
0%
Nemotron banking
0.95/0.65
3%/21%
1/17
+0.449
0%
Appendix
Table 6. Single-run reliability (off/on). Agree: probability that two runs of a task get the same label; noise: probability that one run gets the minority outcome; k90 : runs for a majority label that is right with 90% probability; Δ ASR: true effect of thinking; flip: probability that one run per task and mode reports the opposite sign (5,000 simulations).
Agent
Switch
T , top- p (off / on)
Budget (off / on)
Qwen3-8B, 14B
template
0.7, 0.8 / 0.6, 0.95
768 / 4096
Nemotron-Nano-8B
system line
0.6, 0.95
1024 / 8192
Gemini 2.5 Flash
API effort
0.7, –
provider
Llama-3.1-8B
none
0.6, 0.9
768
SmolLM3-3B
template
0.6, 0.95
768 / 4096
Appendix
Table 7. Agents and decoding. Budgets are new tokens per step. Qwen3 uses its recommended sampling in each mode; all other agents use the same sampling in both modes.