LLM agents take instructions from the operator, repository configuration, user, tool results and saved memory, which often conflict. Instruction-hierarchy benchmarks ask whether the higher-ranked source wins, but a source sets position, role and label at once. First, we build WhoWins, 1,080 code-scored scenarios in five task types, each with up to five mutually exclusive instructions, and seven controlled experiments that vary only how instructions are delivered. Second, we study 14 open-weight models (0.6B to 122B, thinking off and on) over 1,887,356 scored replies, checking main findings on held-out scenarios. Inside one message the first of two conflicting lines wins 72.42 to 94.66%, and across messages it keeps 22.95 to 62.65%. A policy sent first in the system (or developer) message loses most conflicts with later lines in every Qwen model but wins most on gpt-oss when it acts. In tool and file-write tasks it wins less often than the same text sent first as a labelled user message, in every model-mode. Third, we test whether pairs forecast larger conflicts; forecasts improve in every model-mode once they know where the task line is. Finally, by our keyword patterns at most 2.92% of decided pair replies flag or hedge a conflict.
Figures & tables
Figure 1: Overview. A scenario has a task and up to five options that no reply can satisfy together (here, five casings of one key). Each option is placed in a seat, a message with its own position, chat role and label; each experiment varies their delivery. 14 models reply (filled: thinking off; open: on), and code scores the option a reply followed. Right: per family, the median over model-modes and the range, as shares of decided replies (dashed: 50%), and the operator line’s real role minus label in tool and file-write tasks (dashed: 0).
Study
Sources
Order
Role/ label
Msg. split
Sizes
From pairs
Code only
Think off/on
Models
IHEval ( Zhang et al., 2025 )
S, U, H, T
✗
✗
✗
2
✗
✓
✗
13
Control Illusion ( Geng et al., 2026 )
S, U
∼
∼
∼
2
✗
∼
✗
6
IH-Benchmark ( McCauley et al., 2026 )
S, U, T
✗
✗
✗
2
✗
∼
✓
37
ManyIH ( Zhang et al., 2026 )
tags
random
✗
✗
2–5
✗
∼
✓
13
Choose Sides ( Balp-Straffon et al., 2026 )
S, U
✗
✗
∼
2
✗
✓
✗
8
Authority Bias ( Kumar and Chopra, 2026 )
note , U
∼
∼
✗
n/a
✗
∼
✗
8
Table 1: WhoWins against prior studies of conflicting instructions. One row per study, one column per design choice, ours last. Sources : S system or developer message (ours has two seats there), U user, H conversation history, T tool output, M saved memory note; ManyIH ranks lines by privilege tags in the prompt, and Authority Bias compares a “verified source” note with a user’s claim about a factual answer. Order : the same lines in both orders. Role/label : the same text in its real chat role and as labelled text. Msg. split : the same lines in one message and in separate messages. Sizes : conflicting lines in one trial. From pairs : forecasts larger conflicts from pairs. Code only : no LLM in scoring. Think off/on : the same model with thinking off and on. Models : as each paper counts them. ✓ yes; ∼ partly; ✗ no; grey: set by design or not applicable. Appendix Q explains each entry.
Experiment
What moves
Held fixed
Trials
Replies
Where
Ladder
which seats compete (two, three and five at a time) and their order
text; system seats first; each option also asked alone
12,840
38,520
§ 4.2 , § 4.4
Operator
operator line in the real system message or as labelled user text; first or last
text; the opposing seat (user, tool or memory)
4,320 (3,240)
12,960 (9,720)
§ 4.3
Boundary
user line and memory note in separate messages or inside the task’s message; order
text; roles
1,080
3,240
§ 4.1
Context
instructions bare or inside realistic context; order
text; 50 tool and file-write items
1,200 (900)
3,600 (2,700)
§ 4.3
Agreement
two or more seats ask for the same option (3 to 5 seats)
text; seats dealt as in the Ladder
3,240
9,720
App. K
Load
number n of instructions (2 to 12) and of conflicts
length at each n ; one operator message
100
300
App. O
Table 2: The seven experiments: what each varies and what it holds fixed. Trials and replies are per model-mode; each trial is sent three times (Tiers: once). Parentheses: Qwen 3.5, whose template rejects a late system message. Every experiment covers all three model families. Where: the section that reports it.
Position
Role and label
Seats
Rule
Forecast
Model
Think
Later wins
2nd sys. line
First, one msg.
First, sep. msgs.
Operator first
Real − label ∗
Late op. gain
User > tool
Tier rule
Seat gain
Qwen 3
Qwen3-0.6B
off
79.01
29.50
86.48
48.48
8.86
− 48.75
+ 23.91
65.49
51.32
+ 0.009
on
69.36
35.78
80.41
45.74
11.44
− 46.46
+ 19.79
73.33
52.91
+ 0.044
Qwen3-1.7B
off
83.22
30.89
90.29
34.86
7.85
− 31.87
+ 31.27
57.81
57.12
− 0.020
on
76.11
24.24
86.17
35.95
9.98
− 37.81
+ 29.45
70.56
55.94
+ 0.051
Table 3: Who wins, per model-mode. Rates are percentages of decided replies; differences are absolute and paired within the scenario. Operator first: the operator line, first in the Ladder, against a later line. Real − label ∗ : in tool and file-write tasks, the operator line sent first in the real system message minus the same labelled user text. Seat gain: nats per reply that seat strengths add to the position fit. Red shades larger rates and positive differences, blue negative ones; grey: the 95% interval includes 50% or 0. n/a : not possible (the template rejects a late system message). Appendix H defines each column.
Figure 2: The first line wins inside one message and much less often across messages. Share of decided replies that follow the first-listed of two conflicting lines (user against memory note), with both lines inside the task’s message (upper point) or each in its own message before the task (lower point); a thin line joins the two points of a model-mode. Sizes in billions of parameters; filled: thinking off; open: on; colours mark model families; dashed: 50%. Bars: 95% bootstrap intervals.
Figure 3: Qwen 3 follows order, gpt-oss follows the developer message. Each point is a model-mode, with 95% intervals. Horizontal: how often the later of two lines wins when the user, tool and memory seats meet in both orders. Vertical: how often the operator line wins when it comes first. Marker size grows with parameters; open markers are thinking on; colours mark model families. Dashed: one minus the horizontal value, the first line’s share in those free pairs.
Figure 4: In tool and file-write tasks the real system message lowers the operator line’s wins in every model-mode. Win rate of the operator line sent first in the real system (developer) message minus the same labelled text sent first in a user message. Markers: tool-call and file-write tasks, with 95% intervals (filled: thinking off; open: on); grey ticks: all task types. Colours mark families; sizes in billions of parameters.
Figure 5: Seat strengths help on Qwen 3.5 and gpt-oss. Share of the gap from a uniform guess (0%) to the empirical reference (100%) recovered when forecasting three- and five-way conflicts from pairs. Bar: position fit; tick: task-aware fit; marker on a stem: seats fit (filled: thinking off; open: on). Colours mark families; sizes in billions of parameters.
Figure 6: Thinking helps Qwen follow declared tiers and raises the operator line on Qwen 3. Thinking off (filled) to on (open) for every Qwen size, over scenarios decided in both modes: (a) share that follows the rule that the lower number wins; (b) the operator line’s wins when sent first in the real system message (Operator test). Grey: the 95% interval of the change includes zero; colours mark families.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Task type
n
Key read
Example task
Two of its options
Tool
240
first tool call
Produce a coverage report for the billing library.
“Measure with cov_line.” / “Measure with cov_branch.”
File write
240
first write target
Stage the intermediate files while the job runs.
“Stage them in /scratch/a .” / “Stage them in /scratch/b .”
Format
240
a format label
Emit the response body for the /accounts endpoint.
“Name every field in snake_case.” / “Name every field in camelCase.”
Ordering
240
order of three steps
Get the repository into a releasable state.
“Run build, then test, then lint.” / “Run build, then lint, then test.”
Refusal
120
comply or decline
The scratch directory is full and the job is stalling.
“Clear /scratch and continue running.” / “Decline and report disk state.”
Appendix
Table 4: The five task types. Each tool, file-write, format and ordering scenario has five options that set the same key to different values; refusal has two. The tool, file-write and ordering keys are read from the trace of tool calls, the refusal key from the trace or, when neither tool is called, from a refusal in the reply, and the format key from the text the model writes.
Seat
Message
Label line
\seatchip seatOneFillseatOneLineS1 operator
system (developer on gpt-oss)
Operator policy for this deployment.
\seatchip seatTwoFillseatTwoLineS2 repository
the same message
Repository configuration for this workspace.
\seatchip seatThreeFillseatThreeLineS3 user
user
none
\seatchip seatFourFillseatFourLineS4 tool result
tool reply to config_read
Returned by the config_read tool.
\seatchip seatFiveFillseatFiveLineS5 memory
user
Your saved note from an earlier session.
Appendix
Table 5: The five seats. The task arrives as a user message that starts “Task: …” (in the Boundary test’s merged layout it shares that message with \seatchip seatThreeFillseatThreeLineS3 and \seatchip seatFiveFillseatFiveLineS5). It never shares the system message.
Model
Params
Active
Arch.
Precision
vLLM
Thinking
Qwen3-0.6B
0.6B
–
dense
float16
0.8.5.post1
off, on
Qwen3-1.7B
1.7B
–
dense
float16
0.8.5.post1
off, on
Qwen3-4B
4B
–
dense
float16
0.8.5.post1
off, on
Qwen3-8B
8B
–
dense
float16
0.8.5.post1
off, on
Qwen3-14B
14B
–
dense
float16
0.8.5.post1
off, on
Qwen3.5-0.8B
0.8B
–
dense
bfloat16
0.28.0
off, on
Appendix
Table 6: The models and how they were served. Parameters as on the model card; precision is the number format the weights were loaded in; vLLM is the serving software version. Qwen models run with thinking off and on; gpt-oss always reasons. a : the pre-release served the gpt-oss-120b Ladder and Operator experiments.
Qwen, off
Qwen, on
gpt-oss
Thinking
off
on
high effort
Temperature
0.7
0.6
0.6
Top- p
0.8
0.95
0.95
Top- k
20
20
20
Min- p
0.0
0.0
0.0
Presence penalty
0.0
0.0
0.0
Appendix
Table 7: Decoding. One setting per mode for every Qwen size (the Qwen 3 model cards’ settings, with presence penalty 0); gpt-oss has no non-thinking mode and uses the thinking setting at high reasoning effort.
World 1
World 2
P(A beats B)
1/2
1/2
P(A beats C)
7/10
7/10
P(B beats C)
7/10
7/10
P(A,B,C win the triple)
1/2, 1/5, 3/10
1/5, 1/2, 3/10
Winner of the triple
A
B
Appendix
Table 8: Two worlds with the same pairs and different triples.
Qwen 3
Qwen 3.5
gpt-oss
Finding
D
H
Both
n
D
H
Both
n
D
H
Both
n
Later line wins, free Ladder pairs
10
10
10
10
14
14
14
14
0
1
0
2
Second line of one system message loses
8
9
7
10
14
14
14
14
1
2
1
2
First-listed wins inside one message
10
10
10
10
14
14
14
14
2
2
2
2
First-listed loses across messages
8
8
8
10
8
8
8
14
0
0
0
2
One message minus separate messages
10
10
10
10
14
14
14
14
2
2
2
2
Appendix
Table 9: Each finding on the two halves of the scenarios. Model-modes (Qwen sizes, for the thinking rows) whose 95% interval excludes the null in the finding’s direction on the discovery half (D), on the held-out half (H) and on both; n : model-modes with the measure. Composition findings fit and score within each half. n/a : the measure does not apply (Qwen 3.5 has no late system message; gpt-oss has no thinking-off mode).
Model
Think
Later wins
2nd system line wins
Later wins: follows the task
Later wins: both before the task
Qwen 3
Qwen3-0.6B
off
79.01 [72.99, 85.25]
29.50 [16.90, 42.34]
86.29 [79.33, 92.28]
61.60 [46.72, 75.41]
on
69.36 [64.09, 74.53]
35.78 [23.01, 49.52]
74.44 [68.44, 80.39]
54.47 [42.97, 65.08]
Qwen3-1.7B
off
83.22 [78.64, 87.57]
30.89 [20.00, 41.45]
92.51 [88.55, 95.89]
63.22 [52.89, 73.28]
on
76.11 [71.78, 80.49]
24.24 [15.90, 32.65]
81.62 [77.10, 85.88]
63.39 [53.68, 72.18]
Qwen3-4B
off
82.44 [77.93, 86.81]
30.52 [20.70, 41.20]
91.24 [86.86, 95.09]
64.31 [54.12, 73.91]
Appendix
Table 10: Position in the Ladder. Later wins: the later of two lines wins in free pairs (user, tool result, memory note; both orders). 2nd system line: the second of the two system-message lines ( \seatchip seatOneFillseatOneLineS1 and \seatchip seatTwoFillseatTwoLineS2, each order half the time) wins. The last two columns split later wins by where the task line is: the later line follows the task (the task comes first or between the two lines), or both lines come before the task. Grey brackets: 95% bootstrap intervals. The later line wins in 25 of 26 model-modes; the second line of one system message loses in 26 of 26.
Model
Think
User beats tool
Operator wins (decided)
Operator wins (all usable)
Qwen 3
Qwen3-0.6B
off
65.49 [51.77, 78.77]
8.86 [6.15, 11.56]
4.80 [3.40, 6.28]
on
73.33 [63.92, 82.12]
11.44 [8.61, 14.33]
5.99 [4.55, 7.55]
Qwen3-1.7B
off
57.81 [48.05, 67.58]
7.85 [5.80, 10.19]
6.36 [4.55, 8.21]
on
70.56 [62.20, 78.72]
9.98 [7.91, 12.20]
8.22 [6.50, 10.15]
Qwen3-4B
off
54.69 [44.31, 65.12]
10.92 [8.62, 13.49]
9.09 [7.03, 11.21]
Appendix
Table 11: Seats in the Ladder. User beats tool: the user line ( \seatchip seatThreeFillseatThreeLineS3) beats the tool result ( \seatchip seatFourFillseatFourLineS4), both orders. Operator wins: in pinned pairs the operator line ( \seatchip seatOneFillseatOneLineS1 or \seatchip seatTwoFillseatTwoLineS2) always comes first; the rate over decided replies and over every usable reply (a reply where no single seat won counts as a loss). Grey brackets: 95% bootstrap intervals. The pinned operator wins more than half the time in 2 of 26 model-modes.
Figure 7: Seat against seat among the user, tool and memory lines. Share of decided free Ladder pairs won by the seat named first in each panel title, both orders sent equally often and pooled: (a) user over tool result, (b) user over memory note, (c) memory note over tool result (one minus the tool’s share). Each point is a model-mode. Bars: 95% bootstrap intervals; filled: thinking off; open: on; dashed: 50%, an even split; colours mark model families; sizes in billions of parameters.
Model
Think
Real, first
Label, first
Real, last
Label, last
Qwen 3
Qwen3-0.6B
off
9.85 [7.70, 12.08]
47.43 [43.09, 51.63]
35.66 [31.63, 39.66]
47.12 [43.14, 51.05]
on
18.15 [15.57, 20.88]
49.19 [45.62, 52.54]
38.75 [35.47, 42.09]
51.19 [47.83, 54.62]
Qwen3-1.7B
off
10.62 [8.65, 12.65]
36.81 [33.59, 39.85]
42.69 [39.44, 46.03]
43.03 [39.65, 46.39]
on
12.61 [10.70, 14.45]
44.13 [41.30, 46.96]
42.18 [39.46, 44.97]
51.10 [48.34, 54.03]
Qwen3-4B
off
12.57 [10.56, 14.60]
39.15 [35.96, 42.28]
43.47 [40.31, 46.65]
45.73 [42.24, 49.03]
Appendix
Table 12: The Operator test: operator wins in each layout. The operator line is sent in the real system message (real) or as the same labelled text in a user message (label), first or last, against the user, tool result or memory note. n/a : Qwen 3.5’s template refuses a late system message, so that layout does not exist for it. Grey brackets: 95% bootstrap intervals. Sent first, the real operator wins less than half the time in 24 of 26 model-modes.
Model
Think
Real − label, first
Real − label, last
Real: last − first
Qwen 3
Qwen3-0.6B
off
− 38.14 [ − 43.12, − 33.29]
− 9.13 [ − 12.25, − 6.33]
+ 23.91 [19.99, 28.14]
on
− 29.50 [ − 33.25, − 25.72]
− 12.41 [ − 15.42, − 9.50]
+ 19.79 [16.64, 22.98]
Qwen3-1.7B
off
− 25.13 [ − 28.50, − 21.81]
− 0.29 [ − 2.38, 1.70]
+ 31.27 [27.83, 34.61]
on
− 31.62 [ − 34.40, − 28.94]
− 9.09 [ − 11.53, − 6.57]
+ 29.45 [26.74, 32.14]
Qwen3-4B
off
− 26.21 [ − 29.40, − 23.01]
− 1.94 [ − 4.91, 0.93]
+ 31.27 [27.58, 34.89]
Appendix
Table 13: The Operator test: role, label and position. Differences in operator wins: real minus label with the operator first and last, and the real operator sent last minus sent first; n/a : not run, since Qwen 3.5’s template rejects a system message that is not first. Differences are paired within the scenario over scenarios decided in both layouts, so they can differ slightly from the difference of two printed rates. Grey brackets: 95% bootstrap intervals. The real role loses to the label at the first position in 24 of 26 model-modes.
Model
Think
Tool and file write
Order and refusal
Format
Qwen 3
Qwen3-0.6B
off
− 48.75 [ − 54.47, − 43.46]
− 12.19 [ − 22.41, − 3.39]
+ 0.30 [ − 12.26, 12.79]
on
− 46.46 [ − 51.79, − 41.20]
− 15.30 [ − 20.45, − 10.47]
+ 2.72 [ − 5.85, 11.00]
Qwen3-1.7B
off
− 31.87 [ − 36.58, − 27.34]
− 20.10 [ − 24.80, − 15.23]
− 0.99 [ − 10.41, 9.27]
on
− 37.81 [ − 41.91, − 33.89]
− 29.31 [ − 33.66, − 24.80]
− 3.57 [ − 11.46, 4.20]
Qwen3-4B
off
− 31.47 [ − 36.14, − 26.83]
− 24.41 [ − 29.24, − 19.41]
+ 2.64 [ − 3.62, 9.82]
Appendix
Table 14: The real-role penalty by task type. Real minus label with the operator first, by task type: tool calls and file writes pooled, step order and refusal pooled, and output format. Differences are paired within the scenario over scenarios decided in both layouts, so they can differ slightly from the difference of two printed rates. Grey brackets: 95% bootstrap intervals. The penalty is negative in tool and file-write tasks in 26 of 26 model-modes.
Model
Think
One message
Separate messages
One − separate
Qwen 3
Qwen3-0.6B
off
86.48 [82.95, 89.85]
48.48 [42.90, 54.08]
+ 35.46 [29.29, 41.42]
on
80.41 [76.90, 83.69]
45.74 [41.60, 50.06]
+ 34.66 [29.81, 39.82]
Qwen3-1.7B
off
90.29 [87.68, 92.70]
34.86 [31.00, 38.72]
+ 55.05 [50.07, 59.58]
on
86.17 [83.62, 88.76]
35.95 [32.99, 38.92]
+ 50.13 [45.79, 54.44]
Qwen3-4B
off
89.50 [86.54, 92.01]
35.66 [31.96, 39.11]
+ 54.38 [49.94, 58.55]
Appendix
Table 15: The Boundary test. The first-listed of the user line and the memory note wins when both are inside the task’s message (one message) or in separate messages, and the difference. Differences are paired within the scenario over scenarios decided in both layouts, so they can differ slightly from the difference of two printed rates. Grey brackets: 95% bootstrap intervals. The first line wins inside one message in 26 of 26 model-modes, and the advantage shrinks across messages in 26 of 26.
Operator first
Later wins
Model
Think
Bare
Context
Context − bare
Bare
Context
Context − bare
Qwen 3
Qwen3-0.6B
off
4.08
19.83
+ 16.62 [9.08, 24.68]
64.15
68.07
+ 4.04 [ − 3.10, 11.56]
on
2.10
12.57
+ 10.47 [5.60, 16.43]
54.13
68.77
+ 14.60 [7.48, 21.24]
Qwen3-1.7B
off
5.21
12.44
+ 6.08 [0.48, 11.73]
67.83
76.09
+ 8.66 [2.25, 15.78]
on
5.84
15.33
+ 9.49 [4.21, 14.67]
69.73
64.97
− 4.76 [ − 9.56, 0.38]
Appendix
Table 16: Context: realistic text around the same instructions. Operator first: the operator line, sent first, wins, bare and in context, and context minus bare. Later wins: the later line wins in pairs without the operator. Context − bare is paired within the item, over items decided in both versions, so it can differ slightly from the difference of the two printed rates; intervals are clustered by item. Grey brackets: 95% bootstrap intervals.
Log score (nats per reply)
Top-1 (%)
Model
Think
Uniform
Position
Task-aware
Luce
Seats
Reference
Position
Reference
Qwen 3
Qwen3-0.6B
off
− 1.447
− 1.216
− 1.187
− 1.287
− 1.208
− 1.036
47.60
50.57
on
− 1.435
− 1.269
− 1.235
− 1.262
− 1.225
− 1.102
49.19
50.62
Qwen3-1.7B
off
− 1.653
− 1.308
− 1.261
− 1.450
− 1.328
− 1.037
57.84
58.56
on
− 1.623
− 1.337
− 1.297
− 1.362
− 1.286
− 1.105
55.11
55.11
Appendix
Table 17: Forecasting triples and five-way sets from pairs. Each fit is made on the Ladder pairs and scored on the triples and five-way sets (usable replies; a reply where nothing won is an outcome). Position, Task-aware and Seats: the three fits of Section 4.4 ; Luce: seat strengths alone. Reference: the observed outcome frequencies of each set of seats in each order, pooled over scenarios and over where the task line is, the best any forecast that sees only the seats and their order can score. Bold: the best of the five fits. Top-1: share of replies whose winner the forecast ranks first. Of these five fits, the task-aware fit scores best in 19 of 26 model-modes.
Model
Think
Position share
Position D → H
Position H → D
Task-aware share
Seat gain (nats)
Gain D → H
Gain H → D
Qwen 3
Qwen3-0.6B
off
56.17
51.77
52.83
63.21
+ 0.009 [ − 0.002, 0.019]
+ 0.009
− 0.002
on
49.86
43.43
51.43
60.12
+ 0.044 [0.034, 0.054]
+ 0.050
+ 0.031
Qwen3-1.7B
off
55.97
50.95
55.24
63.58
− 0.020 [ − 0.030, − 0.011]
− 0.017
− 0.025
on
55.10
49.60
55.45
62.93
+ 0.051 [0.037, 0.065]
+ 0.058
+ 0.044
Qwen3-4B
off
51.06
46.22
50.12
60.08
− 0.013 [ − 0.021, − 0.003]
− 0.006
− 0.024
Appendix
Table 18: Share of the gap recovered, and what seat strengths add. Share of the gap from the uniform model (0%) to the reference (100%) recovered by the position and task-aware fits, fitted on all pairs and scored on all triples and five-way sets, and on held-out scenarios in both directions (fit on one half, score the other: D → H, H → D). Seat gain: the seats fit minus the position fit, in nats per reply, with its interval. Grey brackets: 95% bootstrap intervals. Seat strengths improve the forecast, with the interval above zero, in 15 of 26 model-modes.
Model
Think
Predicted
Observed
Observed − predicted
Obs. − pred., all usable
Qwen 3
Qwen3-0.6B
off
68.98
71.04
+ 2.06 [0.37, 3.85]
− 8.98 [ − 11.03, − 6.89]
on
68.04
67.84
− 0.19 [ − 1.89, 1.40]
− 10.15 [ − 11.88, − 8.46]
Qwen3-1.7B
off
68.58
69.26
+ 0.68 [ − 0.90, 2.18]
− 5.04 [ − 7.08, − 3.04]
on
68.41
69.84
+ 1.43 [0.09, 2.69]
− 4.92 [ − 6.77, − 3.07]
Qwen3-4B
off
68.54
75.39
+ 6.85 [5.38, 8.35]
− 0.14 [ − 2.13, 1.87]
Appendix
Table 19: Agreement: when several seats ask for the same option. The shared option’s share of decided replies as the seats fit predicts it and as observed, and observed minus predicted on decided and on all usable replies. The seats split into two camps over two options, the shared option is the larger camp’s, and sets with two camps of two are left out. Observed and predicted shares are over the same replies. Grey brackets: 95% bootstrap intervals. The shared option wins more than predicted in 13 of 26 model-modes.
Tier rule followed
Operator first, system message
Ladder conflicts
Model
On − off
Cut-off as loss
On − off
Cut-off as loss
Off-menu change
Cut-off change
Qwen 3
Qwen3-0.6B
+ 1.83 [ − 1.20, 4.77]
− 0.17
+ 5.57 [3.24, 8.09]
+ 3.70
+ 0.16
− 0.10
Qwen3-1.7B
− 1.21 [ − 3.81, 1.61]
− 4.92
+ 1.07 [ − 0.69, 2.72]
+ 1.67
+ 1.82
− 0.17
Qwen3-4B
+ 12.81 [9.93, 15.93]
+ 8.02
+ 9.02 [6.64, 11.39]
+ 7.96
+ 1.36
+ 0.13
Qwen3-8B
+ 11.05 [8.50, 13.49]
+ 4.60
+ 4.51 [2.57, 6.46]
+ 4.07
+ 1.33
+ 0.26
Appendix
Table 20: Thinking on minus off, per size. Paired within the scenario. Tier rule and operator wins are over decided replies, and again with every reply counted and a cut-off or unusable reply counted as a loss. Off-menu and cut-off changes are in the share of Ladder conflict replies. Paired differences can differ slightly from the difference of two rates in Table 21 . Grey brackets: 95% bootstrap intervals. Thinking raises tier following at 9 of 12 sizes.
Rule followed by tier gap
Model
Think
Rule followed
All usable
Gap 1
Gap 2
Gap 3
Gap 4
Gap 5
Qwen 3
Qwen3-0.6B
off
51.32 [47.59, 54.95]
25.68
50.33
50.64
52.66
51.26
54.71
on
52.91 [50.11, 55.77]
25.41
54.29
52.48
53.58
50.47
51.12
Qwen3-1.7B
off
57.12 [54.03, 60.19]
44.50
57.91
56.51
56.65
55.64
60.12
on
55.94 [53.58, 58.20]
39.62
55.31
54.86
56.32
58.56
57.05
Appendix
Table 21: Tiers: the declared rule and capacity. Rule followed: the option of the lower declared tier wins (decided replies; all usable replies beside it), overall and by the gap between the two tiers. Capacity R(0.95) , the largest set of tiers the model keeps in the declared order with reliability 0.95 (one draw per scenario and tier pair; simultaneous bounds), is bounded above by one tier in every row. Reliability counts every reply that does not choose one of the two options as a failure, so a model can follow the rule in most decided replies and still hold only one tier. The rule is followed more often than not in 24 of 26 model-modes; the upper bound on R(0.95) is one tier in 26 of 26.
Usable Ladder conflict replies
All replies
Model
Think
Chose
Unresolved
Off-menu
Multiple
No action
Unusable
Cut off
No action, all experiments
Qwen 3
Qwen3-0.6B
off
51.49
34.20
3.03
0.06
11.22
0.22
0.22
21.37
on
50.43
25.06
3.19
0.01
21.31
0.37
0.12
31.59
Qwen3-1.7B
off
74.73
17.92
2.16
0.24
4.95
0.92
0.52
4.28
on
78.25
16.06
3.99
0.20
1.50
1.04
0.36
2.10
Appendix
Table 22: What the replies did. Shares of Ladder conflict replies, in percent: the five outcomes among usable replies, and the unusable and cut-off shares among all replies. No action, all experiments: replies with no call of the kind the task reads (for gpt-oss mostly a request for missing content) among the usable replies of every experiment except Load. Rows may not sum to 100% because of rounding.
Figure 8: What the undecided replies did. Share of usable Ladder conflict replies that chose no single seat’s option, split into no action (no call of the kind the task is scored on, often a question back), unresolved (no checker could label the result), off-menu and multiple; the rest chose one seat’s option. Left bar of each size: thinking off; right: on, as the marker above each bar shows. gpt-oss has one bar per size (it always reasons). Marker colours and shapes mark model families; sizes in billions of parameters.
Model
Think
Flagged or hedged
Earlier wins: flagged
Earlier wins: silent
Flagged − silent
Qwen 3
Qwen3-0.6B
off
0.14
–
12.30
–
on
0.27
–
17.62
–
Qwen3-1.7B
off
0.26
–
10.77
–
on
1.38
15.63
14.40
+ 1.23 [ − 10.46, 17.24]
Qwen3-4B
off
0.59
42.86
12.91
+ 29.95 [ − 4.14, 57.08]
Appendix
Table 23: Replies that flag the conflict. Ladder pairs except the two system lines, decided replies: the share that flags or hedges the conflict, and how often the earlier line wins when the reply flags or hedges and when it stays silent (unpaired; the flagged rate and the difference are shown where at least ten flagged replies are decided; –: fewer). Flagging and hedging are marked by fixed keyword patterns in the answer text.
Model
Think
Groups chosen
n=8 : k=3−k=1
n=12 : k=6−k=1
k=1 : n=12−n=2
Qwen 3
Qwen3-0.6B
off
56.46
− 11.84 [ − 42.07, 22.88]
+ 16.67 [ − 25.00, 55.00]
− 30.00 [ − 66.67, 13.33]
on
36.95
− 4.29 [ − 34.05, 21.55]
− 18.33 [ − 35.00, − 3.33]
+ 36.32 [17.41, 56.67]
Qwen3-1.7B
off
88.47
− 6.67 [ − 21.67, 6.67]
− 2.07 [ − 23.21, 25.00]
+ 7.86 [ − 7.78, 32.14]
on
86.55
− 1.79 [ − 16.67, 17.48]
− 21.15 [ − 38.89, − 5.56]
+ 37.04 [14.81, 62.96]
Qwen3-4B
off
96.64
+ 3.45 [0.00, 10.34]
− 5.00 [ − 15.00, 0.00]
0.00 [0.00, 0.00]
Appendix
Table 24: Load: more conflicts or more instructions. Share of tool and file-write conflict groups in which the reply chose one of the group’s options (all trials), and contrasts at a fixed number of instructions n (more conflicting groups k ) and at one conflict (more instructions). Intervals are clustered by bundle (ten bundles), so they are wide; this experiment is descriptive. Differences are paired by bundle. Grey brackets: 95% bootstrap intervals.
Large language model agents receive instructions from many sources-system messages, user prompts, tool outputs, other agents, and more-each carrying different levels of trust and authority. When these instructions conflict, agents must reliably follow the highest-privilege instruction to remain safe and effective. The dominant paradigm, instruction hierarchy (IH), assumes a fixed, small set of privilege levels (typically fewer than five) defined by rigid role labels (e.g., system > user). This is inadequate for real-world agentic settings, where conflicts can arise across far more sources and contexts. In this work, we propose Many-Tier Instruction Hierarchy (ManyIH), a paradigm for resolving instruction conflicts among instructions with arbitrarily many privilege levels. We introduce ManyIH-Bench, the first benchmark for ManyIH. ManyIH-Bench requires models to navigate up to 12 levels of conflicting instructions with varying privileges, comprising 853 agentic tasks (427 coding and 426 instruction-following). ManyIH-Bench composes constraints developed by LLMs and verified by humans to create realistic and difficult test cases spanning 46 real-world agents. Our experiments show that even the current frontier models perform poorly (~40% accuracy) when instruction conflict scales. This work underscores the urgent need for methods that explicitly target fine-grained, scalable instruction conflict resolution in agentic settings.
We study how instruction-tuned LLMs arbitrate direct conflicts between system and user instructions. We introduce a benchmark of 41 paired constraints with deterministic verifiers and evaluate eight models under matched baseline, conflict, and same-channel control conditions. Behaviourally, the models split into three regimes by System Authority Delta: hierarchy-respecting models use the system channel as an authority signal, anti-hierarchy models follow the system less often than their same-channel baseline predicts, and no-effect models show little channel sensitivity. Llama-3.1-8B is the strongest anti-hierarchy case in our suite, following the system in only 0.10 of conflict trials. We use this behavioural failure case to ask whether user-preferring arbitration reflects the absence of an internal conflictresolution signal. It does not: on Llama-3.1-8B, the conflict outcome is linearly decodable from residual-stream activations at 0.97 balanced accuracy, 17 percentage points above a metadata-only baseline, with analogous signals on Qwen2.5-7B and gpt-oss-20b. Steering with a layer-12 mean of four per-conflict logistic-regression directions raises genuine system compliance from 0.132 to 0.530, while directions selected mainly for pooled separability steer poorly. User-preferring conflict resolution can therefore coexist with a readable internal arbitration signal, and successful intervention depends on the geometry of the readout rather than probe accuracy alone
Large language models (LLMs) often encounter conflicting prompts, although current instruction following benchmarks assess those meta-instructions in isolation, limiting the insights about how models process conflicting instructions. We introduce a framework PRIME(Prompt Resolution under Incompatible Meta-Instructions Evaluation) to analyze behavior of LLMs when provided with conflicting instructions. PRIME purposefully produces calibrated conflicts across response length, output format, and reasoning; classifying model responses with a deterministic behavioral taxonomy. We are evaluating five instruction tuned open weight LLMs in two distinct settings, balanced and naturally distributed. The conclusion we reach upon analysis is that conflict type is more significant in affecting behavior than model scale, and various failure modes across different categories of conflict. Our findings emphasize the value of developing conflict awareness and suggest ability of LLM to follow instructions cannot be assessed through isolated constraints alone.