Position, Role or Label? A Controlled Study of Instruction Conflicts in LLM Agents

Anish Laddha; Nitesh Pradhan; Gaurav Srivastava

cs.CLOct 23, 2026

Position, Role or Label? A Controlled Study of Instruction Conflicts in LLM Agents

Authors: Anish Laddha, Nitesh Pradhan, Gaurav Srivastava

Organizations: Department of Computer Science and Engineering, LNMIIT, India · Department of Computer Science, Virginia Tech, USA

Abstract

LLM agents take instructions from the operator, repository configuration, user, tool results and saved memory, which often conflict. Instruction-hierarchy benchmarks ask whether the higher-ranked source wins, but a source sets position, role and label at once. First, we build WhoWins, 1,080 code-scored scenarios in five task types, each with up to five mutually exclusive instructions, and seven controlled experiments that vary only how instructions are delivered. Second, we study 14 open-weight models (0.6B to 122B, thinking off and on) over 1,887,356 scored replies, checking main findings on held-out scenarios. Inside one message the first of two conflicting lines wins 72.42 to 94.66%, and across messages it keeps 22.95 to 62.65%. A policy sent first in the system (or developer) message loses most conflicts with later lines in every Qwen model but wins most on gpt-oss when it acts. In tool and file-write tasks it wins less often than the same text sent first as a labelled user message, in every model-mode. Third, we test whether pairs forecast larger conflicts; forecasts improve in every model-mode once they know where the task line is. Finally, by our keyword patterns at most 2.92% of decided pair replies flag or hedge a conflict.

Figures & tables

Figure 1: Overview. A scenario has a task and up to five options that no reply can satisfy together (here, five casings of one key). Each option is placed in a seat, a message with its own position, chat role and label; each experiment varies their delivery. 14 models reply (filled: thinking off; open: on), and code scores the option a reply followed. Right: per family, the median over model-modes and the range, as shares of decided replies (dashed: 50%), and the operator line’s real role minus label in tool and file-write tasks (dashed: 0).

Study	Sources	Order	Role/ label	Msg. split	Sizes	From pairs	Code only	Think off/on	Models
IHEval ( Zhang et al., 2025 )	S, U, H, T	✗	✗	✗	2	✗	✓	✗	13
Control Illusion ( Geng et al., 2026 )	S, U	$\sim$	$\sim$	$\sim$	2	✗	$\sim$	✗	6
IH-Benchmark ( McCauley et al., 2026 )	S, U, T	✗	✗	✗	2	✗	$\sim$	✓	37
ManyIH ( Zhang et al., 2026 )	tags	random	✗	✗	2–5	✗	$\sim$	✓	13
Choose Sides ( Balp-Straffon et al., 2026 )	S, U	✗	✗	$\sim$	2	✗	✓	✗	8
Authority Bias ( Kumar and Chopra, 2026 )	note , U	$\sim$	$\sim$	✗	n/a	✗	$\sim$	✗	8

Table 1: WhoWins against prior studies of conflicting instructions. One row per study, one column per design choice, ours last. Sources : S system or developer message (ours has two seats there), U user, H conversation history, T tool output, M saved memory note; ManyIH ranks lines by privilege tags in the prompt, and Authority Bias compares a “verified source” note with a user’s claim about a factual answer. Order : the same lines in both orders. Role/label : the same text in its real chat role and as labelled text. Msg. split : the same lines in one message and in separate messages. Sizes : conflicting lines in one trial. From pairs : forecasts larger conflicts from pairs. Code only : no LLM in scoring. Think off/on : the same model with thinking off and on. Models : as each paper counts them. ✓ yes;

\sim

partly; ✗ no; grey: set by design or not applicable. Appendix Q explains each entry.

Experiment	What moves	Held fixed	Trials	Replies	Where
Ladder	which seats compete (two, three and five at a time) and their order	text; system seats first; each option also asked alone	12,840	38,520	§ 4.2 , § 4.4
Operator	operator line in the real system message or as labelled user text; first or last	text; the opposing seat (user, tool or memory)	4,320 (3,240)	12,960 (9,720)	§ 4.3
Boundary	user line and memory note in separate messages or inside the task’s message; order	text; roles	1,080	3,240	§ 4.1
Context	instructions bare or inside realistic context; order	text; 50 tool and file-write items	1,200 (900)	3,600 (2,700)	§ 4.3
Agreement	two or more seats ask for the same option (3 to 5 seats)	text; seats dealt as in the Ladder	3,240	9,720	App. K
Load	number $n$ of instructions (2 to 12) and of conflicts	length at each $n$ ; one operator message	100	300	App. O

Table 2: The seven experiments: what each varies and what it holds fixed. Trials and replies are per model-mode; each trial is sent three times (Tiers: once). Parentheses: Qwen 3.5, whose template rejects a late system message. Every experiment covers all three model families. Where: the section that reports it.

		Position				Role and label			Seats	Rule	Forecast
Model	Think	Later wins	2nd sys. line	First, one msg.	First, sep. msgs.	Operator first	Real $-$ label ∗	Late op. gain	User $>$ tool	Tier rule	Seat gain
Qwen 3
Qwen3-0.6B	off	79.01	29.50	86.48	48.48	8.86	$-$ 48.75	$+$ 23.91	65.49	51.32	$+$ 0.009
	on	69.36	35.78	80.41	45.74	11.44	$-$ 46.46	$+$ 19.79	73.33	52.91	$+$ 0.044
Qwen3-1.7B	off	83.22	30.89	90.29	34.86	7.85	$-$ 31.87	$+$ 31.27	57.81	57.12	$-$ 0.020
	on	76.11	24.24	86.17	35.95	9.98	$-$ 37.81	$+$ 29.45	70.56	55.94	$+$ 0.051

Table 3: Who wins, per model-mode. Rates are percentages of decided replies; differences are absolute and paired within the scenario. Operator first: the operator line, first in the Ladder, against a later line. Real

-

label ∗ : in tool and file-write tasks, the operator line sent first in the real system message minus the same labelled user text. Seat gain: nats per reply that seat strengths add to the position fit. Red shades larger rates and positive differences, blue negative ones; grey: the 95% interval includes 50% or 0. n/a : not possible (the template rejects a late system message). Appendix H defines each column.

Figure 2: The first line wins inside one message and much less often across messages. Share of decided replies that follow the first-listed of two conflicting lines (user against memory note), with both lines inside the task’s message (upper point) or each in its own message before the task (lower point); a thin line joins the two points of a model-mode. Sizes in billions of parameters; filled: thinking off; open: on; colours mark model families; dashed: 50%. Bars: 95% bootstrap intervals.

Figure 3: Qwen 3 follows order, gpt-oss follows the developer message. Each point is a model-mode, with 95% intervals. Horizontal: how often the later of two lines wins when the user, tool and memory seats meet in both orders. Vertical: how often the operator line wins when it comes first. Marker size grows with parameters; open markers are thinking on; colours mark model families. Dashed: one minus the horizontal value, the first line’s share in those free pairs.

Figure 4: In tool and file-write tasks the real system message lowers the operator line’s wins in every model-mode. Win rate of the operator line sent first in the real system (developer) message minus the same labelled text sent first in a user message. Markers: tool-call and file-write tasks, with 95% intervals (filled: thinking off; open: on); grey ticks: all task types. Colours mark families; sizes in billions of parameters.

Figure 5: Seat strengths help on Qwen 3.5 and gpt-oss. Share of the gap from a uniform guess (0%) to the empirical reference (100%) recovered when forecasting three- and five-way conflicts from pairs. Bar: position fit; tick: task-aware fit; marker on a stem: seats fit (filled: thinking off; open: on). Colours mark families; sizes in billions of parameters.

Figure 6: Thinking helps Qwen follow declared tiers and raises the operator line on Qwen 3. Thinking off (filled) to on (open) for every Qwen size, over scenarios decided in both modes: (a) share that follows the rule that the lower number wins; (b) the operator line’s wins when sent first in the real system message (Operator test). Grey: the 95% interval of the change includes zero; colours mark families.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Task type	$n$	Key read	Example task	Two of its options
Tool	240	first tool call	Produce a coverage report for the billing library.	“Measure with cov_line.” / “Measure with cov_branch.”
File write	240	first write target	Stage the intermediate files while the job runs.	“Stage them in /scratch/a .” / “Stage them in /scratch/b .”
Format	240	a format label	Emit the response body for the /accounts endpoint.	“Name every field in snake_case.” / “Name every field in camelCase.”
Ordering	240	order of three steps	Get the repository into a releasable state.	“Run build, then test, then lint.” / “Run build, then lint, then test.”
Refusal	120	comply or decline	The scratch directory is full and the job is stalling.	“Clear /scratch and continue running.” / “Decline and report disk state.”

Appendix

Table 4: The five task types. Each tool, file-write, format and ordering scenario has five options that set the same key to different values; refusal has two. The tool, file-write and ordering keys are read from the trace of tool calls, the refusal key from the trace or, when neither tool is called, from a refusal in the reply, and the format key from the text the model writes.

Seat	Message	Label line
\seatchip seatOneFillseatOneLineS1 operator	system (developer on gpt-oss)	Operator policy for this deployment.
\seatchip seatTwoFillseatTwoLineS2 repository	the same message	Repository configuration for this workspace.
\seatchip seatThreeFillseatThreeLineS3 user	user	none
\seatchip seatFourFillseatFourLineS4 tool result	tool reply to config_read	Returned by the config_read tool.
\seatchip seatFiveFillseatFiveLineS5 memory	user	Your saved note from an earlier session.

Appendix

Table 5: The five seats. The task arrives as a user message that starts “Task: …” (in the Boundary test’s merged layout it shares that message with \seatchip seatThreeFillseatThreeLineS3 and \seatchip seatFiveFillseatFiveLineS5). It never shares the system message.

Model	Params	Active	Arch.	Precision	vLLM	Thinking
Qwen3-0.6B	0.6B	–	dense	float16	0.8.5.post1	off, on
Qwen3-1.7B	1.7B	–	dense	float16	0.8.5.post1	off, on
Qwen3-4B	4B	–	dense	float16	0.8.5.post1	off, on
Qwen3-8B	8B	–	dense	float16	0.8.5.post1	off, on
Qwen3-14B	14B	–	dense	float16	0.8.5.post1	off, on
Qwen3.5-0.8B	0.8B	–	dense	bfloat16	0.28.0	off, on

Appendix

Table 6: The models and how they were served. Parameters as on the model card; precision is the number format the weights were loaded in; vLLM is the serving software version. Qwen models run with thinking off and on; gpt-oss always reasons. a : the pre-release served the gpt-oss-120b Ladder and Operator experiments.

	Qwen, off	Qwen, on	gpt-oss
Thinking	off	on	high effort
Temperature	0.7	0.6	0.6
Top- $p$	0.8	0.95	0.95
Top- $k$	20	20	20
Min- $p$	0.0	0.0	0.0
Presence penalty	0.0	0.0	0.0

Appendix

Table 7: Decoding. One setting per mode for every Qwen size (the Qwen 3 model cards’ settings, with presence penalty 0); gpt-oss has no non-thinking mode and uses the thinking setting at high reasoning effort.

	World 1	World 2
$P(A\text{ beats }B)$	1/2	1/2
$P(A\text{ beats }C)$	7/10	7/10
$P(B\text{ beats }C)$	7/10	7/10
$P(A,B,C\text{ win the triple})$	1/2, 1/5, 3/10	1/5, 1/2, 3/10
Winner of the triple	$A$	$B$

Appendix

Table 8: Two worlds with the same pairs and different triples.

	Qwen 3				Qwen 3.5				gpt-oss
Finding	D	H	Both	$n$	D	H	Both	$n$	D	H	Both	$n$
Later line wins, free Ladder pairs	10	10	10	10	14	14	14	14	0	1	0	2
Second line of one system message loses	8	9	7	10	14	14	14	14	1	2	1	2
First-listed wins inside one message	10	10	10	10	14	14	14	14	2	2	2	2
First-listed loses across messages	8	8	8	10	8	8	8	14	0	0	0	2
One message minus separate messages	10	10	10	10	14	14	14	14	2	2	2	2

Appendix

Table 9: Each finding on the two halves of the scenarios. Model-modes (Qwen sizes, for the thinking rows) whose 95% interval excludes the null in the finding’s direction on the discovery half (D), on the held-out half (H) and on both;

n

: model-modes with the measure. Composition findings fit and score within each half. n/a : the measure does not apply (Qwen 3.5 has no late system message; gpt-oss has no thinking-off mode).

Model	Think	Later wins	2nd system line wins	Later wins: follows the task	Later wins: both before the task
Qwen 3
Qwen3-0.6B	off	79.01 [72.99, 85.25]	29.50 [16.90, 42.34]	86.29 [79.33, 92.28]	61.60 [46.72, 75.41]
	on	69.36 [64.09, 74.53]	35.78 [23.01, 49.52]	74.44 [68.44, 80.39]	54.47 [42.97, 65.08]
Qwen3-1.7B	off	83.22 [78.64, 87.57]	30.89 [20.00, 41.45]	92.51 [88.55, 95.89]	63.22 [52.89, 73.28]
	on	76.11 [71.78, 80.49]	24.24 [15.90, 32.65]	81.62 [77.10, 85.88]	63.39 [53.68, 72.18]
Qwen3-4B	off	82.44 [77.93, 86.81]	30.52 [20.70, 41.20]	91.24 [86.86, 95.09]	64.31 [54.12, 73.91]

Appendix

Table 10: Position in the Ladder. Later wins: the later of two lines wins in free pairs (user, tool result, memory note; both orders). 2nd system line: the second of the two system-message lines ( \seatchip seatOneFillseatOneLineS1 and \seatchip seatTwoFillseatTwoLineS2, each order half the time) wins. The last two columns split later wins by where the task line is: the later line follows the task (the task comes first or between the two lines), or both lines come before the task. Grey brackets: 95% bootstrap intervals. The later line wins in 25 of 26 model-modes; the second line of one system message loses in 26 of 26.

Model	Think	User beats tool	Operator wins (decided)	Operator wins (all usable)
Qwen 3
Qwen3-0.6B	off	65.49 [51.77, 78.77]	8.86 [6.15, 11.56]	4.80 [3.40, 6.28]
	on	73.33 [63.92, 82.12]	11.44 [8.61, 14.33]	5.99 [4.55, 7.55]
Qwen3-1.7B	off	57.81 [48.05, 67.58]	7.85 [5.80, 10.19]	6.36 [4.55, 8.21]
	on	70.56 [62.20, 78.72]	9.98 [7.91, 12.20]	8.22 [6.50, 10.15]
Qwen3-4B	off	54.69 [44.31, 65.12]	10.92 [8.62, 13.49]	9.09 [7.03, 11.21]

Appendix

Table 11: Seats in the Ladder. User beats tool: the user line ( \seatchip seatThreeFillseatThreeLineS3) beats the tool result ( \seatchip seatFourFillseatFourLineS4), both orders. Operator wins: in pinned pairs the operator line ( \seatchip seatOneFillseatOneLineS1 or \seatchip seatTwoFillseatTwoLineS2) always comes first; the rate over decided replies and over every usable reply (a reply where no single seat won counts as a loss). Grey brackets: 95% bootstrap intervals. The pinned operator wins more than half the time in 2 of 26 model-modes.

Figure 7: Seat against seat among the user, tool and memory lines. Share of decided free Ladder pairs won by the seat named first in each panel title, both orders sent equally often and pooled: (a) user over tool result, (b) user over memory note, (c) memory note over tool result (one minus the tool’s share). Each point is a model-mode. Bars: 95% bootstrap intervals; filled: thinking off; open: on; dashed: 50%, an even split; colours mark model families; sizes in billions of parameters.

Model	Think	Real, first	Label, first	Real, last	Label, last
Qwen 3
Qwen3-0.6B	off	9.85 [7.70, 12.08]	47.43 [43.09, 51.63]	35.66 [31.63, 39.66]	47.12 [43.14, 51.05]
	on	18.15 [15.57, 20.88]	49.19 [45.62, 52.54]	38.75 [35.47, 42.09]	51.19 [47.83, 54.62]
Qwen3-1.7B	off	10.62 [8.65, 12.65]	36.81 [33.59, 39.85]	42.69 [39.44, 46.03]	43.03 [39.65, 46.39]
	on	12.61 [10.70, 14.45]	44.13 [41.30, 46.96]	42.18 [39.46, 44.97]	51.10 [48.34, 54.03]
Qwen3-4B	off	12.57 [10.56, 14.60]	39.15 [35.96, 42.28]	43.47 [40.31, 46.65]	45.73 [42.24, 49.03]

Appendix

Table 12: The Operator test: operator wins in each layout. The operator line is sent in the real system message (real) or as the same labelled text in a user message (label), first or last, against the user, tool result or memory note. n/a : Qwen 3.5’s template refuses a late system message, so that layout does not exist for it. Grey brackets: 95% bootstrap intervals. Sent first, the real operator wins less than half the time in 24 of 26 model-modes.

Model	Think	Real $-$ label, first	Real $-$ label, last	Real: last $-$ first
Qwen 3
Qwen3-0.6B	off	$-$ 38.14 [ $-$ 43.12, $-$ 33.29]	$-$ 9.13 [ $-$ 12.25, $-$ 6.33]	$+$ 23.91 [19.99, 28.14]
	on	$-$ 29.50 [ $-$ 33.25, $-$ 25.72]	$-$ 12.41 [ $-$ 15.42, $-$ 9.50]	$+$ 19.79 [16.64, 22.98]
Qwen3-1.7B	off	$-$ 25.13 [ $-$ 28.50, $-$ 21.81]	$-$ 0.29 [ $-$ 2.38, 1.70]	$+$ 31.27 [27.83, 34.61]
	on	$-$ 31.62 [ $-$ 34.40, $-$ 28.94]	$-$ 9.09 [ $-$ 11.53, $-$ 6.57]	$+$ 29.45 [26.74, 32.14]
Qwen3-4B	off	$-$ 26.21 [ $-$ 29.40, $-$ 23.01]	$-$ 1.94 [ $-$ 4.91, 0.93]	$+$ 31.27 [27.58, 34.89]

Appendix

Table 13: The Operator test: role, label and position. Differences in operator wins: real minus label with the operator first and last, and the real operator sent last minus sent first; n/a : not run, since Qwen 3.5’s template rejects a system message that is not first. Differences are paired within the scenario over scenarios decided in both layouts, so they can differ slightly from the difference of two printed rates. Grey brackets: 95% bootstrap intervals. The real role loses to the label at the first position in 24 of 26 model-modes.

Model	Think	Tool and file write	Order and refusal	Format
Qwen 3
Qwen3-0.6B	off	$-$ 48.75 [ $-$ 54.47, $-$ 43.46]	$-$ 12.19 [ $-$ 22.41, $-$ 3.39]	$+$ 0.30 [ $-$ 12.26, 12.79]
	on	$-$ 46.46 [ $-$ 51.79, $-$ 41.20]	$-$ 15.30 [ $-$ 20.45, $-$ 10.47]	$+$ 2.72 [ $-$ 5.85, 11.00]
Qwen3-1.7B	off	$-$ 31.87 [ $-$ 36.58, $-$ 27.34]	$-$ 20.10 [ $-$ 24.80, $-$ 15.23]	$-$ 0.99 [ $-$ 10.41, 9.27]
	on	$-$ 37.81 [ $-$ 41.91, $-$ 33.89]	$-$ 29.31 [ $-$ 33.66, $-$ 24.80]	$-$ 3.57 [ $-$ 11.46, 4.20]
Qwen3-4B	off	$-$ 31.47 [ $-$ 36.14, $-$ 26.83]	$-$ 24.41 [ $-$ 29.24, $-$ 19.41]	$+$ 2.64 [ $-$ 3.62, 9.82]

Appendix

Table 14: The real-role penalty by task type. Real minus label with the operator first, by task type: tool calls and file writes pooled, step order and refusal pooled, and output format. Differences are paired within the scenario over scenarios decided in both layouts, so they can differ slightly from the difference of two printed rates. Grey brackets: 95% bootstrap intervals. The penalty is negative in tool and file-write tasks in 26 of 26 model-modes.

Model	Think	One message	Separate messages	One $-$ separate
Qwen 3
Qwen3-0.6B	off	86.48 [82.95, 89.85]	48.48 [42.90, 54.08]	$+$ 35.46 [29.29, 41.42]
	on	80.41 [76.90, 83.69]	45.74 [41.60, 50.06]	$+$ 34.66 [29.81, 39.82]
Qwen3-1.7B	off	90.29 [87.68, 92.70]	34.86 [31.00, 38.72]	$+$ 55.05 [50.07, 59.58]
	on	86.17 [83.62, 88.76]	35.95 [32.99, 38.92]	$+$ 50.13 [45.79, 54.44]
Qwen3-4B	off	89.50 [86.54, 92.01]	35.66 [31.96, 39.11]	$+$ 54.38 [49.94, 58.55]

Appendix

Table 15: The Boundary test. The first-listed of the user line and the memory note wins when both are inside the task’s message (one message) or in separate messages, and the difference. Differences are paired within the scenario over scenarios decided in both layouts, so they can differ slightly from the difference of two printed rates. Grey brackets: 95% bootstrap intervals. The first line wins inside one message in 26 of 26 model-modes, and the advantage shrinks across messages in 26 of 26.

		Operator first			Later wins
Model	Think	Bare	Context	Context $-$ bare	Bare	Context	Context $-$ bare
Qwen 3
Qwen3-0.6B	off	4.08	19.83	$+$ 16.62 [9.08, 24.68]	64.15	68.07	$+$ 4.04 [ $-$ 3.10, 11.56]
	on	2.10	12.57	$+$ 10.47 [5.60, 16.43]	54.13	68.77	$+$ 14.60 [7.48, 21.24]
Qwen3-1.7B	off	5.21	12.44	$+$ 6.08 [0.48, 11.73]	67.83	76.09	$+$ 8.66 [2.25, 15.78]
	on	5.84	15.33	$+$ 9.49 [4.21, 14.67]	69.73	64.97	$-$ 4.76 [ $-$ 9.56, 0.38]

Appendix

Table 16: Context: realistic text around the same instructions. Operator first: the operator line, sent first, wins, bare and in context, and context minus bare. Later wins: the later line wins in pairs without the operator. Context

-

bare is paired within the item, over items decided in both versions, so it can differ slightly from the difference of the two printed rates; intervals are clustered by item. Grey brackets: 95% bootstrap intervals.

		Log score (nats per reply)						Top-1 (%)
Model	Think	Uniform	Position	Task-aware	Luce	Seats	Reference	Position	Reference
Qwen 3
Qwen3-0.6B	off	$-$ 1.447	$-$ 1.216	$-$ 1.187	$-$ 1.287	$-$ 1.208	$-$ 1.036	47.60	50.57
	on	$-$ 1.435	$-$ 1.269	$-$ 1.235	$-$ 1.262	$-$ 1.225	$-$ 1.102	49.19	50.62
Qwen3-1.7B	off	$-$ 1.653	$-$ 1.308	$-$ 1.261	$-$ 1.450	$-$ 1.328	$-$ 1.037	57.84	58.56
	on	$-$ 1.623	$-$ 1.337	$-$ 1.297	$-$ 1.362	$-$ 1.286	$-$ 1.105	55.11	55.11

Appendix

Table 17: Forecasting triples and five-way sets from pairs. Each fit is made on the Ladder pairs and scored on the triples and five-way sets (usable replies; a reply where nothing won is an outcome). Position, Task-aware and Seats: the three fits of Section 4.4 ; Luce: seat strengths alone. Reference: the observed outcome frequencies of each set of seats in each order, pooled over scenarios and over where the task line is, the best any forecast that sees only the seats and their order can score. Bold: the best of the five fits. Top-1: share of replies whose winner the forecast ranks first. Of these five fits, the task-aware fit scores best in 19 of 26 model-modes.

Model	Think	Position share	Position D $\to$ H	Position H $\to$ D	Task-aware share	Seat gain (nats)	Gain D $\to$ H	Gain H $\to$ D
Qwen 3
Qwen3-0.6B	off	56.17	51.77	52.83	63.21	$+$ 0.009 [ $-$ 0.002, 0.019]	$+$ 0.009	$-$ 0.002
	on	49.86	43.43	51.43	60.12	$+$ 0.044 [0.034, 0.054]	$+$ 0.050	$+$ 0.031
Qwen3-1.7B	off	55.97	50.95	55.24	63.58	$-$ 0.020 [ $-$ 0.030, $-$ 0.011]	$-$ 0.017	$-$ 0.025
	on	55.10	49.60	55.45	62.93	$+$ 0.051 [0.037, 0.065]	$+$ 0.058	$+$ 0.044
Qwen3-4B	off	51.06	46.22	50.12	60.08	$-$ 0.013 [ $-$ 0.021, $-$ 0.003]	$-$ 0.006	$-$ 0.024

Appendix

Table 18: Share of the gap recovered, and what seat strengths add. Share of the gap from the uniform model (0%) to the reference (100%) recovered by the position and task-aware fits, fitted on all pairs and scored on all triples and five-way sets, and on held-out scenarios in both directions (fit on one half, score the other: D

\to

H, H

\to

D). Seat gain: the seats fit minus the position fit, in nats per reply, with its interval. Grey brackets: 95% bootstrap intervals. Seat strengths improve the forecast, with the interval above zero, in 15 of 26 model-modes.

Model	Think	Predicted	Observed	Observed $-$ predicted	Obs. $-$ pred., all usable
Qwen 3
Qwen3-0.6B	off	68.98	71.04	$+$ 2.06 [0.37, 3.85]	$-$ 8.98 [ $-$ 11.03, $-$ 6.89]
	on	68.04	67.84	$-$ 0.19 [ $-$ 1.89, 1.40]	$-$ 10.15 [ $-$ 11.88, $-$ 8.46]
Qwen3-1.7B	off	68.58	69.26	$+$ 0.68 [ $-$ 0.90, 2.18]	$-$ 5.04 [ $-$ 7.08, $-$ 3.04]
	on	68.41	69.84	$+$ 1.43 [0.09, 2.69]	$-$ 4.92 [ $-$ 6.77, $-$ 3.07]
Qwen3-4B	off	68.54	75.39	$+$ 6.85 [5.38, 8.35]	$-$ 0.14 [ $-$ 2.13, 1.87]

Appendix

Table 19: Agreement: when several seats ask for the same option. The shared option’s share of decided replies as the seats fit predicts it and as observed, and observed minus predicted on decided and on all usable replies. The seats split into two camps over two options, the shared option is the larger camp’s, and sets with two camps of two are left out. Observed and predicted shares are over the same replies. Grey brackets: 95% bootstrap intervals. The shared option wins more than predicted in 13 of 26 model-modes.

	Tier rule followed		Operator first, system message		Ladder conflicts
Model	On $-$ off	Cut-off as loss	On $-$ off	Cut-off as loss	Off-menu change	Cut-off change
Qwen 3
Qwen3-0.6B	$+$ 1.83 [ $-$ 1.20, 4.77]	$-$ 0.17	$+$ 5.57 [3.24, 8.09]	$+$ 3.70	$+$ 0.16	$-$ 0.10
Qwen3-1.7B	$-$ 1.21 [ $-$ 3.81, 1.61]	$-$ 4.92	$+$ 1.07 [ $-$ 0.69, 2.72]	$+$ 1.67	$+$ 1.82	$-$ 0.17
Qwen3-4B	$+$ 12.81 [9.93, 15.93]	$+$ 8.02	$+$ 9.02 [6.64, 11.39]	$+$ 7.96	$+$ 1.36	$+$ 0.13
Qwen3-8B	$+$ 11.05 [8.50, 13.49]	$+$ 4.60	$+$ 4.51 [2.57, 6.46]	$+$ 4.07	$+$ 1.33	$+$ 0.26

Appendix

Table 20: Thinking on minus off, per size. Paired within the scenario. Tier rule and operator wins are over decided replies, and again with every reply counted and a cut-off or unusable reply counted as a loss. Off-menu and cut-off changes are in the share of Ladder conflict replies. Paired differences can differ slightly from the difference of two rates in Table 21 . Grey brackets: 95% bootstrap intervals. Thinking raises tier following at 9 of 12 sizes.

				Rule followed by tier gap
Model	Think	Rule followed	All usable	Gap 1	Gap 2	Gap 3	Gap 4	Gap 5
Qwen 3
Qwen3-0.6B	off	51.32 [47.59, 54.95]	25.68	50.33	50.64	52.66	51.26	54.71
	on	52.91 [50.11, 55.77]	25.41	54.29	52.48	53.58	50.47	51.12
Qwen3-1.7B	off	57.12 [54.03, 60.19]	44.50	57.91	56.51	56.65	55.64	60.12
	on	55.94 [53.58, 58.20]	39.62	55.31	54.86	56.32	58.56	57.05

Appendix

Table 21: Tiers: the declared rule and capacity. Rule followed: the option of the lower declared tier wins (decided replies; all usable replies beside it), overall and by the gap between the two tiers. Capacity

R(0.95)

, the largest set of tiers the model keeps in the declared order with reliability 0.95 (one draw per scenario and tier pair; simultaneous bounds), is bounded above by one tier in every row. Reliability counts every reply that does not choose one of the two options as a failure, so a model can follow the rule in most decided replies and still hold only one tier. The rule is followed more often than not in 24 of 26 model-modes; the upper bound on

R(0.95)

is one tier in 26 of 26.

		Usable Ladder conflict replies					All replies
Model	Think	Chose	Unresolved	Off-menu	Multiple	No action	Unusable	Cut off	No action, all experiments
Qwen 3
Qwen3-0.6B	off	51.49	34.20	3.03	0.06	11.22	0.22	0.22	21.37
	on	50.43	25.06	3.19	0.01	21.31	0.37	0.12	31.59
Qwen3-1.7B	off	74.73	17.92	2.16	0.24	4.95	0.92	0.52	4.28
	on	78.25	16.06	3.99	0.20	1.50	1.04	0.36	2.10

Appendix

Table 22: What the replies did. Shares of Ladder conflict replies, in percent: the five outcomes among usable replies, and the unusable and cut-off shares among all replies. No action, all experiments: replies with no call of the kind the task reads (for gpt-oss mostly a request for missing content) among the usable replies of every experiment except Load. Rows may not sum to 100% because of rounding.

Figure 8: What the undecided replies did. Share of usable Ladder conflict replies that chose no single seat’s option, split into no action (no call of the kind the task is scored on, often a question back), unresolved (no checker could label the result), off-menu and multiple; the rest chose one seat’s option. Left bar of each size: thinking off; right: on, as the marker above each bar shows. gpt-oss has one bar per size (it always reasons). Marker colours and shapes mark model families; sizes in billions of parameters.

Model	Think	Flagged or hedged	Earlier wins: flagged	Earlier wins: silent	Flagged $-$ silent
Qwen 3
Qwen3-0.6B	off	0.14	–	12.30	–
	on	0.27	–	17.62	–
Qwen3-1.7B	off	0.26	–	10.77	–
	on	1.38	15.63	14.40	$+$ 1.23 [ $-$ 10.46, 17.24]
Qwen3-4B	off	0.59	42.86	12.91	$+$ 29.95 [ $-$ 4.14, 57.08]

Appendix

Table 23: Replies that flag the conflict. Ladder pairs except the two system lines, decided replies: the share that flags or hedges the conflict, and how often the earlier line wins when the reply flags or hedges and when it stays silent (unpaired; the flagged rate and the difference are shown where at least ten flagged replies are decided; –: fewer). Flagging and hedging are marked by fixed keyword patterns in the answer text.

Model	Think	Groups chosen	$n{=}8$ : $k{=}3$ $-$ $k{=}1$	$n{=}12$ : $k{=}6$ $-$ $k{=}1$	$k{=}1$ : $n{=}12$ $-$ $n{=}2$
Qwen 3
Qwen3-0.6B	off	56.46	$-$ 11.84 [ $-$ 42.07, 22.88]	$+$ 16.67 [ $-$ 25.00, 55.00]	$-$ 30.00 [ $-$ 66.67, 13.33]
	on	36.95	$-$ 4.29 [ $-$ 34.05, 21.55]	$-$ 18.33 [ $-$ 35.00, $-$ 3.33]	$+$ 36.32 [17.41, 56.67]
Qwen3-1.7B	off	88.47	$-$ 6.67 [ $-$ 21.67, 6.67]	$-$ 2.07 [ $-$ 23.21, 25.00]	$+$ 7.86 [ $-$ 7.78, 32.14]
	on	86.55	$-$ 1.79 [ $-$ 16.67, 17.48]	$-$ 21.15 [ $-$ 38.89, $-$ 5.56]	$+$ 37.04 [14.81, 62.96]
Qwen3-4B	off	96.64	$+$ 3.45 [0.00, 10.34]	$-$ 5.00 [ $-$ 15.00, 0.00]	0.00 [0.00, 0.00]

Appendix

Table 24: Load: more conflicts or more instructions. Share of tool and file-write conflict groups in which the reply chose one of the group’s options (all trials), and contrasts at a fixed number of instructions

n

(more conflicting groups

k

) and at one conflict (more instructions). Intervals are clustered by bundle (ten bundles), so they are wide; this experiment is descriptive. Differences are paired by bundle. Grey brackets: 95% bootstrap intervals.

LLM Agents

Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.

See all 705 papers

Many-Tier Instruction Hierarchy in LLM Agents
Apr 10, 2026Jingyu Zhang, Tianjian Li, William Jurayj +3LLM Agents AI Agent Safety Benchmarks
Johns Hopkins University
How Language Models Choose Sides: Internal Representations of Instruction Hierarchy
Aug 16, 2026Enrique Balp-Straffon, Chih-Hao Hsu, Rushiraj Gadhvi +3Causal Interventions in Language Models Language Model Probing
Amazon · National Taiwan University · Plaksha University, Mohali, India +3
PRIME: Evaluating Prompt Resolution Under Incompatible Instructions in LLMs
Jun 21, 2026Tehreem Javed, Shumaim Fatimah, Masooma Bakhtiari +2LLM Evaluation Language Model Generation Evaluation
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan

Apr 10, 2026J/K move · Enter open · S save

Many-Tier Instruction Hierarchy in LLM Agents

Jingyu Zhang, Tianjian Li, William Jurayj, Hongyuan Zhan+2

Johns Hopkins University

LLM Agents AI Agent Safety Benchmarks LLM Agent Evaluation

Open paper AskPDF arXiv

Large language model agents receive instructions from many sources-system messages, user prompts, tool outputs, other agents, and more-each carrying different levels of trust and authority. When these instructions conflict, agents must reliably follow the highest-privilege instruction to remain safe and effective. The dominant paradigm, instruction hierarchy (IH), assumes a fixed, small set of privilege levels (typically fewer than five) defined by rigid role labels (e.g., system > user). This is inadequate for real-world agentic settings, where conflicts can arise across far more sources and contexts. In this work, we propose Many-Tier Instruction Hierarchy (ManyIH), a paradigm for resolving instruction conflicts among instructions with arbitrarily many privilege levels. We introduce ManyIH-Bench, the first benchmark for ManyIH. ManyIH-Bench requires models to navigate up to 12 levels of conflicting instructions with varying privileges, comprising 853 agentic tasks (427 coding and 426 instruction-following). ManyIH-Bench composes constraints developed by LLMs and verified by humans to create realistic and difficult test cases spanning 46 real-world agents. Our experiments show that even the current frontier models perform poorly (~40% accuracy) when instruction conflict scales. This work underscores the urgent need for methods that explicitly target fine-grained, scalable instruction conflict resolution in agentic settings.

Abstract

LLM Agents

Explore similar work

Many-Tier Instruction Hierarchy in LLM Agents

How Language Models Choose Sides: Internal Representations of Instruction Hierarchy

PRIME: Evaluating Prompt Resolution Under Incompatible Instructions in LLMs