Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable

Abhinav Rajeev Kumar; Paras Chopra

cs.LGSep 29, 2026

Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable

Authors: Abhinav Rajeev Kumar, Paras Chopra

Organizations: Lossfunk

Abstract

Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on their merits rather than deferring to the user. Yet the same models are far more compliant when a wrong answer is attributed to a verified source, which is how retrieval results, tool outputs, and grounded-search content often present information. We measure this gap across five open-weight families and three closed APIs. A single verified-source note endorsing a wrong answer flips 45-88% of baseline-correct responses in seven of eight models, and compliance rises with how authoritative the note sounds. Source deference and user agreement are not behaviorally interchangeable inside the model: on matched items with the same wrong answer, causal interventions can selectively suppress one without equally affecting the other. In three open-weight families, removing a fitted source direction lowers source compliance by 65-80 percentage points while removing a user or assistant direction has far smaller effects, and removing the user direction shows the reverse preference. A separately fitted intervention derived from source-versus-user cue activations moves compliance in both directions while leaving the prompt text unchanged. An authority direction fitted on trivia also transfers to PIQA and multi-turn SYCON dialogues without refitting, and removing it lowers wrong-source compliance by tens of percentage points in four of five families with no detected change in MMLU-Pro or GSM8K accuracy at our evaluation sizes. Source deference and user agreement therefore need separate evaluation.

Figures & tables

Figure 1: Verified-source cues induce more wrong answers than user cues in the models shown. (A) Accuracy drop when a verified-source cue endorses a wrong answer. (B) Change in wrong-answer rate, in percentage points, for source cues in red and user-domain-expert cues in blue on the same items. “No-cue” denotes the neutral baseline. Figure 4 gives the full eight-model comparison, including OLMo-2 and Gemini-3.1-Pro.

Model	Extraction $N$	Position	Layer
Qwen3.5-27B	1808	endorsed answer	L5
GPT-OSS-20B	791	endorsement end	L18
OLMo-3.1-32B	311	endorsement start	L5
OLMo-2-32B	275	answer position	L16
Gemma-4-26B-A4B	842	endorsement span	L22

Table 1: Per-model extraction setup. “Extraction

N

” is the number of items used to fit

\hat{v}_{\mathrm{auth}}

. Held-out AUROC identifies candidate layers and token positions, and intervention effect breaks ties. Intervention layers can differ and are listed in Appendix A .

	Open-weight					Closed API
Experiment	Qwen3.5	GPT-OSS	OLMo-2	OLMo-3.1	Gemma-4	GPT-5.4	Grok-4.20	Gemini
Behavioral override (Fig. 1 A)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Source-vs-user (Fig. 1 B)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Authority gradient	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	—	—	—
Forward steering (trivia + PIQA)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	inert	—	—	—
Source/user causal $2{\times}2$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	—	—	—	—

Table 2: Experimental scope.

\checkmark

= run; — = not run; inert = the tested direction does not change behavior at any tested layer and strength. Mechanistic experiments require residual-stream access.

Figure 2: Authority compliance is graded. Same item and wrong answer; only source wording changes. Hedged suggestions are weakest, verified attributions strongest. Items per cue level: GPT-OSS 810, Gemma-4 887, OLMo-2 320, Qwen3.5 1813, OLMo-3.1 504.

Model	Layer	$\alpha$	Matched flip
OLMo-3.1-32B	L15	1.0	58.8%
GPT-OSS-20B	L16	1.0	32.7%
OLMo-2-32B	L16	0.5	19.0%
Qwen3.5-27B	L2	1.0	16.1%
Gemma-4-26B-A4B	L15	1.0	10.6%

Table 3: Wrong-answer flips after adding the authority direction. Strongest cell in each model’s sweep.

Model (layer)	Removed vector	Source-wrong $\Delta$ pp [95% CI]	User-wrong $\Delta$ pp [95% CI]
Qwen3.5 (L5)	source	$-69.6$ [ $-74.0,-64.9$ ]	$-16.8$ [ $-23.5,-10.4$ ]
	user	$-10.8$ [ $-15.7,-6.1$ ]	$-41.6$ [ $-47.0,-36.3$ ]
	assistant	$+0.0$ [ $-1.7,+1.7$ ]	$-0.4$ [ $-2.0,+1.1$ ]
GPT-OSS (L16)	source	$-77.5$ [ $-81.2,-73.5$ ]	$+8.4$ [ $+2.3,+14.3$ ]
	user	$-9.4$ [ $-12.9,-5.9$ ]	$-16.1$ [ $-21.4,-10.6$ ]
	assistant	$+0.2$ [ $-1.2,+1.6$ ]	$+1.6$ [ $-1.4,+4.5$ ]

Table 4: Source and user directions have different causal effects. Change in wrong-answer compliance after projection removal at

\alpha=1

, in percentage points. Paired 95% intervals use 10,000 bootstrap samples over items parsed both before and after intervention; paired sample sizes range from 86 to 512. Matched cells are source removal on source-cued prompts and user removal on user-cued prompts. For OLMo-2, assistant removal alone has a large effect, so its source effect cannot be attributed to the source direction.

Model (layer)	Baseline gap	Source $\rightarrow$ user	User $\rightarrow$ source
Qwen3.5 (L5)	20.1 pp	$-11.0$ [ $-11.9,-10.1$ ]	$+11.8$ [ $+10.9,+12.7$ ]
GPT-OSS (L16)	50.0 pp	$-30.5$ [ $-31.6,-29.4$ ]	$+32.0$ [ $+30.8,+33.2$ ]
OLMo-3.1 (L22)	34.0 pp	$-19.4$ [ $-20.5,-18.3$ ]	$+20.3$ [ $+19.1,+21.5$ ]

Table 5: Attribution patching with reported-speech wording. Effects use the wrong-minus-correct margin on 512 held-out items per model. Paired 95% intervals use 10,000 bootstrap samples.

Figure 3: Authority and assistant-axis removal have different effects. Change in wrong-source compliance at $\alpha=1$ .

Model	Capitulation rate	$\Delta$ pp [95% CI]
GPT-OSS-20B	$39\%\to 91\%$	$+52.7$ [ $+33.0,+66.2$ ]
Qwen3.5-27B	$45\%\to 63\%$	$+18.6$ [ $+3.3,+32.7$ ]
OLMo-3.1-32B	$25\%\to 33\%$	$+7.9$ [ $-7.0,+22.1$ ]
OLMo-2-32B	$46\%\to 53\%$	$+7.4$ [ $-8.5,+22.7$ ]
Gemma-4-26B-A4B	$58\%\to 62\%$	$+3.1$ [ $-12.1,+18.1$ ]

Table 6: Transfer to multi-turn SYCON. Capitulation rate before

\to

after steering, at the setting with the largest increase in each model’s sweep. Changes are in percentage points with 95% Newcombe intervals.

Model	Trivia	PIQA
Qwen3.5	$42.7\to 5.2$	$57.3\to 4.5$
GPT-OSS	$59.9\to 44.0$	$87.6\to 73.0$
OLMo-3.1	$83.4\to 64.0$	$96.2\to 78.0$
OLMo-2	$82.8\to 53.3$	$89.0\to 66.5$
Gemma-4	$58.7\to 55.8$	$97.7\to 97.1$

Table 7: Wrong-source compliance under authority removal. Percentages before

\to

after removal at

\alpha=1

, except Gemma-4 at L18,

\alpha=0.5

. Table 29 gives intervals at

\alpha=0.75

for Qwen3.5 and

\alpha=1

for OLMo-2.

Model	Layer, multiplier	Wrong-source	Wrong-user
Qwen3.5	L49, $-2.0$	$72.3\to 61.5$	$52.1\to 51.8$
GPT-OSS	L16, $-1.5$	$75.4\to 72.1$	$25.4\to 25.1$
OLMo-3.1	L29, $-2.0$	$69.2\to 52.2$	$35.2\to 32.4$

Table 8: CAA with its own tuned layer and strength. Compliance percentages before

\to

after steering on 512 unseen items, scored by answer probability.

Model	MMLU-Pro $\Delta$ pp [95% CI]	GSM8K $\Delta$ pp [95% CI]
Qwen3.5	$-1.1$ [ $-8.9,+6.7$ ]	$-4.0$ [ $-13.6,+5.7$ ]
GPT-OSS	$-1.9$ [ $-10.0,+6.2$ ]	$+0.5$ [ $-5.8,+6.8$ ]
OLMo-3.1	$+0.0$ [ $-7.9,+7.9$ ]	$-0.5$ [ $-9.1,+8.1$ ]
OLMo-2	$-0.3$ [ $-7.9,+7.2$ ]	$+0.0$ [ $-6.5,+6.5$ ]
Gemma-4	$+0.1$ [ $-7.7,+8.0$ ]	$-1.5$ [ $-11.2,+8.2$ ]

Table 9: Accuracy changes under authority removal. MMLU-Pro with

n=300

and GSM8K with

n=200

per model. Changes are in percentage points with 95% confidence intervals.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Direction	Fitted from	Used for
$\hat{v}_{\mathrm{auth}}$	Correct- and wrong-source prompts vs. neutral prompts (Eq. 1 )	Steering, SYCON, PIQA transfer
$\hat{v}_{\mathrm{auth}\perp\mathrm{asst}}$	$\hat{v}_{\mathrm{auth}}$ with its assistant-axis component removed	Authority removal (mitigation)
$\hat{v}_{\mathrm{src}}$	Wrong-source prompts vs. neutral prompts	Source/user removal (Table 4 )
$\hat{v}_{\mathrm{usr}}$	Wrong-user prompts vs. neutral prompts	Source/user removal (Table 4 )
$\hat{v}_{\mathrm{asst}}$	Default-assistant vs. role-play activations [ 19 ]	Control
$\hat{v}_{\mathrm{attr}}$	Source-minus-user shift over the cue tokens, shared part removed (Eq. 3 )	Attribution patch

Appendix

Table 10: Directions used in the paper.

Model	Fit	Steer	Causal $2{\times}2$	Affect	Mitigation	SYCON
Qwen3.5	L5	L2	L5	L5	L5	L10
GPT-OSS	L18	L16	L16	L18	L16	L12
OLMo-2	L16	L16	L22	L16	L10	L22
OLMo-3.1	L5	L15	L22	L5	L15	L22
Gemma-4	L22	L15	—	—	L20	L24

Appendix

Table 11: Per-model layer choices across experiments. “Fit” uses extraction AUROC; the remaining columns report intervention layers. Dashes mark experiments not run. For Gemma-4, L20 is the control and capability setting; the mitigation result in Table 7 uses L18 at

\alpha=0.5

Family	Dataset	$n$	Layer	$\alpha$	Reference MF%	Held-out MF%	$\Delta$ pp
Qwen3.5	Trivia	200	L5	0.5	32.7	31.0	$-1.7$
Qwen3.5	PIQA	125	L5	0.7	20.6	20.0	$-0.6$
GPT-OSS	Trivia	222	L16	1.0	32.7	31.5	$-1.2$
GPT-OSS	PIQA	100	L16	1.0	18.8	19.0	$+0.2$
OLMo-2	Trivia	39	L10	1.0	19.0	23.1	$+4.1$
OLMo-2	PIQA	143	L16	0.5	10.4	10.5	$+0.1$

Appendix

Table 12: Held-out forward steering. Matched-flip rates (%) from stored comparisons. Layer and strength describe the held-out run, not necessarily its reference. The archived

n

values count baseline-correct items but cannot be verified against the later summary-only runs.

Family	Removal	Source-cued (ref/held/ $\Delta$ )			User-cued (ref/held/ $\Delta$ )
Qwen3.5	source	16.8	14.0	$-2.8$	42.7	49.0	$+6.3$
Qwen3.5	user	76.5	78.0	$+1.5$	16.9	18.5	$+1.6$
Qwen3.5	assistant	87.2	86.0	$-1.2$	59.3	58.5	$-0.8$
GPT-OSS	source	18.2	18.0	$-0.2$	42.4	41.0	$-1.4$
GPT-OSS	user	86.1	87.0	$+0.9$	17.8	18.0	$+0.2$
GPT-OSS	assistant	95.7	95.0	$-0.7$	35.4	35.0	$-0.4$

Appendix

Table 13: Held-out source/user removal at

\alpha=1

. Wrong-answer rates (%) under each removal, shown as reference / held-out / difference. Layers are Qwen3.5 L5, GPT-OSS L16, OLMo-2 L16, and OLMo-3.1 L15. The OLMo layers differ from the L22 tests in Table 4 .

Family	Variant	Trivia (ref/held/ $\Delta$ )			PIQA (ref/held/ $\Delta$ )
Qwen3.5	authority	5.2	6.6	$+1.4$	4.5	6.0	$+1.5$
Qwen3.5	assistant	43.0	43.4	$+0.4$	43.4	44.0	$+0.6$
GPT-OSS	authority	43.2	44.0	$+0.8$	6.2	7.0	$+0.8$
GPT-OSS	assistant	57.8	57.0	$-0.8$	57.5	58.0	$+0.5$
OLMo-2	authority	29.5	31.9	$+2.4$	28.5	30.0	$+1.5$
OLMo-2	assistant	36.8	37.5	$+0.7$	35.2	36.0	$+0.8$

Appendix

Table 14: Held-out mitigation on wrong-source prompts. Wrong-answer rates (%) at

\alpha=1

, shown as reference / held-out / difference. These stored comparisons are separate from the main mitigation runs in Table 7 .

Figure 4: Behavioral comparison across all eight models. Panel A: fraction of baseline-correct items that flip to the source-endorsed wrong answer; footers report each model’s baseline-correct share. Panel B: source vs. user effect on a matched 1024-item set; effect = wrong%(cue) $-$ wrong%(no cue), with no-cue wrong rate in the footer. Open-weight families are left of the divider, closed APIs right.

Model	Open	N parse%	N wrong%	W parse%	W wrong%	Flip rate
Qwen3.5-27B	yes	99.9	25.1	99.9	50.3	44.9%
GPT-OSS-20B	yes	98.5	22.9	98.0	71.9	65.4%
OLMo-2-32B	yes	91.9	18.2	91.6	66.0	82.8%
OLMo-3.1-32B	yes	95.0	53.2	97.4	90.5	82.9%
Gemma-4-26B-A4B	yes	97.2	10.0	97.8	66.8	62.9%
GPT-5.4	no	98.5	7.9	98.5	49.4	44.7%

Appendix

Table 15: Per-model behavioral numbers and parse rates. “N wrong%” is the model’s wrong rate on the neutral-note baseline, computed on parsed responses. “W wrong%” is the wrong rate when the verified-source endorses a wrong answer. Wrong-authority flip rate is the matched flip on the parsed-all-three subset (items parsed under all three conditions).

Figure 5: Authority projections onto the assistant axis. Projection of $\hat{v}_{\mathrm{auth}}$ onto the assistant axis $\hat{v}_{\mathrm{asst}}$ at the layers reported above, with fictional/persona and service/assistant roles for comparison.

Model	Layer	$\cos(\hat{v}_{\mathrm{auth}},\hat{v}_{\mathrm{val}})$	$\cos(\hat{v}_{\mathrm{auth}},\hat{v}_{\mathrm{aro}})$	VA-plane fraction
Qwen3.5-27B	L5	$+0.053$	$+0.032$	$0.062$
GPT-OSS-20B	L18	$-0.047$	$-0.015$	$0.049$
OLMo-2-32B	L16	$+0.004$	$+0.000$	$0.004$
OLMo-3.1-32B	L15	$-0.011$	$+0.025$	$0.027$
Gemma-4-26B-A4B	L15	$+0.005$	$+0.103$	$0.103$

Appendix

Table 16: Authority direction overlaps with the lexical valence/arousal plane only weakly. Cosines and norm fractions are computed at the layers listed below. “VA-plane fraction” is

\|\Pi_{\mathrm{VA}}\hat{v}_{\mathrm{auth}}\|

, the fraction of the authority direction’s unit norm that lies in the 2D affect subspace.

Intervention	Qwen3.5 (L5)	GPT-OSS (L18)	OLMo-2 (L16)	OLMo-3.1 (L5)
None (baseline)	$39.8$	$57.0$	$82.6$	$84.5$
Affect plane only	$40.6$	$56.8$	$82.8$	$84.5$
Authority $\cap$ affect	$39.2$	$58.2$	$82.8$	$83.9$
Authority $\perp$ affect	$\phantom{0}5.0$	$\phantom{0}5.0$	$30.1$	$12.3$

Appendix

Table 17: Causal lexical-affect control. Wrong-rate (%) on W prompts at

\alpha=1

. “Authority

\cap

affect” removes the authority component inside the valence/arousal plane; “Authority

\perp

affect” removes the component outside it. Parsed-subset sizes: Qwen3.5

n{=}496

–

498

, GPT-OSS

n{=}496

–

512

, OLMo-3.1

n{=}155

, OLMo-2

n{=}92

–

93

Figure 6: Factual-QA forward steering by layer and $\alpha$ . Matched-flip rate as a function of patch strength $\alpha\in\{0,0.3,0.5,0.7,1.0\}$ at three layers per model. Steering takes effect on Qwen, GPT-OSS, OLMo-2, and OLMo-3.1; Gemma-4 stays at baseline at every tested layer.

Figure 7: PIQA transfer by layer and $\alpha$ . Same vector as Figure 6 , applied to baseline-correct PIQA items without refitting. The four working families show monotone-or-near-monotone curves on the strongest layer; Gemma-4 stays near zero. OLMo-3.1’s PIQA parse rate under the dynamic parser starts around 41% and varies across $\alpha$ , so OLMo-3.1 numbers are reported as parsed-subset matched flips.

Model	Layer	$n$	Baseline	Source removal	Without assistant component
Qwen3.5	L5	244	86.5%	13.9%	21.3%
GPT-OSS	L16	244	95.9%	18.4%	26.6%
OLMo-3.1	L22	71	88.7%	23.9%	36.6%
OLMo-2	L22	40	87.5%	30.0%	35.0%

Appendix

Table 18: Source removal still works after taking out its assistant component, on held-out items. Wrong-source compliance at

\alpha=1

, computed on the common parsed subset for the three columns. Lower rates indicate less agreement with the wrong source.

Model	Layer	Source $\rightarrow$ user	User $\rightarrow$ source
Qwen3.5	L5	$-1.328$	$+1.328$
GPT-OSS	L16	$-1.024$	$+1.024$
OLMo-3.1	L22	$-1.212$	$+1.212$

Appendix

Table 19: Fixed attribution-patch settings. Layers match the source/user experiment in Table 4 . Coefficients are fixed from the 800-item fitting split.

Model	Layers	Source $\rightarrow$ user	User $\rightarrow$ source
Qwen3.5	L3 / L4 / L5 / L6 / L7	$-3.9/-8.1/\mathbf{-11.0}/-6.8/-3.6$	$+4.3/+8.8/\mathbf{+11.8}/+7.6/+3.9$
GPT-OSS	L14 / L15 / L16 / L17 / L18	$-12.0/-22.0/\mathbf{-30.5}/-23.0/-12.0$	$+12.6/+23.1/\mathbf{+32.0}/+24.2/+12.6$
OLMo-3.1	L20 / L21 / L22 / L23 / L24	$-7.0/-14.0/\mathbf{-19.4}/-14.0/-7.0$	$+7.4/+14.7/\mathbf{+20.3}/+14.7/+7.4$

Appendix

Table 20: Attribution-patch effects across nearby layers. Values are percentage-point shifts in margin-based compliance on the same 512 evaluation items. Bold marks the layer used in Table 4 .

Model	Wording	Baseline gap	Source $\rightarrow$ user	Gap closed	User $\rightarrow$ source
Qwen3.5	Direct	34.8	$-19.1$	54.9%	$+20.8$
	Reported speech	20.1	$-11.0$	54.7%	$+11.8$
	Structured fields	15.6	$-8.6$	55.1%	$+9.4$
GPT-OSS	Direct	86.5	$-52.4$	60.6%	$+55.1$
	Reported speech	50.0	$-30.5$	61.0%	$+32.0$
	Structured fields	38.9	$-23.3$	59.9%	$+24.6$

Appendix

Table 21: Attribution patch under three prompt wordings. Effects are percentage-point shifts. Every paired 95% interval excludes zero.

Model	Cue position	Baseline gap	Source $\rightarrow$ user	User $\rightarrow$ source	Gap closed
Qwen3.5	After options	20.1	$-11.0$	$+11.8$	55%
	Before question	19.0	$-10.5$	$+11.6$	55%
GPT-OSS	After options	50.0	$-30.5$	$+32.0$	61%
	Before question	47.8	$-28.7$	$+30.4$	60%
OLMo-3.1	After options	34.0	$-19.4$	$+20.3$	57%
	Before question	32.3	$-18.2$	$+19.3$	56%

Appendix

Table 22: Attribution patch with the cue after the options or before the question. Effects are percentage-point shifts on 512 evaluation items.

Model	Layer	$\cos(\hat{v}_{\mathrm{src}},\hat{v}_{\mathrm{asst}})$	$\cos(\hat{v}_{\mathrm{usr}},\hat{v}_{\mathrm{asst}})$	$\cos(\hat{v}_{\mathrm{src}},\hat{v}_{\mathrm{usr}})$
Qwen3.5	L5	$+0.005$	$-0.003$	$0.914$
GPT-OSS	L16	$+0.061$	$+0.052$	$0.995$
GPT-OSS	L18	$+0.056$	$+0.050$	$0.992$
OLMo-2	L10	$-0.018$	$-0.037$	$0.983$
OLMo-2	L16	$+0.005$	$-0.010$	$0.940$
OLMo-2	L22	$+0.011$	$-0.009$	$0.913$

Appendix

Table 23: Per-model per-layer cosines at the layers used for the source/user experiment. Cosines computed from the same per-model extraction checkpoint used for Table 4 . Layers used in Table 4 (Qwen3.5 L5, GPT-OSS L16, OLMo-2 L22, OLMo-3.1 L22) are bolded. Blank cells: extraction checkpoint not retained at that layer.

Model	L	flip 0 %	flip 0.3 %	flip 0.5 %	traj 0	traj 0.3	traj 0.5
Qwen3.5	2	43.5	50.6	40.7	3.20	2.97	3.33
Qwen3.5	5	44.7	47.0	31.0	3.20	3.12	3.41
Qwen3.5	10	44.7	47.1	63.3	3.18	3.16	2.48
GPT-OSS	12	38.8	24.5	91.4	1.95	2.07	0.73
GPT-OSS	16	38.5	45.8	66.7	2.09	1.85	0.97
GPT-OSS	18	34.7	44.7	65.2	2.01	1.75	1.47

Appendix

Table 24: Full SYCON layer

\times

\alpha

table per model. “flip%” is the dialogue-level any-flip rate among round-1-rejecting trajectories; “mean traj” is the mean trajectory score (higher = more rejects).

\alpha=0

is the unperturbed baseline at the corresponding layer.

Figure 8: SYCON multi-turn transfer. Forward-patching the trivia-fit authority direction at each assistant turn changes false-presupposition trajectories, most strongly for GPT-OSS and Qwen3.5.

Run	$n$	agree	agree%	Cohen $\kappa$	DS=1%	Gemini=1%
Qwen3.5	4498	3838	85.3%	0.61	79.4%	71.6%
GPT-OSS	6000	5014	83.6%	0.67	52.5%	55.6%
Gemma-4	4500	3696	82.1%	0.56	77.1%	67.0%
OLMo-2	4500	3925	87.2%	0.72	65.4%	64.0%
OLMo-3.1	4500	3920	87.1%	0.67	75.9%	71.6%

Appendix

Table 25: Cross-judge agreement, DeepSeek-V4-Pro vs. Gemini-3.1-flash-lite. Round-level binary labels (rejects vs. accepts the false presupposition) over all dialogues and rounds in each SYCON run. “DS=1%” and “Gemini=1%” are the marginal reject rates of each judge.

Model	Cell (L, $\alpha$ )	DS baseline $\to$ flip	DS $\Delta$ pp [95% CI]	Gemini $\Delta$ pp
GPT-OSS-20B	L12, 0.5	$35.9\%\to 91.9\%$	$+56.0$ [ $+35.5,+70.2$ ]	$+52.7$
Qwen3.5-27B	L10, 0.5	$31.4\%\to 71.6\%$	$+40.2$ [ $+25.6,+52.4$ ]	$+18.6$
OLMo-3.1-32B	L22, 0.5	$40.0\%\to 40.5\%$	$+0.5$ [ $-14.3,+15.1$ ]	$+7.9$
OLMo-2-32B	L22, 0.5	$56.1\%\to 55.8\%$	$-0.3$ [ $-15.4,+14.8$ ]	$+7.4$
Gemma-4-26B-A4B	L24, 0.5	$44.0\%\to 44.2\%$	$+0.1$ [ $-14.5,+14.7$ ]	$+3.1$

Appendix

Table 26: Selected SYCON settings under both judges. Same layer and strength as Table 6 . DeepSeek baseline and steered flip rates use dialogues rejecting the premise at round 1, with 95% Newcombe intervals for the change. The final column repeats the Gemini estimate for comparison.

Model	Cue surface	baseline	source $\Delta$	user $\Delta$	assistant $\Delta$
Qwen3.5-27B	inserted note	$84.9\%$	$-69.5\pm 0.5$	$-12.0\pm 0.4$	$-0.9\pm 0.6$
	system prompt	$26.8\%$	$-24.0\pm 0.7$	$-3.6\pm 0.0$	$-1.9\pm 0.5$
	RAG document	$41.0\%$	$-31.2\pm 0.6$	$-4.6\pm 0.5$	$-0.7\pm 0.5$
GPT-OSS-20B	inserted note	$98.4\%$	$-62.1\pm 1.8$	$-7.3\pm 0.8$	$-1.3\pm 0.9$
	system prompt	$75.3\%$	$-25.4\pm 1.6$	$-5.4\pm 0.5$	$-2.4\pm 0.8$
	RAG document	$70.3\%$	$-27.9\pm 1.2$	$-4.0\pm 1.2$	$-1.7\pm 0.8$

Appendix

Table 27: Source-direction removal on system-message and RAG cues. Change in wrong-answer rate (pp), mean

\pm

sample standard deviation across three decoding seeds at the selected layers. Baseline rates are averaged across seeds. Evaluation sizes are 256 items for Qwen3.5 and GPT-OSS, 47 for OLMo-2, and 79 for OLMo-3.1.

Model	Layer	MMLU-Pro $\alpha{=}0$ $\rightarrow$ $\alpha{=}1$	$\Delta$	GSM8K $\alpha{=}0$ $\rightarrow$ $\alpha{=}1$	$\Delta$
Qwen3.5-27B (auth)	L5	60.7 $\rightarrow$ 59.6	$-1.1$	46.5 $\rightarrow$ 42.5	$-4.0$
Qwen3.5-27B (asst)	L5	60.7 $\rightarrow$ 60.7	$\phantom{-}0.0$	46.5 $\rightarrow$ 46.5	$\phantom{-}0.0$
GPT-OSS-20B (auth)	L16	52.4 $\rightarrow$ 50.5	$-1.9$	88.5 $\rightarrow$ 89.0	$+0.5$
OLMo-2-32B (auth)	L10	34.7 $\rightarrow$ 34.7	$\phantom{-}0.0$	12.0 $\rightarrow$ 12.5	$+0.5$
OLMo-2-32B (asst)	L10	34.7 $\rightarrow$ 34.0	$-0.7$	12.0 $\rightarrow$ 11.5	$-0.5$
OLMo-3.1-32B (auth)	L15	44.3 $\rightarrow$ 44.7	$+0.3$	26.5 $\rightarrow$ 26.0	$-0.5$

Appendix

Table 28: Capability under projection-removal interventions. MMLU-Pro and GSM8K accuracy at

\alpha=1

and the listed layer. Gemma-4 uses L20 here, not the L18,

\alpha=0.5

setting in Table 7 . “

\Delta

” is the change vs

\alpha=0

Model	Task	Authority removal $\Delta$ pp [95% CI]
Qwen3.5	Trivia	$-37.8$ [ $-42.5,-33.0$ ]
Qwen3.5	PIQA	$-53.1$ [ $-62.5,-42.1$ ]
OLMo-2	Trivia	$-29.5$ [ $-41.5,-16.2$ ]
OLMo-2	PIQA	$-22.5$ [ $-30.4,-13.9$ ]

Appendix

Table 29: Authority-removal effects with 95% Newcombe intervals. Wrong-source compliance change at

\alpha=0.75

for Qwen3.5 and

\alpha=1

for OLMo-2. Qwen3.5 uses a different strength from Table 7 .

Explore similar work

How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement
Sep 7, 2026Riyadh Alnasser, Yusuf Mücahit Çetinkaya, Sumin Zhao +1Disagreement Responses
University of Edinburgh · Middle East Technical University
LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit
Apr 21, 2026Manav PandeySycophancy Confabulation
Georgia Institute of Technology
When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA
Sep 8, 2026Manikandan Ravikiran, Siddharth VohraMultiple-Choice Questions Correct
Indian Institute of Technology Mandi, India · Carnegie Mellon University Amazon Web Services AI Native Pittsburgh, PA, USA

Sep 7, 2026, cs.CLJ/K move · Enter open · S save

How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement

Riyadh Alnasser, Yusuf Mücahit Çetinkaya, Sumin Zhao, Tuğrulcan Elmas

University of Edinburgh · Middle East Technical University

Disagreement Responses Llm-As-A-Judge Artificial Intelligence Models Taxonomy

Open paper AskPDF arXiv

Large language models are increasingly used as sources of advice and information, including in high-stakes settings, yet little is known about how they respond to user disagreement. We study how a model manages its epistemic authority, referring here to its claim to knowledge, competence, or the right to advise, once a user challenges its answer. Building on Conversation Analysis, we introduce a taxonomy of six challenge types and a four-layer framework for analysing each response: whether the original claim is maintained or changed, where authority is located, how the disagreement is socially managed, and what kind of evidential support is offered. We construct a new dataset of 2,310 controlled challenge scenarios and 32,340 corresponding responses from 14 models, and analyse them using our framework with an LLM-as-judge pipeline, providing a vocabulary which future evaluation and benchmark design can build on. We find that models show conflicting behaviour: they validate users in 85% of responses but maintain their original claim in 65%. They explicitly apologise in 33% of responses, yet 59% of those apologies accompany maintenance of the original claim. They transfer authority most often in advice tasks, doing so in 28% of responses and reaching 57% in health advice and 49% in legal advice, compared with 6% in fact and 3% in explanation tasks. Abandonment of the original claim ranges from 0.8% for GPT-5.2 to 40% for DeepSeek 7B, while complete replacement of the original claim is rare overall at 1.5%.

Abstract

Explore similar work

How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement

LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit

When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA