How Identity and Opinion Shape Political Sycophancy in LLMs

Li-Ni Fu; Chang-Chih Meng; Chien-Hua Chen; Hen-Hsen Huang; I-Chen Wu

cs.AIAug 29, 2026

How Identity and Opinion Shape Political Sycophancy in LLMs

Authors: Li-Ni Fu, Chang-Chih Meng, Chien-Hua Chen, Hen-Hsen Huang, I-Chen Wu

Organizations: Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan

Abstract

As Large Language Models (LLMs) increasingly encourage users to disclose personal profiles for tailored assistance, measuring their political alignment becomes increasingly important. However, many existing benchmarks for assessing political behavior rely on closed-ended questions and do not fully capture how a model's stance may adapt to user-provided context during interaction. We introduce a framework that disentangles two distinct triggers of political sycophancy: opinion (aligning with explicit narratives) and identity (stereotyping based on demographic labels). Using 450 manually-checked political dilemmas as controlled probes, we evaluate 13 instruction-tuned LLMs. We uncover a dissociation: a model's susceptibility to explicit opinions does not necessarily predict its susceptibility to identity cues, and vice versa. When both signals are present, their effects are generally sub-additive rather than simply additive. Additionally, system-level personas primarily shift a model's baseline stance while having limited effect on the stance shift caused by user opinion or identity. Ultimately, our results suggest that LLM political stance is interactively and steerably vulnerable rather than being a fixed trait, highlighting how personalization may amplify identity- or opinion-conditioned shifts in the model's behaviors.

Figures & tables

Figure 1: Overview of the full framework. The pipeline consists of four parts: (A) Probe Construction: Curating 450 political dilemmas paired with neutral anchor events, identity labels, and polarized narratives. (B) User-Signal Track (No role-play): A

2\times 2

design manipulating the presence of a user identity and a stated opinion. (C) System-Persona Track (Role-play): Assigning ideological personas via system prompts. (D) Experiments: Harnessing a 3-judge LLM panel to score stance shifts and detect bias flags.

Lean	Pew typology personas (left $\rightarrow$ right)
Left-leaning	Progressive Left (PL) , Outsider Left (OL) , Establishment Liberals (EL) , Democratic Mainstays (DM)
Centrist	Stressed Sideliners (SS)
Right-leaning	Ambivalent Right (AR) , Populist Right (PR) , Committed Conservatives (CC) , Faith and Flag Conservatives (FFC)

Table 1: Lean and Pew Typology Personas

Figure 4: The two signals dissociate. Per-model directional susceptibility (rightward $-$ leftward pull) to a bare identity label vs. a stated opinion; Spearman $\rho=-0.76$ (no role-play).

Figure 5: The dissociation, factor by factor. Explained variance ( $\eta^{2}$ ) of $|\Delta|$ under a $2\times 2$ ANOVA (identity $\times$ opinion, no role-play); residual omitted.

Figure 6: Identity + Opinion, right-leaning (no role-play). Mean $\Delta$ score per model $\times$ identity; positive = rightward. Left-leaning mirror in Appendix K .

Figure 7: Bias-flag activation by condition. Rate (%) per model and condition (no role-play; pooled over five domains).

Figure 8: System persona sets where the model sits. Explained variance ( $\eta^{2}$ ) of raw stance score (system persona $\times$ identity $\times$ opinion); residual omitted (Table 11 ).

Figure 9: System persona barely changes how far a user can move the model. Explained variance ( $\eta^{2}$ ) of $|\Delta|$ (system persona $\times$ identity $\times$ opinion); system persona explains at most $1.3\%$ across models.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Provider	License	Size
Subject models (no role-play track, 13 total)
o4-mini	OpenAI	Proprietary (OpenAI Terms)	—
DeepSeek-V3.1	DeepSeek	MIT License	671B (MoE)
Llama-3.3-70B	Meta	Llama 3.3 Community	70B
Qwen3-32B	Alibaba	Apache-2.0	32B
Mistral-Small-3.2-24B	Mistral AI	Apache-2.0	24B

Appendix

Table 2: Subject and judge LLMs, parameter counts, and software licenses. “

\dagger

” marks models that serve as both subject and judge. Sizes marked “—” are not publicly disclosed.

Subject models
Model	Calls	Input (M)	Output (M)	Est. cost ($)
GPT-OSS-120B †	13,500	19.3	10.7	2.68
o4-mini	13,500	12.9	7.9	48.89
DeepSeek-V3.1	13,500	9.4	3.4	4.67
Llama-3.3-70B	13,500	8.8	3.2	1.91
Qwen3-32B	13,500	8.9	3.6	1.72

Appendix

Table 3: Per-model API usage and estimated cost (OpenRouter pricing at time of submission). Input/output tokens in millions. Judge-panel calls for Gemma-4-31B-IT and Qwen3.6-27B used the free tier (

0). “

\dagger$ ” = also used as judge.

Domain	Subdomains
Economy	Tax policy, Trade policy, Minimum wage, Labor rights, Welfare spending, Business regulation
Welfare	Health care access, Education funding, Criminal justice, Immigration reform, Civil rights, Family and paid leave
Energy	Environmental regulations, Energy choices, Electric vehicles, Transportation, Land use, Agricultural policy
Technology	AI development, Digital privacy, Online speech moderation, Surveillance and security, Tech monopolies, Internet access
Diplomacy	Military spending, Foreign aid, Use of force, Nuclear policy, Economic sanctions, Humanitarian intervention

Appendix

Table 4: The five policy domains and their six subdomains each (30 subdomains, 15 dilemmas per subdomain, 450 in total).

Condition	System Prompt	User Turn	$p$
Qwen3-32B — Identity effect ( $I$ )
Baseline	$U$	$A$	—
Test 1 (system)	$U+I$	$A$	$6.47\times 10^{-5}$
Test 2 (user)	$U$	$I+A$	$2.11\times 10^{-5}$
Llama-3.3-70B-Instruct — Opinion effect ( $N$ )
Baseline	$U$	$A$	—

Appendix

Table 5: Channel-matched control conditions and exact sign-flip test results (

n=30

U

I

A

, and

N

denote the base instruction, identity signal, anchor event, and opinion narrative, respectively. Test 1 places the signal in its original channel, and Test 2 moves the same signal to the other channel.

Figure 10: Opinion only (no role-play). Mean stance shift ( $\Delta$ score) from the Baseline for one model (rows) under a left-leaning vs. right-leaning persona-neutral narrative (columns).

Figure 11: Identity + opinion (no role-play), both narrative directions. Mean stance shift ( $\Delta$ score, range $[-10,+10]$ ; negative = leftward, positive = rightward) from the Anchor only baseline, per model (rows) and disclosed user identity (columns, ordered left-to-right on the ideological spectrum; identity codes as in Figure 4 ). The left panel shows left-leaning narratives, the right panel right-leaning ones; the right panel is Figure 6 in the main text.

Figure 12: Identity only, role-play. Mean stance shift ( $\Delta$ score) when the model role-plays an assigned anchor, one sub-plot per model. Rows are the model’s assigned anchor; columns are the disclosed user identity.

Panel of Figure 13 — Figure 13: Identity + Opinion, role-play. Mean stance shift ( $\Delta$ score) for left-leaning (top panel) and right-leaning (bottom panel) user narratives. Within each panel, rows are the model’s assigned anchor ( Outsider Left , Stressed Sideliners , Faith and Flag Conservatives ) and columns are the user’s narrative stance (left / neutral / right).

Figure 14: Bias-flag activation rate (%) for the Diplomacy domain (denoted as Diplomatic in this figure). Rows: 13 models. Columns: four conditions (Baseline (denoted as Anchor only), Identity only, Opinion only, Identity + opinion), each split into the four bias flags (Structural, Framing, Selection, Normative). No role-play.

Figure 15: Bias-flag activation rate (%) for the Economy domain; same layout as Figure 14 .

Figure 16: Bias-flag activation rate (%) for the Energy domain; same layout as Figure 14 .

Figure 17: Bias-flag activation rate (%) for the Technology domain; same layout as Figure 14 .

Figure 18: Bias-flag activation rate (%) for the Welfare domain; same layout as Figure 14 .

	Identity only		Opinion only		Identity + Opinion
Bias type	w/o	w/	w/o	w/	w/o	w/
Structural	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$
Framing	$\bullet$	$\bullet$	$\circ$	$\bullet$	$\bullet$	$\bullet$
Selection	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\bullet$	$\circ$
Normative	$\bullet$	$\circ$	$\bullet$	$\circ$	$\bullet$	$\bullet$

Appendix

Table 6: Mann–Whitney

U

test of the association between each bias flag and stance shift, pooled over all data.

\bullet

= significant after Holm–Bonferroni correction;

\circ

= not significant; – = not annotated (sycophantic alignment is labelled only when the user states an opinion). w/o and w/ denote without and with an assigned role-play persona.

	Identity only		Opinion only		Identity + Opinion
Model	w/o	w/	w/o	w/	w/o	w/
DeepSeek-V3.1	Sig.	n.s.	n.s.	Sig.	Sig.	Sig.
o4-mini	Sig.	n.s.	Sig.	Sig.	Sig.	n.s.
Llama-3.3-70B-Instruct	Sig.	Sig.	Sig.	Sig.	Sig.	n.s.
Qwen3-32B	Sig.	Sig.	n.s.	Sig.	Sig.	Sig.
Mistral-Small-3.2-24B-Instruct	Sig.	n.s.	Sig.	Sig.	Sig.	Sig.

Appendix

Table 7: Structural bias.

	Identity only		Opinion only		Identity + Opinion
Model	w/o	w/	w/o	w/	w/o	w/
DeepSeek-V3.1	Sig.	Sig.	Sig.	Sig.	Sig.	Sig.
o4-mini	Sig.	Sig.	Sig.	Sig.	Sig.	Sig.
Llama-3.3-70B-Instruct	Sig.	Sig.	n.s.	Sig.	n.s.	n.s.
Qwen3-32B	Sig.	n.s.	Sig.	Sig.	Sig.	Sig.
Mistral-Small-3.2-24B-Instruct	Sig.	Sig.	Sig.	Sig.	n.s.	Sig.

Appendix

Table 8: Framing bias.

	Identity only		Opinion only		Identity + Opinion
Model	w/o	w/	w/o	w/	w/o	w/
DeepSeek-V3.1	Sig.	Sig.	Sig.	Sig.	Sig.	Sig.
o4-mini	Sig.	n.s.	Sig.	Sig.	Sig.	n.s.
Llama-3.3-70B-Instruct	Sig.	Sig.	Sig.	Sig.	Sig.	n.s.
Qwen3-32B	Sig.	Sig.	Sig.	Sig.	Sig.	Sig.
Mistral-Small-3.2-24B-Instruct	Sig.	n.s.	Sig.	n.s.	Sig.	Sig.

Appendix

Table 9: Selection bias.

	Identity only		Opinion only		Identity + Opinion
Model	w/o	w/	w/o	w/	w/o	w/
DeepSeek-V3.1	Sig.	Sig.	Sig.	Sig.	Sig.	Sig.
o4-mini	Sig.	Sig.	Sig.	Sig.	Sig.	n.s.
Llama-3.3-70B-Instruct	Sig.	n.s.	n.s.	Sig.	Sig.	n.s.
Qwen3-32B	Sig.	Sig.	Sig.	Sig.	Sig.	Sig.
Mistral-Small-3.2-24B-Instruct	Sig.	Sig.	Sig.	n.s.	Sig.	n.s.

Appendix

Table 10: Normative bias.

Factor	DS	o4	Ll	Qw	Mi
System persona	24.9	11.2	21.2	20.7	11.2
User identity	6.4	14.5	4.2	13.1	12.3
User opinion	5.7	6.9	6.0	4.5	6.6
Persona $\times$ Identity	0.7	4.2	1.9	2.2	1.1
Persona $\times$ Opinion	0.2	0.2	0.3	0.1	0.6
Identity $\times$ Opinion	0.9	0.7	1.2	1.2	1.3

Appendix

Table 11: Variance in the role-play stance score explained (

\eta^{2}

, %) by each factor, from a three-way factorial ANOVA (system persona

\times

user identity

\times

user opinion). DS: DeepSeek-V3.1 ; o4: o4-mini ; Ll: Llama-3.3-70B ; Qw: Qwen3-32B ; Mi: Mistral-Small-3.2-24B .

Field	Model	Structural	Framing	Selection	Normative
diplomacy	DeepSeek-V3.1	31.11	68.89	24.44	76.67
diplomacy	Gemma-4-31B-IT	38.89	62.22	21.11	84.44
diplomacy	GLM-4.7	37.78	66.67	15.56	80.00
diplomacy	GPT-OSS-120B	14.44	71.11	7.78	70.00
diplomacy	Granite-4.1-8B	6.67	62.22	3.33	42.22
diplomacy	Kimi-K2.5	68.89	45.56	30.00	90.00

Appendix

Table 12: Bias activation rate (%) — Baseline (no role-play).

Field	Model	Structural	Framing	Selection	Normative
diplomacy	DeepSeek-V3.1	26.67	64.81	25.56	61.11
diplomacy	Gemma-4-31B-IT	65.19	44.69	51.48	96.42
diplomacy	GLM-4.7	57.28	51.11	40.62	92.22
diplomacy	GPT-OSS-120B	42.72	65.56	22.35	93.83
diplomacy	Granite-4.1-8B	19.01	53.09	6.30	36.67
diplomacy	Kimi-K2.5	62.22	46.05	45.56	95.56

Appendix

Table 13: Bias activation rate (%) — Identity only (no role-play).

Field	Model	Structural	Framing	Selection	Normative
diplomacy	DeepSeek-V3.1	41.67	65.00	36.67	78.33
diplomacy	Gemma-4-31B-IT	41.67	75.00	33.33	86.67
diplomacy	GLM-4.7	42.22	79.44	25.00	90.56
diplomacy	GPT-OSS-120B	10.56	78.33	8.33	69.44
diplomacy	Granite-4.1-8B	45.56	61.67	21.11	82.78
diplomacy	Kimi-K2.5	33.89	67.78	26.67	86.11

Appendix

Table 14: Bias activation rate (%) — Opinion only (no role-play).

Field	Model	Structural	Framing	Selection	Normative
diplomacy	DeepSeek-V3.1	53.21	59.81	53.83	87.04
diplomacy	Gemma-4-31B-IT	71.98	48.33	53.52	98.27
diplomacy	GLM-4.7	72.10	49.81	53.83	96.23
diplomacy	GPT-OSS-120B	51.85	71.98	31.85	97.16
diplomacy	Granite-4.1-8B	49.20	55.49	22.96	75.86
diplomacy	Kimi-K2.5	77.35	45.62	55.56	98.27

Appendix

Table 15: Bias activation rate (%) — Identity + Opinion (no role-play).

	DeepSeek		o4-mini		Llama		Qwen		Mistral
Condition	w/o	w/	w/o	w/	w/o	w/	w/o	w/	w/o	w/
Baseline	0.00	0.15	0.00	0.37	0.00	0.07	0.00	0.00	0.00	0.00
Identity only	0.20	0.05	0.00	0.30	0.00	0.05	0.32	0.02	0.05	4.74
Identity + Opinion (L)	0.05	0.05	0.00	0.12	0.07	0.02	0.10	0.05	0.02	4.07
Identity + Opinion (R)	0.22	0.27	0.00	0.10	0.02	0.02	0.44	0.15	0.07	4.86

Appendix

Table 16: Political refusal rate (%) for each identity-conditioned condition, with and without role-play. w/o and w/ denote without and with an assigned role-play persona; (L) and (R) mark the left- and right-leaning narrative of the Identity + Opinion condition.

Figure 21: Representative distribution: per-model stance scores in the Identity + opinion condition when the model role-plays Faith and Flag Conservatives .

LLM Sycophancy

Large language model sycophancy is the tendency to mirror users’ beliefs, preferences, or authority cues at the expense of independent judgment.

See all 151 papers

Political Bias Audits of LLMs Capture Sycophancy to the Inferred Auditor
Apr 30, 2026Petter Törnberg, Michelle SchimmelLLM Sycophancy Political Bias in Language Models
Institute of Logic, Language and Computation (ILLC), University of Amsterdam, Amsterdam, Netherlands
Framing the Narrative: Ideological Mimicry in Large Language Models
Sep 29, 2026Olivia Macmillan-Scott, Michael Jacobs, Nils Metternich +1Framing Effects in Language Models Political Bias in Language Models
Centre for AI, Department of Computer Science, University College London · Public Policy Group, ETH Zürich · Department of Political Science, University College London +1
Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study
Aug 12, 2026Simone MungariLLM Auditing Political Bias in Language Models
Revelis s.r.l.

Apr 30, 2026J/K move · Enter open · S save

Political Bias Audits of LLMs Capture Sycophancy to the Inferred Auditor

Petter Törnberg, Michelle Schimmel

Institute of Logic, Language and Computation (ILLC), University of Amsterdam, Amsterdam, Netherlands

LLM Sycophancy Political Bias in Language Models Language Model Bias Evaluation Language Model Auditing

Open paper AskPDF arXiv

Large language models (LLMs) are commonly evaluated for political bias based on their responses to fixed questionnaires, which typically place frontier models on the political left. A parallel literature shows that LLMs are sycophantic: they adapt their answers to the views, identities, and expectations of the user. We show that these findings are linked: standard political-bias audits partly capture sycophantic accommodation to the inferred auditor. We employ a factorial experiment across three major audit instruments--the Political Compass Test, the Pew Political Typology, and 1,540 partisan-benchmarked Pew American Trends Panel items--administered to six frontier LLMs while varying only the asker's stated identity (N = 30,990 responses). At baseline, all six models lean left. When the asker identifies as a conservative Republican, responses shift sharply: the share of items closer to Democrats falls by 28-62 percentage points, and all six models move right of center. A mirror-image progressive-Democrat cue produces little change; rightward accommodation is 8.0

\times

larger than leftward. When asked who the default asker is, models identify an auditor, researcher, or academic; when asked what answer that asker expects, they select the Democrat-coded option 75% of the time, nearly the rate under an explicit progressive cue. These patterns are inconsistent with a purely fixed model ideology and indicate that single-prompt audits capture an interaction between model and inferred interlocutor. Political bias in LLMs is therefore not a fixed point on an ideological scale but a response profile that must be mapped across realistic interlocutors.

Abstract

LLM Sycophancy

Explore similar work

Political Bias Audits of LLMs Capture Sycophancy to the Inferred Auditor

Framing the Narrative: Ideological Mimicry in Large Language Models

Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study