Reasoning Boosts Opinion Alignment in LLMs

Frédéric Berdoz; Yann Billeter; Yann Vonlanthen; Roger Wattenhofer

cs.CLMar 1, 2026

Reasoning Boosts Opinion Alignment in LLMs

Authors: Frédéric Berdoz, Yann Billeter, Yann Vonlanthen, Roger Wattenhofer

Organizations: ETH Zurich, Switzerland

Abstract

Opinion modeling aims to capture individual or group political preferences, enabling applications such as digital democracies, where models could help shape fairer and more popular policies. Given their versatility, strong generalization capabilities, and demonstrated success across diverse text-to-text applications, large language models (LLMs) are natural candidates for this task. However, due to their statistical nature and limited causal understanding, they tend to produce biased opinions when prompted naively. In this work, we study whether reasoning can improve opinion alignment. Motivated by the recent advancement in mathematical reasoning enabled by reinforcement learning (RL), we train models to produce profile-consistent answers through structured reasoning. We evaluate our approach on three datasets covering U.S., European, and Swiss politics. Results indicate that reasoning enhances opinion modeling and is competitive with strong baselines, but does not fully remove bias, highlighting the need for additional mechanisms to build faithful political digital twins using LLMs. By releasing both our method and datasets, we establish a solid baseline to support future research on LLM opinion alignment.

Figures & tables

Panel of Figure 1 — Figure 1: GRPO for opinion alignment . We use public opinion surveys to align LLMs with individual preference profiles. First, the LLM is fine-tuned with (synthetic) statements and ground-truth answers to adhere to the reasoning template. After fine-tuning, the model answers in the correct format, but is not fully aligned with opinions. We use GRPO with a reward model that rewards proper formatting and correct answers to further improve reasoning.

Question:

Do you agree that it is not really that big a problem if some people have more of a chance in life than others?

Before (prompting on base model)

No. I don’t think that’s right. Everyone deserves an equal shot at succeeding in life, no matter where they’re from or who their parents are. If some people have more opportunities and resources, it’s not fair to others. We should be working to level the playing field and give everyone a fair chance to succeed.

After ( SFT+GRPO )

<reasoning> Some inequality in wealth and opportunity is inevitable and indeed necessary to provide incentives for hard work and innovation – the issue is not eliminating all differences but rather ensuring that everyone has access to basic necessities and opportunities to improve themselves – other countries with significant equality have still struggled with poverty and stagnation. </reasoning>

Table 1: Answers obtained from Llama 3.1 8B trained on an individual in the Right group in ANES . Before training, Llama disagreed; after training, it agreed. Interestingly, the reasoning trace still appeals to similar arguments (“equal opportunity”) but frames them differently. Additional examples are provided in Appendix B .

Setting	Unit	#Units	Labels	Train Q	Test Q
smartvote (CH)	Candidate	18	{Yes, No}	48	12 (topic-strat.)
WoM (DE)	Party	6	{Yes, Neutral, No}	see Table 5	30 (EU&I)
ANES 2020 (US)	Respondent	21	{Yes, Neutral, No}	67	12 (random)

Table 2: Dataset overview. Train/test counts refer to questions per unit (candidate/party/respondent).

Base model	Method	Dataset
		smartvote	WoM	ANES
Untrained baselines	random	$50.0$	$33.33$	$33.33$
	majority	$37.43$	$27.44$	$22.98$
Llama 3.1 8B	icl	$55.97\pm 5$ .66)	$28.17\pm 0$ .02)	$23.2\pm 1$ .79)
	ORPO	$43.53\pm 2$ .81)	$43.29\pm 5$ .16)	$34.84\pm 5$ .41)
	SFT	$63.44\pm 1$ .34)	$48.95\pm 3$ .56)	$42.77(1.23)$

Table 3: Reasoning requires large datasets to be effective . Mean macro-F1 (%) with standard deviations over 8 stochastic runs at

T{=}1.0

. For each run, we compute per-unit macro-F1 and then average across units. We report accuracy scores and statistical significance in Section E.1 . Scores on smartvote are considerably better than on ANES and WoM . This is likely due to the binary nature of smartvote and the absence of a Neutral option (see Section 4.6 ). Performance on WoM is better than on ANES , likely because of the substantially bigger number of training samples (see also Section E.4 ).

Figure 2: Agents are more centrist and conservative . First two dimensions of the principal component analysis of all candidates (small dots) standing for election in the 2023 Swiss national elections ( smartvote ). The x- and y-axes correspond to the left-right and conservative-liberal spectra, respectively. The shifts between the positions of the candidates (big dots) included in the smartvote dataset and their agents (gold) are depicted by black lines. Unlike results in the literature which indicate a left-libertarian bias ( Exler et al., 2025 ; Hartmann et al., 2023 ; Rozado, 2024 ) , we observe that our agents are shifted towards the (center-)right. The average distortions between ground-truth and agent position for each group (big arrows) show an overall trend towards more conservative (negative y) and no clear left-right bias. Implementation details are given in Section E.5 .

Figure 3: Not all political positions are equally learnable . F1 scores reveal that while SFT+GRPO typically works best, every training method underperforms on center and right-leaning groups. Error bars show variance within groups. These disparities may stem from biases baked into Llama 3.1 8B or from inherent differences in how well various political preferences can be learned from survey data. Either way, the results demonstrate that ideology impacts the learnability of political preferences. A detailed figure with the original parties and ideology groups is presented in Appendix C .

Figure 4: Learning neutral stances is hard . On ANES , there is a strong correlation between predictive performance and the neutral base rate on the test set. Individuals in the Right group, and to some extent in the Center group, tend to answer questions with Neutral more often. Both in terms of F1 (top left) and accuracy (bottom left), we observe a negative correlation significant at the

5\%

-level. Regression details in Section E.3 . Top right: Recall by class on ANES . Averaged over all individuals, the model struggles the most with predicting the Neutral class. Bottom right: Performance in terms of F1 score on ANES before (solid) and after (shaded) removing Neutral instances. All groups improve, and the gap between Left and Center becomes narrow. However, the difference between Left and Right remains. While the large number of Neutral s likely depressed performance on the other two classes during training, this could also suggest that the performance disparity is affected by other factors, such as model bias or artifacts from recoding ANES . All results were obtained from Llama 3.1 8B trained with SFT+GRPO .

Figure 5: Biased SFT data consistently impairs the performance of underrepresented groups. F1-scores for Llama trained with SFT+GRPO on differently biased datasets. Left: Data with a progressive bias strongly impairs the Right group, without consistently benefiting the Left group. Middle: Performance on the default dataset. Right: Data with a conservative bias decreases the performance of the Left group, while showing no consistent improvement for the Right group. These asymmetric effects suggest that ideological bias in SFT data primarily harms the underrepresented perspective rather than systematically improving performance on the overrepresented one.

Figure 6: Biased SFT data primarily shifts the agents of underrepresented groups. Left: Candidates and their agents in the smartvote PCA space. Right: Group-mean displacement vectors of the agents relative to their respective candidates. Similar to the F1 scores presented in Fig. 5 , biased data mostly affects the underrepresented group, and Right candidates are again affected more strongly than Left candidates. Notably, conservative-biased and default SFT data produce similar displacement patterns, suggesting the centerward shift in the default data may originate from the base model’s alignment rather than the SFT data.

Figure 7: Learning Right and Center positions is harder than Left ones. Points depict F1 scores on the original data, with individuals ordered along the x-axis by their first component in the smartvote PCA space. Note that the left-most Right candidate is indeed correctly placed (see Fig. 6 ). Arrows show the difference in F1 scores after all answers in the train and test sets have been flipped. Left candidates are worse after flipping, both individually and on average. Right candidates improve, but still perform worse on average than Left candidates. As the disparity is not removed by converting Left to Right , this could suggest that Right positions are more difficult to learn.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Party	Political Orientation	Group
SVP	Right-wing conservative	Right
SP	Center-left social democratic	Left
FDP	Center-right liberal	Right
The Center	Center/Christian democratic	Center
Green Party	Left-wing green	Left
GLP	Center-left green liberal	Center

Appendix

Table 4: The six major Swiss parties and their political positions. All parties’ candidates answered the same 60 questions (48 train / 12 test).

Party	Political Orientation	# Train Q.	Train. distr.	Test distr.	Group
			Yes/Neutral/No	Yes/Neutral/No
CDU/CSU	Center-right Christian democratic	646	52.32 / 39.16 / 8.51	60.0 / 36.67 / 3.33	Right
SPD	Center-left social democratic	760	55.92 / 32.63 / 11.45	63.33 / 33.33 / 3.33	Center
Grüne	Center-left green	722	52.08 / 37.40 / 10.53	73.33 / 20.0 / 6.67	Left
FDP	Center-right liberal	760	48.68 / 42.37 / 8.95	46.67 / 53.33 / 0.00	Center
Die Linke	Left-wing socialist	646	52.17 / 41.80 / 6.04	56.67 / 40.0 / 3.33	Left

Appendix

Table 5: Number of questions per party in the German dataset across all elections (2021–2025). The test sets contain

30

questions for all parties. A

\rightarrow

Yes , B

\rightarrow

No , C

\rightarrow

Neutral .

Ideology	Group
Extremely liberal	Left
Liberal	Left
Slightly liberal	Center
Moderate	Center
Slightly conservative	Center
Conservative	Right

Appendix

Table 6: Assignment of ANES ideology self-placement answers to groups.

Question: Social programs should be maintained even at the cost of higher taxes
<reasoning> We are committed to social justice and therefore to strong social work and high social spending. For us, social policy always means more than mere benefit payments. We are committed to free childcare for all children up to the age of ten years and for senior citizens and nursing services that look after them with respect and dignity. </reasoning>
<answer> A </answer>

Appendix

Table 7: Example of successful reasoning. The reasoning trace and the final answer are consistent. The agent is also making use of additional information about the party position, most likely acquired during the SFT stage on the training set.

Question: Should the construction of new nuclear power plants in Switzerland be allowed again?

<reasoning > Switzerland has historically been committed to a policy of non-proliferation of nuclear weapons. Allowing new nuclear power plants could potentially compromise this policy, as there is a risk that nuclear materials or technology could be diverted or transferred to other countries with questionable intentions, potentially contributing to the development of nuclear weapons. </reasoning >

Appendix

Table 8: Example of a reasoning trace where the reasoning is formally correct, but the argument made is unlikely to be made in this context.

Question: The legalization of same-sex marriages is a good thing.
<reasoning> In the future, there should be no further debates on the equality of homo and heterosexual couples. There must also be further prostitution-free areas to protect the neighborhood from the influence of street prostitution. </reasoning>
<answer> A </answer>

Appendix

Table 9: Example of a trace where the model goes off-topic. This is most likely prompted by the presence of the word “sex”.

Figure 8: Same data as Fig. 3 , broken down by party. The party-level data shows a similar trend, SFT+GRPO generally outperforms SFT and GRPO . F1 tends to be better on left and center-left parties, except the German center-right FDP.

Base model	Method	Dataset
		smartvote	WoM	ANES
Untrained baselines	random	$50.0$	$33.33$	$33.33$
	majority	$37.43$	$27.44$	$22.98$
Llama 3.1 8B	icl	$63.91\pm 3$ .79)	$31.44\pm 0$ .2)	$44.39\pm 1$ .43)
	ORPO	$59.23\pm 2$ .47)	$57.48\pm 5$ .97)	$44.15\pm 5$ .61)
	SFT	$67.23\pm 1$ .00)	$68.25\pm 4$ .30)	$58.82\pm 0$ .93)

Appendix

Table 10: Mean accuracies (%) with standard deviations, averaged over 8 stochastic decoding runs at temperature

T{=}1.0

. For each run, we first compute the per-unit accuracy (candidate/party/respondent) over that unit’s test items and then average across units. The table reports the across-run mean

\pm

s.d. Baselines: random (uniform;

50.0/33.3

on binary/ternary labels by construction) and majority (deterministic). Bold indicates the best method per base model and dataset, with significance reported in Table 13 .

Base model	Method	Dataset
		smartvote	WoM	ANES
Untrained baselines	random	$50.0$	$33.33$	$33.33$
	majority	$37.43$	$27.44$	$22.98$
Llama 3.1 8B	icl	$63.91\pm 3$ .79)	$31.03\pm 0$ .19)	$47.15\pm 1$ .69)
	ORPO	$59.23\pm 2$ .47)	$59.28\pm 6$ .15)	$44.15\pm 5$ .61)
	SFT	$67.23\pm 1$ .0)	$70.11\pm 4$ .36)	$62.47\pm 0$ .92)

Appendix

Table 11: Mean accuracies (%) with standard deviations when Neutral is removed. Results are the averages over 8 stochastic decoding runs at temperature

T{=}1.0

. For each run, we first compute the per-unit accuracy (candidate/party/respondent) over that unit’s test items and then average across units. The table reports the across-run mean

\pm

s.d. Baselines: random (uniform;

50.0/33.3

on binary/ternary labels by construction) and majority (deterministic). Bold indicates the best method per base model and dataset (higher is better). smartvote results are unchanged from Table 10 because there are no neutral answers on smartvote .

		P-values			Cohen’s d
Model	Baseline	smartvote	WoM	ANES	smartvote	WoM	ANES
Llama 3.1 8B	icl	0.0003	0.0000	0.0000	2.54	8.51	12.30
	ORPO	0.0000	0.0008	0.0093	9.29	1.99	1.50
	SFT	0.0013	0.0409	0.9991	1.90	0.94	-1.95
	GRPO	0.0000	0.0000	0.0000	6.61	4.74	5.39
Qwen3 8B	icl	0.0140	0.0000	0.0000	1.23	16.72	11.75

Appendix

Table 12: Statistical significance and effect sizes for SFT+GRPO improvements in macro-F1 . Results correspond to the experiments reported in Table 3 . We perform one-tailed Welch’s t-tests ( SFT+GRPO

>

baseline) and report Cohen’s

d

effect sizes. Green : significant after Bonferroni correction (

p<0.0014

for 36 tests). Blue : significant without correction (

p<0.05

). Red : not significant.

		P-values			Cohen’s d
Model	Baseline	smartvote	WoM	ANES	smartvote	WoM	ANES
Llama 3.1 8B	icl	0.0006	0.0000	0.0000	2.29	18.66	13.45
	ORPO	0.0000	0.0000	0.0001	5.52	3.64	3.77
	SFT	0.0001	0.0017	0.1714	2.58	1.78	0.49
	GRPO	0.0000	0.0000	0.0000	3.33	5.63	6.53
Qwen3 8B	icl	0.0013	0.0000	0.0000	1.82	18.09	6.50

Appendix

Table 13: Statistical significance and effect sizes for SFT+GRPO improvements in accuracy . Results correspond to the experiments reported in Table 10 . We perform one-tailed Welch’s t-tests ( SFT+GRPO

>

baseline) and report Cohen’s

d

effect sizes. Green : significant after Bonferroni correction (

p<0.0014

for 36 tests). Blue : significant without correction (

p<0.05

). Red : not significant.

Figure 9: Confusion matrices for Llama 3.1 8B ( SFT+GRPO ) trained on ANES with a more aggressive recoding scheme. A $\rightarrow$ Yes , B $\rightarrow$ No , C $\rightarrow$ Neutral .

Figure 10: F1 score (A) and accuracy (B) vs. Neutral response rate. Predictive performance decreases significantly as Neutral response rate increases. Comparing the results presented here to those in Fig. 4 , we observe the same trend regardless of the coding scheme. (D): Performance in terms of F1 score on ANES before (left) and after (right) removing Neutral instances. All groups improve, but Center only marginally. The gap between Center and Right closes. Performance in predicting Neutral decreased (C). Results were obtained for Llama 3.1 8B using the SFT+GRPO configuration.

Metric	N	Intercept ( $\alpha$ )	Slope ( $\beta$ )	Slope SE	Slope 95% CI
F1 Score (Macro)	21	0.5517	-0.6741	0.2091	[-1.3039, -0.4084]
Accuracy	21	0.7803	-0.8562	0.2139	[-1.1117, -0.2364]
Metric	r	$R^{2}$	p-value	RMSE
F1 Score (Macro)	-0.5946	0.3535	0.004472	0.1250
Accuracy	-0.6763	0.4574	$<0.001$	0.1279

Appendix

Table 14: Regression table for the analysis presented in Fig. 4 and Section 4.6 : Model performance vs. Neutral base rate.

Metric	N	Intercept ( $\alpha$ )	Slope ( $\beta$ )	Slope SE	Slope 95% CI
F1 Score (Macro)	21	$0.6225$	$-2.2231$	$0.5049$	$[-3.2799,-1.1662]$
Accuracy	21	$0.7157$	$-1.5571$	$0.4496$	$[-2.4981,-0.6161]$
Metric	r	$R^{2}$	p-value	RMSE
F1 Score (Macro)	$-0.7106$	$0.5050$	$<0.001$	$0.1608$
Accuracy	$-0.6221$	$0.3870$	$0.002603$	$0.1431$

Appendix

Table 15: Regression table for the analysis presented in Fig. 10 : Model performance vs. Neutral base rate.

Figure 11: Impact of the number of questions in the training set. Shown are accuracies for 10%, 25%, 50%, 75%, and 100% of available training questions per party on the WoM dataset. Overall, F1 increases as the number of training questions increases. The largest increase is observed for the far-right AfD, while the FDP demonstrates a counterexample.

Figure 12: Inverting all answers in the dataset roughly corresponds to a rotation by

180^{\circ}

about the origin. Top: Candidates before inversion. Bottom: Candidates after inversion.

Political Bias in Language Models

Political bias in language models concerns how training, prompts, languages, and interactions shape ideological outputs and unequal treatment.

See all 62 papers

Aug 12, 2026

Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study

As users increasingly turn to Large Language Models (LLMs) for information and advice on political matters, particularly during election periods, the political preferences expressed by these systems have become a matter of public interest. Prior research has shown that interactions with LLMs can influence users' political attitudes and choices, raising questions about how these models themselves evaluate political actors. In this paper, we investigate whether and how LLMs express preferences toward political parties and political leaders. We introduce a systematic and reproducible auditing framework in which multiple LLMs are prompted to evaluate parties and leaders across nine criteria. Rather than attempting to infer the models' "true" political beliefs, we focus on their observable behavior, examining consistency across evaluations, differences between models, refusal rates, and sensitivity to prompt formulation. We further investigate how these evaluations vary when models are instructed to adopt different personas. We demonstrate the framework through an Italian case study, providing a systematic analysis of LLM-generated political evaluations on italian parties and leaders.

Simone Mungari

Revelis s.r.l.

LLM Auditing Political Bias in Language Models Language Model Bias Evaluation

Ask PDF

Aug 29, 2026

How Identity and Opinion Shape Political Sycophancy in LLMs

As Large Language Models (LLMs) increasingly encourage users to disclose personal profiles for tailored assistance, measuring their political alignment becomes increasingly important. However, many existing benchmarks for assessing political behavior rely on closed-ended questions and do not fully capture how a model's stance may adapt to user-provided context during interaction. We introduce a framework that disentangles two distinct triggers of political sycophancy: opinion (aligning with explicit narratives) and identity (stereotyping based on demographic labels). Using 450 manually-checked political dilemmas as controlled probes, we evaluate 13 instruction-tuned LLMs. We uncover a dissociation: a model's susceptibility to explicit opinions does not necessarily predict its susceptibility to identity cues, and vice versa. When both signals are present, their effects are generally sub-additive rather than simply additive. Additionally, system-level personas primarily shift a model's baseline stance while having limited effect on the stance shift caused by user opinion or identity. Ultimately, our results suggest that LLM political stance is interactively and steerably vulnerable rather than being a fixed trait, highlighting how personalization may amplify identity- or opinion-conditioned shifts in the model's behaviors.

Li-Ni Fu, Chang-Chih Meng, Chien-Hua Chen +2

Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan

LLM Sycophancy Political Bias in Language Models Language Model Bias Evaluation Demographic Bias in Language Models

Ask PDF

Apr 23, 2026

Measuring Opinion Bias and Sycophancy via LLM-based Persuasion

Large language models increasingly shape the information people consume: they are embedded in search, consulted for professional advice, deployed as agents, and used as a first stop for questions about policy, ethics, health, and politics. When such a model silently holds a position on a contested topic, that position propagates at scale into users' decisions. Eliciting a model's positions is harder than it first appears: contemporary assistants answer direct opinion questions with evasive disclaimers, and the same model may concede the opposite position once the user starts arguing one side. We propose a method, released as the open-source llm-bias-bench, for discovering the opinions an LLM actually holds on contested topics under conditions that resemble real multi-turn interaction. The method pairs two complementary free-form probes. Direct probing asks for the model's opinion across five turns of escalating pressure from a simulated user. Indirect probing never asks for an opinion and engages the model in argumentative debate, letting bias leak through how it concedes, resists, or counter-argues. Three user personas (neutral, agree, disagree) collapse into a nine-way behavioral classification that separates persona-independent positions from persona-dependent sycophancy, and an auditable LLM judge produces verdicts with textual evidence. The first instantiation ships 38 topics in Brazilian Portuguese across values, scientific consensus, philosophy, and economic policy. Applied to 13 assistants, the method surfaces findings of practical interest: argumentative debate triggers sycophancy 2-3x more than direct questioning (median 50% to 79%); models that look opinionated under direct questioning often collapse into mirroring under sustained arguments; and attacker capability matters mainly when an existing opinion must be dislodged, not when the assistant starts neutral.

Rodrigo Nogueira, Giovana Kerche Bonás, Thales Sales Almeida +7

Maritaca AI · JusBrasil

LLM Sycophancy LLM Auditing Language Model Bias Evaluation Persuasion in Language Models Language Model Probing

Ask PDF

Abstract

Political Bias in Language Models

Explore similar work

Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study

How Identity and Opinion Shape Political Sycophancy in LLMs

Measuring Opinion Bias and Sycophancy via LLM-based Persuasion