Large Language Models Unpack Complex Political Opinions through Target-Stance Extraction

Özgür Togay; Javier Garcia-Bernardo; Florian Kunneman; Anastasia Giachanou

cs.CLMar 7, 2026

Large Language Models Unpack Complex Political Opinions through Target-Stance Extraction

Authors: Özgür Togay, Javier Garcia-Bernardo, Florian Kunneman, Anastasia Giachanou

Organizations: Department of Methodology and Statistics, Utrecht University · Department of Languages, Literature and Communication, Utrecht University

Abstract

Political polarization emerges from a complex interplay of beliefs about policies, figures, and issues. However, most computational analyses reduce discourse to coarse partisan labels, overlooking how these beliefs interact. This is especially evident in online political conversations, which are often nuanced and cover a wide range of subjects, making it difficult to automatically identify the target of discussion and the opinion expressed toward them. In this study, we investigate whether Large Language Models (LLMs) can address this challenge through Target-Stance Extraction (TSE), a recent natural language processing task that combines target identification and stance detection, enabling more granular analysis of political opinions. For this, we construct a dataset of 1,084 Reddit posts from r/NeutralPolitics, covering 138 distinct political targets and evaluate a range of proprietary and open-source LLMs using zero-shot, few-shot, and context-augmented prompting strategies. Our results show that the best models perform comparably to highly trained human annotators and remain robust on challenging posts with low inter-annotator agreement. These findings demonstrate that LLMs can extract complex political opinions with minimal supervision, offering a scalable tool for computational social science and political text analysis.

Figures & tables

	Pair	Count	%
Agreements	Positive - Positive	116	20%
	Neutral - Neutral	196	35%
	Negative - Negative	73	13%
	Total	385	68%
Disagreements	Positive - Neutral	83	15%
	Positive - Negative	34	6%

Table 1: Stance agreement distribution for samples with matching target labels (N = 568 of 1,084). The remaining 516 samples had mismatched targets.

Category	Count	% of Gold Dataset
Policies & Socioeconomic Issues	50	25%
Key Divisive Issues	29	14.5%
Key Politicians	29	14.5%
Trust & Institutions	20	10%
Political Groups	11	5.5%
Technology & Governance Issues	5	2.5%

Table 2: Distribution of target categories for gold-labeled posts. For mapping criteria, see Appendix A.

Model	Target F1	Target Acc.	Stance F1	Stance Acc.
gpt-4.1	0.73	0.70	$0.81$	$0.81$
o3	$0.71$	$0.68$	0.84	0.85
gemini-2.5-pro	$0.71$	$0.67$	$0.84$	$0.84$
gemini-2.5-flash	$0.70$	$0.66$	$0.84$	$0.84$
llama-3.1-405b	$0.66$	$0.66$	$0.80$	$0.80$
gpt-4.1-mini	$0.63$	$0.61$	$0.79$	$0.79$

Table 3: Zero-shot performance of all tested models, sorted by Target F1. Bold indicates the best score per metric.

Model	Target F1	Target Acc.	Stance F1	Stance Acc.
gpt-4.1	$0.74$ ↑	$0.73$ ↑	$0.81$	$0.82$ ↑
gemini2.5-flash	$0.74$ ↑	$0.70$ ↑	$0.86$ ↑	$0.86$ ↑
gemini2.5-pro	$0.72$ ↑	$0.68$ ↑	$0.85$ ↑	$0.85$ ↑
o3	$0.72$ ↑	$0.70$ ↑	$0.84$	$0.85$
llama3-405b	$0.67$ ↑	$0.67$ ↑	$0.81$ ↑	$0.80$
gpt-4.1-mini	$0.65$ ↑	$0.64$ ↑	$0.77$ ↓	$0.77$ ↓

Table 4: TSE metrics using broad target labels, ordered by Target F1. Up arrows (↑) indicate increases from zero-shot performance, down arrows (↓) indicate decreases.

Model	Target F1	Target Acc.	Stance F1	Stance Acc.
gemini-2.5-pro	$0.76$ ↑	$0.73$ ↑	$0.82$ ↓	$0.82$ ↓
o3	$0.75$ ↑	$0.73$ ↑	$0.84$	$0.84$ ↓
gemini-2.5-flash	$0.74$ ↑	$0.71$ ↑	$0.81$ ↓	$0.82$ ↓
gpt-4.1	$0.74$ ↑	$0.72$ ↑	$0.82$ ↑	$0.82$ ↑
gpt-4.1-mini	$0.70$ ↑	$0.68$ ↑	$0.78$ ↓	$0.78$ ↓
gemini-2.5-flash-lite	$0.61$ ↑	$0.59$ ↑	$0.78$ ↑	$0.78$ ↑

Table 5: TSE metrics using few-shot prompting, ordered by Target F1. Arrows indicate changes from zero-shot performance.

Model	Target F1	Target Acc.	Stance F1	Stance Acc.
gemini-2.5-pro	$0.74$ ↑	$0.70$ ↑	$0.84$	$0.84$
o3	$0.73$ ↑	$0.72$ ↑	$0.84$	$0.85$
gemini-2.5-flash	$0.72$ ↑	$0.68$ ↑	$0.84$	$0.84$
llama-4-maverick	$0.62$ ↑	$0.59$ ↑	$0.73$ ↑	$0.73$ ↑
gemini-2.5-flash-lite	$0.60$ ↑	$0.58$ ↑	$0.77$ ↑	$0.77$ ↑
llama-3.1-405b	$0.62$ ↓	$0.61$ ↓	$0.83$ ↑	$0.83$ ↑

Table 6: TSE metrics using conversational context, ordered by Target F1. Arrows indicate changes from zero-shot performance.

Model	Target F1	Target Acc.	Stance F1	Stance Acc.
gemini-2.5-flash	$0.75$ ↑	$0.71$ ↑	$0.82$ ↑	$0.83$ ↑
gemini-2.5-pro	$0.75$ ↑	$0.73$ ↑	$0.84$ ↑	$0.84$ ↑
gpt-4.1	$0.74$ ↑	$0.73$ ↑	$0.81$ ↓	$0.81$ ↓
o3	$0.73$ ↑	$0.72$ ↑	$0.84$	$0.85$
llama-3.1-405b	$0.70$ ↑	$0.68$ ↑	$0.81$ ↑	$0.81$ ↑
gpt-4.1-mini	$0.67$ ↑	$0.64$ ↑	$0.75$ ↓	$0.75$ ↓

Table 7: TSE metrics using informational context, ordered by Target F1. Arrows indicate changes from zero-shot performance.

Model	Target F1	Target Acc.	Stance F1	Stance Acc.
o3	$0.76$ ↑	$0.75$ ↑	$0.87$ ↑	$0.87$ ↑
gemini-2.5-pro	$0.73$ ↓	$0.72$ ↓	$0.84$ ↑	$0.85$ ↑
gemini-2.5-flash	$0.73$ ↓	$0.70$ ↓	$0.83$ ↑	$0.84$ ↑
gpt-4.1	$0.74$ ↑	$0.72$ ↑	$0.82$ ↑	$0.82$ ↑
gpt-4.1-mini	$0.70$	$0.68$	$0.77$ ↑	$0.78$
gemini-2.5-flash-lite	$0.63$ ↑	$0.62$ ↑	$0.75$ ↓	$0.75$ ↓

Table 8: TSE metrics using few-shot prompting with informational context, ordered by Target F1. Arrows indicate changes from few-shot performance.

Figure 2: Target and Stance F1 scores for o3 model across different prompting strategies.

#	Post	Labels
1	I’ve never had an issue with funding social programs, especially compared to the money Republicans pouring into the war machine.	o3: Republican Party, Negative
Gold: Social spending, Positive	Focus on alternate target
2	[Ongoing.]www.cnn.com/politics-flynn-russia-calls-investigation	o3: Belief in Trump-Russia Collusion, Neutral
Gold: None, N/A	Target inferred by URL text
3	Why the Democratic Party doesn’t treat campaign finance reform as a major issue?	o3: Democratic Party, Negative
Gold: Democratic Party, Neutral	Using weak cues

Table 9: Example posts showing o3 prediction errors compared with gold labels. Posts are rephrased to protect anonymity.

Category	Count	F1	Accuracy
International Relations	3	1.00	1.00
Key Politicians	29	0.93	0.90
Key Divisive Issues	29	0.90	0.86
Political Groups	11	0.88	0.91
Policies & Socioeconomic Issues	50	0.86	0.82
No Target	50	0.75	0.60

Table 10: Target identification performance across target categories.

True Stance	Count	F1	Accuracy
Positive	43	0.90	0.93
Negative	40	0.84	0.88
Neutral	36	0.76	0.69

Table 11: Performance of o3 by stance (for posts with matching target).

#	Post	Labels
1	It’s not about raising the minimum wage. There are awful gaps where you lose your benefits if you earn more, so making less ends up better.	o3: Social spending, Negative
A1: Minimum wage, Negative
A2: Social spending, Neutral
EX: Social spending, Neutral
2	Biden sees the Green New Deal as an important foundation for tackling the climate crisis we’re facing.	o3: Joe Biden, Neutral
A1: Joe Biden, Neutral

Table 12: Example posts from disagreement cases. Labels show o3 predictions, human annotator labels (A1, A2), and the expert label (EX).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Variant	Change from zero-shot baseline
Few-shot	Adds Examples section (one per stance) before the target list
Broad labels	Replaces fine-grained target list with merged categories (see Appendix B)
Conversational context	Restructures input format to include surrounding posts; adds Context Weighting section
Few-shot with informational context	Combines few-shot examples with per-target descriptions from the codebook

Appendix

Table 15: Prompt variants and their differences from the zero-shot baseline.

Stance Detection

Stance detection identifies whether people support, oppose, or remain neutral toward a target, claim, or position in text and conversation.

See all 36 papers

How Identity and Opinion Shape Political Sycophancy in LLMs
Aug 29, 2026Li-Ni Fu, Chang-Chih Meng, Chien-Hua Chen +2LLM Sycophancy Political Bias in Language Models
Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan
LLMs Can Infer Political Alignment from Online Conversations
Mar 11, 2026Byunghwee Lee, Sangyeon Kim, Filippo Menczer +3Privacy Leakage in Language Models
School of Data Science, University of Virginia, Charlottesville, Virginia, USA, 22903 · Center for Complex Networks and Systems Research, Luddy School of Informatics, Computing, and Engineering Indiana University, Bloomington, Indiana, USA, 47408 · Division of Communication and Media, Ewha Womans University, Seoul, Republic of Korea, 03760
Reasoning Boosts Opinion Alignment in LLMs
Mar 1, 2026Frédéric Berdoz, Yann Billeter, Yann Vonlanthen +1Political Bias in Language Models Personalized Language Model Alignment
ETH Zurich, Switzerland

Aug 29, 2026J/K move · Enter open · S save

How Identity and Opinion Shape Political Sycophancy in LLMs

Li-Ni Fu, Chang-Chih Meng, Chien-Hua Chen, Hen-Hsen Huang+1

Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan

LLM Sycophancy Political Bias in Language Models Language Model Bias Evaluation Demographic Bias in Language Models

Open paper AskPDF arXiv

As Large Language Models (LLMs) increasingly encourage users to disclose personal profiles for tailored assistance, measuring their political alignment becomes increasingly important. However, many existing benchmarks for assessing political behavior rely on closed-ended questions and do not fully capture how a model's stance may adapt to user-provided context during interaction. We introduce a framework that disentangles two distinct triggers of political sycophancy: opinion (aligning with explicit narratives) and identity (stereotyping based on demographic labels). Using 450 manually-checked political dilemmas as controlled probes, we evaluate 13 instruction-tuned LLMs. We uncover a dissociation: a model's susceptibility to explicit opinions does not necessarily predict its susceptibility to identity cues, and vice versa. When both signals are present, their effects are generally sub-additive rather than simply additive. Additionally, system-level personas primarily shift a model's baseline stance while having limited effect on the stance shift caused by user opinion or identity. Ultimately, our results suggest that LLM political stance is interactively and steerably vulnerable rather than being a fixed trait, highlighting how personalization may amplify identity- or opinion-conditioned shifts in the model's behaviors.

Abstract

Stance Detection

Explore similar work

How Identity and Opinion Shape Political Sycophancy in LLMs

LLMs Can Infer Political Alignment from Online Conversations

Reasoning Boosts Opinion Alignment in LLMs