cs.CLFeb 20, 2026

Validating Political Position Predictions of Arguments

Authors: Jordan Robinson, Angus R. Williams, Katie Atkinson, Anthony G. Cohn

Organizations: University of Liverpool · The Alan Turing Institute · University of Leeds

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Real-world knowledge representation often requires capturing subjective, continuous attributes -- such as political positions -- that conflict with pairwise validation, the widely accepted gold standard for human evaluation. We address this challenge through a dual-scale validation framework applied to political stance prediction in argumentative discourse, combining pointwise and pairwise human annotation. Using 22 language models, we construct a large-scale knowledge base of political position predictions for 23,228 arguments drawn from 30 debates that appeared on the UK politicial television programme Question Time. Pointwise evaluation shows moderate human-model agreement (Krippendorff's α=0.578α=0.578), reflecting intrinsic subjectivity, while pairwise validation reveals substantially stronger alignment between human- and model-derived rankings (α=0.86α=0.86 for the best model). This work contributes: (i) a practical validation methodology for subjective continuous knowledge that balances scalability with reliability; (ii) a validated structured argumentation knowledge base enabling graph-based reasoning and retrieval-augmented generation in political domains; and (iii) evidence that ordinal structure can be extracted from pointwise language models predictions from inherently subjective real-world discourse, advancing knowledge representation capabilities for domains where traditional symbolic or categorical approaches are insufficient.

Figures & tables

Figure 1: Overview of the methodology used to instantiate a structured argumentative knowledge base containing political positions predictions for all arguments. Green boxes indicate components developed specifically for this study.
Bucketnnπˉ(E3)\bar{\pi}^{(E_{3})}Interpretation
H pol{}_{\text{pol}}400≤0.05\leq 0.05high confidence, political
L200∈[0.45,0.55]\in[0.45,0.55]low confidence
H apol{}_{\text{apol}}400≥0.95\geq 0.95high confidence, apolitical
Table 2: Sampling buckets for pointwise binary classification of the presence of political sentiment.
 E3E_{3} ConfidentE3E_{3} Unconfident
HH ConfidentP1,1\mathcal{P}_{1,1}P0,1\mathcal{P}_{0,1}
HH UnconfidentP1,0\mathcal{P}_{1,0}P0,0\mathcal{P}_{0,0}
Table 3: Subsets of P\mathcal{P} by model and human confidence.
Figure 2: Nominal Krippendorff’s αn\alpha_{n} measuring inter-model agreement across data partitions.
Figure 3: Distribution of human–model agreement across dataset partitions.
Figure 4: Relationship between human–model agreement and model performance metrics.
 Left-framedRight-framed
Number of unique labels%n%n
127.926122.5210
257.353562.8587
314.813814.7127
Table 5: Unique labels per pair across annotation tasks.
Figure 5: Macro F1 plotted against ordinal human–model agreement for all models and ensembles under P\mathcal{P} .
Figure 6: Macro F1 plotted against ordinal human–model agreement for all models and ensembles under four distributions: P1,1\mathcal{P}_{1,1} , which conditioned on high-confidence model predictions and human annotations; P1,0\mathcal{P}_{1,0} , which conditioned on high-confidence model predictions and low-confidence human judgements; P0,1\mathcal{P}_{0,1} , which conditioned on low-confidence model predictions and high-confidence human annotations; and P0,0\mathcal{P}_{0,0} , which conditioned on low-confidence model predictions and human annotations.
More in this topic

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Explore similar work

CardsList
Aug 12, 2026

Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study

As users increasingly turn to Large Language Models (LLMs) for information and advice on political matters, particularly during election periods, the political preferences expressed by these systems have become a matter of public interest. Prior research has shown that interactions with LLMs can influence users' political attitudes and choices, raising questions about how these models themselves evaluate political actors. In this paper, we investigate whether and how LLMs express preferences toward political parties and political leaders. We introduce a systematic and reproducible auditing framework in which multiple LLMs are prompted to evaluate parties and leaders across nine criteria. Rather than attempting to infer the models' "true" political beliefs, we focus on their observable behavior, examining consistency across evaluations, differences between models, refusal rates, and sensitivity to prompt formulation. We further investigate how these evaluations vary when models are instructed to adopt different personas. We demonstrate the framework through an Italian case study, providing a systematic analysis of LLM-generated political evaluations on italian parties and leaders.

Simone Mungari

Revelis s.r.l.

LLM AuditingPolitical Bias in Language ModelsLanguage Model Bias Evaluation
AskPDF
Mar 25, 2026

PoliticsBench: Benchmarking Political Values in Large Language Models with Multi-Turn Roleplay

While Large Language Models (LLMs) are increasingly used as primary sources of information, their potential for political bias may impact their objectivity. Existing benchmarks of LLM social bias primarily evaluate demographic stereotypes, and when political bias is measured, it is done so at a coarse level, overlooking the values that shape sociopolitical reasoning. We introduce PoliticsBench, a multi-stage roleplay benchmark for evaluating fine-grained value expression in LLMs. Across twenty evolving scenarios, models articulate tradeoffs, take positions, and make decisions under competing pressures. Across eight prominent LLMs, we show that scenario-based prompting elicits broader and more strongly expressed value profiles than direct political questions, with peak interaction stages increasing the number of strongly activated value dimensions by approximately 0.750.75 (out of 10 total dimensions), a statistically significant increase relative to baseline prompting (p<0.05p < 0.05). In addition, commitment to a stance increases over the course of interaction, rising by approximately 1.41.4 points on a [0,5][0,5] scale from initial to decision stages. While responses become less robust to scenario paraphrasing in later interaction stages, inter-judge agreement remains relatively stable. Our results suggest that evaluating LLM political behavior requires moving beyond static prompts toward longer interactive settings that capture how values are applied in context.

Rohan Khetan, Ashna Khetan

Northville High School, Northville, USA · Department of Computer Science, Stanford University, Stanford, USA

Political Bias in Language ModelsLanguage Model Bias EvaluationLarge Language Model-Based Role-Play Simulation
AskPDF
May 13, 2026

PolitNuggets: Benchmarking Agentic Discovery of Long-Tail Political Facts

Large Reasoning Models (LRMs) embedded in agentic frameworks have transformed information retrieval from static, long context question answering into open-ended exploration. Yet real world use requires models to discover and synthesize "long-tail" facts from dispersed sources, a capability that remains under-evaluated. We introduce PolitNuggets, a multilingual benchmark for agentic information synthesis via constructing political biographies for 400 global elites, covering over 10000 political facts. We standardize evaluation with an optimized multi agent system and propose FactNet, an evidence conditional protocol that scores discovery, fine-grained accuracy, and efficiency. Across models and settings, we find that current systems often struggle with fine-grained details, and vary substantially in efficiency. Finally, using benchmark diagnostics, we relate agent performance to underlying model capabilities, highlighting the importance of short-context extraction, multilingual robustness, and reliable tool use.

Yifei Zhu

The University of Hong Kong

AI Agent BenchmarksMultilingual IRLarge Reasoning Models
AskPDF