cs.CLFeb 20, 2026

Validating Political Position Predictions of Arguments

Authors: Jordan Robinson, Angus R. Williams, Katie Atkinson, Anthony G. Cohn

Organizations: University of Liverpool · The Alan Turing Institute · University of Leeds

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Real-world knowledge representation often requires capturing subjective, continuous attributes -- such as political positions -- that conflict with pairwise validation, the widely accepted gold standard for human evaluation. We address this challenge through a dual-scale validation framework applied to political stance prediction in argumentative discourse, combining pointwise and pairwise human annotation. Using 22 language models, we construct a large-scale knowledge base of political position predictions for 23,228 arguments drawn from 30 debates that appeared on the UK politicial television programme Question Time. Pointwise evaluation shows moderate human-model agreement (Krippendorff's α=0.578α=0.578), reflecting intrinsic subjectivity, while pairwise validation reveals substantially stronger alignment between human- and model-derived rankings (α=0.86α=0.86 for the best model). This work contributes: (i) a practical validation methodology for subjective continuous knowledge that balances scalability with reliability; (ii) a validated structured argumentation knowledge base enabling graph-based reasoning and retrieval-augmented generation in political domains; and (iii) evidence that ordinal structure can be extracted from pointwise language models predictions from inherently subjective real-world discourse, advancing knowledge representation capabilities for domains where traditional symbolic or categorical approaches are insufficient.

Figures & tables

Figure 1: Overview of the methodology used to instantiate a structured argumentative knowledge base containing political positions predictions for all arguments. Green boxes indicate components developed specifically for this study.
Bucketnnπˉ(E3)\bar{\pi}^{(E_{3})}Interpretation
H pol{}_{\text{pol}}400≤0.05\leq 0.05high confidence, political
L200∈[0.45,0.55]\in[0.45,0.55]low confidence
H apol{}_{\text{apol}}400≥0.95\geq 0.95high confidence, apolitical
Table 2: Sampling buckets for pointwise binary classification of the presence of political sentiment.
 E3E_{3} ConfidentE3E_{3} Unconfident
HH ConfidentP1,1\mathcal{P}_{1,1}P0,1\mathcal{P}_{0,1}
HH UnconfidentP1,0\mathcal{P}_{1,0}P0,0\mathcal{P}_{0,0}
Table 3: Subsets of P\mathcal{P} by model and human confidence.
Figure 2: Nominal Krippendorff’s αn\alpha_{n} measuring inter-model agreement across data partitions.
Figure 3: Distribution of human–model agreement across dataset partitions.
Figure 4: Relationship between human–model agreement and model performance metrics.
 Left-framedRight-framed
Number of unique labels%n%n
127.926122.5210
257.353562.8587
314.813814.7127
Table 5: Unique labels per pair across annotation tasks.
Figure 5: Macro F1 plotted against ordinal human–model agreement for all models and ensembles under P\mathcal{P} .
Figure 6: Macro F1 plotted against ordinal human–model agreement for all models and ensembles under four distributions: P1,1\mathcal{P}_{1,1} , which conditioned on high-confidence model predictions and human annotations; P1,0\mathcal{P}_{1,0} , which conditioned on high-confidence model predictions and low-confidence human judgements; P0,1\mathcal{P}_{0,1} , which conditioned on low-confidence model predictions and high-confidence human annotations; and P0,0\mathcal{P}_{0,0} , which conditioned on low-confidence model predictions and human annotations.
More in this topic

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Explore similar work

CardsList

  1. Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study

    Aug 12, 2026Simone MungariLLM AuditingPolitical Bias in Language Models

    Revelis s.r.l.

  2. PoliticsBench: Benchmarking Political Values in Large Language Models with Multi-Turn Roleplay

    Mar 25, 2026Rohan Khetan, Ashna KhetanPolitical Bias in Language ModelsLanguage Model Bias Evaluation

    Northville High School, Northville, USA · Department of Computer Science, Stanford University, Stanford, USA

  3. PolitNuggets: Benchmarking Agentic Discovery of Long-Tail Political Facts

    May 13, 2026Yifei ZhuAI Agent BenchmarksMultilingual IR

    The University of Hong Kong