Language models are increasingly asked where real actors stand: what a party's platform prioritises, what an institution has committed to. Such judgments are gradable, carrying both an order and a magnitude. Nearly every benchmark that tests them asks about one entity at a time, then reads that score as the model's knowledge of the domain. Order and magnitude, however, need not be acquired together. Text about social entities carries far more comparison than measurement, so a model may hold a dependable ordering with no anchored scale beneath it. We ask whether a model can place a documented social position on a scale or only order the actors it compares, and what that means for how such knowledge is measured. We test this on party positions in Japan and South Korea, where the Comparative Manifesto Project's coded data supplies a rare external answer key. Asked to order four parties, models beat the strongest fixed-answer baseline by more than a factor of two; asked to place one party on the same scale, they fall below it. The ordering survives inside those failed placements, and supplying the coded values lifts placement to near ceiling. Comparative and absolute questions about the same fact therefore measure different things here, and for party issue salience a benchmark built from one cannot speak for the other.
Figures & tables
Figure 1: The life cycle of the data. Coded CMP percentages are quantised into ordinal buckets, giving each benchmark cell a gold level and a tie-aware gold ranking; the same cell is then put to a model two ways. The two lanes are separate API calls with no shared history, and both ask a four-alternative question: four emphasis levels in a , four named parties in b . Each lane starts from a structured prompt; the model answers in free prose that ends with one machine-readable tagged line (Section 4.2 ), and both lanes are parsed identically. Dashed paths are the only ones that read the gold column.
Model
Corr.
Part.
Rival
None
NoKn.
DeepSeek-V3
26.4
44.4
4.2
20.8
4.2
Gemini-Flash
25.5
30.9
7.3
36.4
0.0
gpt-oss-120b
25.0
38.9
2.8
20.8
12.5
Cohere-A
22.2
31.9
5.6
25.0
15.3
gpt-oss-20b
21.1
35.2
5.6
32.4
5.6
Claude-Sonnet
20.8
22.2
0.0
2.8
54.2
Table 1: Absolute-placement verdicts by model (%), sorted by accuracy. Corr.=Correct, Part.=one bucket off, Rival/None=two or more buckets off and matching a rival’s bucket / no party’s bucket, NoKn.=No knowledge. Models differ far more in calibration than in accuracy: Claude Sonnet declines on over half of items while others rarely do ( χ2=126.7 , p<10−24 , Cramér’s V=0.51 ).
Task and scoring
Model
Base
Ratio
Order four parties
58.1
22.2
2.62 ×
Place one, exact
22.4
37.9
0.59 ×
Place one, ± 1 bucket
58.8
80.7
0.73 ×
Table 2: The dissociation. “Base” is the best single constant answer under that task’s own scoring rule: always naming one fixed party (ordering) or always giving one fixed bucket (placement). The two 58% figures sit on opposite sides of their baselines, which is why raw accuracies must not be compared directly.
Measure
Ungr.
Gr.
Sig.
Absolute placement
22.4
95.6
—
Self-consistency (highest)
86.2
96.8
p=0.039
Self-consistency (lowest)
77.5
96.9
p=0.0006
Table 3: Effect of grounding. Supplying the coded values lifts placement to near ceiling and significantly raises agreement between a model’s comparative answer and the ordering implied by its own absolute answers (Fisher’s exact). Self-consistency n : 80/80 ungrounded, 63/64 grounded.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Work
Task
Ground truth
Source text given?
Liusie et al. (2024) , Qin et al. (2024)
Judge quality of a supplied text, pairwise vs pointwise
Human preference
Yes
Licht et al. (2025)
Measure a scalar construct in a supplied text
Expert-coded scales
Yes
Lehmann et al. (2026)
Compare two entities on a numerical attribute
Recorded attribute values
No (value is in text corpus)
Calderon et al. (2026)
Recall an encoded fact
Web-grounded fact set
No
Di Leo et al. (2025) , Caliskan et al. (2025) , Benoit and Laver (2026)
Estimate party position or salience
CMP / expert surveys
Yes
This work
Order vs place a named party on a coded issue
CMP coded manifesto data
No
Appendix
Table 4: The closest prior work by task, ground truth, and whether the text to be judged is supplied in the prompt. The last column is the axis this paper occupies alone: the model must answer from parametric knowledge, against an external key, in two framings.
Measure
All 9
Variant 6
Anchor 3
Order highest
58.1
69.6
34.2
chance
25.0
25.0
25.0
Order lowest
48.3
38.5
68.4
tie-aware chance
35.3
25.0
56.6
Place, exact
22.4
21.9
23.6
best constant
37.9
41.9
66.2
Appendix
Table 5: Headline measures split by issue type (%, per408 excluded). Ordering beats chance on the variant issues (highest p≈10−16 , lowest p=0.006 ) but not on the anchor issues (highest p=0.13 , lowest p=0.094 ), where all parties share a level; placement stays below its constant baseline in every subset.
Test
p
Holm α
Holds
Calibration across models, χ2
<10−24
0.0056
yes
Order highest, binomial
3.0×10−14
0.0063
yes
Read-out SC highest, Fisher
0.0001
0.0071
yes
Grounding SC lowest, Fisher
0.0006
0.0083
yes
Order lowest, binomial (tie-aware)
0.0028
0.0100
yes
Grounding SC highest, Fisher
0.039
0.0125
no
Appendix
Table 6: Holm correction over the nine tests reported in Section 5 . The last row is a null result by design: the implied and direct orderings are expected to agree.
Real-world knowledge representation often requires capturing subjective, continuous attributes -- such as political positions -- that conflict with pairwise validation, the widely accepted gold standard for human evaluation. We address this challenge through a dual-scale validation framework applied to political stance prediction in argumentative discourse, combining pointwise and pairwise human annotation. Using 22 language models, we construct a large-scale knowledge base of political position predictions for 23,228 arguments drawn from 30 debates that appeared on the UK politicial television programme Question Time. Pointwise evaluation shows moderate human-model agreement (Krippendorff's α=0.578), reflecting intrinsic subjectivity, while pairwise validation reveals substantially stronger alignment between human- and model-derived rankings (α=0.86 for the best model). This work contributes: (i) a practical validation methodology for subjective continuous knowledge that balances scalability with reliability; (ii) a validated structured argumentation knowledge base enabling graph-based reasoning and retrieval-augmented generation in political domains; and (iii) evidence that ordinal structure can be extracted from pointwise language models predictions from inherently subjective real-world discourse, advancing knowledge representation capabilities for domains where traditional symbolic or categorical approaches are insufficient.
Jordan Robinson, Angus R. Williams, Katie Atkinson +1
University of Liverpool · The Alan Turing Institute · University of Leeds
Language models are frequently employed in settings where they are asked to make value judgments and choices. These observed choices often exhibit intransitivity: A model may prefer item A to B and B to C, while also preferring C to A. Existing work that models LLM preferences treats such inconsistencies as sampling noise around a single latent ordering. We instead propose that intransitivity reflects the aggregation of multiple latent, internally consistent orderings. We first show that observed inconsistencies cannot be explained by a single ordering under any monotone link function. We then introduce a noise-augmented mixture Bradley-Terry (MBT) model that infers latent preference components from repeated pairwise comparisons. Across seven models and four tasks, a mixture of orderings often explains structural inconsistencies better than single-utility models. We find that aggregate preferences often hide underlying preference heterogeneity. A case study on Moral Machine dilemmas shows that models which disagree on aggregate orderings can still share latent components. Together, these results suggest that LLMs reflect plural preferences. Alignment and evaluation pipelines that treat LLM preferences as a single function, therefore, risk averaging over coherent orderings that different users may endorse differently.
Aviral Chawla, William H. W. Thompson, Jean-Gabriel Young
Vermont Complex Systems Institute University of Vermont Burlington, Vermont 05403 · Vermont Complex Systems Institute Department of Statistics University of Vermont Burlington, Vermont 05403
Language models now answer questions alongside other claims: a note in the system prompt, a message from another agent, their own earlier answer. We ask what it takes to talk a model out of an answer it gets right on its own, and separate three factors that earlier studies vary together: where the advice sits in the prompt, who is said to have written it, and what the model has already written. On Qwen3.5-27B and four-option MMLU-Pro items, with the rest of the advice held fixed in each comparison, we find three things. First, position matters far more than the API channel, in opposite directions for a bare claim and an argument: a bare "the answer is X" does most harm just before the model answers, while the same claim with a false rationale does most harm before the question, where it raises error on items the model otherwise almost never misses from 1% to 67%; moving the claim between the system message and the user message at the same position changes error by a few points. Second, attributing the note to an authority rather than to another participant, and phrasing it assertively, each raise error by 10-27 points, while the two authorities we tested differ by under two. Third, replaying a correct worked answer before the advice protects it whether the text is shown as the model's own or as someone else's, and the same replay keeps wrong answers from being corrected by a bare correct note. The position result replicates under an independently written prompt. Code, items and all result tables are released at https://github.com/coolkillercat/safety.