Interpretive Variation and Discourse Structure in Detecting Enthymemes in Controversial Political Discourse

Martial Pastor; Nelleke Oostdijk

cs.CLOct 22, 2026

Interpretive Variation and Discourse Structure in Detecting Enthymemes in Controversial Political Discourse

Authors: Martial Pastor, Nelleke Oostdijk

Organizations: Centre for Language Studies, Radboud University, Nijmegen, The Netherlands

Abstract

Enthymemes are arguments that leave a premise or conclusion unstated, and they are a common persuasion device on social media. Because readers must fill in the missing part themselves, they often disagree on whether a text contains one. This paper asks how to detect and evaluate enthymemes when such disagreement is part of the task. On tweets about vaccination and immigration, each labeled by five annotators, F1 stalls around 0.67 for every fine-tuned model we train, regardless of data size, added features or label type. We therefore ask how these models should be evaluated instead, and propose metrics that compare predictions with the full distribution of annotator votes, together with a model that predicts this distribution directly and captures disagreement better at no cost in F1. We then ask what role discourse structure plays. Models learn to rely on explicit argumentative structure: they reliably recognize tweets without an enthymeme but rarely commit on tweets that contain one. Both the models and a discourse parser miss arguments whose links are left implicit and must be inferred by the reader. Finally, we discuss what these blind spots mean for content moderation systems trained on majority labels.

Figures & tables

Features alone (LogReg)
			None			Enthymeme
System	Acc	Macro-F1	P	R	F1	P	R	F1
RST (24d)	.608	.607	.831	.505	.628	.461	.804	.586
Tropes (9d)	.547	.544	.734	.485	.584	.405	.667	.504
RST + tropes	.588	.582	.765	.536	.630	.438	.686	.534
Classical text (LogReg)

Table 1: Test-set results (148 tweets; 97 none, 51 enthymeme). P = precision, R = recall. The None and Enthymeme columns give per-class scores, treating none and enthymeme in turn as the positive class. All systems were developed on the 1,334-tweet train+dev pool, with selection based on out-of-fold cross-validation scores, and evaluated on the test set once. Shallow models (Logistic Regression) are single deterministic fits; transformers report mean

\pm

standard deviation over 5 seeds, with per-class scores from the seed-averaged predictions, at a fixed 0.5 threshold unless noted. The hard-label row at 0.5 includes one near-degenerate seed, which raises its standard deviation to .06 against .01 to .03 for all other transformer rows; its dev-tuned row repairs the threshold per seed. Best score per column in bold.

Figure 1: Out-of-fold (OOF) macro-F1 as a function of training-pool size (honest cross-validation, per-fold tuned thresholds). The arrow next to the metric name gives the direction of improvement ( $\uparrow$ : higher is better). Shaded bands give $\pm$ 1 sd over repeated draws; the full-size points (1,334) carry no standard deviation, as they use the entire training pool and admit no repeated draws.

Figure 2: Out-of-fold probabilities of the text-only soft baseline by annotator vote band (points jittered vertically within each band; horizontal bars give the band mean). Blue points are counted correct by F1, orange as errors; the dashed line marks the 0.5 threshold. Only the 0/5 band spreads across the scale. The bands from 2/5 to 5/5 all sit between 0.5 and 0.62 and overlap, so contested tweets and unanimous enthymemes receive the same probabilities, a difference F1 does not register.

			Best	Gated	Text-only	Text-only
Metric	Question		classical	RST	hard	soft	Betabin
Hard macro-F1	majority verdict	$\uparrow$	.624	.667	.627	.671	.675
Ambiguity Spearman	where	$\uparrow$	.143	.115	.058	.172	.282
Manhattan distance	depth	$\downarrow$	.331	.346	.339	.323	.282
Soft cross-entropy	depth	$\downarrow$	.656	.627	.674	.597	.584
Soft-ECE	calibration	$\downarrow$	.097	.116	.083	.114	.017

Table 2: Evaluation of the best classical system (TF-IDF + RST + tropes, Logistic Regression with probability outputs) and the transformer systems, grouped by the question each metric answers about the vote distribution. Test set (148 tweets; 5 seeds for transformers, single deterministic fit for the classical system), except soft-ECE and the annotator-conditioned per-vote metric ( consistency , ours, defined in Section 5.3 ), computed out-of-fold (1,334 tweets). Arrows give the direction of improvement; best score per row in bold.

Figure 3: Soft metrics vs. training-pool size (out-of-fold; mean $\pm$ 1 sd over repeated draws; arrows = direction of improvement). The full-size points (1,334) carry no standard deviation, as they use the entire training pool and admit no repeated draws.

Dimension	Easy	Rest	$\boldsymbol{\delta}$	$\boldsymbol{p}$
Easy nones (512 vs. 371)
Spine ratio	.52	.32	$+.48$	$<.001$
Max-relation share	.58	.50	$+.16$	$<.001$
Embedded joint	.12	.35	$-.22$	$<.001$
evaluation satellite	.05	.19	$-.14$	$<.001$
attribution present	.21	.33	$-.12$	$<.001$

Table 3: RST dimensions distinguishing the easy tweets, band-matched within each class. Easy and Rest give the mean value of each dimension in the two groups (for binary dimensions, the share of tweets that have it);

\delta

is Cliff’s delta, the strength of the difference: comparing one easy tweet and one rest tweet at random,

\delta

says how much more often the easy one has the higher value (

0

: no difference;

\pm 1

: one group is always higher, whatever the size of the gap, since

\delta

depends only on the ordering of the values). For binary dimensions

\delta

equals the difference in shares. Green marks enrichment, red depletion; shading intensity is proportional to

|\delta|

p

is the Mann–Whitney (continuous) or Fisher exact (binary) test; values below

.001

are collapsed.

	Where (15)				Depth (158)				Calibration (123)				Consistency (49)
Dimension	set	ref	$\delta$	$p$	set	ref	$\delta$	$p$	set	ref	$\delta$	$p$	set	ref	$\delta$	$p$
attribution present	.53	.25	$+.29$	.031	.28	.25	$+.03$	.48	.26	.24	$+.02$	.73	.24	.25	$-.00$	$>$ .99
elaboration share	.33	.43	$-.27$	.087	.38	.35	$+.03$	.49	.38	.34	$+.05$	.35	.38	.35	$+.07$	.40
Max-relation share	.48	.54	$-.23$	.14	.58	.54	$+.07$	.15	.57	.55	$+.04$	.50	.56	.55	$+.04$	.64
attribution share	.12	.07	$+.23$	.071	.11	.09	$+.03$	.38	.11	.09	$+.02$	.60	.08	.10	$-.01$	.88
evaluation satellite	.33	.13	$+.20$	.054	.06	.11	$-.05$	.10	.07	.10	$-.03$	.38	.12	.09	$+.04$	.43

Table 4: Discourse profiles of the four success sets: 22 of the 24 dimensions (the condition and comparison satellite indicators are not shown) compared against vote-band-matched references (conventions as in Table 3 : set and ref give means, or shares for binary dimensions;

\delta

is Cliff’s delta, equal to the difference in shares for binary dimensions;

p

is the Mann–Whitney or Fisher exact test, values below

.001

collapsed). Green marks enrichment, red depletion; shading intensity is proportional to

|\delta|

and encodes effect size, not significance. With set sizes ranging from 15 to 158, significance tracks sample size as much as effect: small significant effects in the large sets and large non-significant effects in the small where set should be read accordingly.

	Consensus none				Contested				Consensus enth.
	68 (32 $\times$ 0/5, 36 $\times$ 1/5)				18 (11 $\times$ 2/5, 7 $\times$ 3/5)				34 (13 $\times$ 4/5, 21 $\times$ 5/5)
Dimension	set	ref	$\delta$	$p$	set	ref	$\delta$	$p$	set	ref	$\delta$	$p$
Spine ratio	.42	.45	$-.03$	.72	.42	.31	$+.34$	.022	.40	.33	$+.21$	.042
attribution present	.27	.26	$+.01$	.89	.06	.32	$-.26$	.022	.18	.37	$-.19$	.037
Max-relation share	.57	.54	$+.06$	.44	.56	.53	$+.06$	.67	.59	.48	$+.25$	.014
attribution share	.12	.09	$+.02$	.71	.03	.09	$-.25$	.033	.06	.10	$-.17$	.053

Table 5: Discourse profile of the where -failure set — the 120 tweets whose contestedness every system misjudges (uncertainty rank displaced from the vote-entropy rank by more than 30 percentile points in all four transformers) — split by vote band and compared against the remaining tweets of the same bands (same conventions as in previous Tables).

Figure 6: The two construals of the unmarked junction on Tweet #196 (2/5 votes). The parser reads elaboration , a continuation; the annotators who hold the implicit warrant $P_{2}$ read evidence . The structure fixes the site of the argument; whether it is read depends on the reader supplying $P_{2}$ .

	Uncommitted consensus nones				Uncommitted consensus enth.
	299 (185 $\times$ 0/5, 114 $\times$ 1/5) vs. 527				313 (131 $\times$ 4/5, 182 $\times$ 5/5) vs. 71
Dimension	set	ref	$\delta$	$p$	set	ref	$\delta$	$p$
Spine ratio	.32	.51	$-.45$	$<$ .001	.34	.31	$+.06$	.41
Shallowest- arg span	.17	.08	$+.29$	$<$ .001	.17	.18	$-.14$	.053
arg share of satellites	.23	.12	$+.29$	$<$ .001	.21	.32	$-.23$	.001
NN share	.25	.18	$+.22$	$<$ .001	.25	.29	$-.14$	.060

Table 6: The two consensus-pole failure profiles, extracted from the nested depth , calibration and consistency failure sets. Shown is the no-commitment criterion applied to the consensus bands: tweets on which no system places a probability within

.35

of the majority pole, profiled against the committed tweets of the same bands (conventions as in Table 3 ;

\delta

is Cliff’s delta;

p

is Mann–Whitney / Fisher exact; green marks enrichment, red depletion, shading proportional to

|\delta|

; uncorrected screening).

Figure 7: Fine-tuning shift $\Delta$ per probed signature (probe macro-F1 on the fine-tuned encoder minus the pretrained one, last-layer CLS). The shaded band marks the random-signature floor ( $\pm.038$ ); the dashed line marks the label’s own shift ( $+.084$ ).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Relational composition
Dim	Feature	Motivation
1–6	Satellite of each core relation present at depth $\leq 2$ (indicators)	Discourse relations are the feature type that helped most in Chernyavskiy, Ilvovsky, and Nakov ( 2024 ); a satellite with one of these relations gives explicit support to the main claim.
7	Share of core-relation satellites among all satellites	How much of the extra information supports the claim rather than just adding to it.
8	Character span of the shallowest core-relation satellite / text length	How much of the tweet is spent on explicit support.
9	Fraction of core-relation satellites in SN order	Whether the support comes before the claim.
10	Spine ratio: EDUs on an all-nucleus path / number of EDUs	How much of the tweet is essential content (Nucleus) with no extra information attached.

Appendix

Table A.1: The 24-dimensional RST feature encoding. All features are indicators (present or absent) or ratios computed over the automatically generated RST tree. The candidate features were constructed from two motivated sources: the argumentative-satellite patterns hypothesized to distinguish explicit from implicit arguments, and the discourse configurations associated with annotator disagreement in our data. From this pool, we kept the 24 dimensions that remained predictive across training folds. We kept features given nonzero weight by an L1 logistic regression in at least three of five training folds.

Figure B.1: Contested where -success set (Section 6.2.1 ), 2/5 enthymeme votes, reconstruction of annotator 1. The backbone of the profile is visible: an attribution node (“German government answer:”) reports an explanation from an exterior source, and an evaluation satellite (“The vaccine is DEFINITELY not to blame!”) judges it ironically. The conclusion is explicit only through the irony, and the annotators split three ways (premise, conclusion, none). Where a manual re-annotation departs from the parser, both labels are shown (re-annotated above, parser below).

Figure B.2: Uncommitted consensus enthymeme (Section 6.3.2 ), 5/5 votes, implicit premise of annotator 1. Parser tree with manual corrections: where the re-annotation departs from the parser, both labels are shown (re-annotated in red above, parser in blue below), and the question’s two EDUs are merged. The parser tree fits the profile of the set: no argumentative satellite, elaboration on half the nodes, unmarked junctions. It labels the second sentence elaboration where it is the evidence for the rhetorical question, and attaches the sarcasm marker “Oh wait…” to the whole tweet; the warrant linking evidence and question is the implicit premise, and nothing in the parser’s tree encodes it.

Figure B.3: Uncommitted consensus enthymeme (Section 6.3.2 ), 5/5 enthymeme votes. Parser tree with manual corrections (re-annotated in red above, parser in blue below; the list is flattened and the split last clause merged). The parser gives a bare paratactic joint chain with a spine ratio of 1 and no connective: the list of actions is the evidence for the judgment in the last EDU, but nothing in the parser’s tree marks it as such.

Figure B.4: Consistency set (Section 6.2.2 ), 0/5 enthymeme votes. Parser tree with corrections (re-annotated in red above, parser in blue below; the split first question is merged). Both topic–comment satellites are rhetorical questions, which can be rephrased as propositions and used in an argument, a reconstruction step that tends to produce inconsistent annotator behavior: here one annotator built the full argument with a scheme and still voted none . The text-only model only senses that this structure may yield ambiguous votes and stays near the threshold; betabin has learned that an enthymeme reading is possible but most likely an inconsistency, and commits the tweet toward none . Only the per-vote consistency metric can observe this difference.

Annotator Disagreement

Annotator disagreement studies why human labels differ, how to measure and model that variation, and how it should shape AI training and evaluation.

See all 91 papers

A Resource for Enthymeme Detection in Controversial Political Discourse
Jun 10, 2026Martial Pastor, Nelleke OostdijkAnnotator Disagreement Human-in-the-Loop Annotation
Centre for Language Studies, Radboud University, Nijmegen
Making Implicit Premises Explicit in Logical Understanding of Enthymemes
Mar 6, 2026Xuyao Feng, Anthony HunterNatural Language Inference Implicit Reasoning in Language Models
Department of Computer Science, University College London, United Kingdom
High prevalence of antagonizing rhetoric in trending topics on Twitter/X
Oct 15, 2026Armin Pournaki, Eckehard OlbrichPolitical Discourse Analysis Political Communication
Institute of Sociology, University of Bern, Bern, Switzerland · Max Planck Institute for Mathematics in the Sciences, Leipzig, Germany · médialab, SciencesPo, Paris, France

Jun 10, 2026J/K move · Enter open · S save

A Resource for Enthymeme Detection in Controversial Political Discourse

Martial Pastor, Nelleke Oostdijk

Centre for Language Studies, Radboud University, Nijmegen

Annotator Disagreement Human-in-the-Loop Annotation Text Classification Computational Argumentation

Open paper AskPDF arXiv

Enthymemes, arguments with unstated premises or conclusions, are pervasive in persuasive discourse, yet their annotation remains notoriously subjective. We present a resource of 1,482 tweets from politically controversial discourse, annotated by five annotators for the presence of enthymemes and their argument structure, designed to study label variation. We first revisit the definition of enthymemes and propose annotation guidelines anchored in Walton's argumentation schemes, offering a structured and constrained approach that nonetheless preserves room for the interpretive nature of the task. This contrasts with past resources, which tend to eliminate disagreement, obscuring its sources and preventing investigation of its potential benefits for model performance. We further propose a complexity analysis of the task, identifying where annotation imposes high cognitive load and may give rise to inconsistent annotation. Our preliminary experiments show that models trained on annotator disagreement outperform models trained on hard majority-vote labels. We close by reflecting on how structural openness in enthymeme definitions and guidelines enables the study of variation in subjective inferential processes for future resources and downstream NLP applications concerned with human inference.

Abstract

Annotator Disagreement

Explore similar work

A Resource for Enthymeme Detection in Controversial Political Discourse

Making Implicit Premises Explicit in Logical Understanding of Enthymemes

High prevalence of antagonizing rhetoric in trending topics on Twitter/X