Interpretive Variation and Discourse Structure in Detecting Enthymemes in Controversial Political Discourse
Organizations: Centre for Language Studies, Radboud University, Nijmegen, The Netherlands
Abstract
Enthymemes are arguments that leave a premise or conclusion unstated, and they are a common persuasion device on social media. Because readers must fill in the missing part themselves, they often disagree on whether a text contains one. This paper asks how to detect and evaluate enthymemes when such disagreement is part of the task. On tweets about vaccination and immigration, each labeled by five annotators, F1 stalls around 0.67 for every fine-tuned model we train, regardless of data size, added features or label type. We therefore ask how these models should be evaluated instead, and propose metrics that compare predictions with the full distribution of annotator votes, together with a model that predicts this distribution directly and captures disagreement better at no cost in F1. We then ask what role discourse structure plays. Models learn to rely on explicit argumentative structure: they reliably recognize tweets without an enthymeme but rarely commit on tweets that contain one. Both the models and a discourse parser miss arguments whose links are left implicit and must be inferred by the reader. Finally, we discuss what these blind spots mean for content moderation systems trained on majority labels.
Figures & tables
| None | Enthymeme | |||||||
| System | Acc | Macro-F1 | P | R | F1 | P | R | F1 |
| Features alone (LogReg) | ||||||||
| RST (24d) | .608 | .607 | .831 | .505 | .628 | .461 | .804 | .586 |
| Tropes (9d) | .547 | .544 | .734 | .485 | .584 | .405 | .667 | .504 |
| RST + tropes | .588 | .582 | .765 | .536 | .630 | .438 | .686 | .534 |
| Classical text (LogReg) | ||||||||
| Best | Gated | Text-only | Text-only | ||||
| Metric | Question | classical | RST | hard | soft | Betabin | |
| Hard macro-F1 | majority verdict | .624 | .667 | .627 | .671 | .675 | |
| Ambiguity Spearman | where | .143 | .115 | .058 | .172 | .282 | |
| Manhattan distance | depth | .331 | .346 | .339 | .323 | .282 | |
| Soft cross-entropy | depth | .656 | .627 | .674 | .597 | .584 | |
| Soft-ECE | calibration | .097 | .116 | .083 | .114 | .017 |
| Dimension | Easy | Rest | ||
| Easy nones (512 vs. 371) | ||||
| Spine ratio | .52 | .32 | ||
| Max-relation share | .58 | .50 | ||
| Embedded joint | .12 | .35 | ||
| evaluation satellite | .05 | .19 | ||
| attribution present | .21 | .33 | ||
| Where (15) | Depth (158) | Calibration (123) | Consistency (49) | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dimension | set | ref | set | ref | set | ref | set | ref | ||||||||
| attribution present | .53 | .25 | .031 | .28 | .25 | .48 | .26 | .24 | .73 | .24 | .25 | .99 | ||||
| elaboration share | .33 | .43 | .087 | .38 | .35 | .49 | .38 | .34 | .35 | .38 | .35 | .40 | ||||
| Max-relation share | .48 | .54 | .14 | .58 | .54 | .15 | .57 | .55 | .50 | .56 | .55 | .64 | ||||
| attribution share | .12 | .07 | .071 | .11 | .09 | .38 | .11 | .09 | .60 | .08 | .10 | .88 | ||||
| evaluation satellite | .33 | .13 | .054 | .06 | .11 | .10 | .07 | .10 | .38 | .12 | .09 | .43 | ||||
| Consensus none | Contested | Consensus enth. | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 68 (32 0/5, 36 1/5) | 18 (11 2/5, 7 3/5) | 34 (13 4/5, 21 5/5) | ||||||||||
| Dimension | set | ref | set | ref | set | ref | ||||||
| Spine ratio | .42 | .45 | .72 | .42 | .31 | .022 | .40 | .33 | .042 | |||
| attribution present | .27 | .26 | .89 | .06 | .32 | .022 | .18 | .37 | .037 | |||
| Max-relation share | .57 | .54 | .44 | .56 | .53 | .67 | .59 | .48 | .014 | |||
| attribution share | .12 | .09 | .71 | .03 | .09 | .033 | .06 | .10 | .053 | |||
| Uncommitted consensus nones | Uncommitted consensus enth. | |||||||
| 299 (185 0/5, 114 1/5) vs. 527 | 313 (131 4/5, 182 5/5) vs. 71 | |||||||
| Dimension | set | ref | set | ref | ||||
| Spine ratio | .32 | .51 | .001 | .34 | .31 | .41 | ||
| Shallowest- arg span | .17 | .08 | .001 | .17 | .18 | .053 | ||
| arg share of satellites | .23 | .12 | .001 | .21 | .32 | .001 | ||
| NN share | .25 | .18 | .001 | .25 | .29 | .060 | ||
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Dim | Feature | Motivation |
|---|---|---|
| Relational composition | ||
| 1–6 | Satellite of each core relation present at depth (indicators) | Discourse relations are the feature type that helped most in Chernyavskiy, Ilvovsky, and Nakov ( 2024 ); a satellite with one of these relations gives explicit support to the main claim. |
| 7 | Share of core-relation satellites among all satellites | How much of the extra information supports the claim rather than just adding to it. |
| 8 | Character span of the shallowest core-relation satellite / text length | How much of the tweet is spent on explicit support. |
| 9 | Fraction of core-relation satellites in SN order | Whether the support comes before the claim. |
| 10 | Spine ratio: EDUs on an all-nucleus path / number of EDUs | How much of the tweet is essential content (Nucleus) with no extra information attached. |
Annotator Disagreement
Annotator disagreement studies why human labels differ, how to measure and model that variation, and how it should shape AI training and evaluation.
See all 91 papers