Retrieving Relations, Detecting Fallacies: A RAG Approach to Political Debate Analysis

Deborah Dore; Greta Damo; Elena Cabrio; Serena Villata

cs.AIAug 18, 2026

Retrieving Relations, Detecting Fallacies: A RAG Approach to Political Debate Analysis

Authors: Deborah Dore, Greta Damo, Elena Cabrio, Serena Villata

Organizations: Université Côte d’Azur, CNRS, INRIA, I3S, France

Abstract

Fallacies are arguments that employ invalid reasoning, making their automatic detection critical in sensitive contexts such as high-stakes political debates, where public opinion is shaped. Spotting a fallacious argument requires contextual knowledge beyond its pure surface text. This entails world knowledge pertaining to the subject matter under discussion, as well as knowledge of the relationships that exist between arguments within the argumentative discourse. Prior work on fallacy analysis has shown that argumentative discourse structure can beneficially improve classification performance. However, such structure is typically encoded only as static classifier features, limiting its flexibility. Building on this intuition while addressing this limitation, we introduce a guided retrieval-augmented methodology for fallacy detection and classification that leverages argumentative relations of support and attack to dynamically steer the extraction of relevant documents. We evaluate our approach on the ElecDeb60to20 benchmark across 42 retrieval configurations and 14 models, performing retrieval over a 15GB knowledge base of collected political-related documents. Our approach improves macro-F1 up to 0.864 for fallacy detection and up to 0.725 for classification over non-retrieval baselines. These results show that incorporating external knowledge significantly enhances fallacy detection and classification when retrieval is argumentatively guided.

Figures & tables

Figure 1: Proposed pipeline for fallacy analysis using external knowledge.

Collection	Size	# Documents	# Paragraphs
Bills	14 GB	178,383	6,668,178
Geography	516 MB	125,891	125,891
People	297 MB	72,950	72,950
History	89 MB	16,121	16,121
Exemplars	5 MB	1,273	1,284
Definitions	1.3 MB	65	65

Table 1: Statistics of the collected KB documents.

	Detection			Classification
Model	Baseline	RAG	$\Delta$	Baseline	RAG	$\Delta$
RoBERTa-base	$74.7\pm 0.8$	$80.9\pm 1.5$	$+6.2$	$34.5\pm 25.0$	$68.0\pm 2.3$	$+33.5$
DeBERTa-v3-base	$74.2\pm 6.1$	$77.5\pm 2.2$	$+3.4$	$43.5\pm 13.0$	$45.5\pm 19.7$	$+2.0$
ModernBERT-base	$76.4\pm 1.7$	$\mathbf{86.4\pm 1.2}$	$+10.0$	$57.7\pm 2.1$	$\mathbf{72.5\pm 4.0}$	$+14.8$
NeoBERT	$57.2\pm 21.9$	$50.8\pm 24.0$	$-6.4$	$24.9\pm 6.8$	$10.8\pm 9.8$	$-14.1$
LegalBERT	$71.7\pm 1.5$	$77.4\pm 1.7$	$+5.8$	$53.9\pm 23.7$	$60.8\pm 3.7$	$+6.9$

Table 2: Macro-F1 (mean

\pm

std, 5 seeds) results. RAG uses a Sentence-RooseBERT retriever with hybrid search (temporal context for detection, none for classification).

\Delta=

RAG

-

Baseline. Best model per column in bold .

Retrieval query
Configuration	Detection	Classification
Full pipeline	$\mathbf{80.2}$	$\mathbf{71.8}$
$\rightarrow$ sentence+relations	$78.3$	$61.0$
$\rightarrow$ target sentence only	$73.0$	$66.4$
Retriever
$\rightarrow$ BGE	$77.8$	$57.9$

Table 3: Results of the ablation study (Macro F1) with hyperparameters frozen. Single seed (14).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Models	Avg. Macro F1
Llama-3.1-8B-Instruct	80.5 $\pm$ 0.6
DeBERTa-v3-base	78.6 $\pm$ 0.8
gpt-oss-20b	78.5 $\pm$ 0.8
Mistral-7B-Instruct-v0.3	76.6 $\pm$ 2.9
ModernBERT-base	76.4 $\pm$ 1.0
Longformer-base-4096	76.3 $\pm$ 1.2

Appendix

Table 4: Argument Component Relation classification performance (mean

\pm

std over 5 seeds) over LIARArg.

Model	Baseline	RAG	$\Delta$
RoBERTa-base	73.5 $\pm$ 0.6	85.0 $\pm$ 0.8	$+11.5$
DeBERTa-v3-base	74.5 $\pm$ 0.5	42.7 $\pm$ 0.0	$-31.8$
ModernBERT-base	73.3 $\pm$ 0.5	81.3 $\pm$ 0.8	$+8.0$
NeoBERT	53.5 $\pm$ 14.7	42.7 $\pm$ 0.0	$-10.8$
LegalBERT	73.0 $\pm$ 0.4	84.9 $\pm$ 0.3	$+11.9$
Longformer-base-4096	73.7 $\pm$ 0.4	85.5 $\pm$ 1.4	$+11.8$

Appendix

Table 5: Propaganda detection performance (macro-F1, mean

\pm

std over 5 seeds) for baseline models and their RAG-augmented counterparts, with the corresponding

\Delta

improvement.

	Fallacy Detection						Fallacy Classification						Propaganda Detection
	Baseline			RAG			Baseline			RAG			Baseline			RAG
Model	BS	LR	Ep.	BS	LR	Ep.	BS	LR	Ep.	BS	LR	Ep.	BS	LR	Ep.	BS	LR	Ep.
RoBERTa-base	16	$2e^{-5}$	5	8	$3e^{-5}$	5	8	$5e^{-5}$	5	8	$3e^{-5}$	5	8	$2e^{-5}$	3	16	$5e^{-5}$	3
DeBERTa-v3-base	8	$2e^{-5}$	5	16	$2e^{-5}$	3	8	$2e^{-5}$	5	16	$5e^{-5}$	5	16	$2e^{-5}$	3	16	$2e^{-5}$	3
ModernBERT-base	8	$3e^{-5}$	5	8	$5e^{-5}$	5	8	$3e^{-5}$	5	8	$5e^{-5}$	5	16	$2e^{-5}$	5	8	$3e^{-5}$	5
NeoBERT	8	$5e^{-5}$	5	16	$2e^{-5}$	5	8	$3e^{-5}$	5	16	$2e^{-5}$	5	8	$3e^{-5}$	3	16	$5e^{-5}$	3

Appendix

Table 6: Best hyperparameters for fallacy detection and classification (ElecDeb60to20), and for propaganda detection (PROPAGANDA), Baseline vs. RAG settings (5 seeds). BS = batch size, LR = learning rate, Ep. = num. of epochs.

Computational Argumentation

Computational argumentation studies how to represent, analyze, and apply arguments and their relationships using formal and computational methods.

See all 99 papers

Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection
Sep 16, 2026Navyansh Singh, Animesh Pathak, Aarav SinghArgument Mining Computational Argumentation
IIIT Naya Raipur
Who Argues What? Joint Argument-Entity Detection and Classification in Political Debates
Sep 9, 2026Lucio La Cava, Stefano Francesco Monea, Sergio GrecoNamed Entity Recognition Argument Mining
DIMES Dept., University of Calabria v. P. Bucci 44Z, 87036 Rende, CS, Italy
UZHCL_for_the_win at Touche: Encoder Fine-Tuning, Synthetic Augmentation, and an Underpowered Encoder-LLM Comparison
Oct 12, 2026Stylianos PsychiasComputational Argumentation Logical Reasoning
University of Zurich, Zurich, Switzerland

Sep 16, 2026J/K move · Enter open · S save

Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection

Navyansh Singh, Animesh Pathak, Aarav Singh

IIIT Naya Raipur

Argument Mining Computational Argumentation Classification Benchmark Validity

Open paper AskPDF arXiv

Fallacy-detection benchmarks pair fallacy classes with a single "valid" or "none" class that takes everything data collection did not label as a fallacy. A detector has two jobs, deciding whether an argument is fallacious and naming which fallacy it commits, and the false-positive rate is meant to measure the first. We show that what these benchmarks actually score is scheme recognition, the ability behind the second job. Their own test sets already show it: when a classifier misses a fallacy, the error lands on "none" rather than on another fallacy type, so detection is failing while classification holds. The reason is what the valid class lacks. The negatives that separate the two jobs are correct arguments using the same argumentation scheme as a fallacy, and they are scarce: nearly absent from the four benchmarks we examined, and rare even under deliberate search. A detector is therefore never tested where recognizing a scheme and judging its use come apart, and can pass on recognition alone. We construct the missing arguments, together with a control condition from the same pipeline that differs only in scheme, so whatever generation contributes, it contributes to both. The classifier labels the scheme-matched negatives as the source fallacy, and labels the wrong-scheme negatives as the scheme they actually use 85.9% of the time and as the source type 0.4%. The classifier has learned which scheme an argument uses, not whether it uses it correctly. The over-flagging follows: a model that scores 16.6% on CoCoLoFa's own valid class flags 58.9% of the constructed arguments. The same dissociation appears in three zero-shot LLM detectors that never saw these benchmarks. We release the items as Scheme Foils. A reported false-positive rate should not be trusted as a measure of detection until the valid class has been audited for scheme-matched coverage.

Abstract

Computational Argumentation

Explore similar work

Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection

Who Argues What? Joint Argument-Entity Detection and Classification in Political Debates

UZHCL_for_the_win at Touche: Encoder Fine-Tuning, Synthetic Augmentation, and an Underpowered Encoder-LLM Comparison