ProBel: Propaganda Detection with Techniques, Spans, and Explanations

Mohamed Bayan Kmainasi; Ali Ezzat Shahroor; Elisa Sartori; Giovanni Da San Martino; Firoj Alam

cs.CLAug 23, 2026

ProBel: Propaganda Detection with Techniques, Spans, and Explanations

Authors: Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Elisa Sartori, Giovanni Da San Martino, Firoj Alam

Organizations: Qatar Computing Research Institute, Qatar · University of Padova, Italy

Abstract

Propaganda detection includes several related prediction levels, ranging from sentence-level decisions to technique classification and span identification. However, it remains unclear how supervision at these levels interacts when learned jointly across Arabic and English. We present ProBel, an Arabic and English resource that aligns binary labels, multi-label annotations over 23 propaganda techniques grouped into six coarse categories, technique-labeled spans, and reference explanations for the same news sentences. It includes a substantially larger English collection and supports matched binary, coarse-grained, multi-label, and span-level tasks in both languages. We evaluate zero-shot prompting, task-specific fine-tuning, and joint training under a shared setup. A single bilingual multi-task model achieves the best overall performance and remains competitive across tasks and languages. Cross-task analysis shows that transfer depends on the supervision level. Joint classification training preserves binary performance, whereas span-only training can weaken sentence-level prediction. Joint bilingual training yields the most stable results, while monolingual fine-tuning can reduce transfer to the other language. We will release the data, code, and evaluation scripts.

Figures & tables

Figure 1: Overview of the aligned tasks and learning setup. Each Arabic and English sentence includes a binary label, fine-grained techniques grouped into six categories, labeled evidence spans, and a reference explanation, supporting task-specific and joint learning.

Figure 2: Overview of the construction of ProBel . Arabic and English content is represented with a binary label, six coarse categories, 23 fine-grained techniques, annotated text spans, and a reference explanation.

Lang.	PropXplain Train/Dev/Test	ProBel Train/Dev/Test	Added in this work
AR	18,453/1,318/ 1,326	18,453/1,318/ 1,326	Technique labels and annotated spans
EN	4,472/621/ 922	18,775/2,567/ 3,993	Additional data, technique labels, annotated spans, and explanations

Table 1: Comparison ProBel with PropXplain.

Lang.	Train	Dev	Test	Total	Pos. (%)
AR	18,453	1,318	1,326	21,097	63.7
EN	20,077	2,567	3,993	26,637	28.0
Total	38,530	3,885	5,319	47,734	43.8

Table 2: Dataset distribution by language. Pos.: the percentage of propagandistic instances.

Figure 3: Two LLM span representations. Span-tag inserts inline tags, while span-occ outputs one object per span and supports repeated or overlapping spans.

System	AR	EN
No learning
Majority label	0.380	0.419
Random, fair coin	0.499	0.470
Zero-shot, open weights
Fanar-2-27B 26	0.409	0.431
Llama-3.1-8B 11	0.672	0.437

Table 3: Binary macro-F 1 without task-specific training; best open model per language in bold.

Figure 4: Performance of Qwen2.5-7B-Instruct across five tasks and four training settings. Full configurations and ablations appear in Appendices D and F .

Figure 5: Supervision transfer with Qwen2.5-7B-Instruct across prompting and fine-tuning settings. Outlined cells mark trained tasks, and colors are normalized by column. Table 15 reports the full results.

	ProBel	Original
English training data	EN test	EN test
Original, binary only	0.582	0.641
Expanded, binary only	0.711	0.593
Expanded, multi-task bilingual	0.735	0.685

Table 4: English binary macro-F 1 under a fixed training recipe. Data expansion improves ProBel , while only multi-task training generalizes to the original PropXplain test set.

Figure 6: Five-task average by training language, Qwen2.5-7B-Instruct. English-only fine-tuning falls below the zero-shot level on Arabic.

Starting point	Training	Bin	Avg.
Qwen2.5-7B	none (zero-shot)	0.561	0.256
	off-policy, GPT-5 traces	0.697	0.441
	OPSD, best configuration	0.623	0.288
Qwen3-8B	none (zero-shot)	0.490	0.295
	OPSD, hard $\tau=0.06$	0.499	0.298
	OPSD, hard $\tau=0.5$	0.485	0.295

Table 5: Distillation results across starting points. Bin denotes binary macro-F 1 , and Avg. the five-task average across languages. Appendix K reports full results.

Testbed (metric)	$\boldsymbol{\tau}$	Zero-shot	Hard	Soft
ProBel EN binary (macro-F 1 )	0.5	0.449	0.448	0.692
HateXplain (macro-F 1 )	0.5	0.604	0.666	0.687
GSM8K (accuracy)	0.06	0.918	0.900	0.928
AIME/HMMT (Avg@12)	0.05	34.2	41.9	39.8

Table 6: Hard versus saturating (soft) clipping on four testbeds. Zero-shot is the model before OPSD training; each row uses its own metric and threshold

\tau

(the published value on GSM8K and the original AIME/HMMT benchmarks,

\tau{=}0.5

elsewhere). Best per row in bold. Appendix K reports the full grids.

System	Macro-F 1	F 1 BERT ‡
Arabic , same test split
Llama-3.1-8B (base) 13	0.588	0.507
Llama-3.1-8B (FT) 13	0.760	0.706
Mt-Sft (ours; five tasks, two languages)	0.763	0.664
English , original PropXplain test set
Llama-3.1-8B (base) 13	0.562	0.596

Table 7: Comparison with PropXplain on matched test splits. Bold marks the best macro-F 1 . ‡ Explanation BERTScores use different backbones across studies and are not directly comparable.

System	AR	EN
GPT-4, zero-shot 12	0.117	0.111 ∗
Task-specific SFT (ours)	0.421	0.255
Mt-Sft (ours)	0.411	0.241

Table 8: Technique-labeled span detection using overlap-adjusted micro-F 1 . Arabic uses the same test split. ∗ The English GPT-4 score uses the SemEval-2023 development set and is not directly comparable.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Resource	Lang.	Bin.	Tech.	Span	NLE
Da San Martino et al. 8	EN	✓	18	✓	–
SemEval-2020 7	EN	–	14	✓	–
SemEval-2023 21	9L	–	23	–	–
ArPro 12	AR	✓	23	✓	–
SemEval-2024 9	4L	✓	22	–	–
PropXplain 13	AR/EN	✓	–	–	✓

Appendix

Table 9: Task coverage in prior propaganda resources. Bin.: binary sentence- or paragraph-level classification; Tech.: the number of fine-grained techniques; Span: technique-annotated text spans; NLE: natural language explanations. 9L/4L: nine/four languages.

Lang.	Text	Tech.	Span	Multi-T	Multi-S
AR	32.6	1.8	3.0	53.3	70.7
EN	23.8	1.3	1.5	24.9	30.6

Appendix

Table 10: Annotation statistics by language. Text: the mean input length in words. Tech. and Span represent mean counts over propagandistic instances. Multi-T and Multi-S represent the percentages of propagandistic instances with more than one technique or span.

Figure 7: Frequency of the ten most common techniques among propagandistic instances.

Coarse category	Fine-grained techniques
Manipulative Wording	Loaded Language, Exaggeration–Minimisation, Obfuscation–Vagueness–Confusion, Repetition
Reputation	Name Calling–Labeling, Questioning the Reputation, Doubt, Guilt by Association, Appeal to Hypocrisy
Justification	Appeal to Authority, Appeal to Fear–Prejudice, Appeal to Popularity, Appeal to Values, Flag Waving
Simplification	Causal Oversimplification, Consequential Oversimplification, False Dilemma–No Choice
Call	Slogans, Conversation Killer, Appeal to Time
Distraction	Red Herring, Straw Man, Whataboutism

Appendix

Table 11: The shared label inventory: 23 techniques in six coarse categories, identical for both languages.

Parameter	Value
Fine-tuning (identical for every regime)
Backbone	Qwen2.5-7B-Instruct
Adapter	LoRA $r{=}16$ , $\alpha{=}32$
Learning rate	$1\times 10^{-5}$ , warmup ratio 0.05
Epochs / batch	4 / 4 per device (4 GPUs)
Max sequence length	4,096

Appendix

Table 12: Configurations used throughout the paper.

	Arabic			English
System	tag	occ	BIO	tag	occ	BIO
Whole sentence as span	0.082	0.082	0.082	0.038	0.038	0.038
AraBERT-v2 / BERT-base	–	–	0.249	–	–	0.227
Qwen2.5-7B zero-shot	0.058	0.093	–	0.087	0.108	–
Qwen2.5-7B ICL, $k{=}5$	0.128	0.200	–	0.142	0.122	–
GPT-5 (CoT)	–	0.218	–	–	0.180	–

Appendix

Table 13: Technique-labeled span detection (overlap-adjusted micro-F 1 ; best per column in bold). Tag and occ are the generative formats; BIO is the encoders’ token-level tagging. GPT-5 emits one span list, scored under the occ protocol.

Figure 8: Development ranking of retriever–pool combinations at $k{=}3$ ; the outlined cell is the winner.

Configuration	Bin	Coa	Tech	S-tag	S-occ
Arabic
Random, $k{=}5$	0.598	0.483	0.265	0.065	0.162
BGE-M3 cross, $k{=}5$	0.622	0.501	0.268	0.051	0.107
BGE-M3 same, $k{=}1$	0.613	0.512	0.286	0.085	0.170
BGE-M3 same, $k{=}3$	0.688	0.536	0.349	0.114	0.190
BGE-M3 same, $k{=}5$	0.669	0.550	0.376	0.128	0.200

Appendix

Table 14: In-context learning on the full test sets; same / cross is the demonstration-pool language.

	Arabic test					English test
Training	Bin	Coa	Tech	S-tag	S-occ	Bin	Coa	Tech	S-tag	S-occ
none (zero-shot)	0.439	0.437	0.238	0.058	0.093	0.683	0.290	0.125	0.087	0.108
AR, all five tasks	0.767	0.666	0.560	0.408	0.364	0.607	0.361	0.272	0.198	0.177
EN, all five tasks	0.400	0.061	0.047	0.004	0.041	0.712	0.355	0.268	0.139	0.203
AR+EN, all five tasks	0.763	0.682	0.575	0.411	0.362	0.735	0.410	0.272	0.189	0.241
AR, binary only	0.754	0.564	0.254	0.177	0.203	0.638	0.259	0.084	0.108	0.106

Appendix

Table 15: Every training regime evaluated on every task in both languages; best per column in bold.

	Arabic test					English test
System	Bin	Coa	Tech	S-tag	S-occ	Bin	Coa	Tech	S-tag	S-occ
No learning
Majority label	0.380	0.558	0.466	0.000	0.000	0.419	0.196	0.162	0.000	0.000
Random, fair coin	0.499	0.244	0.093	0.000	0.000	0.470	0.101	0.031	0.000	0.000
Whole sentence as span	–	–	–	0.082	0.082	–	–	–	0.038	0.038
Zero-shot, open weights (one prompt per task)

Appendix

Table 16: All baselines on all five tasks (binary macro-F 1 ; coarse and technique micro-F 1 ; spans overlap-adjusted micro-F 1 ).

	BERTScore		BLEU		METEOR
System	AR	EN	AR	EN	AR	EN
Zero-shot	0.381	0.657	0.002	0.035	0.024	0.212
ICL, $k{=}5$	0.626	0.672	0.057	0.050	0.205	0.254
SFT (same lang.)	0.663	0.718	0.102	0.103	0.254	0.327

Appendix

Table 17: Explanation quality against the reference explanations.

	Arabic		English
Technique	Sup.	F 1	Sup.	F 1
Loaded Language	675	0.798	441	0.440
Name Calling–Labeling	206	0.562	291	0.291
Exaggeration–Minimisation	159	0.272	106	0.053
Questioning the Reputation	99	0.521	73	0.115
Causal Oversimplification	58	0.033	40	0.000

Appendix

Table 18: Per-technique F 1 of Mt-Sft with test support. The last row averages the techniques with fewer than 20 test occurrences.

Figure 9: Privileged self-distillation on ProBel . Top: one LLM in two roles: a student given only the news text $x$ , and the same weights frozen with the gold annotation $y^{\star}$ in context. The teacher scores the student’s own rollout, and only the student is updated. Bottom: disagreement concentrates on the wrong label token (the depicted token carries 5.6 nats, ${\approx}90\times$ the published clip $\tau=0.06$ ), exactly where the hard clip is flat and contributes zero gradient, while our saturating clip continues to provide a learning signal.

Init	Configuration	Bin	AVG
Qwen2.5 (ZS 0.561/0.256)	think, hard $\tau{=}.06$	0.519	0.253
	think, no clip	0.514	0.245
	no-think, hard $\tau{=}.06$	0.574	0.257
	no-think, no clip	0.556	0.277
	no-think, no clip, $r{=}16$	0.623	0.288
	think, no clip, $r{=}16$	0.465	0.232

Appendix

Table 19: All 21 trained OPSD configurations (mean of Arabic and English; each block’s initialization in parentheses).

Teacher context	Qwen2.5-7B	Mt-Sft
none (= student)	42.0	70.7
full gold block	93.7	94.3
label lines only	98.3	92.7
explanation only	71.0	83.0
mismatched block	20.3	53.0
copies the mismatched label	73.0	38.7

Appendix

Table 20: Teacher label accuracy (%) under different privileged contexts, on 300 training-pool items; the italic row is how often the teacher repeats a wrong asserted label.

Figure 10: Signal survival under the clip. Left: share of divergence a hard clip keeps,

S(\tau)

, for label tokens (solid) and all tokens (dashed). Right: each clip’s gradient factor over the empirical label-token divergence (gray histogram); hard clips drop to zero at their thresholds. At the published

\tau{=}0.06

, only 1.8% of the total divergence mass and 0.8% of the label-token mass survive.

Figure 11: Hard and saturating clips at the published

\tau

(left) and

\tau{=}0.5

(right); each panel uses its own metric, and dotted lines mark the untrained model. Points show medians when several seeds exist. On the original OPSD benchmark, hard clipping is better; on GSM8K, a larger threshold suffices for either shape; on HateXplain, saturating clipping is better at both thresholds; on ProBel , hard clipping remains near the untrained floor (one of three seeds at the published

\tau

partially escaped; see text), while saturating clipping reaches 0.69 at

\tau{=}0.5

Figure 12: Replication of OPSD on the original benchmarks (Qwen3-1.7B, authors’ code and data). Bands show three seeds; reference lines mark the published base and best results.

Figure 13: Per-token forward KL between the privileged teacher and the student at initialization, on each domain’s OPSD training data (mathematics: Qwen3-1.7B; ProBel : Qwen3-8B). The shaded region above the published

\tau

receives no gradient under the hard clip. The above-

\tau

tail is mainly stylistic on mathematics but consists of decision tokens on ProBel .

Comparison (binary task)	$b$	$c$	$p$
Llama-3.1-8B vs. GPT-5 CoT (AR)	251	197	0.012
Qwen2.5-7B vs. GPT-5 CoT (EN)	746	417	$<$ 0.001
Mt-Sft vs. best prompted (AR)	242	116	$<$ 0.001
Mt-Sft vs. best prompted (EN)	396	225	$<$ 0.001
Mt-Sft vs. zero-shot (AR)	579	179	$<$ 0.001
ICL retrieved vs. random, $k{=}3$ (AR)	298	151	$<$ 0.001

Appendix

Table 21: Exact McNemar tests on paired binary predictions (

b

c

: items only the first/second system classifies correctly). The last row shows the OPSD freeze: only

6{+}6

of 1,326 predictions change, so the trained model is statistically indistinguishable from its initialization.

Explore similar work

Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas
May 13, 2026Lukas Stähelin, Veronika Solopova, Max Upravitelev +6LLM Prompting LLM Fine-Tuning
Technische Universität Berlin, QU Lab, XplaiNLP Group, Berlin, Germany · German Research Center for Artificial Intelligence (DFKI), Berlin, Germany · Centre for European Research in Trusted AI (CERTAIN)
Reasoning for Propaganda Detection via Label-Guided CoT Optimisation
Oct 12, 2026Zeba Afroz, Harsh Vardhan, Md. Shad AkhtarCoT Reasoning Text Classification
Indraprastha Institute of Information Technology Delhi
Unmasking Propaganda: A Comparative Analysis of Masked and Causal Language Models
Oct 2, 2026Claudiu Creanga, Ioachim Lihor, Liviu P. Dinu
Faculty of Mathematics and Informatics, Human Language Technologies Research Center, Interdisciplinary School of Doctoral Studies, University of Bucharest Romania

May 13, 2026J/K move · Enter open · S save

Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas

Lukas Stähelin, Veronika Solopova, Max Upravitelev, David Kaplan+5

Technische Universität Berlin, QU Lab, XplaiNLP Group, Berlin, Germany · German Research Center for Artificial Intelligence (DFKI), Berlin, Germany · Centre for European Research in Trusted AI (CERTAIN)

LLM Prompting LLM Fine-Tuning Language Model Robustness

Open paper AskPDF arXiv

Propaganda detection in social media is challenging due to noisy, short texts and low annotation agreements. We introduce a new intent-focused taxonomy of propaganda techniques and compare it against an established, higher-agreement schema. Along three dimensions (model portfolio, schema effects, and prompting strategy) we evaluate the taxonomies as a classification task with the help of four language models (GPT-4.1-nano, Phi-4 14B, Qwen2.5-14B, Qwen3-14B). Our results show that fine-tuning is essential, since it transforms weak zero-shot baselines into competitive systems and reveals methodological differences that are hidden using base models. Across schemas, the Qwen models achieve the strongest overall performance, and Phi-4 14B consistently outperforms GPT-4.1-nano. Our hierarchical prompting method (HiPP), which predicts fine-grained techniques before aggregating them, is especially beneficial after fine-tuning and on the more ambiguous, low-agreement taxonomy, while remaining competitive on the simpler schema. The HQP dataset, annotated with the new intent-based labels, provides a richer lens on propaganda's strategic goals and a challenging benchmark for future work on robust, real-world detection.

Abstract

Explore similar work

Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas

Reasoning for Propaganda Detection via Label-Guided CoT Optimisation

Unmasking Propaganda: A Comparative Analysis of Masked and Causal Language Models