Unmasking Propaganda: A Comparative Analysis of Masked and Causal Language Models

Claudiu Creanga; Ioachim Lihor; Liviu P. Dinu

cs.CLOct 2, 2026

Unmasking Propaganda: A Comparative Analysis of Masked and Causal Language Models

Authors: Claudiu Creanga, Ioachim Lihor, Liviu P. Dinu

Organizations: Faculty of Mathematics and Informatics, Human Language Technologies Research Center, Interdisciplinary School of Doctoral Studies, University of Bucharest Romania

Abstract

Propaganda detection is an essential task in natural language processing (NLP), particularly in the context of manipulative political communications. However, identifying specific propaganda techniques presents a significant challenge due to their often subtle nature and reliance on context, making them difficult to distinguish from legitimate persuasive language. Propaganda often involves highlighting certain facts while downplaying or ignoring others to create a desired perception. This biased communication aims to influence attitudes, beliefs, or behaviors towards a particular cause or position. This paper explores advances in detecting propaganda techniques through a comparative analysis of modern language models, using the SemEval-2020 Task 11 dataset. We evaluated both masked language models (based on XLM-RoBERTa or DeBERTa V3) and causal models (from OpenAI, Google, Mistral, Anthropic and Meta), employing two prompting strategies: base and chain-of-thought prompting. Our results demonstrate improvements over state-of-the-art models, with the best-performing MLM achieving an F1 score of 63.18 in technique classification and the best causal model achieving 63.62. We also observed that certain models excel in specific techniques, such as loaded language and name-calling, while struggling with others like bandwagon and black-and-white fallacy. These findings suggest that fine-tuning, ensemble modeling, and the use of larger datasets can further enhance propaganda detection capabilities.

Figures & tables

Figure 1. Labels distribution in the corpus, split by train and test. On the y-axis the number of articles and on the x-axis the labels. Loaded language is the most frequent technique.

Model	F1-Score	Precision	Recall
Gemini Flash 2.5	63.62	53.75	77.93
Gemini Flash 2.0	63.10	60.51	65.92
GPT-4.1	60.96	53.60	70.67
GPT-4o	59.44	49.91	73.46
GPT-4o-mini	57.67	51.42	65.64
gpt-3.5-ft	55.48	71.68	45.25

Table 1. Model performance with Base Prompting (gpt-3.5-ft is a gpt-3.5-turbo fine-tuned model). The fine-tuned model was trained using 7,178,874 tokens over 3 epochs with a batch size of 2, resulting in 2246 steps, an LR multiplier of 2 and a seed of 477055734 (id: ft:gpt-3.5-turbo-0125:personal::9K750VBH).

Model	F1-Score	Precision	Recall
Gemini Flash 2.5	61.55	50.63	78.49
Gemini Flash 2.0	59.77	55.05	65.36
GPT-4o-mini	58.67	51.82	67.60
GPT-4o	56.58	43.64	80.45
gpt-4-turbo	55.31	52.10	58.94
GPT-4.1	52.88	36.68	94.69

Table 2. Model performance with Chain-of-thought prompting. Best performing model is Gemini Flash 2.5, but the model with the highest recall was GPT-4o.

Techniques	Claude sonnet	OpenAI gpt-3.5	OpenAI GPT-4o	OpenAI gpt-4	Meta llama3	Mistral large	Mistral medium	Gemini 1.0-pro	Gemini 1.5-pro	Gemini Flash 2.0	Gemini Flash 2.5
appeal_to_authority	0.38	0.00	0.37	0.00	0.14	0.36	0.21	0.10	0.43	0.29	0.33
appeal_to_fear	0.53	0.35	0.48	0.49	0.07	0.45	0.29	0.51	0.62	0.63	0.64
bandwagon	0.12	0.00	0.16	0.21	0.10	0.00	0.00	0.09	0.14	0.18	0.18
black_and_white	0.13	0.00	0.35	0.35	0.00	0.30	0.31	0.22	0.26	0.47	0.29
oversimplification	0.29	0.00	0.31	0.41	0.00	0.32	0.37	0.27	0.36	0.41	0.46
doubt	0.35	0.31	0.30	0.54	0.00	0.48	0.59	0.38	0.51	0.52	0.56

Table 3. Comparative analysis of causal language models’ performance (F1 macro scores) in detecting specific propaganda techniques.

Figure 2. This confusion matrix visualizes the performance for the model GPT-4o, using base prompting. (class 0 is the existence of propaganda technique and class 1 is the absence of propaganda technique)

Figure 3. This confusion matrix visualizes the performance for the model GPT-4o-mini, using chain-of-thought prompting.

Explore similar work

ProBel: Propaganda Detection with Techniques, Spans, and Explanations
Aug 23, 2026Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Elisa Sartori +2Text Classification Cross-Lingual Transfer
Qatar Computing Research Institute, Qatar · University of Padova, Italy
Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas
May 13, 2026Lukas Stähelin, Veronika Solopova, Max Upravitelev +6LLM Prompting LLM Fine-Tuning
Technische Universität Berlin, QU Lab, XplaiNLP Group, Berlin, Germany · German Research Center for Artificial Intelligence (DFKI), Berlin, Germany · Centre for European Research in Trusted AI (CERTAIN)
Reasoning for Propaganda Detection via Label-Guided CoT Optimisation
Oct 12, 2026Zeba Afroz, Harsh Vardhan, Md. Shad AkhtarCoT Reasoning Text Classification
Indraprastha Institute of Information Technology Delhi

Aug 23, 2026J/K move · Enter open · S save

ProBel: Propaganda Detection with Techniques, Spans, and Explanations

Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Elisa Sartori, Giovanni Da San Martino+1

Qatar Computing Research Institute, Qatar · University of Padova, Italy

Text Classification Cross-Lingual Transfer Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Propaganda detection includes several related prediction levels, ranging from sentence-level decisions to technique classification and span identification. However, it remains unclear how supervision at these levels interacts when learned jointly across Arabic and English. We present ProBel, an Arabic and English resource that aligns binary labels, multi-label annotations over 23 propaganda techniques grouped into six coarse categories, technique-labeled spans, and reference explanations for the same news sentences. It includes a substantially larger English collection and supports matched binary, coarse-grained, multi-label, and span-level tasks in both languages. We evaluate zero-shot prompting, task-specific fine-tuning, and joint training under a shared setup. A single bilingual multi-task model achieves the best overall performance and remains competitive across tasks and languages. Cross-task analysis shows that transfer depends on the supervision level. Joint classification training preserves binary performance, whereas span-only training can weaken sentence-level prediction. Joint bilingual training yields the most stable results, while monolingual fine-tuning can reduce transfer to the other language. We will release the data, code, and evaluation scripts.

Abstract

Explore similar work

ProBel: Propaganda Detection with Techniques, Spans, and Explanations

Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas

Reasoning for Propaganda Detection via Label-Guided CoT Optimisation