RooseBERT: A New Deal For Political Language Modelling

Deborah Dore; Elena Cabrio; Serena Villata

cs.CLAug 5, 2025

RooseBERT: A New Deal For Political Language Modelling

Authors: Deborah Dore, Elena Cabrio, Serena Villata

Organizations: Universit´e Cˆote d’Azur CNRS, INRIA, I3S Sophia Antipolis, France

Abstract

The increasing amount of political debates and politics-related discussions calls for the definition of novel computational methods to automatically analyse such content with the final goal of lightening up political deliberation to citizens. However, the specificity of the political language and the argumentative form of these debates (employing hidden communication strategies and leveraging implicit arguments) make this task very challenging, even for current general-purpose pre-trained Language Models (PLMs). To address this, we introduce a novel PLM for political discourse language called RooseBERT. Pre-training a language model on a specialised domain presents different technical and linguistic challenges, requiring extensive computational resources and large-scale data. RooseBERT has been trained on large political debate and speech corpora (11GB) in English. To evaluate its performances, we fine-tuned it on multiple downstream tasks related to political debate analysis, i.e., stance detection, sentiment analysis, argument component detection and classification, argument relation prediction and classification, policy classification, named entity recognition (NER). Our results show improvements over general-purpose PLMs on the majority of these tasks, highlighting how domain-specific pre-training enhances performance in political debate analysis. We release RooseBERT for the research community: https://huggingface.co/collections/MARIANNE-INRIA/roosebert.

Figures & tables

Word	BERT vocabulary	RooseBERT vocabulary
annuity	[’an’, ’##nu’, ’##ity’]	annuity
bureaucrat	[’bureau’, ’##crat’]	bureaucrat
deterrent	[’de’, ’##ter’, ’##rent’]	deterrent
undermining	[’under’, ’##mini’, ’##ng’]	undermining
endorse	[’end’, ’##ors’, ’##e’]	endorse
statutorily	[’s’, ’##tat’, ’##uto’, ’##rily’]	statutorily

TABLE I: Example of key political terms and their encoding in BERT and RooseBERT vocabularies.

BERT’s tokenizer	BERT	ModernBERT	ConfliBERT-cont	ConfliBERT-scr	PoliBERTweet	RooseBERT-cont	RooseBERT-scr
cased	22.11	811048014.47	4.37	790959.65	255326.23	2.61	26320.05
uncased	9.60	N/A	5.00	3270380.04	N/A	2.71	446395381.08

TABLE II: Perplexity evaluated on a held-out corpus representing the overall distribution of political debates.

Models	Sentiment Analysis		Stance Detection		Arg. Component Det. & Class.		Arg. Relation Pred. & Class.		Mot. Policy Class.	NER
Models	ParlVote	HanDeSeT	ConVote	AusHansard	ElecDeb	ArgUNSC	ElecDeb	ArgUNSC	ParlVote+	NEREx
BERT-cased	0.69 $\pm$ 0.02**	0.67 $\pm$ 0.04**	0.72 $\pm$ 0.01**	0.54 $\pm$ 0.01**	0.61 $\pm$ 0.00**	0.61 $\pm$ 0.01	0.58 $\pm$ 0.01**	0.57 $\pm$ 0.04**	0.54 $\pm$ 0.01**	0.92 $\pm$ 0.01
BERT-uncased	0.67 $\pm$ 0.08**	0.66 $\pm$ 0.04**	0.73 $\pm$ 0.01**	0.55 $\pm$ 0.02**	0.61 $\pm$ 0.00**	0.60 $\pm$ 0.01**	0.58 $\pm$ 0.01**	0.64 $\pm$ 0.03**	0.55 $\pm$ 0.01**	0.90 $\pm$ 0.01*
ModernBERT	0.77 $\pm$ 0.03	0.65 $\pm$ 0.02**	0.73 $\pm$ 0.04	0.58 $\pm$ 0.05	0.62 $\pm$ 0.00	0.58 $\pm$ 0.03	0.60 $\pm$ 0.01	0.60 $\pm$ 0.14	0.66 $\pm$ 0.00	0.91 $\pm$ 0.01
ConfliB CC	0.76 $\pm$ 0.01**	0.65 $\pm$ 0.01**	0.75 $\pm$ 0.01**	0.61 $\pm$ 0.01**	0.62 $\pm$ 0.00*	0.61 $\pm$ 0.00*	0.61 $\pm$ 0.01**	0.55 $\pm$ 0.18*	0.58 $\pm$ 0.01**	0.90 $\pm$ 0.02*
ConfliB CU	0.75 $\pm$ 0.01**	0.65 $\pm$ 0.01**	0.75 $\pm$ 0.01**	0.57 $\pm$ 0.01**	0.62 $\pm$ 0.00**	0.60 $\pm$ 0.01**	0.61 $\pm$ 0.00**	0.62 $\pm$ 0.01**	0.55 $\pm$ 0.01**	0.91 $\pm$ 0.00**

TABLE III: Average

\pm

standard deviation on downstream tasks. Binary tasks are evaluated with Accuracy; all other tasks use Macro F1. Statistical significance: **

p<0.05

, *

p<0.1

w.r.t. the best model (in bold ). CC = cont cased, CU = cont uncased, SC = scr cased, SU = scr uncased, RooseB = RooseBERT, ConfliB = ConfliBERT.

Clusters	Sentiment Analysis		Stance Detection		Arg. Component Det. & Class.		Arg. Relation Pred. & Class.		Mot. Policy Class.	NER
Clusters	ParlVote	HanDeSeT	ConVote	AusHansard	ElecDeb	ArgUNSC	ElecDeb	ArgUNSC	ParlVote+	NEREx
#0	0.79 $\pm$ 0.00**	0.69 $\pm$ 0.03	0.76 $\pm$ 0.01	0.62 $\pm$ 0.01**	0.63 $\pm$ 0.00	0.61 $\pm$ 0.01	0.63 $\pm$ 0.00**	0.70 $\pm$ 0.01**	0.63 $\pm$ 0.01**	0.89 $\pm$ 0.01
#1	0.79 $\pm$ 0.01**	0.73 $\pm$ 0.02*	0.78 $\pm$ 0.00**	0.62 $\pm$ 0.01**	0.62 $\pm$ 0.00	0.61 $\pm$ 0.01	0.63 $\pm$ 0.01**	0.72 $\pm$ 0.02*	0.62 $\pm$ 0.00**	0.88 $\pm$ 0.01
#2	0.78 $\pm$ 0.00	0.67 $\pm$ 0.03	0.76 $\pm$ 0.01	0.59 $\pm$ 0.01	0.62 $\pm$ 0.00*	0.60 $\pm$ 0.01*	0.62 $\pm$ 0.00**	0.71 $\pm$ 0.03*	0.60 $\pm$ 0.01**	0.90 $\pm$ 0.02
#3	0.69 $\pm$ 0.09*	0.70 $\pm$ 0.04	0.72 $\pm$ 0.02**	0.57 $\pm$ 0.02*	0.52 $\pm$ 0.01**	0.59 $\pm$ 0.01**	0.61 $\pm$ 0.00**	0.71 $\pm$ 0.03*	0.60 $\pm$ 0.01**	0.91 $\pm$ 0.02
RB SU	0.78 $\pm$ 0.00	0.69 $\pm$ 0.02	0.75 $\pm$ 0.01	0.59 $\pm$ 0.01	0.63 $\pm$ 0.01	0.61 $\pm$ 0.01	0.62 $\pm$ 0.01	0.65 $\pm$ 0.05	0.58 $\pm$ 0.01	0.88 $\pm$ 0.01

TABLE IV: Average

\pm

standard deviation on downstream tasks. Binary tasks are evaluated with Accuracy; all other tasks use Macro F1. Statistical significance: **

p<0.05

, *

p<0.1

w.r.t. to RB SU (RooseBERT-scr-uncased).

Fig. 1: Cosine similarity among pre-training datasets. Dataset abbreviations: G/SA (Ghana and South Africa Parliamentary debates), AUS (Australian Parliamentary debates), CA (Canadian Parliamentary debates), EU (European Parliamentary debates), IR (Irish Parliamentary debates), NZ (New Zealand Parliamentary debates), SC (Scottish Parliamentary debates), UNGDC (United Nations General Debate Corpus), UK (United Kingdom Parliamentary debates), US (United States Debates), and UNSC (United Nations Security Council).

Explore similar work

BanglaRhet: Benchmarking Classical and Transformer Models for Rhetorical and Persuasion Detection in Bangla Political Speech
Oct 7, 2026Rohit Kumar Sen, Anik Chowdhury
Department of Computer Science and Engineering North East University Bangladesh Sylhet, Bangladesh
Who Argues What? Joint Argument-Entity Detection and Classification in Political Debates
Sep 9, 2026Lucio La Cava, Stefano Francesco Monea, Sergio GrecoDebate Political Discourse
DIMES Dept., University of Calabria v. P. Bucci 44Z, 87036 Rende, CS, Italy
TTLab at Daleel 2026: STAR-Ar, Sequence Tagging for Argument Recognition in Arabic
Sep 30, 2026Bhuvanesh Verma, Ali Abusaleh, Alexander MehlerArabic Natural Language Processing Arabic
Text Technology Lab (TTLab), Goethe University Frankfurt

Oct 7, 2026, cs.CLJ/K move · Enter open · S save

BanglaRhet: Benchmarking Classical and Transformer Models for Rhetorical and Persuasion Detection in Bangla Political Speech

Rohit Kumar Sen, Anik Chowdhury

Department of Computer Science and Engineering North East University Bangladesh Sylhet, Bangladesh

Open paper AskPDF arXiv

Political discourse often uses rhetorical and persuasive language to frame narratives, influence public opinion, and mobilize audiences. While Bangla natural language processing has made progress in sentiment analysis and opinion mining, systematic benchmarking of transformer models for fine-grained rhetorical and persuasion technique detection in Bangla political speech remains largely underexplored. This paper presents a benchmark study of transformer-based models for detecting rhetorical form and persuasive intent in Bangla political discourse. Using BanglaRhet, a manually annotated corpus of 30,289 Bangla political speech segments collected from publicly available political news sources, we formulate two supervised single-label classification tasks: rhetorical technique detection (contrast, repetition, exaggeration, metaphor, rhetorical questions) and persuasion technique detection (blame assignment, call to action, unity call, moral, emotional, and logical appeals). We evaluate four transformer-based models, BanglaBERT, BanglaBERT-Base, SahajBERT, and XLM-RoBERTa-Base, against classical TF-IDF baselines. BanglaBERT achieves the highest performance, with 65.40% macro-F1 for rhetorical technique detection and 66.46% for persuasion technique detection, outperforming the best tuned classical baseline by 19.2 and 13.8 macro-F1 points, respectively. Class-level analysis indicates that errors are mainly associated with semantic overlap among labels, figurative language, and class imbalance. The results provide initial benchmark baselines for Bangla rhetorical and persuasion-aware political discourse analysis and highlight the need for context-aware and multi-label modeling.

Abstract

Explore similar work

BanglaRhet: Benchmarking Classical and Transformer Models for Rhetorical and Persuasion Detection in Bangla Political Speech

Who Argues What? Joint Argument-Entity Detection and Classification in Political Debates

TTLab at Daleel 2026: STAR-Ar, Sequence Tagging for Argument Recognition in Arabic