Build, Borrow, or Just Fine-Tune? A Political Scientist's Guide to Choosing NLP Models

Shreyas Meher

cs.CLMar 10, 2026

Build, Borrow, or Just Fine-Tune? A Political Scientist's Guide to Choosing NLP Models

Authors: Shreyas Meher

Organizations: School of Social and Behavioural Sciences, Erasmus University Rotterdam

Abstract

Political scientists increasingly face a consequential choice when adopting natural language processing tools: build a domain-specific model from scratch, borrow and adapt an existing one, or simply fine-tune a general-purpose model on task data? Each approach occupies a different point on the spectrum of performance, cost, and required expertise, yet the discipline has offered little empirical guidance on how to navigate this trade-off. This paper provides such guidance. Using conflict event classification as a test case, I fine-tune ModernBERT on the Global Terrorism Database (GTD) to create Confli-mBERT and systematically compare it against ConfliBERT, a domain-specific pretrained model that represents the current gold standard. Confli-mBERT achieves 75.46% accuracy compared to ConfliBERT's 79.34%. Critically, the four-percentage-point gap is not uniform: on high-frequency attack types such as Bombing/Explosion (F1 = 0.95 vs. 0.96) and Kidnapping (F1 = 0.92 vs. 0.91), the models are nearly indistinguishable. Performance differences concentrate in rare event categories comprising fewer than 2% of all incidents. I use these findings to develop a practical decision framework for political scientists considering any NLP-assisted research task: when does the research question demand a specialized model, and when does an accessible fine-tuned alternative suffice? The answer, I argue, depends not on which model is "better" in the abstract, but on the specific intersection of class prevalence, error tolerance, and available resources. The model, training code, and data are publicly available on Hugging Face.

Figures & tables

Attack Type	Count	Percentage
Bombing/Explosion	14,578	35.9%
Armed Assault	10,365	25.5%
Unknown	4,328	10.6%
Facility/Infrastructure Attack	4,176	10.3%
Hostage Taking (Kidnapping)	3,523	8.7%
Assassination	2,990	7.4%

Table 1: Attack Type Distribution in the GTD Test Set (2017+)

Attack Type	Training Count ( $n_{j}$ )	Weight ( $w_{j}$ )
Bombing/Explosion	83,710	2.04
Armed Assault	43,425	3.93
Assassination	18,575	9.19
Facility/Infrastructure Attack	11,111	15.35
Hostage Taking (Kidnapping)	10,789	15.81
Unknown	6,485	26.31

Table 2: Inverse-Frequency Class Weights Used in Training

Parameter	Value
Base model	answerdotai/ModernBERT-base (149M params.)
Number of output labels	9
Maximum sequence length	512 tokens
Training epochs	5
Per-device batch size	16
Learning rate	$3\times 10^{-5}$

Table 3: Training Hyperparameters for Confli-mBERT

Metric	ConfliBERT	Confli-mBERT	ConflLlama
Overall Accuracy	79.34%	75.46%	72.41%

Table 4: Overall Model Performance on the GTD Test Set

Attack Type	ConfliBERT	Confli-mBERT	ConflLlama
Bombing/Explosion	0.96	0.95	0.91
Armed Assault	0.74	0.72	0.69
Unknown	0.61	0.59	0.47
Kidnapping	0.91	0.92	0.84
Assassination	0.79	0.66	0.66
Facility/Infrastructure	0.73	0.73	0.67

Table 5: Per-Class F1 Scores by Attack Type

Attack Type	ConfliBERT AUC	Confli-mBERT AUC	Difference
Bombing/Explosion	0.9905	0.9688	+0.0217
Hostage Taking (Kidnapping)	0.9860	0.9585	+0.0275
Assassination	0.9613	0.7814	+0.1799
Armed Assault	0.9244	0.8468	+0.0776
Unknown	0.9225	0.7640	+0.1585
Facility/Infrastructure Attack	0.9193	0.8211	+0.0982

Table 6: Per-Class AUC-ROC Scores

Panel of Table 7 — Table 7: Categorized Performance Analysis: AUC Gap by Class Prevalence

Panel of Figure 1 — Figure 1: AUC performance of ConfliBERT and Confli-mBERT across attack types, plotted against class prevalence on a logarithmic scale. The two models converge in performance as class size increases, but ConfliBERT mantains a strict advantage.

Panel of Figure 2 — Figure 2: The AUC difference between ConfliBERT and Confli-mBERT, plotted against class prevalence (log scale). The dashed red line shows the logarithmic trend. The performance gap narrows toward zero for common attack types.

Attack Type	ConfliBERT	Confli-mBERT	Difference	Diff %
Bombing/Explosion	14,086	13,780	+306	+2.2%
Armed Assault	7,662	8,056	$-$ 394	$-$ 5.1%
Hostage Taking (Kidnapping)	3,096	3,243	$-$ 147	$-$ 4.7%
Facility/Infrastructure Attack	2,639	2,731	$-$ 92	$-$ 3.5%
Unknown	2,381	2,463	$-$ 82	$-$ 3.4%
Assassination	2,203	1,720	+483	+21.9%

Table 8: True Positive Counts from Confusion Matrices

Error Type	ConfliBERT	Confli-mBERT	Difference	Better Model
Armed misclassified as Assassination	232	294	+62	ConfliBERT
Armed misclassified as Unknown	820	1,050	+230	ConfliBERT
Assassination misclassified as Armed	350	840	+490	ConfliBERT
Bombing misclassified as Armed	88	405	+317	ConfliBERT
Bombing misclassified as Unknown	60	234	+174	ConfliBERT
Kidnapping misclassified as Armed	107	641	+534	ConfliBERT

Table 9: Major Misclassification Patterns from Confusion Matrices

Model & Prevalence Tier	True Pos. Rate	False Pos. Rate	Precision	F1 Score
ConfliBERT – High Prevalence	83.0%	3.7%	86.2%	0.8458
Confli-mBERT – High Prevalence	84.0%	5.6%	80.8%	0.8235
ConfliBERT – Medium Prevalence	67.6%	1.1%	86.1%	0.7572
Confli-mBERT – Medium Prevalence	62.1%	1.6%	79.9%	0.6899
ConfliBERT – Low Prevalence	53.3%	0.1%	73.8%	0.6188
Confli-mBERT – Low Prevalence	25.9%	0.3%	36.6%	0.3030

Table 10: Model Calibration by Class Prevalence Tier

Type	Model	Accuracy	Micro F1	Macro F1
Fine-tuned	ConfliBERT (110M)	79.70%	0.831	0.708
	Confli-mBERT (177M)	75.80%	0.797	0.580
	ConflLlama-Q8 (8B)	72.85%	0.759	0.610
Commercial API	Gemini 3 Flash †	65.85%	0.730	0.602
	Claude Haiku 4.5 †	60.95%	0.664	0.526
	DeepSeek V3.2	56.45%	0.665	0.492

Table 11: Classification Performance: Fine-Tuned Models vs. Zero-Shot LLMs (2,000-row stratified sample)

Panel of Figure 3 — Figure 3: Overall classification accuracy across nine models evaluated on a stratified 2,000-event sample from the GTD test set. The three fine-tuned models (dark bars) form a distinct performance tier above the commercial APIs (medium bars), which in turn outperform the locally deployed open-source models used without any fine-tuning (light bars). The gap between the weakest fine-tuned model (ConflLlama, 72.9%) and the strongest commercial API (Gemini Flash, 65.9%) is nearly seven percentage points.

Panel of Figure 4 — Figure 4: Model size (total parameters, log scale) plotted against Micro F1 classification performance. Fine-tuned models ( $\bigstar$ ) cluster in the upper left: small models with high performance. Commercial APIs ( $\blacklozenge$ ) and open-source models ( $\bullet$ ) occupy the lower right: large models with mediocre to poor performance. ConfliBERT, at 110 million parameters, outperforms DeepSeek V3.2, a model roughly 6,200 times its size. The relationship between size and performance is negative across model families, demonstrating that task-specific fine-tuning overwhelmingly dominates model scale for classification tasks.

Model	Input	Output	2,000	37,709	170,623
	($/M tok.)	($/M tok.)	rows	rows a	rows b
Claude Haiku 4.5	$1.00	$5.00	$1.00	$18.85	$85.31
Gemini 3 Flash	$0.50	$3.00	$0.53	$9.99	$45.22
DeepSeek V3.2	$0.26	$0.38	$0.20	$3.77	$17.06
3-model total	—	—	$1.73	$32.61	$147.59

Table 12: Commercial API Pricing and Projected Classification Costs (February 2026)

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 753 papers

RooseBERT: A New Deal For Political Language Modelling
Aug 5, 2025Deborah Dore, Elena Cabrio, Serena VillataLanguage Model Pretraining Argument Mining
Universit´e Cˆote d’Azur CNRS, INRIA, I3S Sophia Antipolis, France
Clash of the models: Comparing performance of BERT-based variants for generic news frame detection
Mar 27, 2026Vihang JumlePolitical Communication Transformer
Do We Still Need Fine Tuning? Turkish Sentiment Analysis in the Era of Large Language Model
Jun 28, 2026Sercan Karakaş, Yusuf ŞimşekSupervised Fine-Tuning LLM Evaluation
University of Chicago · Fırat University

Aug 5, 2025J/K move · Enter open · S save

RooseBERT: A New Deal For Political Language Modelling

Deborah Dore, Elena Cabrio, Serena Villata

Universit´e Cˆote d’Azur CNRS, INRIA, I3S Sophia Antipolis, France

Language Model Pretraining Argument Mining Stance Detection Political Discourse Analysis Computational Social Science Language Modeling

Open paper AskPDF arXiv

The increasing amount of political debates and politics-related discussions calls for the definition of novel computational methods to automatically analyse such content with the final goal of lightening up political deliberation to citizens. However, the specificity of the political language and the argumentative form of these debates (employing hidden communication strategies and leveraging implicit arguments) make this task very challenging, even for current general-purpose pre-trained Language Models (PLMs). To address this, we introduce a novel PLM for political discourse language called RooseBERT. Pre-training a language model on a specialised domain presents different technical and linguistic challenges, requiring extensive computational resources and large-scale data. RooseBERT has been trained on large political debate and speech corpora (11GB) in English. To evaluate its performances, we fine-tuned it on multiple downstream tasks related to political debate analysis, i.e., stance detection, sentiment analysis, argument component detection and classification, argument relation prediction and classification, policy classification, named entity recognition (NER). Our results show improvements over general-purpose PLMs on the majority of these tasks, highlighting how domain-specific pre-training enhances performance in political debate analysis. We release RooseBERT for the research community: https://huggingface.co/collections/MARIANNE-INRIA/roosebert.

Abstract

LLM Fine-Tuning

Explore similar work

RooseBERT: A New Deal For Political Language Modelling

Clash of the models: Comparing performance of BERT-based variants for generic news frame detection

Do We Still Need Fine Tuning? Turkish Sentiment Analysis in the Era of Large Language Model