cs.CLMar 10, 2026

Build, Borrow, or Just Fine-Tune? A Political Scientist's Guide to Choosing NLP Models

Authors: Shreyas Meher

Organizations: School of Social and Behavioural Sciences, Erasmus University Rotterdam

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Political scientists increasingly face a consequential choice when adopting natural language processing tools: build a domain-specific model from scratch, borrow and adapt an existing one, or simply fine-tune a general-purpose model on task data? Each approach occupies a different point on the spectrum of performance, cost, and required expertise, yet the discipline has offered little empirical guidance on how to navigate this trade-off. This paper provides such guidance. Using conflict event classification as a test case, I fine-tune ModernBERT on the Global Terrorism Database (GTD) to create Confli-mBERT and systematically compare it against ConfliBERT, a domain-specific pretrained model that represents the current gold standard. Confli-mBERT achieves 75.46% accuracy compared to ConfliBERT's 79.34%. Critically, the four-percentage-point gap is not uniform: on high-frequency attack types such as Bombing/Explosion (F1 = 0.95 vs. 0.96) and Kidnapping (F1 = 0.92 vs. 0.91), the models are nearly indistinguishable. Performance differences concentrate in rare event categories comprising fewer than 2% of all incidents. I use these findings to develop a practical decision framework for political scientists considering any NLP-assisted research task: when does the research question demand a specialized model, and when does an accessible fine-tuned alternative suffice? The answer, I argue, depends not on which model is "better" in the abstract, but on the specific intersection of class prevalence, error tolerance, and available resources. The model, training code, and data are publicly available on Hugging Face.

Figures & tables

Attack TypeCountPercentage
Bombing/Explosion14,57835.9%
Armed Assault10,36525.5%
Unknown4,32810.6%
Facility/Infrastructure Attack4,17610.3%
Hostage Taking (Kidnapping)3,5238.7%
Assassination2,9907.4%
Table 1: Attack Type Distribution in the GTD Test Set (2017+)
Attack TypeTraining Count ( njn_{j} )Weight ( wjw_{j} )
Bombing/Explosion83,7102.04
Armed Assault43,4253.93
Assassination18,5759.19
Facility/Infrastructure Attack11,11115.35
Hostage Taking (Kidnapping)10,78915.81
Unknown6,48526.31
Table 2: Inverse-Frequency Class Weights Used in Training
ParameterValue
Base modelanswerdotai/ModernBERT-base (149M params.)
Number of output labels9
Maximum sequence length512 tokens
Training epochs5
Per-device batch size16
Learning rate3×10−53\times 10^{-5}
Table 3: Training Hyperparameters for Confli-mBERT
MetricConfliBERTConfli-mBERTConflLlama
Overall Accuracy79.34%75.46%72.41%
Table 4: Overall Model Performance on the GTD Test Set
Attack TypeConfliBERTConfli-mBERTConflLlama
Bombing/Explosion0.960.950.91
Armed Assault0.740.720.69
Unknown0.610.590.47
Kidnapping0.910.920.84
Assassination0.790.660.66
Facility/Infrastructure0.730.730.67
Table 5: Per-Class F1 Scores by Attack Type
Attack TypeConfliBERT AUCConfli-mBERT AUCDifference
Bombing/Explosion0.99050.9688+0.0217
Hostage Taking (Kidnapping)0.98600.9585+0.0275
Assassination0.96130.7814+0.1799
Armed Assault0.92440.8468+0.0776
Unknown0.92250.7640+0.1585
Facility/Infrastructure Attack0.91930.8211+0.0982
Table 6: Per-Class AUC-ROC Scores
Table 7: Categorized Performance Analysis: AUC Gap by Class Prevalence
Figure 1: AUC performance of ConfliBERT and Confli-mBERT across attack types, plotted against class prevalence on a logarithmic scale. The two models converge in performance as class size increases, but ConfliBERT mantains a strict advantage.
Figure 2: The AUC difference between ConfliBERT and Confli-mBERT, plotted against class prevalence (log scale). The dashed red line shows the logarithmic trend. The performance gap narrows toward zero for common attack types.
Attack TypeConfliBERTConfli-mBERTDifferenceDiff %
Bombing/Explosion14,08613,780+306+2.2%
Armed Assault7,6628,056−- 394−- 5.1%
Hostage Taking (Kidnapping)3,0963,243−- 147−- 4.7%
Facility/Infrastructure Attack2,6392,731−- 92−- 3.5%
Unknown2,3812,463−- 82−- 3.4%
Assassination2,2031,720+483+21.9%
Table 8: True Positive Counts from Confusion Matrices
Error TypeConfliBERTConfli-mBERTDifferenceBetter Model
Armed misclassified as Assassination232294+62ConfliBERT
Armed misclassified as Unknown8201,050+230ConfliBERT
Assassination misclassified as Armed350840+490ConfliBERT
Bombing misclassified as Armed88405+317ConfliBERT
Bombing misclassified as Unknown60234+174ConfliBERT
Kidnapping misclassified as Armed107641+534ConfliBERT
Table 9: Major Misclassification Patterns from Confusion Matrices
Model & Prevalence TierTrue Pos. RateFalse Pos. RatePrecisionF1 Score
ConfliBERT – High Prevalence83.0%3.7%86.2%0.8458
Confli-mBERT – High Prevalence84.0%5.6%80.8%0.8235
ConfliBERT – Medium Prevalence67.6%1.1%86.1%0.7572
Confli-mBERT – Medium Prevalence62.1%1.6%79.9%0.6899
ConfliBERT – Low Prevalence53.3%0.1%73.8%0.6188
Confli-mBERT – Low Prevalence25.9%0.3%36.6%0.3030
Table 10: Model Calibration by Class Prevalence Tier
TypeModelAccuracyMicro F1Macro F1
Fine-tunedConfliBERT (110M)79.70%0.8310.708
Confli-mBERT (177M)75.80%0.7970.580
ConflLlama-Q8 (8B)72.85%0.7590.610
Commercial APIGemini 3 Flash †65.85%0.7300.602
Claude Haiku 4.5 †60.95%0.6640.526
DeepSeek V3.256.45%0.6650.492
Table 11: Classification Performance: Fine-Tuned Models vs. Zero-Shot LLMs (2,000-row stratified sample)
Figure 3: Overall classification accuracy across nine models evaluated on a stratified 2,000-event sample from the GTD test set. The three fine-tuned models (dark bars) form a distinct performance tier above the commercial APIs (medium bars), which in turn outperform the locally deployed open-source models used without any fine-tuning (light bars). The gap between the weakest fine-tuned model (ConflLlama, 72.9%) and the strongest commercial API (Gemini Flash, 65.9%) is nearly seven percentage points.
Figure 4: Model size (total parameters, log scale) plotted against Micro F1 classification performance. Fine-tuned models ( ★\bigstar ) cluster in the upper left: small models with high performance. Commercial APIs ( ⧫\blacklozenge ) and open-source models ( ∙\bullet ) occupy the lower right: large models with mediocre to poor performance. ConfliBERT, at 110 million parameters, outperforms DeepSeek V3.2, a model roughly 6,200 times its size. The relationship between size and performance is negative across model families, demonstrating that task-specific fine-tuning overwhelmingly dominates model scale for classification tasks.
ModelInputOutput2,00037,709170,623
 ($/M tok.)($/M tok.)rowsrows arows b
Claude Haiku 4.5$1.00$5.00$1.00$18.85$85.31
Gemini 3 Flash$0.50$3.00$0.53$9.99$45.22
DeepSeek V3.2$0.26$0.38$0.20$3.77$17.06
3-model total——$1.73$32.61$147.59
Table 12: Commercial API Pricing and Projected Classification Costs (February 2026)
More in this topic

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 753 papers

Explore similar work

CardsList
Aug 5, 2025

RooseBERT: A New Deal For Political Language Modelling

The increasing amount of political debates and politics-related discussions calls for the definition of novel computational methods to automatically analyse such content with the final goal of lightening up political deliberation to citizens. However, the specificity of the political language and the argumentative form of these debates (employing hidden communication strategies and leveraging implicit arguments) make this task very challenging, even for current general-purpose pre-trained Language Models (PLMs). To address this, we introduce a novel PLM for political discourse language called RooseBERT. Pre-training a language model on a specialised domain presents different technical and linguistic challenges, requiring extensive computational resources and large-scale data. RooseBERT has been trained on large political debate and speech corpora (11GB) in English. To evaluate its performances, we fine-tuned it on multiple downstream tasks related to political debate analysis, i.e., stance detection, sentiment analysis, argument component detection and classification, argument relation prediction and classification, policy classification, named entity recognition (NER). Our results show improvements over general-purpose PLMs on the majority of these tasks, highlighting how domain-specific pre-training enhances performance in political debate analysis. We release RooseBERT for the research community: https://huggingface.co/collections/MARIANNE-INRIA/roosebert.

Deborah Dore, Elena Cabrio, Serena Villata

Universit´e Cˆote d’Azur CNRS, INRIA, I3S Sophia Antipolis, France

Language Model PretrainingArgument MiningStance DetectionPolitical Discourse AnalysisComputational Social ScienceLanguage Modeling
AskPDF
Mar 27, 2026

Clash of the models: Comparing performance of BERT-based variants for generic news frame detection

Framing continues to remain one of the most extensively applied theories in political communication. Developments in computation, particularly with the introduction of transformer architecture and more so with large language models (LLMs), have naturally prompted scholars to explore various novel computational approaches, especially for deductive frame detection, in recent years. While many studies have shown that different transformer models outperform their preceding models that use bag-of-words features, the debate continues to evolve regarding how these models compare with each other on classification tasks. By placing itself at this juncture, this study makes three key contributions: First, it comparatively performs generic news frame detection and compares the performance of five BERT-based variants (BERT, RoBERTa, DeBERTa, DistilBERT and ALBERT) to add to the debate on best practices around employing computational text analysis for political communication studies. Second, it introduces various fine-tuned models capable of robustly performing generic news frame detection. Third, building upon numerous previous studies that work with US-centric data, this study provides the scholarly community with a labelled generic news frames dataset based on the Swiss electoral context that aids in testing the contextual robustness of these computational approaches to framing analysis.

Vihang Jumle

Political CommunicationTransformer
AskPDF
Jun 28, 2026

Do We Still Need Fine Tuning? Turkish Sentiment Analysis in the Era of Large Language Model

This study examines whether supervised fine-tuning remains necessary for Turkish sentiment analysis in the era of large language models. We compare classical machine learning methods, fine-tuned pretrained language models, and prompted large language models on a Turkish e-commerce review dataset with negative, neutral, and positive labels. Fine-tuned BERTurk models perform best overall and outperform all prompted large language models in the full three-class task. The neutral class emerges as the main difficulty: while several large language models are much more competitive in binary positive--negative classification, they degrade substantially in the three-class setting by collapsing neutral reviews into polarized categories. The findings suggest that, in realistic Turkish sentiment classification, prompted large language models do not yet match supervised fine-tuning in the zero-shot setting, and that including the neutral class is crucial for robust evaluation.

Sercan Karakaş, Yusuf Şimşek

University of Chicago · Fırat University

Supervised Fine-TuningLLM EvaluationZero-Shot Text ClassificationSentiment AnalysisLarge Language Model-Based Text Classification
AskPDF