Not All Pretraining are Created Equal: Threshold Tuning and Class Weighting for Imbalanced Polarization Tasks in Low-Resource Settings

Abass Oguntade

cs.CLMar 8, 2026

Not All Pretraining are Created Equal: Threshold Tuning and Class Weighting for Imbalanced Polarization Tasks in Low-Resource Settings

Authors: Abass Oguntade

Organizations: African Institute of Mathematical Sciences, South Africa

Abstract

This paper describes my submission to the Polarization Shared Task at SemEval-2025, which addresses polarization detection and classification in social media text. I develop Transformer-based systems for English and Swahili across three subtasks: binary polarization detection, multi-label target type classification, and multi-label manifestation identification. The approach leverages multilingual and African language-specialized models (mDeBERTa-v3-base, SwahBERT, AfriBERTa-large), class-weighted loss functions, iterative stratified data splitting, and per-label threshold tuning to handle severe class imbalance. The best configuration, mDeBERTa-v3-base, achieves 0.8032 macro-F1 on validation for binary detection, with competitive performance on multi-label tasks (up to 0.556 macro-F1). Error analysis reveals persistent challenges with implicit polarization, code-switching, and distinguishing heated political discourse from genuine polarization.

Figures & tables

Figure 1: English dataset polarization class distribution for Subtask 1. The dataset shows moderate imbalance with 64% non-polarized instances.

Figure 2: Swahili dataset polarization class distribution for Subtask 1. Nearly balanced with 50% representation in each class.

Figure 3: Word cloud for English dataset showing dominance of political terms (trump, ukraine, gaza) and geopolitical discourse markers.

Figure 4: Word cloud for Swahili dataset revealing frequent vulgar and offensive terms indicating different polarization expression patterns.

Figure 5: Subtask 2 label distribution across five target types showing severe imbalance. Political dominates while gender/sexual remains extremely rare.

Figure 6: Class distribution per label for Subtask 2 showing positive vs negative instances. All labels exhibit severe imbalance favoring negative class.

Figure 7: Correlation heatmap for Subtask 2 labels. Weak correlations indicate mostly independent label relationships, except for racial/ethnic and religious (r=0.34).

Figure 8: Subtask 3 manifestation label distribution.

Figure 9: Distribution of number of labels per instance in Subtask 3. Most instances (63.5%) have exactly 0 number of labels, indicating sparse multi-label patterns.

Model	English	Swahili
TwHIN-BERT	0.7279	0.7048
DistilBERT-multilingual (Baseline)	0.7279 *	0.8022 *
mDeBERTa-v3-base	0.8032	0.7850
SwahBERT	0.6375	0.8050
AfriBERTa-large	0.7333	0.8075
AfroXLMR-large	0.5317	0.5788

Table 1: Subtask 1 validation macro-F1 scores. * Early stopping at epoch 8. Best scores in bold.

Figure 10: Training loss curves for six models on English dataset (Subtask 1). All models show stable convergence.

Figure 11: Validation F1-macro curves for six models on English dataset (Subtask 1). mDeBERTa-v3-base achieves highest scores with clear separation from alternatives.

Figure 12: Training loss curves for six models on Swahili dataset (Subtask 1). Steeper descent reflects larger corpus providing richer gradient signals.

Figure 13: Validation F1-macro curves for six models on Swahili dataset (Subtask 1). Smoother trajectories and reduced oscillation compared to English.

Figure 14: Training loss curves for combined English+Swahili dataset (Subtask 1). Comparable convergence to Swahili-only despite increased linguistic diversity.

Figure 15: Validation F1-macro curves for combined dataset (Subtask 1). Performance degradation indicates negative transfer from naive multilingual training.

English
Model	Before	After
DistilBERT-multilingual	0.245	0.464
AfriBERTa-large	0.330	0.437
Swahili
AfriBERTa-large	0.132	0.556
SwahBERT	0.210	0.325

Table 2: Subtask 2 validation macro-F1 before/after threshold tuning.

Figure 16: Training loss for Subtask 2 configurations. Swahili models achieve lower final losses consistent with larger training sets.

Figure 17: Validation F1-macro for Subtask 2. Dramatic 20–25 percentage point gap between Swahili and English models reflects data volume effects.

Figure 18: Training loss for Subtask 3 configurations. Lower absolute losses (0.2–0.6) than Subtask 2 despite apparent increased task difficulty.

Figure 19: Validation F1-macro for Subtask 3. Compressed range (0.45–0.60) represents 15–25 percentage point decline from Subtask 2, indicating fundamental task difficulty.

Multi-Label Classification

Multi-label classification assigns several potentially related tags to each example, from medical findings and topics to objects and user interests.

See all 137 papers

Multilingual Polarization Detection Using Transformer-Based Models with Class Weighting and Threshold Tuning
Jun 29, 2026Aaron Bundi AnampiuMulti-Label Classification
African Institute for Mathematical Sciences, South Africa
YEZE at SemEval-2026 Task 9: Detecting Multilingual, Multicultural and Multievent Online Polarization via Heterogeneous Ensembling
May 7, 2026Fengze Guo, Yue ChangSocial Media Analysis Multilingual Text Classification
University of Tübingen
Polarization Detection: A Hybrid Approach with AfroXLMR-Social and DeBERTa for Low- and High-Resource Settings
Jul 11, 2026Muhammad Abdullahi Said
African Institute for Mathematical Sciences (AIMS)

Jun 29, 2026J/K move · Enter open · S save

Multilingual Polarization Detection Using Transformer-Based Models with Class Weighting and Threshold Tuning

Aaron Bundi Anampiu

African Institute for Mathematical Sciences, South Africa

Multi-Label Classification

Open paper AskPDF arXiv

This paper describes our submission to SemEval-2026 Task 9 on detecting multilingual, multicultural, and multievent online polarization. We address all three subtasks: binary polarization detection, polarization type classification, and manifestation identification for English and Swahili. Our approach leverages transformer-based models (RoBERTa-base for English, AfroXLMR-base for Swahili) with class-weighted loss functions to address severe label imbalance and per-label threshold tuning to optimize multi-label classification. On the test set, we achieve F1 macro scores of 0.7901 (English) and 0.7910 (Swahili) for Subtask 1, 0.4615 (English) and 0.4808 (Swahili) for Subtask 2 and 0.4791 (English) and 0.5830 (Swahili) for Subtask 3, which give competitive performance on the leaderboard, demonstrating the effectiveness of our methods for handling imbalanced multi-label polarization detection. Our error analysis reveals that models struggle with dehumanization detection and lack of empathy.

Abstract

Multi-Label Classification

Explore similar work

Multilingual Polarization Detection Using Transformer-Based Models with Class Weighting and Threshold Tuning

YEZE at SemEval-2026 Task 9: Detecting Multilingual, Multicultural and Multievent Online Polarization via Heterogeneous Ensembling

Polarization Detection: A Hybrid Approach with AfroXLMR-Social and DeBERTa for Low- and High-Resource Settings