The POLAR SemEval-2026 Shared Task aims to detect online polarization and focuses on the classification and identification of multilingual, multicultural, and multi-event polarization. Accurate computational detection of online polarization is challenging due to nuanced rhetoric, implicit framing, and the high cost of human-in-the-loop annotation. Building on recent findings that contextual prompting enables large language models to function as strong polarization detectors, we present a two-stage approach for detecting polarization in social media text that combines structured supervised fine tuning with Direct Preference Optimization (DPO) refinement. We fine tune Qwen 2.5-7B-Instruct with LoRA using an interpretable slot-filling template (target, claim type, manifestation checklist, and justification). We then apply DPO with automatically generated preference pairs to reduce costly false negatives. Our submitted system achieves 0.7664 Macro-F1 on the English test set. Post-submission experiments with Mistral-Nemo-Instruct-2407 and LLM-judge-filtered preference pairs further improve to 0.8162 Macro-F1 (not submitted to CodaBench), surpassing the organiser baseline of 0.7802.
Figures & tables
Figure 1: Overview of our two-stage pipeline for Subtask 1 polarization detection: structured supervised fine-tuning (SFT) first generates schema-consistent outputs, and DPO refinement then learns from preference pairs to produce the final prediction.
Split
Total
Polarized
Non-polarized
Train
3,222
1,175 (36.5%)
2,047 (63.5%)
Dev
160
59 (36.9%)
101 (63.1%)
Test
1,452
533 (36.7%)
919 (63.3%)
Total
4,834
1,767 (36.6%)
3,067 (63.4%)
Table 1: English dataset statistics and label distribution across official splits.
Method
English Dev F1
Zero-shot baseline
0.7105
SLMs
0.7149
SFT
0.738
SFT + DPO
0.7893
Table 2: F1 scores on the English development set. SLMs = Small Language Models baseline (DistilBERT fine-tuned for sequence classification). Weighted loss during SFT did not improve performance.
System
Rank (out of 60)
F1
Highest-ranked system
1
0.8252
POLAR baseline
47
0.7802
Our system
52
0.7664
Table 3: Unofficial English Subtask 1 leaderboard comparison (60 teams).
Metric
SFT
SFT + DPO
Accuracy
0.7812
0.8000
Precision
0.8333
0.7077
Recall
0.5085
0.7797
F1 (Binary)
0.6316
0.7419
F1 (Macro)
0.7380
0.7893
F1 (Micro)
0.7812
0.8000
Table 4: English development set metrics comparing SFT vs. SFT + DPO.
Model
mF1
Acc.
P(1)
R(1)
Mistral-Nemo
0.7963
0.8030
0.7928
0.8121
Qwen2.5-7B
0.7835
0.7899
0.7811
0.8006
Table 5: Label-only SFT performance on the English test set ( n=1,452 ) trained on the original unmodified labels without the Rejudged Sonnet dataset.
Mode
F1
FNs
FPs
SFT
0.7795
158
137
SFT with DPO (330 pairs)
0.7889
132
155
SFT with DPO (721 pairs)
0.7637
64
274
Table 6: Effect of DPO preference pair count on F1, false negatives (FNs), and false positives (FPs). SFT baseline is the 10-epoch full-training-set model; development-set results appear in Tables 2 and 4 .
Condition
Acc.
P(1)
R(1)
mF1
Label-only SFT
0.792
0.777
0.7884
0.781
Label-only + DPO
0.720
0.618
0.625
0.699
Reasoning SFT
0.793
0.745
0.662
0.771
Reasoning + DPO
0.802
0.732
0.704
0.789
Table 7: Structured rationale ablation on the English test set. P(1) and R(1) = precision and recall for the polarized class; mF1 = Macro-F1. Best in bold.
System
mF1
Qwen2.5-7B Label-Only SFT (Rejudged)
0.7811
Mistral-Nemo Label-Only SFT (Rejudged)
0.8019
Mistral-Nemo SFT (Rejudged)
0.8097
Mistral-Nemo DPO (FP:FN 62:38, β =0.3)
0.8162
Table 8: Post-submission results on the English test set ( n=1,452 ). None submitted to CodaBench. mF1 = Macro-F1.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Post-submission pipeline. Rejudged Sonnet training data produces an SFT checkpoint, which then generates completions at varied temperatures to form candidate preference pairs. DeepSeek-R1 filters these into a balanced set (62:38 FP:FN ratio). DPO then refines the same SFT checkpoint (dashed arrow) using the filtered pairs, producing the best post-submission result of 0.8162 Macro-F1. The Rejudged Sonnet training data was produced by generating reasoning with GPT OSS 120B Nitro and filtering labels with Claude 3.5 Sonnet as a judge (see Appendix A.7 ).
β
mF1
Acc.
P(1)
R(1)
0.10
0.8127
0.8230
0.739
0.801
0.15
0.8125
0.8230
0.740
0.799
0.20
0.8136
0.8237
0.738
0.805
0.25
0.8108
0.8209
0.734
0.803
0.30
0.8162
0.8230
0.736
0.809
0.35
0.8079
0.8182
0.731
0.799
Appendix
Table 9: Beta sweep on 299 DeepSeek-R1-filtered DPO pairs (62:38 FP:FN ratio), Mistral-Nemo-Instruct-2407, 2 epochs. mF1 = Macro-F1; P(1)/R(1) = precision/recall for the polarized class.
Input text ( eng_03a6df98 )
“Ending 2024 with an illadvised text conversation with my brother that ended with ‘Really little bro? I can deal with the atheism, but socialism?”’
Gold label: 1 (Lack of Empathy: present; Invalidation: present)
SFT output (label: 0 — false negative)
Target: specific individual Claim type: evaluative opinion
Stereotype: absent Vilification: absent
Dehumanization: absent Extreme Language: absent
Appendix
Table 10: Qualitative comparison of SFT (Mistral-Nemo baseline) vs. best DPO model (299 DeepSeek-R1-filtered pairs, β =0.3) on a false-negative example from the test set. Bold fields differ between the two outputs. SFT misreads the implicit ideological framing as a personal anecdote and predicts 0. DPO recovers the correct label (1) by identifying Lack of Empathy and Invalidation, though it also flags Stereotype and Vilification as present, which are absent in the gold annotation.
Parameter
Value
Max Length
1024
Train Batch Size
1
Eval Batch Size
4
Gradient Accumulation Steps
8
Learning Rate
5e-5
Num Epochs
3
Appendix
Table 11: Pre-submission SFT hyperparameters (Qwen2.5-7B-Instruct). Class weights [1.0, 1.5] were used in both pre- and post-submission runs but did not massively improve performance over unweighted training (see Section 5 ).
Parameter
Value
Max Length
1024
Max Prompt Length
512
Train Batch Size
1
Gradient Accumulation Steps
8
Learning Rate
5e-6
Num Epochs
2
Appendix
Table 12: DPO hyperparameters. Pre-submission uses Qwen2.5-7B-Instruct with β =0.1; post-submission uses Mistral-Nemo-Instruct-2407 with β =0.3.
Configuration
FP:FN
Total
mF1
Recall
SFT baseline
—
—
0.8097
0.827
Original 190/70
73:27
260
0.8012
0.747
Subset 150/70
68:32
220
0.8024
0.745
Subset 120/70
63:37
190
0.8005
0.779
Subset 100/70
59:41
170
0.7986
0.739
Appendix
Table 13: DPO subset sweep on the stale unfiltered preference pairs: effect of subsampling class-0 (FP-correcting) pairs while holding class-1 (FN-correcting) pairs fixed at 70. Section 6.3 ), confirming that pair quality and volume rather than FP:FN ratio alone drive the performance gap. Matching the test-set class distribution (120/70, 63:37) did not improve performance.
We present SemEval-2026 Task 9, a shared task on online polarization detection, covering 22 languages and comprising over 110K annotated instances. Each data instance is multi-labeled with the presence of polarization, polarization type, and polarization manifestation. Participants were asked to predict labels in three sub-tasks: (1) detecting the presence of polarization, (2) identifying the type of polarization, and (3) recognizing the polarization manifestation. The three tasks attracted over 1,000 participants worldwide and more than 10k submission on Codabench. We received final submissions from 67 teams and 73 system description papers. We report the baseline results and analyze the performance of the best-performing systems, highlighting the most common approaches and the most effective methods across different subtasks and languages. The dataset of this task is publicly available.
Usman Naseem, Robert Geislinger, Juan Ren +31
Macquarie University · University of Hamburg · Zayed University +23
SemEval-2026 Task 9 is focused on multilingual polarization detection. Specifically, it covers the identification of multilingual, multicultural and multievent polarization along three axes (in subtasks), namely detection, type, and manifestation. Online polarization presents a concern, because it is often followed by hate speech, offensive discourse, and social fragmentation. Therefore, its detection before it escalates is crucial for a safer and more inclusive online space. We have coped with this SemEval task by finetuning mid-size LLMs for the sequence-classification task using the QLoRA parameter-efficient finetuning technique. The training data augmented the multilingual (22 languages) training sets by anonymized, lower-cased, upper-cased, and homoglyphied counterparts, making the detection more robust.
Dominik Macko, Alok Debnath, Jakub Simko
Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia · ADAPT Centre, School of Computer Science and Statistics, Trinity College Dublin
This paper presents our system for SemEval-2026 Task 9: Detecting Multilingual, Multicultural and Multievent Online Polarization, which identifies polarized social media content in 22 languages through three subtasks: binary detection, target classification, and manifestation identification. We propose a heterogeneous ensemble of multilingual pretrained models, combining XLM-RoBERTa-large and mDeBERTa-v3-base. We investigate techniques such as multi-task learning, translation-based data augmentation, and class weighting to improve classification performance under severe label imbalance. Our findings indicate that independent task modeling combined with class weighting is more effective.