Multilingual Multi-Label Emotion Classification at Scale with Synthetic Data

Vadim Borisov

cs.CLApr 14, 2026

Multilingual Multi-Label Emotion Classification at Scale with Synthetic Data

Authors: Vadim Borisov

Abstract

Emotion classification in multilingual settings remains constrained by the scarcity of annotated data: existing corpora are predominantly English, single-label, and cover few languages. We address this gap by constructing a large-scale synthetic training corpus of over 1M multi-label samples (50k per language) across 23 languages: Arabic, Bengali, Dutch, English, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Mandarin, Polish, Portuguese, Punjabi, Russian, Spanish, Swahili, Tamil, Turkish, Ukrainian, Urdu, and Vietnamese, covering 11 emotion categories using culturally-adapted generation and programmatic quality filtering. We train and compare six multilingual transformer encoders, from DistilBERT (135M parameters) to XLM-R-Large (560M parameters), under identical conditions. On our in-domain test set, XLM-R-Large achieves 0.868 F1-micro and 0.987 AUC-micro. To validate against human-annotated data, we evaluate all models zero-shot on GoEmotions (English) and SemEval-2018 Task 1 E-c (English, Arabic, Spanish). On threshold-free ranking metrics, XLM-R-Large matches or exceeds English-only specialist models, tying on AP-micro (0.636) and LRAP (0.804) while surpassing on AUC-micro (0.810 vs. 0.787), while natively supporting all 23 languages. The best base-sized model is publicly available at https://huggingface.co/tabularisai/multilingual-emotion-classification

Figures & tables

Emotion	Count	Share (%)
sadness	111,059	19.0
anger	107,058	18.3
frustration	104,456	17.9
surprise	99,942	17.1
disgust	92,825	15.9
love	84,870	14.5

Table 1: Emotion class distribution in the training set.

Model	Params	Epochs	Train (min)
DistilBERT-multi [ 8 ]	135M	3	14.0
mBERT [ 9 ]	178M	3	27.4
XLM-R-Base [ 10 ]	278M	3	31.5
Twitter-XLM-R [ 11 ]	278M	3	69.8
mDeBERTa-v3 [ 12 ]	278M	3	69.9
XLM-R-Large [ 10 ]	560M	3	130.8

Table 2: Models trained. All share the same head, loss, and hyperparameters.

Model	F1-mic	F1-mac	Jacc.	AUC	AP	LRAP
XLM-R-Large	.868	.868	.830	.987	.946	.952
XLM-R-Base	.840	.839	.794	.980	.923	.936
Twitter-XLM-R	.840	.839	.794	.980	.922	.936
mDeBERTa-v3	.838	.837	.790	.980	.920	.934
mBERT	.817	.816	.765	.975	.903	.921
DistilBERT	.790	.789	.728	.969	.881	.904

Table 3: In-domain test results (23 languages, 11 emotions,

\tau=0.5

). Best in bold .

Figure 1: In-domain test performance across four key metrics. XLM-R-Large leads on all metrics; the three base models cluster tightly.

Figure 2: Per-language F1-micro on the test set. Languages are sorted by mean F1 across models (hardest at top).

	GoE (EN)		SE (EN)		SE (AR)		SE (ES)
Model	F1	AUC	F1	AUC	F1	AUC	F1	AUC
XLM-R-Large	.534	.877	.436	.806	.503	.863	.466	.835
Tw-XLM-R	.518	.859	.401	.789	.520	.860	.457	.824
XLM-R-B	.517	.855	.389	.770	.506	.842	.429	.799
mDeBERTa	.494	.846	.409	.784	.503	.837	.467	.821
mBERT	.458	.819	.371	.741	.392	.785	.373	.747

Table 4: Cross-benchmark transfer (intersection label space). Best per benchmark in bold . GoE = GoEmotions, SE = SemEval-2018 E-c.

bhadresh’s 6-label subset
Model	F1-mic	AUC-mic	AP-mic	LRAP
bhadresh †	.557	.787	.636	.804
XLM-R-Large (ours)	.467	.810	.636	.804
mDeBERTa-v3 (ours)	.438	.792	.616	.793
j-hartmann’s 7-label subset
j-hartmann †	.461	.700	.518	.712

Table 5: Head-to-head on SemEval EN test in each specialist’s label subset.

{\dagger}

= English-only single-label specialist. Highest per metric in bold .

Figure 3: Compute vs. quality Pareto frontier. Training time (min, log scale) vs. test Jaccard. XLM-R-Base dominates among base models due to its 2

\times

speed advantage.

Figure 4: Micro-averaged precision-recall curves on the in-domain test set. AP-micro values are shown in the legend.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Our label	GoEmotions labels mapped
anger	anger, annoyance
contempt	disapproval
disgust	disgust
fear	fear, nervousness, embarrassment
frustration	disappointment
gratitude	gratitude

Appendix

Table 6: GoEmotions label projection.

Our label	SemEval labels mapped
anger	anger
disgust	disgust
fear	fear
joy	joy, optimism
love	love, trust
sadness	sadness, pessimism

Appendix

Table 7: SemEval-2018 E-c label projection. anticipation is dropped (no cognate in our taxonomy).

Model	F1-mic	F1-mac	Jacc.	AUC	AP	LRAP
XLM-R-Large	.357	.314	.312	.750	.326	.542
Twitter-XLM-R	.349	.314	.293	.748	.331	.531
XLM-R-Base	.345	.305	.296	.740	.332	.523
mDeBERTa-v3	.337	.308	.278	.740	.330	.526
mBERT	.317	.285	.266	.709	.304	.489
DistilBERT	.306	.272	.249	.704	.300	.483

Appendix

Table 8: GoEmotions test—projected label space.

	English		Arabic		Spanish
Model	F1-mic	AUC	F1-mic	AUC	F1-mic	AUC
XLM-R-Large	.366	.756	.424	.804	.366	.782
Twitter-XLM-R	.342	.743	.446	.806	.366	.777
XLM-R-Base	.331	.727	.441	.798	.344	.758
mDeBERTa-v3	.348	.739	.435	.792	.377	.776
mBERT	.318	.705	.329	.734	.304	.719

Appendix

Table 9: SemEval-2018 E-c test—projected (11-label space), by language.

Explore similar work

Fine-Grained Emotion Classification from Mobile App Reviews: An Empirical Study with Large Language Models
Oct 1, 2026Quim Motger, Carlota Catot, Marc OriolEmotion Recognition Sentiment Analysis
Department of Service and Information System Engineering, Universitat Politècnica de Catalunya, Spain
Do Speech Emphasis Models Generalize across Languages and Emotions?
Jun 26, 2026Megan Wei, Deepali Aneja, Jiaqi Su +3Speech Prosody Cross-Lingual Transfer
Adobe Research, USA · Brown University, USA
Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection
Oct 8, 2026Vinko Sabolčec, Bettina Messmer, Yassine Turki +1Language Model Pretraining Cross-Lingual Transfer
EPFL

Oct 1, 2026J/K move · Enter open · S save

Fine-Grained Emotion Classification from Mobile App Reviews: An Empirical Study with Large Language Models

Quim Motger, Carlota Catot, Marc Oriol

Department of Service and Information System Engineering, Universitat Politècnica de Catalunya, Spain

Emotion Recognition Sentiment Analysis LLM Fine-Tuning Class-Imbalanced Learning

Open paper AskPDF arXiv

Context: Fine-grained emotion classification of mobile app reviews enables requirements engineering activities that go beyond polarity-based opinion mining, including emotionally informed issue prioritisation and feature-oriented feedback analysis. However, automatic fine-grained emotion extraction from app reviews remains understudied. Objectives: Building on a previously published annotation framework and human-labelled ground truth adapted from Plutchik's taxonomy, this paper investigates how large language models can be leveraged for automatic multi-label emotion classification under severe class imbalance. Methods: We compare encoder-only fine-tuning under multi-label and binary-ensemble formulations, decoder-only zero- and few-shot prompting across open-source and proprietary models, and a catalogue of imbalance mitigation strategies (loss reweighting, resampling, generative data augmentation), with the synthetic-review generator and prompting strategy selected via an intrinsic augmentation-utility ranking. Results: Fine-tuned encoders trail the best decoder-only few-shot prompting (macro-F1 0.642) by a wide margin at baseline (multi-label: 0.387; binary ensemble: 0.450); pairing the best multi-label encoder with generative data augmentation and positive-weighted loss closes most of this gap (+0.204) at up to three orders of magnitude lower inference latency than the decoders, with the largest gains on the rarest emotions, from undetected to gains of up to +0.501 F1. Conclusion: Large language models make fine-grained, multi-label emotion classification of app reviews feasible for requirements engineering pipelines, with modest macro-F1, and the best formulation and mitigation strategy are backbone- and formulation-dependent. We release the experimental pipeline, synthetic corpora, and fine-tuned checkpoints for replication and reuse.

Abstract

Explore similar work

Fine-Grained Emotion Classification from Mobile App Reviews: An Empirical Study with Large Language Models

Do Speech Emphasis Models Generalize across Languages and Emotions?

Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection