Fine-Grained Emotion Classification from Mobile App Reviews: An Empirical Study with Large Language Models

Quim Motger; Carlota Catot; Marc Oriol

cs.CLOct 1, 2026

Fine-Grained Emotion Classification from Mobile App Reviews: An Empirical Study with Large Language Models

Authors: Quim Motger, Carlota Catot, Marc Oriol

Organizations: Department of Service and Information System Engineering, Universitat Politècnica de Catalunya, Spain

Abstract

Context: Fine-grained emotion classification of mobile app reviews enables requirements engineering activities that go beyond polarity-based opinion mining, including emotionally informed issue prioritisation and feature-oriented feedback analysis. However, automatic fine-grained emotion extraction from app reviews remains understudied. Objectives: Building on a previously published annotation framework and human-labelled ground truth adapted from Plutchik's taxonomy, this paper investigates how large language models can be leveraged for automatic multi-label emotion classification under severe class imbalance. Methods: We compare encoder-only fine-tuning under multi-label and binary-ensemble formulations, decoder-only zero- and few-shot prompting across open-source and proprietary models, and a catalogue of imbalance mitigation strategies (loss reweighting, resampling, generative data augmentation), with the synthetic-review generator and prompting strategy selected via an intrinsic augmentation-utility ranking. Results: Fine-tuned encoders trail the best decoder-only few-shot prompting (macro-F1 0.642) by a wide margin at baseline (multi-label: 0.387; binary ensemble: 0.450); pairing the best multi-label encoder with generative data augmentation and positive-weighted loss closes most of this gap (+0.204) at up to three orders of magnitude lower inference latency than the decoders, with the largest gains on the rarest emotions, from undetected to gains of up to +0.501 F1. Conclusion: Large language models make fine-grained, multi-label emotion classification of app reviews feasible for requirements engineering pipelines, with modest macro-F1, and the best formulation and mitigation strategy are backbone- and formulation-dependent. We release the experimental pipeline, synthetic corpora, and fine-tuned checkpoints for replication and reuse.

Figures & tables

Resource	Lang.	Labels	Unit	Size	Annotation	Guidelines	Avail.
Motger et al. (2025b)	EN	Plutchik (8) + Neutral	Sentence	1,090	Human ( $n=5$ )	Publicly released guidelines + examples	Yes
Riccosan and Saputra (2023)	ID	Shaver (5) + Neutral	Sentence	21,697	Human ( $n=2$ )	Paper-level description	Yes
Siqueira et al. (2024)	PT	Ekman (6) + Neutral	Review	3,000	Human ( $n=3$ )	Paper-level description	Yes
Singh and Suri (2022)	EN	Plutchik	Review	2,000	Automatic (lexicon-based)	Tool-oriented	No ‡
Keertipati et al. (2016)	EN	LIWC negative emotions	Review	4,442	Automatic (lexicon-based)	Tool-oriented	No ‡
Malgaonkar et al. (2019)	EN	Ekman (6)	Review	52,705	Automatic (lexicon-based)	Tool-oriented	No ‡

Table 1: Comparison of emotion-oriented resources for (mobile) app or closely related software reviews. “Avail.” indicates whether a reusable emotion-labelled corpus is publicly released.

Factor	RQ	Levels (summary)
Encoder backbone	RQ1, RQ4	See Table 3 (base/large variants)
Classification head	RQ1, RQ4	Multi-label (single model) vs. binary ensemble
Encoder training hyperparameters	RQ1, RQ4	Learning rate (tuned per backbone); epochs 3; batch size 8; max length 512
Decoder model	RQ2, RQ3	See Table 3 (open-source and proprietary)
Prompting regime	RQ2, RQ3	Zero-shot; few-shot (guidelines); few-shot (guidelines + dataset samples)
Decoding temperature	RQ2	Sensitivity grid $\{0.0,0.3,0.7\}$ (default $0.0$ for primary tables)

Table 2: Main experimental factors.

Architecture	Access	Family	Variant	Identifier	Role
Encoder-only	Open-source	BERT	base	bert-base-cased	Fine-tune (RQ1, RQ4)
	Open-source	BERT	large	bert-large-cased	Fine-tune (RQ1)
	Open-source	RoBERTa	base	roberta-base	Fine-tune (RQ1)
	Open-source	RoBERTa	large	roberta-large	Fine-tune (RQ1, RQ4)
	Open-source	DistilBERT	base	distilbert-base-cased	Fine-tune (RQ1)
	Open-source	DeBERTa-v3	base	microsoft/deberta-v3-base	Fine-tune (RQ1)

Table 3: Large language models used in this study. Encoder-only models are fine-tuned for classification (RQ1); the best backbone per formulation is also used in RQ4. Decoder-only open-source models are used for zero-/few-shot classification (RQ2). Proprietary decoder-only models are used for classification (RQ2) and as teachers for synthetic augmentation (RQ3). mistral-large-2512 is released with open weights but is accessed through the provider’s API and grouped with the proprietary cohort.

	Multi-label				Binary ensemble
Encoder	P	R	Macro-F1	Micro-F1	P	R	Macro-F1	Micro-F1
BERT ${}_{\text{base}}$	0.332	0.337	0.329 $\pm$ 0.039	0.644	0.484	0.450	0.450 $\pm$ 0.077	0.670
BERT ${}_{\text{large}}$	0.417	0.383	0.387 $\pm$ 0.047	0.671	0.463	0.448	0.440 $\pm$ 0.063	0.673
RoBERTa ${}_{\text{base}}$	0.405	0.381	0.380 $\pm$ 0.041	0.683	0.326	0.348	0.327 $\pm$ 0.052	0.641
RoBERTa ${}_{\text{large}}$	0.410	0.390	0.387 $\pm$ 0.032	0.697	0.413	0.415	0.404 $\pm$ 0.054	0.696
DistilBERT	0.380	0.351	0.353 $\pm$ 0.050	0.633	0.406	0.401	0.395 $\pm$ 0.046	0.638

Table 4: RQ1 aggregate performance across encoder backbones (10-fold CV, mean

\pm

std for macro-F1), listed as in Table 3 . P/R: macro-averaged precision/recall. Bold marks the best backbone per formulation (by macro-F1).

		Multi-label			Binary ensemble
Emotion	Support	P	R	F1	P	R	F1
Joy	371	0.826	0.786	0.804	0.763	0.808	0.784
Sadness	303	0.673	0.861	0.755	0.702	0.784	0.738
Anticipation	181	0.798	0.794	0.791	0.728	0.820	0.765
Trust	152	0.669	0.731	0.694	0.685	0.617	0.644
Neutral	88	0.528	0.255	0.322	0.520	0.341	0.384

Table 5: RQ1 per-emotion precision (P), recall (R) and F1 for the best backbone per formulation (RoBERTa

{}_{\text{large}}

, multi-label; BERT

{}_{\text{base}}

, binary ensemble). Support: positive sentences in the full 1,090-sentence corpus (encoder cross-validation uses the 938-sentence pool); sentences may carry multiple emotions. Per-emotion values are means over folds, so F1 need not equal the harmonic mean of the reported P and R. Sorted by support, descending.

Encoder	Multi-label (ms)	Binary ensemble (ms)
BERT ${}_{\text{base}}$	0.38	3.45
BERT ${}_{\text{large}}$	1.10	9.90
RoBERTa ${}_{\text{base}}$	0.38	3.43
RoBERTa ${}_{\text{large}}$	1.08	9.74
DistilBERT	0.21	1.88
DeBERTa-v3 ${}_{\text{base}}$	0.72	4.73

Table 6: RQ1 inference latency per sentence (ms), over the 1,090 sentences with batch size 16 on the RTX 4090. Binary ensemble: nine times the latency of a single binary classifier.

Decoder	Strategy	P	R	Macro-F1	Micro-F1
Proprietary
Gemini-3.5-Flash	Zero-shot	0.603	0.647	0.605	0.744
	Few-shot (guid.)	0.595	0.638	0.594	0.753
	Few-shot (dataset)	0.637	0.685	0.642	0.767
GPT-5.2	Zero-shot	0.538	0.575	0.528	0.669
	Few-shot (guid.)	0.559	0.582	0.542	0.681

Table 7: RQ2 precision (P), recall (R), macro-F1 and micro-F1 by decoder and prompting strategy (

T{=}0

). P/R: macro-averaged. Few-shot (guid.): guidelines; Few-shot (dataset): guidelines + dataset exemplars. Decoders are sorted by best macro-F1 within each cohort. Bold marks the best strategy (by macro-F1) per decoder.

Decoder	$T{=}0$	$T{=}0.3$	$T{=}0.7$	Range
Gemini-3.5-Flash	0.642	0.649	0.628	0.021
GPT-5.2	0.577	0.573	0.570	0.007
Claude-Opus-4.6	0.544	0.545	0.545	0.001
Mistral-Large	0.504	0.505	0.509	0.005
GPT-OSS-20B	0.526	0.508	0.532	0.024
Qwen3-8B	0.461	0.467	0.439	0.028

Table 8: RQ2 temperature sensitivity: macro-F1 at each decoder’s best strategy. Decoders listed as in Table 7 .

Decoder	Prompt tok.	Compl. tok.	Latency (s)
Gemini-3.5-Flash	571	31	1.21
GPT-5.2	540	19	0.23
Claude-Opus-4.6	598	28	0.24
Mistral-Large	567	24	0.42
GPT-OSS-20B	542	130	0.80
Qwen3-8B	447	145	1.13

Table 9: RQ2 cost profile per sentence (best strategy,

T{=}0

, batched

b{=}10

). Decoders listed as in Table 7 .

		Gemini-3.5-Flash			GPT-OSS-20B
Emotion	Support	P	R	F1	P	R	F1
Joy	371	0.839	0.820	0.827	0.829	0.714	0.762
Sadness	303	0.814	0.845	0.829	0.787	0.859	0.820
Anticipation	181	0.803	0.895	0.844	0.788	0.773	0.771
Trust	152	0.728	0.828	0.764	0.728	0.540	0.611
Neutral	88	0.591	0.764	0.661	0.476	0.683	0.552

Table 10: RQ2 per-emotion precision (P), recall (R) and F1 for the best proprietary (Gemini-3.5-Flash) and open-source (GPT-OSS-20B) decoder, each at its own best strategy,

T{=}0

. Support as in Table 5 , sorted by support, descending.

Rank	Strategy	Generator	Div.	Nov. (synth)	Nov. (GT)	On-emo.	Borda
1	Few-shot (dataset)	claude-opus-4-6	0.743	0.539	0.513	0.431	38
2	Few-shot (guidelines)	mistral-large-2512	0.739	0.504	0.570	0.357	34
3	Few-shot (guidelines)	claude-opus-4-6	0.727	0.518	0.511	0.429	33
4	Few-shot (dataset)	gemini-3.5-flash	0.716	0.499	0.559	0.384	30
5	Few-shot (guidelines)	gemini-3.5-flash	0.702	0.502	0.534	0.422	29
6	Zero-shot	claude-opus-4-6	0.695	0.476	0.553	0.395	27

Table 11: RQ3 augmentation-utility ranking across the four generators and three prompting strategies. Div. : diversity_intra ; Nov. (synth) : novelty_vs_synth ; Nov. (GT) : novelty_vs_gt ; On-emo. : on_emotion ; higher is better for all four metrics. Borda aggregates their per-metric ranks; overall rank sorts by Borda score, with the tie at ranks 11–12 broken by alphabetical order of the generator identifier. Bold marks the top-ranked configuration.

	Multi-label				Binary ensemble
Configuration	P	R	Macro-F1	Micro-F1	P	R	Macro-F1	Micro-F1
Baseline (none)	0.410	0.390	0.387 $\pm$ 0.032	0.697	0.484	0.450	0.450 $\pm$ 0.077	0.670
Loss reweighting
BCE + pos. weight (L1)	0.242	0.527	0.303 $\pm$ 0.151	0.443	0.361	0.378	0.359 $\pm$ 0.044	0.632
BCE + class weight (L2)	0.206	0.214	0.185 $\pm$ 0.120	0.476	0.355	0.360	0.345 $\pm$ 0.023	0.635
Focal loss (L3)	0.396	0.728	0.462 $\pm$ 0.021	0.553	0.442	0.551	0.461 $\pm$ 0.039	0.606

Table 12: RQ4 imbalance mitigation sweep on the best RQ1 backbone per formulation (RoBERTa

{}_{\text{large}}

, multi-label; BERT

{}_{\text{base}}

, binary ensemble; 10-fold CV, mean

\pm

std; P/R: macro-averaged). L1–L4: loss reweighting;

\tau

: undersampling cut-off;

n

: injected synthetic reviews per emotion. Bold marks the best configuration per formulation (by macro-F1).

		Baseline			Best config.
Emotion	Support	P	R	F1	P	R	F1	$\Delta$ F1
Joy	371	0.826	0.786	0.804	0.750	0.878	0.806	+0.002
Sadness	303	0.673	0.861	0.755	0.713	0.842	0.771	+0.016
Anticipation	181	0.798	0.794	0.791	0.764	0.885	0.815	+0.024
Trust	152	0.669	0.731	0.694	0.639	0.847	0.720	+0.026
Neutral	88	0.528	0.255	0.322	0.537	0.538	0.507	+0.185

Table 13: RQ4 per-emotion precision (P), recall (R) and F1, baseline vs. best configuration (RoBERTa

{}_{\text{large}}

, multi-label; GenAI aug.

n{=}100

+ BCE positive weight). Support as in Table 5 , sorted by support, descending.

Figure 1: RQ4: per-emotion F1 as a function of the generative-augmentation injection size

n

(RoBERTa

{}_{\text{large}}

, multi-label; pure augmentation, no reweighting);

n{=}0

is the no-mitigation baseline.

Explore similar work

Multilingual Multi-Label Emotion Classification at Scale with Synthetic Data
Apr 14, 2026Vadim BorisovEmotion Recognition Multi-Label Classification
Extracting Consumer Insight from Text: A Large Language Model Approach to Emotion and Evaluation Measurement
Feb 17, 2026Stephan Ludwig, Peter J. Danaher, Xiaohao Yang +4Sentiment Analysis LLM Information Extraction
Monash University, Australia · University of New South Wales, Australia · Flinders University, Australia +3
Quantifying the Affective Gap: A Zero-Shot Evaluation of LLMs on Fine-Grained Emotion Taxonomies
Jul 1, 2026Lawrence Obiuwevwi, Krzysztof J. Rechowicz, Jessica M. Johnson +3LLM Evaluation Zero-Shot Text Classification
Old Dominion University, Norfolk, VA, USA

Apr 14, 2026J/K move · Enter open · S save

Multilingual Multi-Label Emotion Classification at Scale with Synthetic Data

Vadim Borisov

Emotion Recognition Multi-Label Classification Multilingual Language Models

Open paper AskPDF arXiv

Emotion classification in multilingual settings remains constrained by the scarcity of annotated data: existing corpora are predominantly English, single-label, and cover few languages. We address this gap by constructing a large-scale synthetic training corpus of over 1M multi-label samples (50k per language) across 23 languages: Arabic, Bengali, Dutch, English, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Mandarin, Polish, Portuguese, Punjabi, Russian, Spanish, Swahili, Tamil, Turkish, Ukrainian, Urdu, and Vietnamese, covering 11 emotion categories using culturally-adapted generation and programmatic quality filtering. We train and compare six multilingual transformer encoders, from DistilBERT (135M parameters) to XLM-R-Large (560M parameters), under identical conditions. On our in-domain test set, XLM-R-Large achieves 0.868 F1-micro and 0.987 AUC-micro. To validate against human-annotated data, we evaluate all models zero-shot on GoEmotions (English) and SemEval-2018 Task 1 E-c (English, Arabic, Spanish). On threshold-free ranking metrics, XLM-R-Large matches or exceeds English-only specialist models, tying on AP-micro (0.636) and LRAP (0.804) while surpassing on AUC-micro (0.810 vs. 0.787), while natively supporting all 23 languages. The best base-sized model is publicly available at https://huggingface.co/tabularisai/multilingual-emotion-classification

Abstract

Explore similar work

Multilingual Multi-Label Emotion Classification at Scale with Synthetic Data

Extracting Consumer Insight from Text: A Large Language Model Approach to Emotion and Evaluation Measurement

Quantifying the Affective Gap: A Zero-Shot Evaluation of LLMs on Fine-Grained Emotion Taxonomies