Fine-Grained Emotion Classification from Mobile App Reviews: An Empirical Study with Large Language Models

Quim Motger; Carlota Catot; Marc Oriol

cs.CLOct 1, 2026

Fine-Grained Emotion Classification from Mobile App Reviews: An Empirical Study with Large Language Models

Authors: Quim Motger, Carlota Catot, Marc Oriol

Organizations: Department of Service and Information System Engineering, Universitat Politècnica de Catalunya, Spain

Abstract

Context: Fine-grained emotion classification of mobile app reviews enables requirements engineering activities that go beyond polarity-based opinion mining, including emotionally informed issue prioritisation and feature-oriented feedback analysis. However, automatic fine-grained emotion extraction from app reviews remains understudied. Objectives: Building on a previously published annotation framework and human-labelled ground truth adapted from Plutchik's taxonomy, this paper investigates how large language models can be leveraged for automatic multi-label emotion classification under severe class imbalance. Methods: We compare encoder-only fine-tuning under multi-label and binary-ensemble formulations, decoder-only zero- and few-shot prompting across open-source and proprietary models, and a catalogue of imbalance mitigation strategies (loss reweighting, resampling, generative data augmentation), with the synthetic-review generator and prompting strategy selected via an intrinsic augmentation-utility ranking. Results: Fine-tuned encoders trail the best decoder-only few-shot prompting (macro-F1 0.642) by a wide margin at baseline (multi-label: 0.387; binary ensemble: 0.450); pairing the best multi-label encoder with generative data augmentation and positive-weighted loss closes most of this gap (+0.204) at up to three orders of magnitude lower inference latency than the decoders, with the largest gains on the rarest emotions, from undetected to gains of up to +0.501 F1. Conclusion: Large language models make fine-grained, multi-label emotion classification of app reviews feasible for requirements engineering pipelines, with modest macro-F1, and the best formulation and mitigation strategy are backbone- and formulation-dependent. We release the experimental pipeline, synthetic corpora, and fine-tuned checkpoints for replication and reuse.

Figures & tables

Resource	Lang.	Labels	Unit	Size	Annotation	Guidelines	Avail.
Motger et al. (2025b)	EN	Plutchik (8) + Neutral	Sentence	1,090	Human ( $n=5$ )	Publicly released guidelines + examples	Yes
Riccosan and Saputra (2023)	ID	Shaver (5) + Neutral	Sentence	21,697	Human ( $n=2$ )	Paper-level description	Yes
Siqueira et al. (2024)	PT	Ekman (6) + Neutral	Review	3,000	Human ( $n=3$ )	Paper-level description	Yes
Singh and Suri (2022)	EN	Plutchik	Review	2,000	Automatic (lexicon-based)	Tool-oriented	No ‡
Keertipati et al. (2016)	EN	LIWC negative emotions	Review	4,442	Automatic (lexicon-based)	Tool-oriented	No ‡
Malgaonkar et al. (2019)	EN	Ekman (6)	Review	52,705	Automatic (lexicon-based)	Tool-oriented	No ‡

Table 1: Comparison of emotion-oriented resources for (mobile) app or closely related software reviews. “Avail.” indicates whether a reusable emotion-labelled corpus is publicly released.

Factor	RQ	Levels (summary)
Encoder backbone	RQ1, RQ4	See Table 3 (base/large variants)
Classification head	RQ1, RQ4	Multi-label (single model) vs. binary ensemble
Encoder training hyperparameters	RQ1, RQ4	Learning rate (tuned per backbone); epochs 3; batch size 8; max length 512
Decoder model	RQ2, RQ3	See Table 3 (open-source and proprietary)
Prompting regime	RQ2, RQ3	Zero-shot; few-shot (guidelines); few-shot (guidelines + dataset samples)
Decoding temperature	RQ2	Sensitivity grid $\{0.0,0.3,0.7\}$ (default $0.0$ for primary tables)

Table 2: Main experimental factors.

Architecture	Access	Family	Variant	Identifier	Role
Encoder-only	Open-source	BERT	base	bert-base-cased	Fine-tune (RQ1, RQ4)
	Open-source	BERT	large	bert-large-cased	Fine-tune (RQ1)
	Open-source	RoBERTa	base	roberta-base	Fine-tune (RQ1)
	Open-source	RoBERTa	large	roberta-large	Fine-tune (RQ1, RQ4)
	Open-source	DistilBERT	base	distilbert-base-cased	Fine-tune (RQ1)
	Open-source	DeBERTa-v3	base	microsoft/deberta-v3-base	Fine-tune (RQ1)

Table 3: Large language models used in this study. Encoder-only models are fine-tuned for classification (RQ1); the best backbone per formulation is also used in RQ4. Decoder-only open-source models are used for zero-/few-shot classification (RQ2). Proprietary decoder-only models are used for classification (RQ2) and as teachers for synthetic augmentation (RQ3). mistral-large-2512 is released with open weights but is accessed through the provider’s API and grouped with the proprietary cohort.

	Multi-label				Binary ensemble
Encoder	P	R	Macro-F1	Micro-F1	P	R	Macro-F1	Micro-F1
BERT ${}_{\text{base}}$	0.332	0.337	0.329 $\pm$ 0.039	0.644	0.484	0.450	0.450 $\pm$ 0.077	0.670
BERT ${}_{\text{large}}$	0.417	0.383	0.387 $\pm$ 0.047	0.671	0.463	0.448	0.440 $\pm$ 0.063	0.673
RoBERTa ${}_{\text{base}}$	0.405	0.381	0.380 $\pm$ 0.041	0.683	0.326	0.348	0.327 $\pm$ 0.052	0.641
RoBERTa ${}_{\text{large}}$	0.410	0.390	0.387 $\pm$ 0.032	0.697	0.413	0.415	0.404 $\pm$ 0.054	0.696
DistilBERT	0.380	0.351	0.353 $\pm$ 0.050	0.633	0.406	0.401	0.395 $\pm$ 0.046	0.638

Table 4: RQ1 aggregate performance across encoder backbones (10-fold CV, mean

\pm

std for macro-F1), listed as in Table 3 . P/R: macro-averaged precision/recall. Bold marks the best backbone per formulation (by macro-F1).

		Multi-label			Binary ensemble
Emotion	Support	P	R	F1	P	R	F1
Joy	371	0.826	0.786	0.804	0.763	0.808	0.784
Sadness	303	0.673	0.861	0.755	0.702	0.784	0.738
Anticipation	181	0.798	0.794	0.791	0.728	0.820	0.765
Trust	152	0.669	0.731	0.694	0.685	0.617	0.644
Neutral	88	0.528	0.255	0.322	0.520	0.341	0.384

Table 5: RQ1 per-emotion precision (P), recall (R) and F1 for the best backbone per formulation (RoBERTa

{}_{\text{large}}

, multi-label; BERT

{}_{\text{base}}

, binary ensemble). Support: positive sentences in the full 1,090-sentence corpus (encoder cross-validation uses the 938-sentence pool); sentences may carry multiple emotions. Per-emotion values are means over folds, so F1 need not equal the harmonic mean of the reported P and R. Sorted by support, descending.

Encoder	Multi-label (ms)	Binary ensemble (ms)
BERT ${}_{\text{base}}$	0.38	3.45
BERT ${}_{\text{large}}$	1.10	9.90
RoBERTa ${}_{\text{base}}$	0.38	3.43
RoBERTa ${}_{\text{large}}$	1.08	9.74
DistilBERT	0.21	1.88
DeBERTa-v3 ${}_{\text{base}}$	0.72	4.73

Table 6: RQ1 inference latency per sentence (ms), over the 1,090 sentences with batch size 16 on the RTX 4090. Binary ensemble: nine times the latency of a single binary classifier.

Decoder	Strategy	P	R	Macro-F1	Micro-F1
Proprietary
Gemini-3.5-Flash	Zero-shot	0.603	0.647	0.605	0.744
	Few-shot (guid.)	0.595	0.638	0.594	0.753
	Few-shot (dataset)	0.637	0.685	0.642	0.767
GPT-5.2	Zero-shot	0.538	0.575	0.528	0.669
	Few-shot (guid.)	0.559	0.582	0.542	0.681

Table 7: RQ2 precision (P), recall (R), macro-F1 and micro-F1 by decoder and prompting strategy (

T{=}0

). P/R: macro-averaged. Few-shot (guid.): guidelines; Few-shot (dataset): guidelines + dataset exemplars. Decoders are sorted by best macro-F1 within each cohort. Bold marks the best strategy (by macro-F1) per decoder.

Decoder	$T{=}0$	$T{=}0.3$	$T{=}0.7$	Range
Gemini-3.5-Flash	0.642	0.649	0.628	0.021
GPT-5.2	0.577	0.573	0.570	0.007
Claude-Opus-4.6	0.544	0.545	0.545	0.001
Mistral-Large	0.504	0.505	0.509	0.005
GPT-OSS-20B	0.526	0.508	0.532	0.024
Qwen3-8B	0.461	0.467	0.439	0.028

Table 8: RQ2 temperature sensitivity: macro-F1 at each decoder’s best strategy. Decoders listed as in Table 7 .

Decoder	Prompt tok.	Compl. tok.	Latency (s)
Gemini-3.5-Flash	571	31	1.21
GPT-5.2	540	19	0.23
Claude-Opus-4.6	598	28	0.24
Mistral-Large	567	24	0.42
GPT-OSS-20B	542	130	0.80
Qwen3-8B	447	145	1.13

Table 9: RQ2 cost profile per sentence (best strategy,

T{=}0

, batched

b{=}10

). Decoders listed as in Table 7 .

		Gemini-3.5-Flash			GPT-OSS-20B
Emotion	Support	P	R	F1	P	R	F1
Joy	371	0.839	0.820	0.827	0.829	0.714	0.762
Sadness	303	0.814	0.845	0.829	0.787	0.859	0.820
Anticipation	181	0.803	0.895	0.844	0.788	0.773	0.771
Trust	152	0.728	0.828	0.764	0.728	0.540	0.611
Neutral	88	0.591	0.764	0.661	0.476	0.683	0.552

Table 10: RQ2 per-emotion precision (P), recall (R) and F1 for the best proprietary (Gemini-3.5-Flash) and open-source (GPT-OSS-20B) decoder, each at its own best strategy,

T{=}0

. Support as in Table 5 , sorted by support, descending.

Rank	Strategy	Generator	Div.	Nov. (synth)	Nov. (GT)	On-emo.	Borda
1	Few-shot (dataset)	claude-opus-4-6	0.743	0.539	0.513	0.431	38
2	Few-shot (guidelines)	mistral-large-2512	0.739	0.504	0.570	0.357	34
3	Few-shot (guidelines)	claude-opus-4-6	0.727	0.518	0.511	0.429	33
4	Few-shot (dataset)	gemini-3.5-flash	0.716	0.499	0.559	0.384	30
5	Few-shot (guidelines)	gemini-3.5-flash	0.702	0.502	0.534	0.422	29
6	Zero-shot	claude-opus-4-6	0.695	0.476	0.553	0.395	27

Table 11: RQ3 augmentation-utility ranking across the four generators and three prompting strategies. Div. : diversity_intra ; Nov. (synth) : novelty_vs_synth ; Nov. (GT) : novelty_vs_gt ; On-emo. : on_emotion ; higher is better for all four metrics. Borda aggregates their per-metric ranks; overall rank sorts by Borda score, with the tie at ranks 11–12 broken by alphabetical order of the generator identifier. Bold marks the top-ranked configuration.

	Multi-label				Binary ensemble
Configuration	P	R	Macro-F1	Micro-F1	P	R	Macro-F1	Micro-F1
Baseline (none)	0.410	0.390	0.387 $\pm$ 0.032	0.697	0.484	0.450	0.450 $\pm$ 0.077	0.670
Loss reweighting
BCE + pos. weight (L1)	0.242	0.527	0.303 $\pm$ 0.151	0.443	0.361	0.378	0.359 $\pm$ 0.044	0.632
BCE + class weight (L2)	0.206	0.214	0.185 $\pm$ 0.120	0.476	0.355	0.360	0.345 $\pm$ 0.023	0.635
Focal loss (L3)	0.396	0.728	0.462 $\pm$ 0.021	0.553	0.442	0.551	0.461 $\pm$ 0.039	0.606

Table 12: RQ4 imbalance mitigation sweep on the best RQ1 backbone per formulation (RoBERTa

{}_{\text{large}}

, multi-label; BERT

{}_{\text{base}}

, binary ensemble; 10-fold CV, mean

\pm

std; P/R: macro-averaged). L1–L4: loss reweighting;

\tau

: undersampling cut-off;

n

: injected synthetic reviews per emotion. Bold marks the best configuration per formulation (by macro-F1).

		Baseline			Best config.
Emotion	Support	P	R	F1	P	R	F1	$\Delta$ F1
Joy	371	0.826	0.786	0.804	0.750	0.878	0.806	+0.002
Sadness	303	0.673	0.861	0.755	0.713	0.842	0.771	+0.016
Anticipation	181	0.798	0.794	0.791	0.764	0.885	0.815	+0.024
Trust	152	0.669	0.731	0.694	0.639	0.847	0.720	+0.026
Neutral	88	0.528	0.255	0.322	0.537	0.538	0.507	+0.185

Table 13: RQ4 per-emotion precision (P), recall (R) and F1, baseline vs. best configuration (RoBERTa

{}_{\text{large}}

, multi-label; GenAI aug.

n{=}100

+ BCE positive weight). Support as in Table 5 , sorted by support, descending.

Figure 1: RQ4: per-emotion F1 as a function of the generative-augmentation injection size

n

(RoBERTa

{}_{\text{large}}

, multi-label; pure augmentation, no reweighting);

n{=}0

is the no-mitigation baseline.

Explore similar work

Apr 14, 2026

Multilingual Multi-Label Emotion Classification at Scale with Synthetic Data

Emotion classification in multilingual settings remains constrained by the scarcity of annotated data: existing corpora are predominantly English, single-label, and cover few languages. We address this gap by constructing a large-scale synthetic training corpus of over 1M multi-label samples (50k per language) across 23 languages: Arabic, Bengali, Dutch, English, French, German, Hindi, Indonesian, Italian, Japanese, Korean, Mandarin, Polish, Portuguese, Punjabi, Russian, Spanish, Swahili, Tamil, Turkish, Ukrainian, Urdu, and Vietnamese, covering 11 emotion categories using culturally-adapted generation and programmatic quality filtering. We train and compare six multilingual transformer encoders, from DistilBERT (135M parameters) to XLM-R-Large (560M parameters), under identical conditions. On our in-domain test set, XLM-R-Large achieves 0.868 F1-micro and 0.987 AUC-micro. To validate against human-annotated data, we evaluate all models zero-shot on GoEmotions (English) and SemEval-2018 Task 1 E-c (English, Arabic, Spanish). On threshold-free ranking metrics, XLM-R-Large matches or exceeds English-only specialist models, tying on AP-micro (0.636) and LRAP (0.804) while surpassing on AUC-micro (0.810 vs. 0.787), while natively supporting all 23 languages. The best base-sized model is publicly available at https://huggingface.co/tabularisai/multilingual-emotion-classification

Vadim Borisov

Emotion Recognition Multi-Label Classification Multilingual Language Models

Ask PDF

Feb 17, 2026

Extracting Consumer Insight from Text: A Large Language Model Approach to Emotion and Evaluation Measurement

Accurately measuring consumer emotions and evaluations from unstructured text remains a core challenge for marketing research and practice. This study introduces the Linguistic eXtractor (LX), a fine-tuned, large language model trained on consumer-authored text that also has been labeled with consumers' self-reported ratings of 16 consumption-related emotions and four evaluation constructs: trust, commitment, recommendation, and sentiment. LX consistently outperforms leading models, including GPT-4 Turbo, RoBERTa, and DeepSeek, achieving 81% macro-F1 accuracy on open-ended survey responses and greater than 95% accuracy on third-party-annotated Amazon and Yelp reviews. An application of LX to online retail data, using seemingly unrelated regression, affirms that review-expressed emotions predict product ratings, which in turn predict purchase behavior. Most emotional effects are mediated by product ratings, though some emotions, such as discontent and peacefulness, influence purchase directly, indicating that emotional tone provides meaningful signals beyond star ratings. To support its use, a no-code, cost-free, LX web application is available, enabling scalable analyses of consumer-authored text. In establishing a new methodological foundation for consumer perception measurement, this research demonstrates new methods for leveraging large language models to advance marketing research and practice, thereby achieving validated detection of marketing constructs from consumer data.

Stephan Ludwig, Peter J. Danaher, Xiaohao Yang +4

Monash University, Australia · University of New South Wales, Australia · Flinders University, Australia +3

Sentiment Analysis LLM Information Extraction

Ask PDF

Jul 1, 2026

Quantifying the Affective Gap: A Zero-Shot Evaluation of LLMs on Fine-Grained Emotion Taxonomies

Emotion recognition in natural language is a foundational challenge in affective computing, with critical implications for human-computer interaction, mental health support, and conversational AI. This paper presents a rigorous, unified zero-shot evaluation of three leading commercial large language models: Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash). The models were queried through their respective production APIs as of April 2026 on a fine-grained 13-class emotion classification task. Using a stratified 1,000-sentence sample from the boltuix/emotions dataset, which comprises 131,306 sentences across 13 categories, a single uniform prompt with no exemplars was applied identically across all models. Gemini achieves the highest accuracy (39.9%) and macro-F1 score (0.363), followed by GPT-5.4 (38.8%, macro-F1 = 0.291) and Claude (38.0%, macro-F1 = 0.159). All models excel on sarcasm and desire while consistently failing on love, confusion, and shame. McNemar tests reveal no statistically significant pairwise differences (p > 0.10), suggesting convergence at a shared zero-shot ceiling. Claude's markedly lower macro-F1 score exposes a class-imbalance prediction bias. These findings highlight the current limitations of frontier AI systems in zero-shot fine-grained emotion classification.

Lawrence Obiuwevwi, Krzysztof J. Rechowicz, Jessica M. Johnson +3

Old Dominion University, Norfolk, VA, USA

LLM Evaluation Zero-Shot Text Classification Emotion Recognition

Ask PDF