Speech emotion recognition (SER) systems can exhibit gender-related performance disparities, but how such bias manifests in multilingual speech LLMs across languages and modalities is unclear. We introduce a novel multilingual, multimodal benchmark built on MELD-ST, spanning English, Japanese, and German, to quantify language-specific SER performance and gender gaps. We find bias is strongly language-dependent, and multimodal fusion does not reliably improve fairness. To address these, we propose ERM-MinMaxGAP, a fairness-informed training objective, which augments empirical risk minimization (ERM) with a proposed adaptive fairness weight mechanism and a novel MinMaxGAP regularizer on the maximum male-female loss gap within each language and modality. Building upon the Qwen2-Audio backbone, our ERM-MinMaxGAP approach improves multilingual SER performance by 5.5% and 5.0% while reducing the overall gender bias gap by 0.1% and 1.4% in the unimodal and multimodal settings, respectively.
Figures & tables
Figure 1: Architecture of the proposed method. The method consists of (1) empirical risk minimization for overall SER improvement, (2) MinMaxGAP for minimizing the language-wise gender gap, and (3) adaptive fairness-weight adjustment for fairness-aware SER.
Language
Gender
Train
Valid
Test
Total
English
Female
3546
464
501
4511
Male
3850
448
459
4757
Total
7396
912
960
9268
Japanese
Female
3546
464
501
4511
Male
3850
448
459
4757
Total
7396
912
960
9268
Table 1: Statistics of the MELD-ST dataset.
Model
Unimodal (Speech only)
Multimodal (Speech + Transcription)
SER Result
Gender Bias Gap
SER Result
Gender Bias Gap
W-F1 ↑
ACC ↑
TPR ↓
FPR ↓
W-F1 ↓
ACC ↓
AVG ↓
W-F1 ↑
ACC ↑
TPR ↓
FPR ↓
W-F1 ↓
ACC ↓
AVG ↓
Multilingual
Qwen2-Audio
34.89
33.31
9.78
4.26
3.36
4.67
5.51
34.62
30.79
8.66
2.08
3.67
3.35
4.44 ↓1.07
Voxtral-Mini-3B
44.78
44.52
9.28
2.74
4.97
5.22
5.55
50.04
55.03
7.27
1.90
2.18
1.84
3.30 ↓2.25
gpt4o-mini-audio
45.89
44.57
9.61
1.48
2.61
4.03
4.43
52.65
51.76
8.09
2.08
4.38
5.25
4.95 ↑0.52
Table 2: Comparison of unimodal and multimodal performance, with SER results and gender bias gap metrics [%]. Relative improvements and degradations of the multimodal input over the unimodal input are shown in green , and red , respectively. Top two results are highlighted in bold and underline , respectively.
Model
Unimodal (Speech only)
Multimodal (Speech + Transcription)
SER Result
Gender Bias Gap
SER Result
Gender Bias Gap
W-F1 ↑
ACC ↑
TPR ↓
FPR ↓
W-F1 ↓
ACC ↓
AVG ↓
W-F1 ↑
ACC ↑
TPR ↓
FPR ↓
W-F1 ↓
ACC ↓
AVG ↓
Main Components
Qwen2-Audio
34.89
33.31
9.78
4.26
3.36
4.67
5.51
34.62
30.79
8.66
2.08
3.67
3.35
4.44 ↓1.07
+ ERM (SFT)
47.50
46.07
11.28
2.64
2.87
3.08
4.97
56.13
54.77
9.50
2.03
0.93
1.22
3.42 ↓1.55
ERM-MinMaxGAP (Ours)
51.38
54.32
6.38
3.01
3.52
4.44
4.34
57.68
58.65
7.08
2.69
1.84
2.53
3.53 ↓0.80
Table 3: Ablation study [%]. Relative improvements and degradations of the multimodal input over the unimodal input are shown in green , and red , respectively.