Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore · AI Singapore, National University of Singapore · Institute for Infocomm Research (I2R), A∗STAR, Singapore
Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction, but are difficult to train with supervised fine-tuning, requiring large, task-specific speech corpora. While recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector, these models under-perform when scaled to multilingual settings due to language interference in the shared projector. We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector. Our approach shows gains of 14% over matched multilingual distillation baselines on instruction following. We further synthesize Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions. Our best model improves over existing Speech LLM baselines by 32% on Audio-MLQA.
Figures & tables
Figure 1: End-to-end model architecture. Trainable components are colored orange.
Dataset
Language
Hours
#Samples
CV21 [ 26 ]
EN
1750
1.10M
ViVoice [ 27 ]
VI
900
0.785M
YODAS2 [ 28 ]
ID
950
0.996M
MagicData [ 29 ]
ZH
755
0.500M
CV24 [ 26 ]
ZH
100
0.030M
CV24 [ 26 ]
ES
515
0.357M
Table 1: Breakdown of ASR training data by language, number of hours, and sample count. CV is CommonVoice.
Model
Close-ended (A-MLQA) ( ↑ )
Open-ended ( ↑ )
EN
VI
ES
DE
ZH
Avg
ZH
EN
ID
AE
AE
OH
Avg
AE
OH
Avg
Text-only reference
SEA-LION-v3-8B-IT
4.34
4.10
4.19
4.08
3.98
4.14
4.21
4.59
4.44
4.52
4.19
3.85
4.02
Cascaded
Whisper-v3 + SEA-LION-v3-8B-IT
4.20
3.75
4.09
4.01
3.94
3.99
3.95
3.99
4.17
4.08
3.68
3.61
3.65
Table 2: GPT-4.1 Model-as-Judge scores for open-ended instruction following and close-ended spoken QA. Higher is better ( ↑ ). To keep comparison fair, if a model does not support a language, the score has been left blank. EN-DiVA and ML-DiVA are matched DiVA [ 12 ] baseline models trained on English-only ASR data and Multilingual ASR data respectively from our pool of training data.
Variant
LOUT↓
LIN↓
LID Acc. ↑
L=64
57.92
8.63
–
L=128
40.69
1.16
–
L=256
39.47
0.97
–
+ Conv. DS + Pool Gate
37.31
0.84
95.15%
+ Attn. Pool Gate
36.93
0.94
94.97%
Table 3: Ablations on query length L and gating design. We report validation losses for input and output distillation ( LIN , LOUT ); for gating variants we additionally report LID accuracy.