Organizations: Shanghai Engineering Research Center of Intelligent Education and Bigdata, Shanghai Normal University, Shanghai, 200234, China. · Unisound AI Technology Co., Ltd., Beijing, China. · SHNU-Unisound Natural Human-Computer Interaction Lab, Shanghai Normal University.
Speech Large Language Models (Speech-LLMs) have emerged as a powerful approach for automatic speech recognition (ASR) by aligning speech encoders with large language models. However, adapting these systems to multilingual settings with imbalanced data distributions remains challenging. In such scenarios, a stability-plasticity dilemma often arises: fully shared Parameter-Efficient Fine-Tuning (PEFT) can cause negative inter-lingual interference for under-represented languages, while fully language-specific tuning limits the cross-lingual beneficial knowledge transfer needed for low-resource tasks. To address this, we propose Zipper-LoRA, a novel rank-level decoupling framework with three variants (Static, Hard, and Soft) that dynamically synthesizes LoRA updates from shared and language-specific subspaces. By using a lightweight language-conditioned router, Zipper-LoRA dynamically controls the contribution of each subspace at the LoRA rank level, enabling fine-grained sharing where languages are compatible and strict decoupling when conflicts occur. To further stabilize optimization under imbalanced data, we propose a two-stage training strategy with an Initial-B warm start that significantly accelerates convergence. Experiments on a 12-language mixed-resource setting show that Zipper-LoRA consistently outperforms both fully shared and independent baselines, particularly in extremely low-resource scenarios. Moreover, we demonstrate that these gains are robust across both chunked and non-chunked encoder configurations, confirming the framework's reliability for practical, large-scale multilingual ASR. Our code and data will be available at https://github.com/YuCeong-May/Zipper-LoRA for reproducibility.
Figures & tables
Fig. 1: Overall Speech-LLM backbone consisting of a speech encoder, a modality projector, and a decoder-only LLM.
Fig. 2: Language-specific prompts. All these prompts have the same meaning: “Please transcribe the audio content into text.” but are written in specific languages based on the language given for a speech.
Fig. 3: Illustration of three representative PEFT frameworks for multilingual ASR Speech-LLM adaptation: Vanilla-LoRA (a), Independent-LoRA (b), and FlyLoRA (c).
Fig. 4: Overview of the proposed Zipper-LoRA. A language-aware router outputs rank-wise mixing weights from language embeddings to construct Bmerged(l) for multilingual ASR adaptation.
Setting
Dataset
Language
Train(h)
Test (h)
Baseline (source)
MSR86k [ 53 ]
English (en)
3000
14.72
French (fr)
3000
12.60
Thai (th)
3000
8.37
WenetSpeech (WS) [ 54 ]
Chinese (zh)
3000
15.18 ∣ 23.06
SFT (target)
MLS [ 55 ]
German (de)
1000
14.29
Spanish (es)
1000
10.00
TABLE I: Dataset statistics (duration in hours).
MSR86k
WenetSpeech
en
fr
th
Meeting
Net
Chunked
4.65
9.24
4.20
16.55
9.60
Non-chunked
4.59
9.29
4.05
17.16
10.22
TABLE II: Foundation baseline WER/CER (%) comparison under chunked vs. non-chunked settings.
Methods
Shared
Specific
LID Router
Dynamic Rank
Soft Mixing
Initial-B
Routing
Vanilla-LoRA
✓
–
–
–
–
–
FlyLoRA
✓
–
–
✓
–
–
Independent-LoRA
–
✓
–
–
–
–
Zipper-LoRA-Static
✓
✓
–
–
–
–
Zipper-LoRA-Hard
✓
✓
✓
✓
–
–
TABLE III: Component configurations of the compared LoRA-based adaptation methods.
Methods
MSR86k
WenetSpeech
en
fr
th
Meeting
Net
Chunked
Vanilla-LoRA
4.88
10.74
13.58
18.51
12.12
FlyLoRA
4.87
10.65
7.99
17.57
11.86
Independent-LoRA
4.86
11.07
7.72
17.24
11.63
Zipper-LoRA-Static
4.91
11.35
7.85
18.12
12.11
TABLE IV: Source-domain performance after SFT on MSR86k (en/fr/th) and WenetSpeech (Meeting/Net) under chunked and non-chunked encoder settings.
Fig. 5: Performance comparison of Zipper-LoRA-Soft + Initial-B and other LoRA-based methods on the 12 target languages in the SFT stage under the non-chunked setting, using (1−WER/CER)% as the metric. Detailed numerical results are provided in Table V .
Methods
ar
ja
ko
pt
Chunked
Vanilla-LoRA
62.32
48.93
38.37
33.49
FlyLoRA
65.20
49.75
39.02
33.58
Independent-LoRA
45.24
37.84
25.98
25.15
Zipper-LoRA-Static
42.36
37.51
25.65
23.26
Zipper-LoRA-Hard
43.07
37.10
25.80
23.86
TABLE VII: WER/CER (%) on low-resource Common Voice languages under chunked and non-chunked settings.
Fig. 6: Performance gap visualization from extremely-low to low-resource target languages. Bars show relative performance change vs. Vanilla-LoRA; positive values indicate improvement and negative values indicate degradation. The 1-hour setting uses 1 hour per language as in Table VII ; for the 30-hour setting, Arabic, Japanese, and Korean are from Common Voice (Korean uses 6.51 hours due to the limited official data size), while Portuguese is from MLS.
Chunk Size
Encoder Peak
Encoder Extra Peak
Extra Mem.
Relative
GPU Mem.
GPU Mem.
Reduction
Speed
Non-chunked
6.314 GB
4.947 GB
–
1.00 ×
8s chunks
4.382 GB
3.011 GB
39.14%
1.41 ×
4s chunks
4.010 GB
2.638 GB
46.68%
1.53 ×
2s chunks
3.645 GB
2.273 GB
54.05%
1.56 ×
1s chunks
3.580 GB
2.208 GB
55.36%
1.56 ×
TABLE VIII: Encoder efficiency under different chunk sizes.
Methods
Encoder
Encoder PEFT
PEFT Param.
Params.
Params.
Ratio
Vanilla-LoRA
746.48 M
23.60 M
3.16%
FlyLoRA
746.48 M
23.60 M
3.16%
Independent-LoRA
1006.02 M
283.13 M
28.14%
Zipper-LoRA-Static
811.38 M
88.47 M
10.90%
Zipper-LoRA-Hard
896.41 M
173.53 M
19.36%
TABLE IX: Encoder-side parameter comparison of PEFT methods.
Fig. 7: Cosine-similarity heatmap of Whisper language-ID (LID) embeddings for the 12 target languages.
Fig. 8: t-SNE visualization of encoder output representations across the 12 target languages.
Language
Vanilla-LoRA
Zipper-LoRA-Soft
Relative Reduction
ar
180.19
245.13
-36.04%
de
225.98
79.25
64.93%
en
209.74
68.11
67.53%
es
310.25
51.48
83.41%
fr
421.29
46.56
88.95%
it
90.13
182.22
-102.17%
TABLE X: Per-language covariance determinant values computed from the 2-D t-SNE coordinates. Lower values indicate more compact language-specific clusters.
We present a comprehensive benchmark of Federated Learning (FL) for multilingual Automatic Speech Recognition (ASR), evaluating four Speech-LLM architectures on the Multilingual LibriSpeech dataset. We compare FedAvg and FedProx across frozen and unfrozen encoder configurations, demonstrating that optimized learning rates are critical for performance. Specifically, independently tuning the learning rates for the speech encoder, connector, and decoder yields the lowest error rates, with full three-component adaptation (LoRA for encoder and decoder, full training for the connector) producing the best FL results. We observe that FedProx efficacy is architecture-dependent, providing notable advantages in multilingual pre-trained architectures (e.g., EuroLLM over TinyLlama when keeping the encoder fixed); this indicates that LLM backbone capacity plays a key role in mediating resilience to heterogeneous data distributions. These findings offer concrete design guidance for deploying multilingual Speech-LLMs in privacy-sensitive, distributed environments.
Jordi Luque, Aleix Sant, Fernando López
Telefónica Innovación Digital, Scientific Research
Adapting multilingual speech foundation models to low-resource languages remains difficult, especially for languages that are poorly represented during pre-training. While parameter-efficient fine-tuning (PEFT) reduces the cost of adapting large models, conventional approaches such as LoRA rely on generic low-rank parameterizations and do not explicitly use downstream task information to define the adaptation subspace. To investigate whether task-informed PEFT can better support low-resource ASR, we apply Fisher-Whitened Cross-Covariance Analysis (FCCA) to Whisper and Qwen3-ASR, and introduce two complementary extensions: Asymmetric-Coupled FCCA (AC-FCCA), which exploits structured cross-layer sharing, and Adaptive-Rank FCCA (AR-FCCA), which reallocates adaptation capacity across projection matrices under a fixed parameter budget. Under controlled multilingual experiments, we evaluate these approaches on languages that are poorly represented or unsupported during pre-training alongside well-represented languages. Standard FCCA is competitive with, and usually outperforms, trainable-parameter-budget-matched LoRA. AR-FCCA provides the most consistent improvement over standard FCCA across both model architectures, with statistically significant gains in several evaluation settings, while retaining the same number of trainable parameters. These results show that task-informed subspace construction can be effective for low-resource speech adaptation, and that adaptive rank allocation provides a robust way to improve parameter efficiency without increasing model capacity.
Asmee Mishra, Mengjie Qian, Brechtje Post +1
Department of Engineering, University of Cambridge, United Kingdom · Phonetics Laboratory, University of Cambridge, United Kingdom
Extending large-scale multilingual automatic speech recognition (ASR) models to low-resource languages remains challenging. Model performance is skewed toward high-resource languages and degrades sharply for languages with limited labeled data and pre-training exposure. To address this, we investigate parameter-efficient approaches for transferring knowledge from resource-rich source languages to low-resource target languages on Whisper. Alongside warm initialization and attention-based fusion, we propose Sequential Adapter Stacking, which places a trainable target-language adapter on top of a frozen source-language adapter. Under controlled experiments, these approaches are evaluated on three target languages unsupported by Whisper -- Asturian, Assamese, and Xhosa -- using source languages with varying degrees of relatedness. Sequential Adapter Stacking with the closest related source consistently and significantly outperforms full fine-tuning across the three targets, with 5--8% relative WER reductions. These gains largely persist with only one hour of target training data.
Thai Thi Thanh Thao Dang, Mengjie Qian, Kate Knill
Department of Engineering, University of Cambridge, UK