Preserve to Expand: Source-Aware Adaptation and Model Merging for Multilingual LLM-Based ASR

Yuchen Zhang; Haralambos Mouratidis; Ravi Shekhar

cs.CLOct 26, 2026

Preserve to Expand: Source-Aware Adaptation and Model Merging for Multilingual LLM-Based ASR

Authors: Yuchen Zhang, Haralambos Mouratidis, Ravi Shekhar

Organizations: Institute for Analytics and Data Science, University of Essex · School of Computer Science and Electronic Engineering, University of Essex

Abstract

Expanding multilingual ASR systems to new languages is challenging when adaptation must preserve previously supported languages without replaying source-language training data. We study this problem in connector-based Speech LLMs, where a trainable connector maps speech representations into a frozen large language model. We evaluate target-update construction and merging strategies for adapting the connector while preserving source-language performance. In single-target expansion, target-only fine-tuning and standard TIES merging learn new languages but cause severe source-language forgetting, while SSU-Wanda+TIES gives the best source-preserving trade-off. In multi-target expansion, update merging provides stable source retention, whereas recursive expansion can improve target adaptation under favorable orders but is highly order-sensitive. These results show that multilingual ASR expansion requires balancing target acquisition, source retention, and interactions among target-language updates.

Figures & tables

Figure 1: Overview of source-aware multilingual ASR expansion.

Group	Languages
Germanic (GEM)	English (EN), German (DE), Dutch (NL), Danish (DA), Swedish (SV)
Romance (ROA)	French (FR), Spanish (ES), Portuguese (PT), Galician (GL), Catalan (CA)
Slavic (SLA)	Russian (RU), Ukrainian (UK), Belarusian (BE), Bulgarian (BG), Macedonian (MK)
Targets	Norwegian (NO), Italian (IT), Serbian (SR), Polish (PL), Slovenian (SL), Kyrgyz (KY)

Table 1: Source-language groups and target languages.

Type	Base	Target
G+/S+	Germanic	Norwegian (NO)
	Romance	Italian (IT)
	Slavic	Serbian (SR)
G+/S-	Slavic	Polish (PL)
G+/S-	Slavic	Slovenian (SL)
G-/S+	Germanic	Italian (IT)

Table 2: Single-target base–target configurations.G+/G- indicate language-group match/mismatch, and S+/S- indicate script match/mismatch.

Base	Method	Source	Target	All	$\boldsymbol{\Delta}$ Source
GEM	Base	22.57	102.70	35.93	+0.00
	Target FT	69.73	27.86	62.75	+47.15
	SSU-Wanda+FT	67.46	28.47	60.96	+44.89
	TIES	69.08	27.71	62.18	+46.50
	SSU-Rand+TIES	29.42	30.30	29.56	+6.84
	SSU-Mag+TIES	40.83	29.52	38.95	+18.26

Table 3: Macro-averaged WER (%)

\downarrow

for single-target expansion by base.

Target	Type	Method	Source	Target	All	$\boldsymbol{\Delta}$ Source
IT	G-/S-	Base	36.93	116.08	50.12	+0.00
		Target FT	129.79	20.11	111.51	+92.86
		SSU-Wanda+FT	115.24	23.36	99.93	+78.31
		TIES	128.32	21.07	110.45	+91.40
		SSU-Rand+TIES	52.82	39.52	50.60	+15.89
		SSU-Mag+TIES	82.62	34.28	74.56	+45.70

Table 4: Detailed single-target expansion WER (%)

\downarrow

for the Slavic base.

Base	Method	Source	Target	All
GEM	Base	22.57	102.70	52.62
	TIES	22.11	64.93	38.17
	Hard SSU+TIES	22.02	86.76	46.30
	Incremental dynamic SSU	21.64	84.26	45.12
	Recursive rebased SSU	37.13	45.39	40.23
	Recursive FT then SSU	28.97	57.34	39.61

Table 5: Final-stage WER (%) for multi-target expansion. Lower is better.

Base	Target Order	Source	Target	All
GEM	NO $\rightarrow$ IT $\rightarrow$ SR	28.97	57.34	39.61
	IT $\rightarrow$ NO $\rightarrow$ SR	36.03	62.12	45.81
	SR $\rightarrow$ IT $\rightarrow$ NO	24.30	53.74	35.34
	IT $\rightarrow$ SR $\rightarrow$ NO	25.05	51.72	35.05
	NO $\rightarrow$ SR $\rightarrow$ IT	28.00	48.72	35.77
ROA	NO $\rightarrow$ IT $\rightarrow$ SR	50.37	70.21	57.81

Table 6: Final-stage WER (%)

\downarrow

for Recursive FT then SSU under different orders.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Original source languages					Target	Source retention
Method	DA	NL	EN	DE	SV	Target $\downarrow$	Source $\downarrow$	$\Delta$ Source $\downarrow$
Added language: Norwegian
Base (zero-shot target)	27.76	19.37	19.17	23.61	22.96	62.60	22.57	+0.00
Target FT	44.52	19.99	17.48	24.26	62.70	33.12	33.79	+11.21
SSU-Wanda + FT (50%)	39.13	19.83	19.11	24.05	57.41	35.46	31.91	+9.34
TIES (no protection)	45.08	19.94	17.72	24.26	61.88	33.08	33.78	+11.20

Appendix

Table 7: Germanic single-target expansion results. Values are WER (%).

	Original source languages					Target	Source retention
Method	CA	FR	GL	PT	ES	Target $\downarrow$	Source $\downarrow$	$\Delta$ Source $\downarrow$
Added language: Norwegian
Base (zero-shot target)	26.11	20.20	22.69	13.28	12.48	124.03	18.95	+0.00
Target FT	59.24	61.23	38.67	73.86	72.41	36.86	61.08	+42.13
SSU-Wanda + FT (50%)	31.41	32.66	26.57	33.10	24.28	41.04	29.61	+10.66
TIES (no protection)	55.88	58.52	36.78	71.82	70.67	36.41	58.73	+39.78

Appendix

Table 8: Romance single-target expansion results. Values are WER (%).

	Original source languages					Target	Source retention
Method	RU	UK	BE	BG	MK	Target $\downarrow$	Source $\downarrow$	$\Delta$ Source $\downarrow$
Added language: Italian
Base (zero-shot target)	27.79	52.88	37.52	31.48	34.97	116.08	36.93	+0.00
Target FT	131.42	127.99	159.59	115.63	114.32	20.11	129.79	+92.86
SSU-Wanda + FT (50%)	108.31	113.37	133.94	111.24	109.35	23.36	115.24	+78.31
TIES (no protection)	128.18	126.42	157.57	115.33	114.13	21.07	128.32	+91.40

Appendix

Table 9: Slavic single-target expansion results. Values are WER (%).

Method	Stage	Accepted targets	DA $\downarrow$	NL $\downarrow$	EN $\downarrow$	DE $\downarrow$	SV $\downarrow$	Source mean $\downarrow$	NO $\downarrow$	IT $\downarrow$	SR $\downarrow$	Target mean $\downarrow$	All $\downarrow$
Base	0	Zero-shot	27.76	19.37	19.17	23.61	22.96	22.57	62.60	109.69	135.82	102.70	52.62
Order: NO $\rightarrow$ IT $\rightarrow$ SR (configured)
TIES (joint)	1	NO	45.08	19.94	17.72	24.26	61.88	33.78	33.08	–	–	33.08	33.66
TIES (joint)	2	NO $\rightarrow$ IT	27.88	18.84	17.46	23.48	24.44	22.42	36.99	40.15	–	38.57	27.03
TIES (joint)	3	NO $\rightarrow$ IT $\rightarrow$ SR	27.91	18.52	16.83	22.25	25.03	22.11	36.41	40.65	117.72	64.93	38.17
Hard SSU+TIES	1	NO	28.09	19.07	19.16	22.86	26.30	23.09	37.13	–	–	37.13	25.43

Appendix

Table 10: Germanic multi-target expansion results under evaluated target orders. Values are WER (%).

Method	Stage	Accepted targets	CA $\downarrow$	FR $\downarrow$	GL $\downarrow$	PT $\downarrow$	ES $\downarrow$	Source mean $\downarrow$	NO $\downarrow$	IT $\downarrow$	SR $\downarrow$	Target mean $\downarrow$	All $\downarrow$
Base	0	Zero-shot	26.11	20.20	22.69	13.28	12.48	18.95	124.03	83.57	125.88	111.16	53.53
Order: NO $\rightarrow$ IT $\rightarrow$ SR (configured)
TIES (joint)	1	NO	55.88	58.52	36.78	71.82	70.67	58.73	36.41	–	–	36.41	55.01
TIES (joint)	2	NO $\rightarrow$ IT	24.56	19.58	22.43	13.91	17.38	19.57	71.83	18.17	–	45.00	26.83
TIES (joint)	3	NO $\rightarrow$ IT $\rightarrow$ SR	24.21	19.55	22.53	14.59	17.04	19.58	68.00	19.13	114.52	67.22	37.45
Hard SSU+TIES	1	NO	24.41	19.78	22.42	12.73	12.18	18.30	46.94	–	–	46.94	23.08

Appendix

Table 11: Romance multi-target expansion results under evaluated target orders. Values are WER (%).

Method	Stage	Accepted targets	RU $\downarrow$	UK $\downarrow$	BE $\downarrow$	BG $\downarrow$	MK $\downarrow$	Source mean $\downarrow$	IT $\downarrow$	SR $\downarrow$	PL $\downarrow$	SL $\downarrow$	KY $\downarrow$	Target mean $\downarrow$	All $\downarrow$
Base	0	Zero-shot	27.79	52.88	37.52	31.48	34.97	36.93	116.08	66.58	107.36	110.70	150.01	110.15	73.54
Order: IT $\rightarrow$ SR $\rightarrow$ PL $\rightarrow$ SL $\rightarrow$ KY (configured)
TIES (joint)	1	IT	128.18	126.42	157.57	115.33	114.13	128.32	21.07	–	–	–	–	21.07	110.45
TIES (joint)	2	IT $\rightarrow$ SR	23.17	34.90	39.05	34.31	35.79	33.44	85.05	43.92	–	–	–	64.49	42.31
TIES (joint)	3	IT $\rightarrow$ SR $\rightarrow$ PL	34.41	68.00	47.44	35.09	41.76	45.34	71.05	69.67	42.62	–	–	61.11	51.26
TIES (joint)	4	IT $\rightarrow$ SR $\rightarrow$ PL $\rightarrow$ SL	24.78	40.90	38.96	33.32	38.27	35.25	99.18	60.48	90.83	95.93	–	86.60	58.07

Appendix

Table 12: Slavic multi-target expansion results under evaluated target orders. Values are WER (%).

Romance
Method	Stage	Accepted targets	CA $\downarrow$	FR $\downarrow$	GL $\downarrow$	PT $\downarrow$	ES $\downarrow$	Source mean $\downarrow$	NO $\downarrow$	IT $\downarrow$	SR $\downarrow$	Target mean $\downarrow$	All $\downarrow$
Base	0	Zero-shot	26.11	20.20	22.69	13.28	12.48	18.95	124.03	83.57	125.88	111.16	53.53
Easy-to-hard: NO $\rightarrow$ SR $\rightarrow$ IT
Recursive FT then SSU	1	NO	24.21	19.50	22.59	13.05	12.29	18.33	46.80	–	–	46.80	23.07
Recursive FT then SSU	2	NO $\rightarrow$ SR	42.22	48.04	52.57	88.80	65.94	59.52	108.24	–	32.83	70.53	62.66
Recursive FT then SSU	3	NO $\rightarrow$ SR $\rightarrow$ IT	26.58	21.12	27.72	19.33	24.37	23.83	72.73	17.00	73.38	54.37	35.28

Appendix

Table 13: Stage-by-stage Recursive FT then SSU results under additional target orders. Values are WER (%).

Base	Target A	Target B	Cosine	Sign Disagreement	Support Jaccard
Germanic	Norwegian	Italian	0.053	0.473	0.667
Germanic	Norwegian	Serbian	0.037	0.481	0.667
Germanic	Italian	Serbian	0.047	0.475	0.667
Romance	Norwegian	Italian	0.064	0.466	0.668
Romance	Norwegian	Serbian	0.062	0.468	0.667
Romance	Italian	Serbian	0.048	0.475	0.667

Appendix

Table 14: Pairwise target-update conflict after TIES trimming.

Multilingual ASR

Multilingual automatic speech recognition (ASR) transcribes speech across languages, dialects, scripts, and code-switching settings.

See all 108 papers

Sequential Adapter Stacking for Cross-Lingual Low-Resource ASR
Sep 14, 2026Thai Thi Thanh Thao Dang, Mengjie Qian, Kate KnillLow-Resource Speech Recognition Parameter-Efficient Fine-Tuning
Department of Engineering, University of Cambridge, UK
Merging the Knowledge of LLMs for Automatic Speech Recognition
Sep 14, 2026Hayato Futami, Tatsuya KawaharaDomain Adaptation for ASR Automatic Speech Recognition
Graduate School of Informatics, Kyoto University, Japan
Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling
Jun 9, 2026Guodong Lin, Ziqi Chen, Yuxiang Fu +2Audio-Language Models Speech Language Models
Department of Electronic Engineering, Tsinghua University, Beijing, China · Beijing Haitian Ruisheng Science Technology Ltd., China

Sep 14, 2026J/K move · Enter open · S save

Sequential Adapter Stacking for Cross-Lingual Low-Resource ASR

Thai Thi Thanh Thao Dang, Mengjie Qian, Kate Knill

Department of Engineering, University of Cambridge, UK

Low-Resource Speech Recognition Parameter-Efficient Fine-Tuning Cross-Lingual Transfer Multilingual ASR

Open paper AskPDF arXiv

Extending large-scale multilingual automatic speech recognition (ASR) models to low-resource languages remains challenging. Model performance is skewed toward high-resource languages and degrades sharply for languages with limited labeled data and pre-training exposure. To address this, we investigate parameter-efficient approaches for transferring knowledge from resource-rich source languages to low-resource target languages on Whisper. Alongside warm initialization and attention-based fusion, we propose Sequential Adapter Stacking, which places a trainable target-language adapter on top of a frozen source-language adapter. Under controlled experiments, these approaches are evaluated on three target languages unsupported by Whisper -- Asturian, Assamese, and Xhosa -- using source languages with varying degrees of relatedness. Sequential Adapter Stacking with the closest related source consistently and significantly outperforms full fine-tuning across the three targets, with 5--8% relative WER reductions. These gains largely persist with only one hour of target training data.

Abstract

Multilingual ASR

Explore similar work

Sequential Adapter Stacking for Cross-Lingual Low-Resource ASR

Merging the Knowledge of LLMs for Automatic Speech Recognition

Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling