Model merging is a scalable alternative to multi-task training that combines the capabilities of multiple specialised models into a single model. This is particularly attractive for large speech foundation models, which are typically adapted through domain-specific fine-tuning, resulting in multiple customised checkpoints, for which repeating full fine-tuning when new data becomes available is computationally prohibitive. In this work, we study model merging for multi-domain ASR and benchmark 11 merging algorithms for 10 European Portuguese domains, evaluating in-domain accuracy, robustness under distribution shift, as well as English and multilingual performance. We further propose BoostedTSV-M, a new merging algorithm based on TSV-M that mitigates rank collapse via singular-value boosting and improves numerical stability. Overall, our approach outperforms full fine-tuning on European Portuguese while preserving out-of-distribution generalisation in a single model.
Figures & tables
Train
Test
Corpus
#hours
#speakers
#hours
#speakers
ID
Alert (AL) [ 40 ]
45.6
1,356
6.6
175
BD-Publico (BDP) [ 41 ]
21.8
100
2.0
10
Fala Bracarense (FP) [ 42 ]
66.1
75
6.1
8
Lectra (LE) [ 43 ]
22.0
7
2.6
7
MLS_extended (MLSE) [ 44 ]
54.8
12
1.0
10
Table 1: European Portuguese (EP) corpora used in this work: in-domain (ID) training data and ID/out-of-domain(OOD) evaluation sets, with hours of speech and number of speakers per split. OOD corpora are used for evaluation only (train shown as “–”).
Group
System
EP ID
EP OOD
EP Full Avg.
AAP
BP
OpenASR-HF
Fleurs
zero-shot
15.62
25.21
18.82
29.00
22.17
7.17
7.43
Full-FT
8.54
17.65
11.58
23.96
27.00
8.83
9.63
PS
SOUP w/o pt.
10.15
17.60
12.63
23.04
21.43
7.14
7.40
SOUP
11.25
17.97
13.65
23.88
21.55
7.14
6.98
Model Stock
15.02
24.49
18.18
28.45
22.27
7.20
6.79
Karcher Mean
10.23
17.63
12.70
22.88
20.99
7.12
7.43
Table 2: ASR error rates (%) on in-domain EP (ID) and out-of-distribution evaluation sets (EP, AAP and BP), as well as OpenASR-HF and FLEURS. We report WER for all datasets except FLEURS, where we report the macro-average error rate across the selected languages (WER for languages with explicit word boundaries; CER otherwise).
ID
OOD
System
AL
BDP
FB
LE
MLSE
MV
PP
PTA
PTE
SpD
Avg.
CV
PTC1
PTC2
PTF
VoxPT
Avg.
zero-shot
8.21
4.52
29.55
20.53
9.21
12.72
18.04
18.94
26.7
7.76
15.62
9.98
42.25
12.15
49.12
12.56
25.21
Full-FT
5.00
1.71
16.70
9.46
9.32
13.91
14.91
2.33
7.67
4.40
8.54
10.35
25.26
4.40
39.27
8.95
17.65
ID-FT
4.85
2.42
15.61
8.74
8.94
11.55
15.48
2.95
8.16
5.15
8.39
–
–
–
–
–
–
TSV-M w/ NS
5.36
3.01
18.84
10.02
9.37
14.97
14.11
2.97
9.76
5.65
9.41
9.85
20.64
3.98
37.61
8.29
16.07
BoostedTSV-M
5.17
2.58
18.78
9.88
9.41
15.41
13.93
2.71
9.42
5.42
9.27
9.97
20.73
4.00
37.46
8.4
16.11
Table 3: WER (%) per EP corpus on in-domain (ID) and out-of-distribution (OOD) test sets. “Avg.” is the mean across corpora within each split. For ID-FT, we report the score of the model fine-tuned on each corresponding ID corpus (one model per domain), which serves as an upper bound for what a single merged model could achieve on ID performance.
Figure 1: ID/OOD EP WER (%) for BoostedTSV-M as a function of β . TSV-M w/ NS performance is shown as horizontal lines.
Automatic speech recognition (ASR) systems, trained on paired speech-text data, have been improved by leveraging language models (LMs) trained on text-only data. LM fusion methods such as shallow fusion and density ratio are well-established methods that incorporate external LMs during ASR decoding. However, they incur additional computational costs due to LM inference, which is particularly problematic for recent larger LMs. In this study, we propose incorporating external LMs via model merging. This method integrates the LMs directly into the parameters of an LLM-based ASR model, requiring no additional computational cost at inference. We formulate domain extension and transfer via arithmetic operations on LoRA parameters. Experimental evaluations were conducted for the domain adaptation of LLM-based ASR trained on CSJ and LibriSpeech. We show that our LM merging consistently improved the ASR performance in the target domains, without degrading inference speed or memory footprint.
Hayato Futami, Tatsuya Kawahara
Graduate School of Informatics, Kyoto University, Japan
Leading multilingual speech recognition models like Whisper transcribe diverse, low-resource languages without language-specific training but are computationally expensive to deploy. Token merging mitigates this inefficiency by dynamically combining redundant features, shortening the sequence length during inference without requiring retraining. In this paper, we systematically evaluate token merging on the Whisper model family across sixteen diverse languages and three different model sizes. We also test how token merging interacts with fine-tuning (DoRA) on low-resource languages. Our findings show that merging tokens increases computational efficiency with almost no loss in transcription accuracy across most low-resource languages and model sizes, and it works even after the model has been fine-tuned. Our results demonstrate that token merging is a highly practical method for making multilingual speech recognition faster and cheaper to deploy.
Expanding multilingual ASR systems to new languages is challenging when adaptation must preserve previously supported languages without replaying source-language training data. We study this problem in connector-based Speech LLMs, where a trainable connector maps speech representations into a frozen large language model. We evaluate target-update construction and merging strategies for adapting the connector while preserving source-language performance. In single-target expansion, target-only fine-tuning and standard TIES merging learn new languages but cause severe source-language forgetting, while SSU-Wanda+TIES gives the best source-preserving trade-off. In multi-target expansion, update merging provides stable source retention, whereas recursive expansion can improve target adaptation under favorable orders but is highly order-sensitive. These results show that multilingual ASR expansion requires balancing target acquisition, source retention, and interactions among target-language updates.
Yuchen Zhang, Haralambos Mouratidis, Ravi Shekhar
Institute for Analytics and Data Science, University of Essex · School of Computer Science and Electronic Engineering, University of Essex