Residual Expert Routing for Text-Only Domain Adaptation of Factorized Transducer ASR
Authors: Shashi Kumar, Esaú Villatoro-Tello, Thibault Bañeras-Roux, Sergio Burdisso, Petr Motlicek, Kadri Hacioglu, Andreas Stolcke
Organizations: Idiap Research Institute, Martigny, Switzerland · EPFL, Lausanne, Switzerland · Brno University of Technology, Czech Republic · Uniphore, U.S.A.
Fine-tuning the non-blank predictor of a factorized transducer on target-domain text improves recognition on that domain but degrades performance on the source domain. A single interpolation weight between base and adapted predictors cannot resolve this: the adapted predictor helps at some prefixes and hurts at others, often within one utterance. We propose conservative residual routing: the adapted predictor becomes a residual correction to the base, and a 16k-parameter gate decides at each prefix how much of it to admit. Closing the gate reproduces the base exactly. On a 40-hour conversational corpus, routing retains essentially all of the text-only gain on two held-out domains (19.89 to 16.74, 20.91 to 16.21 WER) while leaving source-domain WER unchanged (12.90 to 12.93); the adapted predictor alone costs 1.08 WER. Routing needs no per-domain tuning on target-domain audio. Analysis reveals the gate does not behave as a simple domain-level switch: it modulates intervention through the disagreement between the two predictors, suppressing it in-domain and amplifying it out-of-domain.
Figures & tables
Figure 1: Residual routing inside the non-blank branch of a factorized transducer. Only the shaded gate is trained; everything else is frozen. The blank branch is unchanged and omitted.
Trained
OOD
In-dom.
System
params
Retail
Telco
DefinedAI
Factorized transducer (base)
all
19.89
20.91
12.90
+ text-only adaptation (expert)
pred.
16.88
16.10
13.98
+ static interpolation ( γ=0.4 )
none
17.16
17.07
12.89
Residual routing (default)
gate
16.86
16.57
12.91
λsp=0.0
gate
16.87
16.53
12.95
Table 1: WER (%) on two out-of-domain (OOD) test sets and the in-domain test set. Static interpolation holds γu at a constant chosen by best combined development WER, which needs target-domain audio; routed systems train only the 16 k-parameter gate and are selected on target text. Best result in each column in bold.
States with D≥0.7
Flip rate
Test set
E[γ]
E[I]
Agree
decoder
fused
Retail (OOD)
0.65
0.55
0.73
0.234
0.040
Telco (OOD)
0.69
0.58
0.77
0.245
0.047
DefinedAI (in-dom.)
0.36
0.29
0.38
0.168
0.014
Table 2: Routing behaviour by test set, the first three columns restricted to high-disagreement decoder states ( D≥0.7 ). “Agree” is the fraction of those states where the mixed and expert distributions share an argmax; “Flip” is the fraction of emitted symbols differing from the base decision, at the decoder output and after acoustic fusion and blank competition.
Figure 2: Mean trust conditioned on base-expert disagreement, over all decoder states visited during decoding. The curves share a starting point, since the gate has no input to act on where the two predictors agree, and then separate: trust grows with disagreement out of domain and shrinks with it in domain. The gate reads the same signal as specialization on target-like data and as risk on source-like data.
Pretrained automatic speech recognition (ASR) models such as Whisper perform well but still need domain adaptation to handle unseen parlance. In many real-world settings, collecting speech data is impractical, necessitating text-only adaptation. We propose WhisTLE, a deeply supervised, text-only adaptation method for pretrained encoder-decoder ASR models. WhisTLE trains a variational autoencoder (VAE) to model encoder outputs from text and fine-tunes the decoder using the learned text-to-latent encoder, optionally combined with text-to-speech (TTS) adaptation. At inference, the original encoder is restored, incurring no extra runtime cost. Across four datasets and four ASR models, WhisTLE alone helps in 28 of 32 (88%) cases, with an average relative WER drop of 22%. Using WhisTLE alone or in combination with other adaptation approaches helps in 116 of 144 (81%) cases, with an average additional WER drop of 15%. Given all treatments that improve on baseline scores, WhisTLE helps in 45 of 55 (82%) cases, with an average additional relative WER drop of 16%. WhisTLE with TTS reduces word error rate (WER) by a relative 49% and outperforms all non-WhisTLE baselines in 100 of 112 scenarios. We also find that WhisTLE additively complements any combination of other domain adaptation approaches; we thus recommend the inclusion of WhisTLE during standard processes for adapting encoder-decoder ASR models. Our code is at https://github.com/akshat0123/WhisTLE
Adapting large language model (LLM)-based automatic speech recognition (ASR) systems to new domains using text-only data is a significant yet underexplored challenge. Standard fine-tuning of the LLM on the target domain text often disrupts the critical alignment between the speech and text modality learned by the projector, degrading performance. We introduce a novel text-only adaptation method that frames this process as a text denoising task. Our approach trains the LLM to recover clean transcripts from noisy inputs. This process effectively adapts the model to a target domain while preserving cross-modal alignment. Our solution is lightweight, requiring no architectural changes or additional parameters. Extensive evaluation on two datasets demonstrates up to 22.1% relative improvement, outperforming recent state-of-the-art text-only adaptation methods.
Conventional end-to-end automatic speech recognition (ASR) systems rely on paired speech-text data for domain adaptation. Recent LLM-based ASR architectures connect a speech encoder to a large language model via a projection module, enabling adaptation with text-only data. However, this introduces a modality gap, as the LLM is not exposed to the noisy representations produced by the speech projector. We investigate whether small amounts of speech can mitigate this mismatch. We compare three strategies: text-only adaptation, paired speech-text adaptation, and mixed batching (MB), which combines both. Experiments in in-domain and out-of-domain settings show that even limited speech consistently improves performance. Notably, MB using only 10% of the target-domain (less than 4 hours) speech achieves word error rates comparable to, or better than, conventional ASR fine-tuning with the full dataset, indicating that small amounts of speech provide a strong modality-alignment signal.