Residual Expert Routing for Text-Only Domain Adaptation of Factorized Transducer ASR

Shashi Kumar; Esaú Villatoro-Tello; Thibault Bañeras-Roux; Sergio Burdisso; Petr Motlicek; Kadri Hacioglu; Andreas Stolcke

cs.CLOct 23, 2026

Residual Expert Routing for Text-Only Domain Adaptation of Factorized Transducer ASR

Authors: Shashi Kumar, Esaú Villatoro-Tello, Thibault Bañeras-Roux, Sergio Burdisso, Petr Motlicek, Kadri Hacioglu, Andreas Stolcke

Organizations: Idiap Research Institute, Martigny, Switzerland · EPFL, Lausanne, Switzerland · Brno University of Technology, Czech Republic · Uniphore, U.S.A.

Abstract

Fine-tuning the non-blank predictor of a factorized transducer on target-domain text improves recognition on that domain but degrades performance on the source domain. A single interpolation weight between base and adapted predictors cannot resolve this: the adapted predictor helps at some prefixes and hurts at others, often within one utterance. We propose conservative residual routing: the adapted predictor becomes a residual correction to the base, and a 16k-parameter gate decides at each prefix how much of it to admit. Closing the gate reproduces the base exactly. On a 40-hour conversational corpus, routing retains essentially all of the text-only gain on two held-out domains (19.89 to 16.74, 20.91 to 16.21 WER) while leaving source-domain WER unchanged (12.90 to 12.93); the adapted predictor alone costs 1.08 WER. Routing needs no per-domain tuning on target-domain audio. Analysis reveals the gate does not behave as a simple domain-level switch: it modulates intervention through the disagreement between the two predictors, suppressing it in-domain and amplifying it out-of-domain.

Figures & tables

Panel of Figure 1 — Figure 1: Residual routing inside the non-blank branch of a factorized transducer. Only the shaded gate is trained; everything else is frozen. The blank branch is unchanged and omitted.

	Trained	OOD		In-dom.
System	params	Retail	Telco	DefinedAI
Factorized transducer (base)	all	19.89	20.91	12.90
+ text-only adaptation (expert)	pred.	16.88	16.10	13.98
+ static interpolation ( $\gamma\!=\!0.4$ )	none	17.16	17.07	12.89
Residual routing (default)	gate	16.86	16.57	12.91
$\lambda_{sp}=0.0$	gate	16.87	16.53	12.95

Table 1: WER (%) on two out-of-domain (OOD) test sets and the in-domain test set. Static interpolation holds

\gamma_{u}

at a constant chosen by best combined development WER, which needs target-domain audio; routed systems train only the

16

k-parameter gate and are selected on target text. Best result in each column in bold.

	States with $D\geq 0.7$			Flip rate
Test set	$\mathbb{E}[\gamma]$	$\mathbb{E}[I]$	Agree	decoder	fused
Retail (OOD)	0.65	0.55	0.73	0.234	0.040
Telco (OOD)	0.69	0.58	0.77	0.245	0.047
DefinedAI (in-dom.)	0.36	0.29	0.38	0.168	0.014

Table 2: Routing behaviour by test set, the first three columns restricted to high-disagreement decoder states (

D\!\geq\!0.7

). “Agree” is the fraction of those states where the mixed and expert distributions share an argmax; “Flip” is the fraction of emitted symbols differing from the base decision, at the decoder output and after acoustic fusion and blank competition.

Figure 2: Mean trust conditioned on base-expert disagreement, over all decoder states visited during decoding. The curves share a starting point, since the gate has no input to act on where the two predictors agree, and then separate: trust grows with disagreement out of domain and shrinks with it in domain. The gate reads the same signal as specialization on target-like data and as risk on source-like data.

Domain Adaptation for ASR

Domain adaptation for automatic speech recognition (ASR) tailors speech models to new speakers, domains, languages, and acoustic conditions.

See all 51 papers

WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformers
Sep 12, 2025Akshat Pandey, Karun Kumar, Raphael TangDomain Adaptation for ASR Domain Adaptation
Comcast · Speech AI · Washington, D.C., USA +3
Text-only adaptation in LLM-based ASR through text denoising
Jan 28, 2026Andrés Carofilis, Sergio Burdisso, Esaú Villatoro-Tello +8Domain Adaptation for ASR Domain Adaptation
Idiap Research Institute · EPFL, Lausanne · Uniphore +2
Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR
Apr 7, 2026Thibault Bañeras-Roux, Sergio Burdisso, Esaú Villatoro-Tello +9Domain Adaptation for ASR Cross-Modal Alignment
Idiap Research Institute, Switzerland · Laboratoire d’Informatique et des Syst`emes, France · EPFL, Switzerland +2

Sep 12, 2025J/K move · Enter open · S save

WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformers

Akshat Pandey, Karun Kumar, Raphael Tang

Comcast · Speech AI · Washington, D.C., USA+3

Domain Adaptation for ASR Domain Adaptation Automatic Speech Recognition

Open paper AskPDF arXiv

Pretrained automatic speech recognition (ASR) models such as Whisper perform well but still need domain adaptation to handle unseen parlance. In many real-world settings, collecting speech data is impractical, necessitating text-only adaptation. We propose WhisTLE, a deeply supervised, text-only adaptation method for pretrained encoder-decoder ASR models. WhisTLE trains a variational autoencoder (VAE) to model encoder outputs from text and fine-tunes the decoder using the learned text-to-latent encoder, optionally combined with text-to-speech (TTS) adaptation. At inference, the original encoder is restored, incurring no extra runtime cost. Across four datasets and four ASR models, WhisTLE alone helps in 28 of 32 (88%) cases, with an average relative WER drop of 22%. Using WhisTLE alone or in combination with other adaptation approaches helps in 116 of 144 (81%) cases, with an average additional WER drop of 15%. Given all treatments that improve on baseline scores, WhisTLE helps in 45 of 55 (82%) cases, with an average additional relative WER drop of 16%. WhisTLE with TTS reduces word error rate (WER) by a relative 49% and outperforms all non-WhisTLE baselines in 100 of 112 scenarios. We also find that WhisTLE additively complements any combination of other domain adaptation approaches; we thus recommend the inclusion of WhisTLE during standard processes for adapting encoder-decoder ASR models. Our code is at https://github.com/akshat0123/WhisTLE

Abstract

Domain Adaptation for ASR

Explore similar work

WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformers

Text-only adaptation in LLM-based ASR through text denoising

Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR