Text-Utilization for Encoder-dominated Speech Recognition Models

Albert Zeyer; Tim Posielek; Ralf Schlüter; Hermann Ney

cs.CLApr 29, 2026

Text-Utilization for Encoder-dominated Speech Recognition Models

Authors: Albert Zeyer, Tim Posielek, Ralf Schlüter, Hermann Ney

Organizations: 1Machine Learning and Human Language Technology Group, RWTH Aachen University, Germany · 2AppTek, Germany

Abstract

This paper investigates efficient methods for utilizing text-only data to improve speech recognition, focusing on encoder-dominated models that facilitate faster recognition. We provide a comprehensive comparison of techniques to integrate text-only data, including modality matching and dynamic downsampling to reach text-level representations within the encoder. Our experiments on the LibriSpeech corpus show that a larger encoder with a smaller decoder can equal or surpass the performance of architectures with larger decoders. We demonstrate that simple configurations, such as random duration models, are often more effective than complex alternatives, significantly simplifying the training pipeline. All code and recipes are made publicly available.

Explore similar work

WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformers
Sep 12, 2025Akshat Pandey, Karun Kumar, Raphael TangDomain Adaptation Variational Autoencoder
Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model
Jul 14, 2026Harsha Vardhan Khurdula, Abhinav Kumar Singh, Yoeven D Khemlani +1Speech Encoder Librispeech
Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition
Apr 23, 2026Srishti Ginjala, Eric Fosler-Lussier, Christopher W. Myers +1Large Language Model Decoder Prior Knowledge

Sep 12, 2025 · cs.CLJ/K move · Enter open · S save

WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformers

Akshat Pandey, Karun Kumar, Raphael Tang

Comcast · Speech AI · Washington, D.C., USA · University College London · Department of Computer Science

Pretrained automatic speech recognition (ASR) models such as Whisper perform well but still need domain adaptation to handle unseen parlance. In many real-world settings, collecting speech data is impractical, necessitating text-only adaptation. We propose WhisTLE, a deeply supervised, text-only adaptation method for pretrained encoder-decoder ASR models. WhisTLE trains a variational autoencoder (VAE) to model encoder outputs from text and fine-tunes the decoder using the learned text-to-latent encoder, optionally combined with text-to-speech (TTS) adaptation. At inference, the original encoder is restored, incurring no extra runtime cost. Across four datasets and four ASR models, WhisTLE alone helps in 28 of 32 (88%) cases, with an average relative WER drop of 22%. Using WhisTLE alone or in combination with other adaptation approaches helps in 116 of 144 (81%) cases, with an average additional WER drop of 15%. Given all treatments that improve on baseline scores, WhisTLE helps in 45 of 55 (82%) cases, with an average additional relative WER drop of 16%. WhisTLE with TTS reduces word error rate (WER) by a relative 49% and outperforms all non-WhisTLE baselines in 100 of 112 scenarios. We also find that WhisTLE additively complements any combination of other domain adaptation approaches; we thus recommend the inclusion of WhisTLE during standard processes for adapting encoder-decoder ASR models. Our code is at https://github.com/akshat0123/WhisTLE

Domain Adaptation Variational Autoencoder

Open paper Ask about it PDF

Abstract

Explore similar work

WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformers

Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model

Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition