Whisper-AuT: Domain-Adapted Audio Encoder for Efficient Audio-LLM Training

Jielin Qiu; Ming Zhu; Wenting Zhao; Zhiwei Liu; Liangwei Yang; Zixiang Chen; Roshan Ram; Akshara Prabhakar

cs.SDApr 12, 2026

Whisper-AuT: Domain-Adapted Audio Encoder for Efficient Audio-LLM Training

Authors: Jielin Qiu, Ming Zhu, Wenting Zhao, Zhiwei Liu, Liangwei Yang, Zixiang Chen, Roshan Ram, Akshara Prabhakar,

Organizations: Salesforce AI Research

Abstract

Audio-native large language models (audio-LLMs) commonly use Whisper as their audio encoder. However, Whisper was trained exclusively on speech data, producing weak representations for music and environmental sound. This forces downstream audio-LLMs to compensate through extensive training on large-scale non-speech data. We present Whisper-AuT, a domain-adapted audio encoder obtained by fine-tuning Whisper-large-v3 on a curated mixture of speech (80%), environmental sound (10%), and music (10%) totaling approximately 20M samples. The full encoder-decoder is trained end-to-end with a seq2seq captioning objective; the decoder is then discarded and only the encoder is retained. Linear probe evaluations show that Whisper-AuT achieves +23.0% on ESC-50 (environmental sound), +5.0% on GTZAN (music genre), and +0.7% on Speech Commands (keyword spotting) compared to the original Whisperlarge-v3 encoder. Whisper-AuT is designed as a drop-in replacement for Whisper in audio-LLM architectures, with the goal of reducing downstream training cost by providing stronger initial audio representations for non-speech domains.

Figures & tables

Domain (%)	Datasets	$\sim$ Samples
Speech (80%)	GigaSpeech [ 2 ] , People’s Speech, Common Voice,	16M
	MLS English, LibriHeavy, and proprietary speech data	16M
Sound (10%)	AudioSet [ 6 ] and proprietary sound data	2M
Music (10%)	NSynth [ 5 ] , MusicBench [ 9 ] , MusicCaps,	2M
	and proprietary music data	2M
Total		$\sim$ 20M

Table 1: Training data composition. Non-speech captions are generated by commercial models and filtered to

\leq

448 tokens (Whisper’s decoder limit).

Benchmark	Whisper-large-v3	Whisper-AuT	$\Delta$
ESC-50 (sound)	54.50	77.50	+23.00
Speech Commands (speech)	87.60	88.32	+0.72
GTZAN (music)	81.00	86.00	+5.00

Table 2: Linear probe accuracy (%) comparing Whisper-AuT to the original Whisper-large-v3 encoder.

\Delta

shows absolute improvement.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Base model	openai/whisper-large-v3 (1.55B)
Hardware	8 $\times$ NVIDIA H200 (143GB)
Precision	bfloat16
Distributed strategy	DeepSpeed ZeRO-2 [ 15 ]
Optimizer	AdamW [ 8 ] ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.999$ )
Learning rate	$1\times 10^{-5}$ (cosine decay)

Appendix

Table 3: Training hyperparameters.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 207 papers

USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding
Jun 4, 2026Heng-Jui Chang, Alexander H. Liu, Saurabhchand Bhati +4Audio Representation Learning Audio Understanding
MIT CSAIL, USA · Amazon, USA
Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs
Sep 16, 2026Mohan Shi, Zilai Wang, Natarajan Balaji Shankar +3Domain Adaptation for ASR Adapter Tuning
Department of Electrical and Computer Engineering University of California Los Angeles Los Angeles, USA
UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation
Feb 25, 2026Yuxuan Chen, Peize He, Haoyuan Yu +1Audio Representation Learning Audio Understanding
Jilin University · University of Electronic Science and Technology of China · Hunan University +1

Jun 4, 2026J/K move · Enter open · S save

USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding

Heng-Jui Chang, Alexander H. Liu, Saurabhchand Bhati, Mrudula Athi+3

MIT CSAIL, USA · Amazon, USA

Audio Representation Learning Audio Understanding Knowledge Distillation

Open paper AskPDF arXiv

Audio encoders are critical to modern audio applications as large language models (LLMs) increasingly rely on a single encoder for diverse inputs. While self-supervised learning (SSL) has yielded strong domain-specific encoders like speech or music experts, multi-domain approaches like USAD and SPEAR remain limited in coverage and evaluation. Recent studies also suggest supervised encoders align better with audio LLMs. We present USAD 2.0, a universal encoder integrating knowledge from both SSL and supervised foundation models. USAD 2.0 introduces domain-aware distillation to address teacher mismatch, extends coverage to the music domain, and adds second-stage supervised distillation for downstream use. We further scale the model to one billion parameters via depth scaling. Experiments show USAD 2.0 achieves strong or state-of-the-art performance across probing and LLM-based evaluations.

Abstract

Audio Representation Learning

Explore similar work

USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation