UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation

Yuxuan Chen; Peize He; Haoyuan Yu; Junzi Zhang

cs.SDFeb 25, 2026

UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation

Authors: Yuxuan Chen, Peize He, Haoyuan Yu, Junzi Zhang

Organizations: Jilin University · University of Electronic Science and Technology of China · Hunan University · Shandong University

Abstract

A universal audio representation should capture fine-grained speech cues and high-level semantics for environmental sounds and music in a single encoder. Existing encoders often excel in one domain but degrade in others. We propose UniWhisper, an efficient continual multi-task training framework that casts heterogeneous audio tasks into a unified instruction and answer format. This enables standard next-token training without task-specific heads and losses. We train it on 38k hours of public audio and assess the encoder using shallow MLP probes and k-nearest neighbors (kNN) on 20 tasks spanning speech, environmental sound, and music. UniWhisper reaches normalized weighted averages of 0.81 with MLP probes and 0.61 with kNN, compared to 0.64 and 0.46 for Whisper, while retaining strong speech performance.

Figures & tables

Figure 1: Normalized per-task performance of UniWhisper on our 20-task extended HEAR [ 14 ] spanning speech, environmental sound, and music. Full results are reported in Table 3 .

Type	Name	Task	Duration
General	AudioCaps [ 16 ]	Captioning	142.5 h
	AudioSet [ 17 ]	Tagging	5.8k h
	LAION-Audio [ 18 ]	Captioning	4.3k h
	WavCaps [ 19 ]	Captioning	7.6k h
Speech	AISHELL-1 [ 20 ]	ASR & SID	178 h
	GigaSpeech [ 21 ]	ASR	10k h

Table 1: Public datasets used for continual multi-task training, grouped by domain. Duration denotes the total audio hours used after filtering and deduplication.

Type	Task	Metric	Weight
Type	Task	Metric	MLP	kNN
Speech	ASV2015	Acc	2000	2000
	CREMA-D	Acc	1116	1116
	Fluent Speech Commands	Acc	2000	2000
	LibriCount	Acc	1144	1144
	LibriSpeech-100h	iWER	10000	-

Table 2: Evaluation tasks, metrics, and weights used to compute normalized weighted averages. Weights are proportional to the number of test examples.

Domain	Task	wav2vec 2.0		HuBERT		WavLM		BEATs		CLAP		Whisper		CLAP-U		Whisper-U		Whisper-U-3		Ours
Domain	Task	MLP	kNN	MLP	kNN	MLP	kNN	MLP	kNN	MLP	kNN	MLP	kNN	MLP	kNN	MLP	kNN	MLP	kNN	MLP	kNN
Speech	ASV2015	$94.0$	$\mathbf{95.8}$	$95.1$	$88.5$	$97.4$	$94.6$	$91.5$	$81.0$	$95.1$	$71.2$	$97.5$	$90.3$	$92.3$	$82.9$	$98.3$	$95.0$	$94.7$	$90.6$	$\mathbf{99.0}$	$92.9$
	CREMA-D	$55.5$	$21.8$	$64.5$	$34.0$	$71.3$	$28.0$	$66.5$	$39.7$	$36.9$	$25.4$	$57.5$	$38.2$	$54.3$	$49.0$	$64.3$	$17.8$	$66.3$	$30.0$	$\mathbf{84.4}$	$\mathbf{67.9}$
	FSC	$47.2$	$1.6$	$96.5$	$3.6$	$97.5$	$4.3$	$47.3$	$5.5$	$3.6$	$1.6$	$97.8$	$23.4$	$17.7$	$5.4$	$97.2$	$1.0$	$\mathbf{98.0}$	$\mathbf{53.1}$	$82.7$	$14.6$
	LibriCount	$58.0$	$20.8$	$58.2$	$18.2$	$65.1$	$38.8$	$\mathbf{68.2}$	$37.8$	$38.8$	$28.1$	$59.4$	$45.2$	$60.9$	$\mathbf{54.0}$	$62.8$	$9.1$	$53.7$	$41.4$	$64.4$	$47.2$
	LS-100h	$16.0$	–	$83.4$	–	$64.1$	–	$15.8$	–	$0.2$	–	$62.5$	–	$1.5$	–	$79.0$	–	$89.9$	–	$\mathbf{91.6}$	–

Table 3: Per-task normalized results for MLP and kNN on 20 tasks. Scores are normalized to

[0,1]

using task-specific bounds so higher is better and reported as percentages (×100). CLAP-U replaces the Whisper encoder with CLAP-HASAT under the same training recipe. Whisper-U and Whisper-U-3 use the original Whisper Large-v3 decoder and train with one pass and three-pass replay over the same data, respectively. Highest score is in bold .

Task	Whisper		UniWhisper		CLAP-Uni		Whisper-Uni-1		Whisper-Uni-3
Task	MLP	kNN	MLP	kNN	MLP	kNN	MLP	kNN	MLP	kNN
ASV2015	$\scriptstyle\pm$ 1.7	$\scriptstyle\pm$ 1.4	$\scriptstyle\pm$ 2.0	$\scriptstyle\pm$ 1.8	$\scriptstyle\pm$ 1.1	$\scriptstyle\pm$ 1.3	$\scriptstyle\pm$ 2.5	$\scriptstyle\pm$ 2.4	$\scriptstyle\pm$ 1.4	$\scriptstyle\pm$ 1.3
CREMA-D	$\scriptstyle\pm$ 0.9	$\scriptstyle\pm$ 0.8	$\scriptstyle\pm$ 1.4	$\scriptstyle\pm$ 1.2	$\scriptstyle\pm$ 0.9	$\scriptstyle\pm$ 1.1	$\scriptstyle\pm$ 1.7	$\scriptstyle\pm$ 0.3	$\scriptstyle\pm$ 1.3	$\scriptstyle\pm$ 0.4
FSC	$\scriptstyle\pm$ 1.8	$\scriptstyle\pm$ 0.4	$\scriptstyle\pm$ 1.6	$\scriptstyle\pm$ 0.2	$\scriptstyle\pm$ 0.4	$\scriptstyle\pm$ 0.1	$\scriptstyle\pm$ 1.8	$\scriptstyle\pm$ 0.3	$\scriptstyle\pm$ 2.5	$\scriptstyle\pm$ 0.7
LibriCount	$\scriptstyle\pm$ 0.9	$\scriptstyle\pm$ 0.8	$\scriptstyle\pm$ 1.0	$\scriptstyle\pm$ 0.9	$\scriptstyle\pm$ 1.5	$\scriptstyle\pm$ 0.6	$\scriptstyle\pm$ 1.6	$\scriptstyle\pm$ 0.2	$\scriptstyle\pm$ 1.2	$\scriptstyle\pm$ 0.8
LS-100h	$\scriptstyle\pm$ 1.0	–	$\scriptstyle\pm$ 1.8	–	$\scriptstyle\pm$ 0.3	–	$\scriptstyle\pm$ 1.9	–	$\scriptstyle\pm$ 1.8	–

Table 4: Reported deviations correspond to the maximum absolute difference between the mean and the endpoints of the 95% percentile CI, estimated using hierarchical bootstrap resampling over seeds and evaluation examples with replacement.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 207 papers

Whisper-AuT: Domain-Adapted Audio Encoder for Efficient Audio-LLM Training
Apr 12, 2026Jielin Qiu, Ming Zhu, Wenting Zhao +11Audio Representation Learning Domain Adaptation
Salesforce AI Research
USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding
Jun 4, 2026Heng-Jui Chang, Alexander H. Liu, Saurabhchand Bhati +4Audio Representation Learning Audio Understanding
MIT CSAIL, USA · Amazon, USA
UniAE-MoE: A Unified Audio Encoder via Mixture of Experts
Sep 30, 2026Shengbo Cai, Zhisheng Zhang, Zichao Nie +3Audio Representation Learning Mixture of Experts
Tsinghua University, China

Apr 12, 2026J/K move · Enter open · S save

Whisper-AuT: Domain-Adapted Audio Encoder for Efficient Audio-LLM Training

Jielin Qiu, Ming Zhu, Wenting Zhao, Zhiwei Liu+10

Salesforce AI Research

Audio Representation Learning Domain Adaptation Audio-Language Models

Open paper AskPDF arXiv

Audio-native large language models (audio-LLMs) commonly use Whisper as their audio encoder. However, Whisper was trained exclusively on speech data, producing weak representations for music and environmental sound. This forces downstream audio-LLMs to compensate through extensive training on large-scale non-speech data. We present Whisper-AuT, a domain-adapted audio encoder obtained by fine-tuning Whisper-large-v3 on a curated mixture of speech (80%), environmental sound (10%), and music (10%) totaling approximately 20M samples. The full encoder-decoder is trained end-to-end with a seq2seq captioning objective; the decoder is then discarded and only the encoder is retained. Linear probe evaluations show that Whisper-AuT achieves +23.0% on ESC-50 (environmental sound), +5.0% on GTZAN (music genre), and +0.7% on Speech Commands (keyword spotting) compared to the original Whisperlarge-v3 encoder. Whisper-AuT is designed as a drop-in replacement for Whisper in audio-LLM architectures, with the goal of reducing downstream training cost by providing stronger initial audio representations for non-speech domains.

Abstract

Audio Representation Learning

Explore similar work

Whisper-AuT: Domain-Adapted Audio Encoder for Efficient Audio-LLM Training

USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding

UniAE-MoE: A Unified Audio Encoder via Mixture of Experts