DashengTokenizer: One layer is enough for unified audio understanding and generation

Heinrich Dinkel; Xingwei Sun; Gang Li; Jiahao Mei; Yadong Niu; Jizhong Liu; Xiyang Li; Yifan Liao

cs.SDFeb 27, 2026

DashengTokenizer: One layer is enough for unified audio understanding and generation

Authors: Heinrich Dinkel, Xingwei Sun, Gang Li, Jiahao Mei, Yadong Niu, Jizhong Liu, Xiyang Li, Yifan Liao,

Organizations: MiLM Plus, Xiaomi Inc., Beijing, China

Abstract

This paper introduces DashengTokenizer, a continuous audio tokenizer engineered for joint use in both understanding and generation tasks. Unlike conventional approaches, which train acoustic tokenizers and subsequently integrate frozen semantic knowledge, our method inverts this paradigm: we leverage frozen semantic features and inject acoustic information. In linear evaluation across 22 diverse tasks, our method outperforms previous audio codec and audio encoder baselines by a significant margin while maintaining competitive audio reconstruction quality. Notably, we demonstrate that this acoustic injection improves performance for tasks such as speech emotion recognition, music understanding, and acoustic scene classification. We further evaluate the tokenizer's generative performance on text-to-audio (TTA), text-to-music (TTM), and speech enhancement (SE). Our approach surpasses standard variational autoencoder (VAE)-based methods on TTA and TTM tasks, while its effectiveness on SE underscores its capabilities as a general-purpose audio encoder. Finally, our results challenge the prevailing assumption that VAE-based architectures are a prerequisite for audio synthesis. Checkpoints are available at https://huggingface.co/mispeech/dashengtokenizer.

Figures & tables

Approach	Type	Dimension	Representative work	Understanding	Generation
Audio Codec	Discrete	Low	VQ-VAE Van Den Oord et al. (2017)	${\color[rgb]{0.9336,0.25,0.1563}\mathbf{\times}}$	${\color[rgb]{0.0117,0.0117,0.0117}\mathbf{\sim}}$
Audio Codec + Semantic	Discrete	Low	Mimi Défossez et al. (2024)	${\color[rgb]{0.0117,0.0117,0.0117}\mathbf{\sim}}$	${\color[rgb]{0.0117,0.0117,0.0117}\mathbf{\sim}}$
Audio Encoders	Continuous	High	Whisper Radford et al. (2023)	✓	${\color[rgb]{0.9336,0.25,0.1563}\mathbf{\times}}$
Acoustic Tokenizers	Continuous	Low	AudioLDM Liu et al. (2023)	${\color[rgb]{0.9336,0.25,0.1563}\mathbf{\times}}$	✓
Ours	Continuous	High	Ours	✓	✓

Table 1: A comparison of our work in contrast to previous audio encoders and tokenizers.

Panel of Figure 1 — Figure 1: A summarization of DashengTokenizer capabilities for understanding and generation tasks.

Figure 2: The proposed DashengTokenizer compared to prior approaches: [A] standard acoustic Acoustic modeling using VAE, and [B] semantically distilled (VQ-)VAEs. In contrast, our approach eliminates the multi-stage training required by [B] and does not rely on a semantic decoder that is discarded during inference, thereby avoiding a train-test mismatch.

Attribute	Semantic Encoder	Acoustic Encoder	Decoder
Parameters	630 M	0.66 M	173 M
Architecture	32 Layer Transformer	2D Conv	Vocos
Hidden Dim	1280	1280	1280
Input	64-bin Mel	128-bin Mel	Unified ( $d=1280$ )
Frame Rate	25 Hz	25 Hz	25 Hz (Upsample to 50 Hz)

Table 2: DashengTokenizer architecture.

Model	Framerate	Seed-TTS (ZH)		Seed-TTS (EN)
		PESQ $\uparrow$	STOI $\uparrow$	PESQ $\uparrow$	STOI $\uparrow$
SNAC Siuzdak et al. (2024)	-	1.841	0.862	1.804	0.870
Mimi Défossez et al. (2024)	12.5	2.050	0.890	2.010	0.890
XCodec 2.0 Ye et al. (2025)	50	2.190	0.920	2.370	0.930
XY-Tokenizer Gong et al. (2025)	12.5	2.270	0.900	2.140	0.900
DAC Kumar et al. (2023)	50	3.860	0.967	3.763	0.969

Table 3: Speech reconstruction performance across different tokenizers. Codecs are highlighted in blue , while continuous tokenizers are highlighted in red . For all metrics higher is better and best are in bold and second best are underlined .

Model	Framerate	Audioset		MUSDB18
		Mel-16k $\downarrow$	STFT-16k $\downarrow$	Mel-16k $\downarrow$	STFT-16k $\downarrow$
XY-Tokenizer	12.5	0.901	2.361	0.607	1.817
DAC	50	0.664	1.986	0.965	2.352
SNAC	-	1.195	2.690	1.242	2.655
XCodec 2.0	50	1.253	2.925	1.273	2.855
EzAudio	50	0.349	1.434	0.258	1.168

Table 4: General audio (Audioset) and music (MUSDB) reconstruction performance measured with Mel and STFT distances on 16 kHz audio. Codecs are highlighted in blue , while continuous tokenizers are highlighted in red . For all metrics lower is better and best are in bold and second best are underlined .

Model	ASV	Clo	DES	ESC	FSD50	F18K	Urb8
DAC	93.46	0.73	20.20	31.20	7.80	16.44	50.26
Encodec	95.14	0.57	3.82	16.80	4.63	6.94	40.34
SemantiCodec	93.69	1.89	48.57	81.75	34.84	26.69	84.54
FlexiCodec	93.00	0.00	17.00	30.00	0.00	0.00	0.00
WavTokenizer	95.00	0.00	16.00	19.00	0.00	0.00	0.00
OpenBeats	0.00	4.00	55.20	86.80	38.00	68.90	86.30

Table 6: Understanding Performance in the Sound Domain on the X-ARES benchmark. Codecs are highlighted in blue , while continuous tokenizers are highlighted in red and encoder models are highlighed in yellow . For all metrics higher is better and best are in bold and second best are underlined .

Model	FMA	GTZAN	MAESTRO	NSynth
DAC	35.43	54.06	12.78	38.57
Encodec	28.80	35.14	23.37	32.59
SemantiCodec	57.94	66.67	8.62	68.14
FlexiCodec	0.00	0.00	0.00	0.00
WavTokenizer	0.00	0.00	0.00	0.00
OpenBeats	62.40	85.90	0.00	58.90

Table 7: Understanding Performance in the Music Domain on the X-ARES benchmark. Codecs are highlighted in blue , while continuous tokenizers are highlighted in red and encoder models are highlighed in yellow . For all metrics higher is better and best are in bold and second best are underlined .

Dataset	Model	PESQ $\uparrow$	STOI $\uparrow$	DNSMOS $\uparrow$				NISQAv2 $\uparrow$	SpkSim $\uparrow$
Dataset	Model	PESQ $\uparrow$	STOI $\uparrow$	P808	SIG	BAK	OVL	NISQAv2 $\uparrow$	SpkSim $\uparrow$
Valentini	Noisy	1.97	0.92	3.09	3.32	3.11	2.68	3.04	0.888
	LMS	1.77	0.85	3.45	3.12	4.07	2.90	4.25	0.718
	Whisper	1.31	0.80	3.39	3.47	4.12	3.22	3.83	0.406
	WavLM	1.27	0.81	3.49	3.49	4.13	3.26	4.12	0.408
	Dasheng Denoiser	2.32	0.90	3.52	3.45	4.14	3.22	4.51	0.783

Table 8: Speech enhancement results on the Valentini and DNS1 datasets. Baselines are taken from Sun et al. (2025) . Higher is better and best is in bold .

Task	Model	FAD $\downarrow$	FD $\downarrow$	KL $\downarrow$	CLAP Score $\uparrow$
TTA	VAE	4.27	24.72	1.84	0.412
TTA	Ours	3.06	21.23	1.56	0.438
TTM	VAE	4.35	23.68	1.94	0.258
TTM	Ours	3.47	22.74	2.00	0.261

Table 9: Flow-matching generation performance for TTA and TTM, compared with the VAE from UniFlow-Audio. Best results are marked in bold .

Figure 3: Text-to-Audio and Text-to-Music training progress of our proposed framework compared with the VAE from UniFlow-Audio.

Domain	Task	Semantic	Acoustic	Unified
Speech	CREMA-D	80.10	44.44	80.55
	FSC	98.26	2.71	83.39
	LibCnt	80.79	52.34	79.98
	LS100h	80.32	0.00	68.70
	LSMF	98.47	88.14	98.06
	RAV	87.15	33.54	83.06

Table 10: Audio understanding performance on the X-ARES benchmark for the individual acoustic, semantic and proposed unified features. Higher is better and best is in bold .

Feature	Audioset		MUSDB18		SEED-ZH	SEED-EN
Feature	Mel-16k $\downarrow$	STFT-16k $\downarrow$	Mel-16k $\downarrow$	STFT-16k $\downarrow$	PESQ $\uparrow$
Acoustic	0.268	1.497	0.267	1.410	4.178	4.131
Semantic	3.310	4.978	3.290	5.540	1.040	1.040
Unified	0.320	1.585	0.293	1.458	4.163	4.125

Table 11: Reconstruction performance for semantic, acoustic and the proposed unified features. Best are in bold .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

$\lambda_{\text{sem}}$	$\lambda_{\text{mel}}$	LibriTTS-clean	Audioset	MusicCaps
		Mel-16k $\downarrow$
0	45	0.33	0.37	0.41
10	45	0.35	0.44	0.52
30	45	0.35	0.47	0.56
45	45	0.35	0.49	0.59
60	45	0.48	0.61	0.68

Appendix

Table 12: Reconstruction Performance metrics for different

\lambda_{\text{sem}},\lambda_{\text{mel}}

. Configuration used in this paper is highlighted.

$\lambda_{\text{sem}}$	$\lambda_{\text{mel}}$	LibriTTS-clean	Audioset	MusicCaps
0	45	-6%	-24%	-31%
10	45	0%	-10%	-12%
30	45	0%	-4%	-5%
45	45	0.35	0.49	0.59
60	45	+37%	+24%	+15%
45	0	+320%	+747%	+741%

Appendix

Table 13: Relative reconstruction performance change (

\Delta

vs.

\lambda_{\text{sem}}=45,\lambda_{\text{mel}=45}

). Configuration used in this paper is highlighted.

$\lambda_{\text{sem}}$	$\lambda_{\text{mel}}$	ESC	FSD50	GTZAN	LS100h	NSynth	SPV1	Urb8	VocS	VoxC	VoxL33
0	45	35.20	8.20	54.16	0.00	38.67	24.77	51.97	50.29	21.62	10.07
10	45	85.25	53.11	81.78	78.43	76.56	97.23	79.11	92.99	30.94	90.30
30	45	90.25	55.26	85.89	80.23	76.90	97.29	81.18	93.13	33.67	90.55
45	45	91.85	55.94	86.89	81.89	77.05	97.29	81.83	93.15	37.20	91.11
60	45	91.60	56.09	86.88	85.39	77.17	97.40	82.12	92.79	35.40	91.30
45	0	95.35	60.43	89.09	83.67	77.86	97.57	85.49	93.71	32.61	91.86

Appendix

Table 14: Downstream understanding performance for different

\lambda_{\text{sem}},\lambda_{\text{mel}}

$\lambda_{\text{sem}}$	$\lambda_{\text{mel}}$	ESC	FSD50	GTZAN	LS100h	NSynth	SPV1	Urb8	VocS	VoxC	VoxL33
0	45	-62%	-85%	-38%	-100%	-50%	-75%	-36%	-46%	-42%	-89%
10	45	-7%	-3%	-5%	-3%	-0.5%	-0.1%	-3%	-0.2%	-6%	-0.8%
30	45	-2%	-1%	-1%	-2%	-0.2%	0.0%	-0.8%	0.0%	-9%	-0.6%
45	45	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%	0.0%
60	45	-0.3%	+0.2%	0.0%	+4%	+0.1%	+0.1%	+0.3%	-0.4%	-2%	+0.2%
45	0	+4%	+8%	+3%	+2%	+1%	+0.3%	+4%	+0.6%	-12%	+0.8%

Appendix

Table 15: Relative performance change in percent for downstream tasks (Ablation

\Delta

vs.

\lambda_{\text{sem}}=45,\lambda_{\text{mel}}=45

Audio Tokenization

Audio tokenization converts speech, music, and other sounds into compact units for machine learning, generation, and understanding.

See all 42 papers

F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
Jun 4, 2026Dinghao Zhou, Xingchen Song, Di Wu +3Audio Representation Learning Autoencoders
2WeNet Open Source Community · ∗Equal contribution · 1Nanjing University, China
EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement
Jun 1, 2026Hui Li, Yangfan Gao, Junlin Shang +4Audio Representation Learning Neural Audio Codecs
Fudan University
UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception
May 29, 2026Yuhan Song, Linhao Zhang, Aiwei Liu +6Audio Representation Learning Audio Understanding
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Basic Model Technology Center, WeChat AI, Tencent Inc.

Jun 4, 2026J/K move · Enter open · S save

F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation

Dinghao Zhou, Xingchen Song, Di Wu, Pengyu Cheng+2

2WeNet Open Source Community · ∗Equal contribution · 1Nanjing University, China

Audio Representation Learning Autoencoders Audio Understanding Audio Tokenization Tokenizer Adaptation Audio Generation

Open paper AskPDF arXiv

Continuous audio autoencoders reconstruct waveforms well but often produce latents with weak structure for understanding, while self-supervised audio encoders capture semantics but are not directly decodable. This mismatch complicates a single audio tokenizer that must support both understanding and generation. We adapt continuous autoencoder latents to this setting with two components: a noise-regularized autoencoder bottleneck and a latent-side representation encoder. The bottleneck uses channel normalization and stochastic perturbation instead of KL-based variational training, yielding scale-controlled continuous latents for reconstruction and autoregressive generation. The representation encoder is trained on frozen autoencoder latents with RQ-MTP and frozen-LLM supervision. The resulting tokenizer provides high-dimensional representations for understanding while preserving normalized continuous latents as generation targets

Abstract

Audio Tokenization

Explore similar work

F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation

EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement

UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception