SemanticVocoder: Bridging Audio Generation and Audio Understanding via Semantic Latents

Zeyu Xie; Chenxing Li; Qiao Jin; Xuenan Xu; Guanrou Yang; Wenfu Wang; Mengyue Wu; Dong Yu

cs.SDFeb 26, 2026

SemanticVocoder: Bridging Audio Generation and Audio Understanding via Semantic Latents

Authors: Zeyu Xie, Chenxing Li, Qiao Jin, Xuenan Xu, Guanrou Yang, Wenfu Wang, Mengyue Wu, Dong Yu,

Organizations: Peking University, Shenzhen · Tencent AI Lab, Shenzhen · Shanghai AI Lab, Shanghai · SJTU, Shanghai

Abstract

Recent audio generation models typically rely on Variational Autoencoders (VAEs) and perform generation within the VAE latent space. Although VAEs excel at compression and reconstruction, their latents inherently encode low-level acoustic details rather than semantically discriminative information, leading to entangled event semantics and complicating the training of generative models. To address these issues, we discard VAE acoustic latents and introduce semantic encoder latents, thereby proposing SemanticVocoder, a generative vocoder that directly synthesizes waveforms from semantic latents. Equipped with SemanticVocoder, our text-to-audio generation model achieves a Frechet Distance of 12.823 and a Frechet Audio Distance of 1.709 on the AudioCaps test set, as the introduced semantic latents exhibit superior discriminability compared to acoustic VAE latents. Beyond improved generation performance, it also serves as a promising attempt towards unifying audio understanding and generation within a shared semantic space. Generated samples are available at https://zeyuxie29.github.io/SemanticVocoder/.

Figures & tables

Figure 1: The SemanticVocoder pioneers the generation of waveforms directly from semantic latents, thereby bridging understanding-oriented representations and generation tasks. (Left): Three sub-tasks from the HEAR benchmark are employed to evaluate the latent representations, in which linear classifiers are trained on fixed latents. The semantic latents exhibit a more discriminative semantic structure than the acoustic VAE latents used in previous work. (Right): For the downstream text-to-audio task, a text-to-latent model predicts latents conditioned on input text. The predicted latents are then fed into SemanticVocoder for audio synthesis, yielding superior performance.

Figure 2: An overview of SemanticVocoder training, downstream TTA training, and downstream task inference. ( $\rightarrow$ Blue arrow) SemanticVocoder training: the input audio is fed into a semantic encoder to extract semantic latents, which serve as conditions to train the flow-matching network for waveform prediction. ( $\rightarrow$ Red arrow) Generative audio DiT training: the input text is processed by a text encoder to obtain textual features, which are used to train the DiT model for generating semantic latents. ( $\rightarrow$ Black arrow) Downstream task inference: equipped with SemanticVocoder, both audio generation and understanding tasks can be performed within the same semantic latent space.

Figure 3: Visualization of different latents on HEAR-ESC50, where the 10 most frequent categories are presented. Each audio feature is aggregated by mean pooling along the temporal axis and projected into 2D space via t-SNE. Compared to VAE acoustic latents used in baseline models, semantic latents exhibit a more discriminative structure and superior semantic disentanglement.

System	FAD ${}_{\text{VGG}}$ $\downarrow$	FD ${}_{\text{PANNs}}$ $\downarrow$	KL $\downarrow$	IS $\uparrow$	CLAP ${}_{\text{LAION}}$ $\uparrow$	CLAP ${}_{\text{MS}}$ $\uparrow$
EzAudio [ 1 ]	2.262	15.738	1.322	11.018	0.458	0.423
AudioLDM2 [ 2 ]	2.957	24.387	1.545	9.105	0.395	0.556
TangoFlux [ 3 ]	2.338	19.319	1.183	12.365	0.482	0.468
MakeAnAudio [ 4 ]	2.455	18.169	1.506	8.820	0.406	0.574
StableAudio [ 5 ]	3.409	44.806	2.152	9.446	0.216	0.345
MMAudio [ 6 ]	6.184	18.631	1.337	12.144	0.416	0.446

Table 1: Text-to-audio generation performance on AudioCaps. The best FAD and FD scores reflect that the audio distribution generated by SemanticVocoder is closest to that of the reference audio.

System	Dimension	DCASE2016 $\uparrow$	ESC50 $\uparrow$	FSD50k $\uparrow$
StableAudio [ 5 ] /TangoFlux [ 3 ]	64	60.523	42.450	21.134
EzAudio [ 1 ]	128	21.616	22.850	10.768
AudioLDM2 [ 2 ]	8*16	20.821	36.150	16.221
MakeAnAudio [ 4 ]	4*10	68.771	35.250	15.694
MMAudio [ 6 ]	40	56.770	35.600	15.831
SemanticVocoder	768	93.690	83.400	51.583

Table 2: Evaluation scores of acoustic and semantic latents on the HEAR audio understanding benchmark. TangoFlux adopts the same VAE as StableAudio.

System	Sample Rate	ViSQOL $\uparrow$	Mel $\downarrow$	STFT $\downarrow$	Waveform $\downarrow$
StableAudio [ 5 ] /TangoFlux [ 3 ]	44.1K	4.381	0.766	2.001	0.047
EzAudio [ 1 ]	24K	4.550	1.115	2.267	0.038
AudioLDM2 [ 2 ]	16K	3.514	1.488	3.332	0.084
MakeAnAudio [ 4 ]	16K	3.214	1.498	3.831	0.089
MMAudio [ 6 ]	44.1K	4.481	0.620	2.003	0.099
SemanticVocoder	24K	3.239	1.304	3.283	0.096

Table 3: Reconstruction evaluation on AudioCaps test set. Despite SemanticVocoder being optimized for generation based on semantic information, it exhibits comparable reconstruction performance. TangoFLux adopts the same VAE from StableAudio.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Latent Type	FAD ${}_{\text{VGG}}$ $\downarrow$	FD ${}_{\text{PANNs}}$ $\downarrow$	KL $\downarrow$	IS $\uparrow$	CLAP ${}_{\text{LAION}}$ $\uparrow$	CLAP ${}_{\text{MS}}$ $\uparrow$
VAE acoustic latents	2.449	16.358	1.531	10.250	0.419	0.391
Mel acoustic latents	5.087	22.295	1.790	8.475	0.410	0.421
Semantic latents	1.956	14.110	1.292	11.038	0.438	0.541

Appendix

Table 4: Ablation studies on replacing semantic latents with VAE latents or mel-spectrograms.

Strategy	Architecture	FAD ${}_{\text{VGG}}$ $\downarrow$	FD ${}_{\text{PANNs}}$ $\downarrow$	KL $\downarrow$	IS $\uparrow$	CLAP ${}_{\text{LAION}}$ $\uparrow$
Reconstruction	VAE	4.781	18.962	1.212	8.240	0.403
Generation	Flow-Matching	1.709	12.823	1.154	11.286	0.454

Appendix

Table 5: Text-to-audio generation results on AudioCaps. The same text-to-semantic latent model is used, while different training strategies are adopted for the latent-to-waveform module.

Figure 4: The influence of inference steps and Class-Free guidance on TTA performance.

System	FAD ${}_{\text{VGG}}$ $\downarrow$	FD ${}_{\text{PANNs}}$ $\downarrow$	KL $\downarrow$	IS $\uparrow$	CLAP ${}_{\text{LAION}}$ $\uparrow$	CLAP ${}_{\text{MS}}$ $\uparrow$
EzAudio [ 1 ]	1.887	20.499	2.581	10.476	0.264	0.473
AudioLDM2 [ 2 ]	2.677	21.257	2.188	8.754	0.289	0.422
TangoFlux [ 3 ]	2.503	21.899	2.462	10.833	0.318	0.495
MakeAnAudio [ 4 ]	3.368	23.032	2.631	8.173	0.236	0.384
StableAudio [ 5 ]	2.994	40.731	2.515	9.168	0.285	0.481
MMAudio [ 6 ]	2.257	20.143	2.501	9.223	0.356	0.515

Appendix

Table 6: Cross-domain evaluation results. Experiments are conducted on the Clotho test set, where the first caption is used as input and each audio clip is truncated to 10 seconds.

Neural Vocoders

See all 9 papers

SemanticAudio: Audio Generation and Editing in Semantic Space
Jan 29, 2026Zheqi Dai, Guangyan Zhang, Haolin He +5Text-to-Audio Generation Audio Editing
The Chinese University of Hong Kong · LIGHTSPEED · Shanghai Jiao Tong University +1
WavFlow: Audio Generation in Waveform Space
May 18, 2026Feiyan Zhou, Luyuan Wang, Shoufa Chen +6Text-to-Audio Generation Audio-Video Generation
Meta AI · Northeastern University
AudioGAR: Bridging Reconstruction and Generation in Latent Audio Generative Models
Oct 5, 2026Xianghong Fang, Geeyang Tay, Wentao Ma +2Neural Audio Codecs Audio Diffusion Models
University of Toronto · Vijil

Jan 29, 2026J/K move · Enter open · S save

SemanticAudio: Audio Generation and Editing in Semantic Space

Zheqi Dai, Guangyan Zhang, Haolin He, Xiquan Li+4

The Chinese University of Hong Kong · LIGHTSPEED · Shanghai Jiao Tong University+1

Text-to-Audio Generation Audio Editing Audio Generation

Open paper AskPDF arXiv

In recent years, Text-to-Audio Generation has achieved remarkable progress, offering sound creators powerful tools to transform textual inspirations into vivid audio. However, existing models predominantly operate directly in the acoustic latent space of a Variational Autoencoder (VAE), often leading to suboptimal alignment between generated audio and textual descriptions. In this paper, we introduce SemanticAudio, a novel framework that conducts both audio generation and editing directly in a high-level semantic space. We define this semantic space as a compact representation capturing the global identity and temporal sequence of sound events, distinct from fine-grained acoustic details. SemanticAudio employs a two-stage Flow Matching architecture: the Semantic Planner first generates these compact semantic features to sketch the global semantic layout, and the Acoustic Synthesizer subsequently produces high-fidelity acoustic latents conditioned on this semantic plan. Leveraging this decoupled design, we further introduce a training-free text-guided editing mechanism that enables precise attribute-level modifications on general audio without retraining. Specifically, this is achieved by steering the semantic generation trajectory via the difference of velocity fields derived from source and target text prompts. Extensive experiments demonstrate that SemanticAudio surpasses existing mainstream approaches in semantic alignment. Demo available at: https://semanticaudio1.github.io/

Abstract

Neural Vocoders

Explore similar work

SemanticAudio: Audio Generation and Editing in Semantic Space

WavFlow: Audio Generation in Waveform Space

AudioGAR: Bridging Reconstruction and Generation in Latent Audio Generative Models