Recent audio generation models typically rely on Variational Autoencoders (VAEs) and perform generation within the VAE latent space. Although VAEs excel at compression and reconstruction, their latents inherently encode low-level acoustic details rather than semantically discriminative information, leading to entangled event semantics and complicating the training of generative models. To address these issues, we discard VAE acoustic latents and introduce semantic encoder latents, thereby proposing SemanticVocoder, a generative vocoder that directly synthesizes waveforms from semantic latents. Equipped with SemanticVocoder, our text-to-audio generation model achieves a Frechet Distance of 12.823 and a Frechet Audio Distance of 1.709 on the AudioCaps test set, as the introduced semantic latents exhibit superior discriminability compared to acoustic VAE latents. Beyond improved generation performance, it also serves as a promising attempt towards unifying audio understanding and generation within a shared semantic space. Generated samples are available at https://zeyuxie29.github.io/SemanticVocoder/.
Figures & tables
Figure 1: The SemanticVocoder pioneers the generation of waveforms directly from semantic latents, thereby bridging understanding-oriented representations and generation tasks. (Left): Three sub-tasks from the HEAR benchmark are employed to evaluate the latent representations, in which linear classifiers are trained on fixed latents. The semantic latents exhibit a more discriminative semantic structure than the acoustic VAE latents used in previous work. (Right): For the downstream text-to-audio task, a text-to-latent model predicts latents conditioned on input text. The predicted latents are then fed into SemanticVocoder for audio synthesis, yielding superior performance.
Figure 2: An overview of SemanticVocoder training, downstream TTA training, and downstream task inference. ( → Blue arrow) SemanticVocoder training: the input audio is fed into a semantic encoder to extract semantic latents, which serve as conditions to train the flow-matching network for waveform prediction. ( → Red arrow) Generative audio DiT training: the input text is processed by a text encoder to obtain textual features, which are used to train the DiT model for generating semantic latents. ( → Black arrow) Downstream task inference: equipped with SemanticVocoder, both audio generation and understanding tasks can be performed within the same semantic latent space.
Figure 3: Visualization of different latents on HEAR-ESC50, where the 10 most frequent categories are presented. Each audio feature is aggregated by mean pooling along the temporal axis and projected into 2D space via t-SNE. Compared to VAE acoustic latents used in baseline models, semantic latents exhibit a more discriminative structure and superior semantic disentanglement.
System
FAD VGG↓
FD PANNs↓
KL ↓
IS ↑
CLAP LAION↑
CLAP MS↑
EzAudio [ 1 ]
2.262
15.738
1.322
11.018
0.458
0.423
AudioLDM2 [ 2 ]
2.957
24.387
1.545
9.105
0.395
0.556
TangoFlux [ 3 ]
2.338
19.319
1.183
12.365
0.482
0.468
MakeAnAudio [ 4 ]
2.455
18.169
1.506
8.820
0.406
0.574
StableAudio [ 5 ]
3.409
44.806
2.152
9.446
0.216
0.345
MMAudio [ 6 ]
6.184
18.631
1.337
12.144
0.416
0.446
Table 1: Text-to-audio generation performance on AudioCaps. The best FAD and FD scores reflect that the audio distribution generated by SemanticVocoder is closest to that of the reference audio.
System
Dimension
DCASE2016 ↑
ESC50 ↑
FSD50k ↑
StableAudio [ 5 ] /TangoFlux [ 3 ]
64
60.523
42.450
21.134
EzAudio [ 1 ]
128
21.616
22.850
10.768
AudioLDM2 [ 2 ]
8*16
20.821
36.150
16.221
MakeAnAudio [ 4 ]
4*10
68.771
35.250
15.694
MMAudio [ 6 ]
40
56.770
35.600
15.831
SemanticVocoder
768
93.690
83.400
51.583
Table 2: Evaluation scores of acoustic and semantic latents on the HEAR audio understanding benchmark. TangoFlux adopts the same VAE as StableAudio.
System
Sample Rate
ViSQOL ↑
Mel ↓
STFT ↓
Waveform ↓
StableAudio [ 5 ] /TangoFlux [ 3 ]
44.1K
4.381
0.766
2.001
0.047
EzAudio [ 1 ]
24K
4.550
1.115
2.267
0.038
AudioLDM2 [ 2 ]
16K
3.514
1.488
3.332
0.084
MakeAnAudio [ 4 ]
16K
3.214
1.498
3.831
0.089
MMAudio [ 6 ]
44.1K
4.481
0.620
2.003
0.099
SemanticVocoder
24K
3.239
1.304
3.283
0.096
Table 3: Reconstruction evaluation on AudioCaps test set. Despite SemanticVocoder being optimized for generation based on semantic information, it exhibits comparable reconstruction performance. TangoFLux adopts the same VAE from StableAudio.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Latent Type
FAD VGG↓
FD PANNs↓
KL ↓
IS ↑
CLAP LAION↑
CLAP MS↑
VAE acoustic latents
2.449
16.358
1.531
10.250
0.419
0.391
Mel acoustic latents
5.087
22.295
1.790
8.475
0.410
0.421
Semantic latents
1.956
14.110
1.292
11.038
0.438
0.541
Appendix
Table 4: Ablation studies on replacing semantic latents with VAE latents or mel-spectrograms.
Strategy
Architecture
FAD VGG↓
FD PANNs↓
KL ↓
IS ↑
CLAP LAION↑
Reconstruction
VAE
4.781
18.962
1.212
8.240
0.403
Generation
Flow-Matching
1.709
12.823
1.154
11.286
0.454
Appendix
Table 5: Text-to-audio generation results on AudioCaps. The same text-to-semantic latent model is used, while different training strategies are adopted for the latent-to-waveform module.
Figure 4: The influence of inference steps and Class-Free guidance on TTA performance.
System
FAD VGG↓
FD PANNs↓
KL ↓
IS ↑
CLAP LAION↑
CLAP MS↑
EzAudio [ 1 ]
1.887
20.499
2.581
10.476
0.264
0.473
AudioLDM2 [ 2 ]
2.677
21.257
2.188
8.754
0.289
0.422
TangoFlux [ 3 ]
2.503
21.899
2.462
10.833
0.318
0.495
MakeAnAudio [ 4 ]
3.368
23.032
2.631
8.173
0.236
0.384
StableAudio [ 5 ]
2.994
40.731
2.515
9.168
0.285
0.481
MMAudio [ 6 ]
2.257
20.143
2.501
9.223
0.356
0.515
Appendix
Table 6: Cross-domain evaluation results. Experiments are conducted on the Clotho test set, where the first caption is used as input and each audio clip is truncated to 10 seconds.
In recent years, Text-to-Audio Generation has achieved remarkable progress, offering sound creators powerful tools to transform textual inspirations into vivid audio. However, existing models predominantly operate directly in the acoustic latent space of a Variational Autoencoder (VAE), often leading to suboptimal alignment between generated audio and textual descriptions. In this paper, we introduce SemanticAudio, a novel framework that conducts both audio generation and editing directly in a high-level semantic space. We define this semantic space as a compact representation capturing the global identity and temporal sequence of sound events, distinct from fine-grained acoustic details. SemanticAudio employs a two-stage Flow Matching architecture: the Semantic Planner first generates these compact semantic features to sketch the global semantic layout, and the Acoustic Synthesizer subsequently produces high-fidelity acoustic latents conditioned on this semantic plan. Leveraging this decoupled design, we further introduce a training-free text-guided editing mechanism that enables precise attribute-level modifications on general audio without retraining. Specifically, this is achieved by steering the semantic generation trajectory via the difference of velocity fields derived from source and target text prompts. Extensive experiments demonstrate that SemanticAudio surpasses existing mainstream approaches in semantic alignment. Demo available at: https://semanticaudio1.github.io/
Zheqi Dai, Guangyan Zhang, Haolin He +5
The Chinese University of Hong Kong · LIGHTSPEED · Shanghai Jiao Tong University +1
Modern audio generation predominantly relies on latent-space compression, introducing additional complexity and potential information loss. In this work, we challenge this paradigm with WavFlow, a framework that generates high-fidelity audio directly in raw waveform space without intermediate representations. To overcome the inherent difficulties of modeling high-dimensional and low-energy signals, we reshape audio into 2D token grids through waveform patchify and introduce amplitude lifting to align signal scales, enabling stable optimization via direct x-prediction in flow matching. To capture complex semantic alignment and temporal synchronization, we leverage an automated data pipeline to curate 5 million high-quality video-text-audio triplets, allowing the model to learn fine-grained acoustic patterns from scratch. Experimental results show that WavFlow achieves competitive performance on the video-to-audio benchmark VGGSound (FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44) and the text-to-audio benchmark AudioCaps (FD_PANNs: 10.63, IS_PANNs: 12.62), matching or exceeding the performance of established latent-based methods. Our work demonstrates that intermediate compression is not a prerequisite for high-quality synthesis, offering a simpler and more scalable alternative for multimodal audio generation.
Latent audio generative models are typically trained in two stages: an audio codec is learned first, followed by a latent generative model. This decomposition leads to a decoder train-generation mismatch: the codec decoder is trained on encoder-induced latents but deployed on generator-produced latents at inference time. Across diverse datasets and latent generative models, we observe clear reconstruction-generation gaps under both FD and FAD, showing that strong reconstruction quality does not necessarily translate into strong end-to-end generation quality. A natural remedy is to adapt the decoder on generation-produced latents, but generated latents lack correspondence with source audio and therefore cannot directly provide the paired supervision used for decoder fine-tuning. We introduce AudioGAR, which constructs intermediate latents by perturbing encoder latents and denoising them through the frozen latent diffusion model. These latents form a trajectory from reconstruction toward generation, with lower-noise latents retaining source correspondence and supporting paired decoder fine-tuning. We fine-tune only the codec decoder on these latents, while keeping the codec encoder and latent generative model frozen. When applied to AudioX, AudioGAR substantially improves generative performance. It requires only 1.5% of the original training audio hours and 0.26% of the original training cost.