Recent advancements in audio generation have been largely driven by Transformer-based diffusion models. However, these models suffer from quadratic complexity of self-attention, severely bottlenecking the long-form audio synthesis. To overcome this limitation, we propose MFlowAudio, a novel latent audio generation framework that synergizes the continuous-time dynamics of Flow Matching with a custom-designed TCMamba backbone. TCMamba uses an innovative dual-scan mechanism: a TimeMamba module to capture long-range causal dependencies with linear complexity, and a ChannelMamba module to model cross-channel correlations such as harmonic structures. Exploiting this structural foundation, we formulate the Stateful Flow Matching (SFM) paradigm. This framework inherently enables chunk-wise training and streaming generation, maintaining an O(1) caching complexity without incurring extra computational overhead. To enable fine-grained controllable synthesis, we devise a novel guidance mechanism that neutralizes vector field collisions precipitated by on-the-fly semantic transitions of prompts. Comprehensive empirical evaluations confirm that MFlowAudio yields generative fidelity on par with state-of-the-art baselines while establishing significant computational efficiency, achieving above 14.5% acceleration in generation and facilitating perceptually seamless streaming synthesis with a constant latency of 1.8 seconds. Demo: https://huggingface.co/spaces/mflowaudio/MFlowAudio
Figures & tables
Figure 1: The key components of MFlowAudio: (left) the workflow of MFlowAudio, audio is encoded and decoded into latent using VAE and vocoder; (middle) the architecture of the TCMamba block; and (right) the dual-scanning mechanism of Mamba.
Figure 2: Beyond conventional monolithic training and generation, MFlowAudio also facilitates (a) chunked AR-style training and (b) chunked generation to support streaming.
Figure 3: Switching prompts across variable-sized chunks to control the exact timing of events.
Duration
Model
Parameters
FD ↓
KL ↓
IS ↑
FAD v ↓
CLAP ↑
FAD openl3 ↓
10s
AudioGen (16K)
1.5B
38.7259
2.5699
6.6716
6.6353
0.1950
192.4442
AudioLDM2-48K
1.7B
36.2054
2.0261
5.442
5.4774
0.3827
148.1751
Stable Audio Open
1B
32.0744
2.052
10.1462
3.7894
0.3442
109.0337
TangoFlux-base
516M
16.9351
1.1724
12.0486
3.2514
0.4722
113.6228
MFlowAudio-base
581M
10.9178
1.1588
10.8935
2.9612
0.4655
95.6402
TangoFlux
516M
15.8429
1.1164
12.5489
2.1963
0.4990
102.8903
Table 1: Performance comparison across baselines on AudioCaps testset.
Duration
AudioLDM2-48K
Stable-Audio-Open
TangoFlux
MFlowAudio
10s
22.23 ± 3.1 s
5.23 ± 13 ms
3.51 ± 3 ms
1.93 ± 13 ms
20s
32.17 ± 4.3 s
5.31 ± 21 ms
3.54 ± 12 ms
2.17 ± 17 ms
30s
97.82 ± 7.5 s
5.45 ± 25 ms
3.55 ± 5 ms
3.10 ± 18 ms
Table 2: Generation latency comparison (s)
Duration
BatchSize
One-time
TTFC
Speedup
Reduction
10.0
1
1.9 ± 19 ms
1.82 ± 9 ms
1.1x
+5.8%
10.0
2
2.0 ± 13 ms
1.85 ± 21 ms
1.1x
+10.6%
20.0
1
2.1 ± 26 ms
1.81 ± 17 ms
1.2x
+16.8%
20.0
2
3.5 ± 32 ms
1.85 ± 31 ms
1.9x
+47.4%
30.0
1
3.1 ± 18 ms
1.80 ± 29 ms
1.7x
+41.7%
30.0
2
5.1 ± 57 ms
1.85 ± 37 ms
2.8x
+63.9%
Table 3: One-time Latency v.s. Time-to-First-Chunk (s)
Model
FD ↓
KL ↓
IS ↑
FAD v ↓
CLAP ↑
FAD openl3 ↓
BBR ↓
TangoFlux (OneTime)
21.2885
1.1381
9.9338
2.1743
0.5208
110.7247
-
TangoFlux (Dyna-P)
22.6873
1.2788
6.2233
3.9077
0.4191
114.2842
11.6443
MFlowAudio (OneTime)
16.3070
1.1416
9.2517
2.2228
0.5186
96.8583
-
MFlowAudio (Dyna-P)
15.8575
1.2190
7.5717
2.8448
0.5087
101.2825
4.3287
Table 4: Performance comparison of dynamic conditional injection
Model
FD ↓
KL ↓
IS ↑
FAD v ↓
CLAP ↑
FAD openl3 ↓
BBR ↓
w/o state
19.3239
1.3015
6.8395
3.5030
0.4925
99.4748
7.4001
non-state prompt
16.9351
1.2767
7.2420
3.1169
0.4996
99.2386
5.4953
vanilla
16.0940
1.2536
7.6370
2.9659
0.5060
102.8763
4.2633
w/ naive sub
16.0056
1.2494
7.6201
2.8947
0.5075
102.3913
4.3578
w/ sdcg
15.8575
1.2190
7.5717
2.8448
0.5087
101.2825
4.3287
Table 5: Ablation experiments of SDCG
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: evaluation platform for subjective evaluations.
Model
REL ↑
OVL ↑
AudioLDM2-48K
2.61
2.57
Stable-Audio-Open
2.82
3.28
TangoFlux
4.15
3.52
MFlowAudio
4.09
3.59
Appendix
Table 6: Subjective evaluations
Model
BAS ↑
CPO ↓
TN ↑
TangoFlux (Dyna-P)
4.08
0.76
2.04
MFlowAudio (Dyna-P)
4.12
0.02
4.18
Appendix
Table 7: Naturalness evaluations with dynamic prompting
L (dur.)
MFlowAudio SSM state / fwd
Transformer KV cache / fwd
128 ( 6 s)
18.0 MB
18.8 MB
256 ( 12 s)
18.0 MB
37.5 MB
512 ( 24 s)
18.0 MB
75.0 MB
scaling
constant, O(1)
4× , O(L)
Appendix
Table 8: Cache size comparison between MFlowAudio and Transformer-based diffusion models
overlap W
lookahead
rel-L1 vs full
peak error
0 (naive per-chunk)
0 ms
4.54%
0.62
4
182 ms
0.39%
0.15
8
364 ms
0.049%
0.003
16
727 ms
0.047%
0.002
Appendix
Table 9: Lookahead Analysis in Stable Audio Open Oobleck VAE
Table 11: Performance comparison across baselines on Clotho testset.
Figure 5: Spectrum generated by TangoFlux (top) and MFlowAudio (bottom) via dynamic prompting. Prompts: ”0, people are talking” ”3, a crowd of people talking” ”6, a horse neighing as a lawnmower engine runs in the background”. There is a noticeable spectral jump in TangoFlux at the third and sixth seconds (a bright vertical line).
Text-to-Audio Generation creates speech, music, and sound effects from natural-language prompts, aiming to match their content, timing, and acoustic qualities.
While recent few-step sampling text-to-audio generation models like MeanAudio substantially accelerate generation by modeling average velocities, their strict one-step generation quality still lags significantly behind multi-step counterparts. We propose FdAudio to bridge this gap. Unlike MeanAudio, which relies solely on regression against target velocity fields, our post-training approach optimizes the final one-step distribution directly across pre-trained embedding spaces via a multi-representation Fréchet-distance (FD) loss. Crucially, to prevent the multi-step degradation that naive post-training with FD-loss causes, we introduce a MeanFlow consistency objective as a structural anchor. Results demonstrate that FdAudio establishes state-of-the-art one-step T2A generation quality among few-step systems, yielding an 11.4% reduction in FD score and a 28.8% improvement in FAD score relative to the baseline MeanAudio framework. Notably, we solve FD post-training's naive multi-step degradation issue by proposing the MeanFlow anchor, enabling a 25-step sampling path to maintain high-fidelity audio synthesis that matches or surpasses strong multi-step models at a fraction of their computational latency.
Text-to-audio (TTA) generation aims to synthesize realistic audio that faithfully reflects natural-language descriptions. Most TTA systems adopt a two-stage latent paradigm: an audio tokenizer is optimized for reconstruction and then frozen, after which a generative model is trained in the resulting latent space. However, reconstruction-oriented representations may be suboptimal for generation, motivating joint representation and generative learning. To this end, we introduce Unite-Audio, to our knowledge, is the first to jointly learn continuous audio representations and latent flow matching for TTA. By coupling reconstruction with self-supervised generative prediction, Unite-Audio allows the generative objective to directly shape the latent space rather than treating it as a fixed intermediate representation. We further employ Flow-GRPO post-training to improve text-conditioned generation. Experiments show competitive TTA performance with a compact latent flow model, while ablation studies confirm the benefit of jointly learning the audio representation and generative model. Audio samples are available at https://runwushi.github.io/Unite-Audio.
Runwu Shi, Kai Li, Yujin Wang +8
Institute of Science Tokyo · Tsinghua University · Wuhan University +2
Modern audio generation predominantly relies on latent-space compression, introducing additional complexity and potential information loss. In this work, we challenge this paradigm with WavFlow, a framework that generates high-fidelity audio directly in raw waveform space without intermediate representations. To overcome the inherent difficulties of modeling high-dimensional and low-energy signals, we reshape audio into 2D token grids through waveform patchify and introduce amplitude lifting to align signal scales, enabling stable optimization via direct x-prediction in flow matching. To capture complex semantic alignment and temporal synchronization, we leverage an automated data pipeline to curate 5 million high-quality video-text-audio triplets, allowing the model to learn fine-grained acoustic patterns from scratch. Experimental results show that WavFlow achieves competitive performance on the video-to-audio benchmark VGGSound (FD_PaSST: 59.98, IS_PANNs: 17.40, DeSync: 0.44) and the text-to-audio benchmark AudioCaps (FD_PANNs: 10.63, IS_PANNs: 12.62), matching or exceeding the performance of established latent-based methods. Our work demonstrates that intermediate compression is not a prerequisite for high-quality synthesis, offering a simpler and more scalable alternative for multimodal audio generation.