AudioTurbo: Fast Text-to-Audio Generation with Rectified Diffusion
Authors: Junqi Zhao, Jinheng Zhao, Haohe Liu, Yun Chen, Lu Han, Haoyang Zhang, Xubo Liu, Mark Plumbley, Wenwu Wang
Organizations: Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK · Laboratory of Noise and Audio Research, Institute of Acoustics, Chinese Academy of Sciences, China
Diffusion models have significantly improved the quality and diversity of text-to-audio (TTA) generation but are hindered by slow inference speed. Recent flow-based approaches have been explored for accelerating TTA generation. However, these approaches typically rely on dedicated flow-based generative frameworks and do not directly exploit existing pre-trained diffusion-based TTA models. To address this limitation while leveraging the strong generative capabilities of pre-trained diffusion models, we integrate rectified diffusion into these models to improve TTA generation efficiency. Specifically, we propose AudioTurbo, which learns first-order approximate ordinary differential equation (ODE) paths from deterministic noise-sample pairs generated by a pre-trained TTA model. Experiments show that our approach outperforms a flow-matching-based acceleration model in both generation quality and inference speed.
Figures & tables
Figure 1: An overview of AudioTurbo architecture. Note that the trainable parameters are initialized using the pretrained TTA model, Auffusion.
Model
Params
Steps
RTF
Text
Objective Metrics
Subjective Metrics
FD ↓
KL ↓
IS ↑
CLAP (%) ↑
OVL ↑
REL ↑
Ground Truth
−
−
−
−
−
−
−
−
91.30
92.03
Tango
866 M
200
1.789
✓
21.97
1.52
7.37
23.6
80.87
82.30
AudioLDM2
1.1 B
200
2.070
✗
36.95
1.82
6.87
22.3
79.63
75.97
LAFMA
272 M
200
1.009
✓
29.59
1.56
7.29
22.9
78.00
78.73
AudioTurbo
1.1 B
5
0.093
✓
22.18
1.30
8.88
29.2
−
−
Table 1: The comparison between our AudioTurbo and baseline TTA models on the AudioCaps dataset.
Model
Steps
Guidance
FD ↓
KL ↓
IS ↑
CLAP(%) ↑
AudioTurbo
25
1.0
23.20
1.35
8.95
28.9
1.5
21.69
1.31
9.74
29.6
2.5
21.98
1.36
9.86
28.8
5.0
26.18
1.51
9.16
25.1
7.5
32.49
1.72
7.56
20.4
Table 2: Impact on objective evaluation metrics with varying levels of classifier-free guidance.
Metrics
Model
Inference Steps
3
5
10
25
200
FD ↓
LAFMA
91.53
53.46
31.26
27.37
29.59
Auffusion
55.66
36.53
25.17
21.21
21.31
AudioTurbo
29.96
22.18
20.65
21.69
21.75
KL ↓
LAFMA
3.30
2.13
1.61
1.58
1.56
Auffusion
2.75
1.98
1.65
1.27
1.30
Table 3: Objective metric values for LAFMA, Auffusion, and AudioTurbo models at different inference steps.
Model
Steps
FD ↓
KL ↓
IS ↑
CLAP (%) ↑
AudioLDM
200
35.04
1.81
7.49
18.4
AudioLDM
25
38.3
1.95
7.15
15.6
AudioLDM
10
47.96
2.16
5.93
12.5
AudioLDM
5
63.46
2.68
4.23
7.8
AudioLDM-Turbo
25
32.34
1.69
7.24
17.4
AudioLDM-Turbo
10
31.99
1.70
7.28
17.2
Table 4: Evaluation of our method’s generalizability to AudioLDM on the AudioCaps test set.
Text-to-Audio Generation creates speech, music, and sound effects from natural-language prompts, aiming to match their content, timing, and acoustic qualities.