In this paper, we propose Universal Holistic Audio Generation (UniHAGen), a task for synthesizing comprehensive auditory scenes that include both on-screen and off-screen sounds across diverse domains (e.g., ambient events, musical instruments, and human speech). Prior video-conditioned audio generation models typically focus on producing on-screen environmental sounds that correspond to visible sounding events, neglecting off-screen auditory events. While recent holistic joint text-video-to-audio generation models aim to produce auditory scenes with both on- and off-screen sound but they are limited to non-speech sounds, lacking the ability to generate or integrate human speech. To overcome these limitations, we introduce OmniSonic, a flow-matching-based diffusion framework jointly conditioned on video and text. It features a TriAttn-DiT architecture that performs three cross-attention operations to process on-screen environmental sound, off-screen environmental sound, and speech conditions simultaneously, with a Mixture-of-Experts (MoE) gating mechanism that adaptively balances their contributions during generation. Furthermore, we construct UniHAGen-Bench, a new benchmark with over one thousand samples covering three representative on/off-screen speech-environment scenarios. Extensive experiments show that OmniSonic consistently outperforms state-of-the-art approaches on both objective metrics and human evaluations, establishing a strong baseline for universal and holistic audio generation. Project page: https://weiguopian.github.io/OmniSonic_webpage/
Figures & tables
Figure 1: Illustration of the proposed Universal Holistic Audio Generation (UniHAGen) task. The example depicts the scenario of on-screen speech with off-screen environmental sound. Our model is able to generate audio that is temporally and semantically aligned with the video, consistent with the environmental captions, and faithful to the given speech transcription.
Figure 2: (A) Overview of our proposed OmniSonic, which mainly consists of an environmental text encoder (FLAN-T5), a speech transcription encoder (SpeechT5), a visual encoder (CLIP visual encoder), an audio VAE, and our specially designed TriAttn-DiT blocks. The input example demonstrates the scenario of on-screen speech with off-screen environmental sound. The input conditions include visual frames, speech transcription, an on-screen environmental sound caption (represented by a placeholder "" ), and an off-screen environmental sound caption. (B) Details of our proposed TriAttn-DiT block.
Model
Condition Modality
Generation Quality
Semantic Alignment
Speech Correctness
Temporal Alignment
Text
Visual
FAD ↓
MKL ↓
AT score ↑
AV score ↑
Mean ↑
WER ↓
CER ↓
PER ↓
DeSync ↓
AudioLDM 2 [ 28 ]
✓
✗
10.28
6.37
20.17
6.62
13.40
1.00
1.01
0.97
1.39
AudioLDM 2-gigaspeech [ 28 ]
✓
✗
14.09
9.33
24.26
7.65
15.96
0.66
0.50
0.51
1.04
VoiceLDM [ 24 ]
✓
✗
3.58
5.74
21.25
6.80
14.03
0.15
0.08
0.09
1.25
VinTAGe [ 22 ]
✓
✓
8.30
5.00
20.60
8.36
14.48
1.99
1.46
1.36
1.35
MMAudio [ 6 ]
✓
✓
5.82
5.60
23.37
11.12
17.25
1.50
1.31
1.19
0.51
Table 1: Experimental results of our OmniSonic and compared baselines on the proposed UniHAGen-Bench using objective evaluation metrics. Bold values denote the best performance in each column.
Model
MOS-Q ↑
MOS-EF ↑
MOS-SF ↑
MOS-T ↑
VoiceLDM [ 24 ]
3.13
3.40
4.05
2.54
MMAudio [ 6 ]
3.74
3.24
1.15
3.71
HunyuanVideo-Foley [ 39 ]
3.85
3.88
1.17
3.86
OmniSonic (Ours)
4.35
4.42
4.74
4.29
Table 2: Experimental results of our OmniSonic and compared baselines using objective evaluation metrics. Bold values denote the best performance in each column.
Figure 3: Visualization of the spectrograms of generated audios and the ground-truth.
Model
Generation Quality
Semantic Alignment
Speech Correctness
Temporal Alignment
FAD ↓
MKL ↓
AT score ↑
AV score ↑
Mean ↑
WER ↓
CER ↓
PER ↓
DeSync ↓
OmniSonic
3.07
2.79
28.07
9.01
18.54
0.14
0.05
0.06
0.72
w/o MoE Gating
6.12
5.25
24.76
7.11
15.94
0.56
0.49
0.49
1.23
Table 3: Ablation study on the MoE Gating. Bold values denote the best performance in each column.
Figure 4: Ablation study on the MoE Gating module using in-the-wild samples. In this example, sounds are generated with different gating configurations. Top row: MoE Gating (full model); Middle row: reduced weight for the speech branch; Bottom row: reduced weight for the off-screen environmental branch.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Interface for the subjective evaluation.
λtxton
λtxtsp.
FAD ↓
Mean AV AT ↑
WER ↓
5.0
2.5
3.40
19.62
0.15
5.0
7.5
5.39
19.50
0.14
7.5
2.5
4.14
19.02
0.16
7.5
5.0
4.98
19.32
0.15
7.5
7.5
5.04
19.28
0.15
9.5
2.5
4.23
18.94
0.17
Appendix
Table 4: Parameter study for Scenario 1 (on-screen environmental sound + off-screen human speech). We fix λtxtoff=0.5 and examine the effect of varying λtxton and λtxtsp. on the results.
λtxtsp.
λtxtoff
FAD
Mean AV AT
WER
2.5
2.5
2.63
18.13
0.11
2.5
5.5
4.77
17.26
0.12
2.5
7.5
5.32
17.19
0.10
5.5
2.5
2.33
17.95
0.12
5.5
5.5
4.42
17.55
0.11
5.5
7.5
5.36
17.18
0.11
Appendix
Table 5: Parameter study for Scenario 2 (on-screen human speech + off-screen environmental sound). We fix λtxton=0.5 and examine the effect of varying λtxtsp. and λtxtoff on the results.
λtxton
λtxtoff
λtxtsp.
FAD ↓
Mean AV AT ↑
WER ↓
5.0
2.5
2.5
3.39
18.26
0.16
5.0
2.5
3.5
3.97
18.55
0.15
5.0
5.0
2.5
3.66
17.00
0.16
5.0
5.0
3.5
3.98
17.50
0.14
7.5
2.5
2.5
3.88
18.30
0.15
7.5
2.5
3.5
4.34
18.54
0.18
Appendix
Table 6: Parameter study for Scenario 3 (on-screen environmental sound + off-screen environmental sound + off-screen human speech). We examine the effect of varying λtxton , λtxtoff , and λtxtsp. on the results.
Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.