Video-conditioned audio generation, including Video-to-Sound (V2S) and Visual Text-to-Speech (VisualTTS), has traditionally been studied as two separate tasks despite their shared objective of generating audio conditioned on visual content. Existing methods rely on task-specific architectures and training paradigms, while the possibility of a unified generative framework remains largely unexplored due to heterogeneous conditioning signals and limited joint training data. In this paper, we propose ViSSFlow, a unified flow-matching framework for both V2S and VisualTTS. To effectively integrate diverse conditioning modalities within a Diffusion Transformer (DiT), we introduce a disentangled condition aggregation mechanism that exploits the complementary roles of attention layers, using cross-attention to model semantic conditions and self-attention to capture temporally dense visual cues. Furthermore, contrary to the common assumption that joint optimization of V2S and VisualTTS causes task interference, we show that a single model can be trained end-to-end without sacrificing task-specific performance. To address the scarcity of paired training data covering both sound and speech generation, we further propose a simple yet effective feature-level data synthesis strategy that enables robust joint learning. Extensive experiments on V2S, VisualTTS, and joint generation benchmarks demonstrate that ViSSFlow consistently outperforms state-of-the-art task-specific baselines while providing a unified solution for video-conditioned audio generation. Code and demos are provided in Supplementary Material.
Figures & tables
Figure 1: ViSSFlow: A unified generative model for video-conditioned sound and speech synthesis. (a) Video-to-Sound generation given a silent video. (b) Visual-TTS generation given a silent talking video and speech transcripts. (c) Sound-Speech Joint generation given a silent video and transcripts.
Figure 2: Architecture of ViSSFlow. ViSSFlow employs cross-attention-based DiT blocks and a flow-matching paradigm, which can handle multiple conditional inputs via cross-attention or concatenation with latents. Temporally dense signals like sound synchronization, speech synchronization, and speech transcript features are introduced by concatenation, while video semantic features are incorporated through cross-attention.
Sound Quality
Synch.
Seman.
Method
FAD↓
FAD↓
KL↓
KL↓
DeSync
IB-VA
(pas.)
(pann.)
(pas.)
(pann.)
↓
↑
SpecVQ.
270.07
28.62
3.20
3.31
1.23
13.77
Im2Wav
242.05
18.34
2.27
2.50
1.22
19.59
V-AURA
218.50
14.80
2.76
2.42
0.65
27.60
VAB
192.07
18.26
2.36
2.51
1.17
25.82
Table 1: V2S evaluation results on the VGGSound benchmark. For each metric, the highest score is shown in bold , and the second-highest is underlined .
Speech Quality
Synch.
Method
WER
Spk.
UTMOS
MCD
LSE-C
LSE-D
↓
Sim. ↑
↑
↓
↑
↓
Benchmark – Chem
DSU
37.3
72.9
3.33
10.52
5.88
7.86
HPMDubbing
22.3
44.6
3.11
11.54
7.30
7.58
StyleDubber
16.3
73.7
3.14
14.46
3.58
11.08
Table 2: VisualTTS evaluation results on the Chem and GRID benchmarks.
Speech
Audio
Method
WER ↓
Spk. Sim. ↑
UTMOS ↑
MCD ↓
LSE-C ↑
LSE-D ↓
FAD pas.↓
FAD pann.↓
KL pas.↓
KL pann.↓
LoVA+Speaker
25.98
20.02
1.33
23.77
1.28
10.66
285.84
8.68
1.27
0.97
LoVA+Style
37.56
19.58
1.30
23.05
1.27
10.65
296.64
9.28
1.31
1.05
MMAudio+Speaker
41.47
18.39
1.32
24.15
1.91
10.95
229.65
4.32
1.14
0.89
MMAudio+Style
56.86
17.78
1.30
23.33
1.92
10.86
240.70
4.48
1.16
0.91
DeepAudio
39.20
12.30
1.31
27.76
1.85
10.98
218.01
8.72
1.43
1.35
Table 3: Video-conditioned sound-speech joint generation evaluation results on the V2C benchmark.
Speech
Sound
Method
Qual. ↑
Sync. ↑
Spk. Sim. ↑
Qual. ↑
Sync. ↑
LoVA+Speaker
2.65
2.57
2.63
2.59
2.43
MMAudio+Style
2.91
2.96
2.89
2.88
2.69
DeepAudio
2.90
2.92
2.95
3.05
2.84
ViSSFlow (Ours)
3.57
3.23
3.44
3.48
3.30
Table 4: Human evaluation on 30 joint generation samples from V2C and out-of-domain Veo3-generated videos.
Figure 3: Performance comparison of different conditioning mechanisms. (a) demonstrates the overview of different model architectures. (b.1) and (b.2) show WER and LSE-D metrics for the VisualTTS task, while (b.3) - (b.5) present FAD, VA-IB, and DeSync metrics for the V2S task. The checkmarks and crosses in the legend of (b) indicate whether the variant effectively incorporates the specific condition. (c) is the visualization of the attention weights in self- and cross-attention layers of DiT.
Figure 4: Impact of joint learning on sound and speech generation. Panels (a.1), (a.2) display FAD and DeSync metrics for the V2S task over training epochs, while (a.3) presents WER for VisualTTS. The joint learning does not compromise convergence speed or final synthesis quality.
Training Data
WER ↓
UTMOS ↑
LSE-C ↑
FAD ↓
KL ↓
V2C
36.5
1.45
2.42
158.01
1.01
V2C + Synth. Data
22.6
1.76
2.58
116.08
1.00
Synthesized Data
19.4
2.07
2.00
177.16
1.07
Table 5: Experiment on training data configurations for video-conditioned audio-speech joint generation on the V2C benchmark. The introduction of synthetic data significantly improves both sound and speech generation quality.
Condition
WER ↓
LSE-C ↑
FAD ↓
DeSync ↓
Full
14.2
8.79
6.68
0.58
w/o CLIP
14.1 (-0.1)
9.16 (+0.37)
20.42 (+13.74)
0.63 (+0.05)
w/o Synch
14.8 (+0.6)
8.77 (-0.02)
10.57 (+3.89)
1.24 (+0.66)
w/o Text
54.4 (+40.2)
8.88 (+0.09)
6.68 (–)
0.58 (–)
w/o Lip
15.7 (+1.5)
8.02 (-0.77)
6.68 (–)
0.58 (–)
w/ CAVP
14.7 (+0.5)
8.72 (-0.07)
11.47 (+4.79)
0.99 (+0.41)
Table 6: Ablation on different conditions. Values in parentheses denote the performance change relative to the full model.
Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.
Thanh V. T. Tran, Ngoc-Son Nguyen, Luong Tran +4
FPT Software AI Center, Vietnam · NVIDIA Corporation, USA
Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis. However, existing conditional diffusion-based V2A methods typically enhance visual conditioning with additional audio-visual supervision, acoustic structure prediction, or reasoning from large multimodal models, requiring extra networks or strong inductive biases. Inspired by recent advances in visual representation learning, we introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A, enriching visual conditioning with minimal architectural modification. We first investigate TD at both the frame and feature levels to identify the most effective representation level at which TD complements visual representations. Based on these findings, we develop a hierarchically continual learning strategy and an annealed temporal differences guidance method to progressively learn and exploit TD information during diffusion training and sampling process, respectively. Extensive experiments on benchmark datasets demonstrate that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality, even outperforming dedicated V2A representations such as contrastive audio-visual pretraining.
Zehua Chen, Junyou Wang, Yuxuan Jiang +5
Tsinghua University, Beijing, China · Hefei University of Technology, Hefei, China · Monash University, Australia +1
This study focuses on a challenging yet promising task, Text-to-Sounding-Video (T2SV) generation, which aims to generate a video with synchronized audio from text conditions, meanwhile ensuring both modalities are aligned with text. Despite progress in joint audio-video training, two critical challenges still remain unaddressed: (1) a single, shared text caption where the text for video is equal to the text for audio often creates modal interference, confusing the pretrained backbones, and (2) the optimal mechanism for cross-modal feature interaction remains unclear. To address these challenges, we first propose the Hierarchical Visual-Grounded Captioning (HVGC) framework that generates pairs of disentangled captions, a video caption, and an audio caption, eliminating interference at the conditioning stage. Based on HVGC, we further introduce BridgeDiT, a novel dual-tower diffusion transformer, which employs a Dual CrossAttention (DCA) mechanism that acts as a robust ``bridge" to enable a symmetric, bidirectional exchange of information, achieving both semantic and temporal synchronization. Extensive experiments on three benchmark datasets, supported by human evaluations, demonstrate that our method achieves state-of-the-art results on most metrics. Comprehensive ablation studies further validate the effectiveness of our contributions, offering key insights for the future T2SV task. All the codes and checkpoints will be publicly released.