ViSSFlow: Unifying Video-Conditioned Sound and Speech Generation via Joint Learning
Organizations: Renmin University of China · Apple
Abstract
Video-conditioned audio generation, including Video-to-Sound (V2S) and Visual Text-to-Speech (VisualTTS), has traditionally been studied as two separate tasks despite their shared objective of generating audio conditioned on visual content. Existing methods rely on task-specific architectures and training paradigms, while the possibility of a unified generative framework remains largely unexplored due to heterogeneous conditioning signals and limited joint training data. In this paper, we propose ViSSFlow, a unified flow-matching framework for both V2S and VisualTTS. To effectively integrate diverse conditioning modalities within a Diffusion Transformer (DiT), we introduce a disentangled condition aggregation mechanism that exploits the complementary roles of attention layers, using cross-attention to model semantic conditions and self-attention to capture temporally dense visual cues. Furthermore, contrary to the common assumption that joint optimization of V2S and VisualTTS causes task interference, we show that a single model can be trained end-to-end without sacrificing task-specific performance. To address the scarcity of paired training data covering both sound and speech generation, we further propose a simple yet effective feature-level data synthesis strategy that enables robust joint learning. Extensive experiments on V2S, VisualTTS, and joint generation benchmarks demonstrate that ViSSFlow consistently outperforms state-of-the-art task-specific baselines while providing a unified solution for video-conditioned audio generation. Code and demos are provided in Supplementary Material.
Figures & tables
| Sound Quality | Synch. | Seman. | ||||
|---|---|---|---|---|---|---|
| Method | DeSync | IB-VA | ||||
| (pas.) | (pann.) | (pas.) | (pann.) | |||
| SpecVQ. | 270.07 | 28.62 | 3.20 | 3.31 | 1.23 | 13.77 |
| Im2Wav | 242.05 | 18.34 | 2.27 | 2.50 | 1.22 | 19.59 |
| V-AURA | 218.50 | 14.80 | 2.76 | 2.42 | 0.65 | 27.60 |
| VAB | 192.07 | 18.26 | 2.36 | 2.51 | 1.17 | 25.82 |
| Speech Quality | Synch. | |||||
| Method | WER | Spk. | UTMOS | MCD | LSE-C | LSE-D |
| Sim. | ||||||
| Benchmark – Chem | ||||||
| DSU | 37.3 | 72.9 | 3.33 | 10.52 | 5.88 | 7.86 |
| HPMDubbing | 22.3 | 44.6 | 3.11 | 11.54 | 7.30 | 7.58 |
| StyleDubber | 16.3 | 73.7 | 3.14 | 14.46 | 3.58 | 11.08 |
| Speech | Audio | |||||||||
| Method | WER | Spk. Sim. | UTMOS | MCD | LSE-C | LSE-D | FAD | FAD | KL | KL |
| LoVA+Speaker | 25.98 | 20.02 | 1.33 | 23.77 | 1.28 | 10.66 | 285.84 | 8.68 | 1.27 | 0.97 |
| LoVA+Style | 37.56 | 19.58 | 1.30 | 23.05 | 1.27 | 10.65 | 296.64 | 9.28 | 1.31 | 1.05 |
| MMAudio+Speaker | 41.47 | 18.39 | 1.32 | 24.15 | 1.91 | 10.95 | 229.65 | 4.32 | 1.14 | 0.89 |
| MMAudio+Style | 56.86 | 17.78 | 1.30 | 23.33 | 1.92 | 10.86 | 240.70 | 4.48 | 1.16 | 0.91 |
| DeepAudio | 39.20 | 12.30 | 1.31 | 27.76 | 1.85 | 10.98 | 218.01 | 8.72 | 1.43 | 1.35 |
| Speech | Sound | ||||
|---|---|---|---|---|---|
| Method | Qual. | Sync. | Spk. Sim. | Qual. | Sync. |
| LoVA+Speaker | 2.65 | 2.57 | 2.63 | 2.59 | 2.43 |
| MMAudio+Style | 2.91 | 2.96 | 2.89 | 2.88 | 2.69 |
| DeepAudio | 2.90 | 2.92 | 2.95 | 3.05 | 2.84 |
| ViSSFlow (Ours) | 3.57 | 3.23 | 3.44 | 3.48 | 3.30 |
| Training Data | WER | UTMOS | LSE-C | FAD | KL |
|---|---|---|---|---|---|
| V2C | 36.5 | 1.45 | 2.42 | 158.01 | 1.01 |
| V2C + Synth. Data | 22.6 | 1.76 | 2.58 | 116.08 | 1.00 |
| Synthesized Data | 19.4 | 2.07 | 2.00 | 177.16 | 1.07 |
| Condition | WER | LSE-C | FAD | DeSync |
|---|---|---|---|---|
| Full | 14.2 | 8.79 | 6.68 | 0.58 |
| w/o CLIP | 14.1 (-0.1) | 9.16 (+0.37) | 20.42 (+13.74) | 0.63 (+0.05) |
| w/o Synch | 14.8 (+0.6) | 8.77 (-0.02) | 10.57 (+3.89) | 1.24 (+0.66) |
| w/o Text | 54.4 (+40.2) | 8.88 (+0.09) | 6.68 (–) | 0.58 (–) |
| w/o Lip | 15.7 (+1.5) | 8.02 (-0.77) | 6.68 (–) | 0.58 (–) |
| w/ CAVP | 14.7 (+0.5) | 8.72 (-0.07) | 11.47 (+4.79) | 0.99 (+0.41) |
Audio-Video Generation
Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.
See all 164 papers