eess.ASSep 29, 2025

ViSSFlow: Unifying Video-Conditioned Sound and Speech Generation via Joint Learning

Authors: Xin Cheng, Yuyue Wang, Xihua Wang, Yihan Wu, Kaisi Guan, Yijing Chen, Peng Zhang, Xiaojiang Liu, 

Organizations: Renmin University of China · Apple

Abstract

Video-conditioned audio generation, including Video-to-Sound (V2S) and Visual Text-to-Speech (VisualTTS), has traditionally been studied as two separate tasks despite their shared objective of generating audio conditioned on visual content. Existing methods rely on task-specific architectures and training paradigms, while the possibility of a unified generative framework remains largely unexplored due to heterogeneous conditioning signals and limited joint training data. In this paper, we propose ViSSFlow, a unified flow-matching framework for both V2S and VisualTTS. To effectively integrate diverse conditioning modalities within a Diffusion Transformer (DiT), we introduce a disentangled condition aggregation mechanism that exploits the complementary roles of attention layers, using cross-attention to model semantic conditions and self-attention to capture temporally dense visual cues. Furthermore, contrary to the common assumption that joint optimization of V2S and VisualTTS causes task interference, we show that a single model can be trained end-to-end without sacrificing task-specific performance. To address the scarcity of paired training data covering both sound and speech generation, we further propose a simple yet effective feature-level data synthesis strategy that enables robust joint learning. Extensive experiments on V2S, VisualTTS, and joint generation benchmarks demonstrate that ViSSFlow consistently outperforms state-of-the-art task-specific baselines while providing a unified solution for video-conditioned audio generation. Code and demos are provided in Supplementary Material.

Figures & tables

Explore similar work

CardsList
  1. Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space

    Jul 7, 2026Thanh V. T. Tran, Ngoc-Son Nguyen, Luong Tran +4Audio-Video GenerationAudio-Visual Synchronization

  2. Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation

    Aug 5, 2026Zehua Chen, Junyou Wang, Yuxuan Jiang +5Audio-Video GenerationAudio Diffusion Models

  3. Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction

    Oct 3, 2025Kaisi Guan, Xihua Wang, Zhengfeng Lai +5Audio-Video GenerationVideo Diffusion Models