cs.CVSep 30, 2026

Soundwich: Video Generation with Layered and Controllable Audio

Authors: Zhuo Ning, AmirHossein Naghi Razlighi, Sagi Polaczek, Daniel Cohen-Or, Ali Mahdavi-Amiri

Organizations: Simon Fraser University · University of Cyprus · CYENS Centre of Excellence · Tel Aviv University

Abstract

Recent joint audio-video generative models can synthesize realistic videos with synchronized sound, but typically generate audio as a single mixed track. This limits source-level control and differs from practical audiovisual workflows, where speech, music, sound effects, and ambient sounds are represented as separate editable tracks. We introduce Soundwich, a training-free framework that transforms a frozen joint audio-video flow-matching model into a generator of multiple synchronized, independently editable audio stems coupled to a shared video. Soundwich generates separate audio stems with explicit control over their temporal activity. To keep separately generated sounds coherent, we introduce a shared scene representation that communicates global audiovisual context across stems while preserving their source-level separation. We further route cross-modal interactions between each audio stem and its corresponding visual source, improving audiovisual consistency. The resulting stems remain synchronized with the video and can be independently retimed, muted, replaced, or remixed. Experiments and human evaluations show improved temporal control, source separation, and naturalness, while enabling flexible source-level editing within coherent audiovisual generation. Code is available at https://github.com/CodyNing/Soundwich.

Explore similar work

CardsList
  1. Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance

    Jun 26, 2025Akio Hayakawa, Masato Ishii, Takashi Shibuya +1Audio-Video GenerationAudio Understanding

  2. DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution

    Aug 31, 2026Jiashu Zhu, Yanhao Zheng, Ruitian Tian +7Audio-Video GenerationMultimodal Generation

  3. SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing

    May 24, 2026Sen Liang, Cong Wang, Fengbin Guan +6Audio-Visual ConsistencyAudio-Video Generation