cs.SDNov 4, 2026

LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds

Authors: Yusheng Dai, Vladimir Iashin, Kangdi Wang, Yuxuan Jiang, Ilpo Viertola, Weiqiang Wang, Qiuhong Ke, Min Zhao, 

Organizations: Monash University · ShengShu Technology · Tampere University · Alibaba Token Hub, Alibaba Group · 

Abstract

Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation

    Apr 26, 2026Chunyu Li, Jiaye Li, Ruiqiao Mei +4Audio-Video GenerationVideo Diffusion Models

  2. Vorch-Streamer: Extending Human Audio-Visual Generation to Real-Time Long-Form Streaming

    Aug 6, 2026Menglin Han, Yang Ding, Yulei Lu +6T2V GenerationLong-Horizon Video Generation

  3. AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer

    Mar 16, 2026Pengjun Fang, Yingqing He, Yazhou Xing +3Audio-Video GenerationAudio Generation