LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds
Authors: Yusheng Dai, Vladimir Iashin, Kangdi Wang, Yuxuan Jiang, Ilpo Viertola, Weiqiang Wang, Qiuhong Ke, Min Zhao, Jianfei Cai
Organizations: Monash University · ShengShu Technology · Tampere University · Alibaba Token Hub, Alibaba Group · Tsinghua University · Nanjing University
Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/
Figures & tables
Figure 1: (a) User study: the minimum audio delay at which participants notice the sound lagging behind the video, for action sounds, speech, and ambient sounds in free roaming (mean ± 95% CI). (b) Empirical study of streaming visual frontends on the V-AURA [ 54 ] backbone: the traditional chunked window (Figure 2 a) cuts latency by shrinking the window w at a sharp cost in audio-visual alignment, while our trailing sliding window (Figure 2 b) shortens the hop s and preserves alignment.
Figure 2: Streaming visual frontend. Each dot denotes the token at one window position. (a) Chunked window: shrinking the window w lowers latency but also narrows the receptive field, which degrades audio-visual alignment. (b) Trailing sliding window: anchoring the window tail to the latest frame reduces look-ahead to one hop while preserving the full receptive field w and hence the alignment. Black dots are emitted tokens at newly arrived frames. Gray and dashed dots correspond to earlier frames and first-frame padding, respectively; these frames provide context only within the encoder, and their tokens are discarded. (c) Unified Varhop Framework: a Synchronizer maps tokens from different hop sizes into a shared latent space, allowing a single backend to switch among hop sizes.
Figure 3: Continuous autoregressive audio generation backbone. Left: at each 12.5 Hz step, the causal LM fuses the current visual token vi and the LocEnc embedding ei−1 of the previous clean patch xi−10 into a hidden state hi , on which Independent LocDiT denoises the current patch xit in isolation. Right: History LocDiT (top) adds clean patches xi−20 and xi−10 as context without extra look-ahead. Future-Warmup LocDiT (bottom) jointly denoises xit and a warmup patch xi+1t , which is conditioned on hi and vi+1 and discarded at inference.
Type
Method
Frontend Look-ahead (s)
Distribution Matching
Audio Quality
Modality Alignment
FDPANNs ↓
FDPaSST ↓
KLPANNs ↓
IS ↑
PQ ↑
DeSync ↓
IB ↑
Offline
Frieren [ 57 ]
8.00
12.13
123.75
2.73
11.32
5.82
0.86
0.21
AudioX [ 51 ]
10.00
12.73
121.82
2.96
13.34
6.17
1.22
0.24
MMAudio-L [ 4 ]
8.00
8.48
74.16
2.18
11.62
6.18
0.47
0.31
Online
MMAudio (online)
0.64
9.82
175.92
2.11
10.11
5.93
0.71
0.29
V-AURA [ 54 ]
0.64
16.43
245.25
2.28
10.82
5.74
0.69
0.28
Table 1: System-level comparison on the VGGSound test set. Frontend look-ahead is the duration of video that must be buffered before the visual encoder can encode a new chunk or hop; offline models see the entire clip before generating. MMAudio (online) is our training-free online adaptation of MMAudio-L (Appendix D ). Bold and underline mark the best and second-best scores within each group (offline and online); gray entries are quoted from the original paper (no official checkpoint) and are not ranked. The frontend look-ahead of WorldSonus (0.10 s) equals three frames at 30 fps. LiveFoley uses History LocDiT as the default with s=1 for both the 16 kHz and 48 kHz variants.
Figure 4: Subjective evaluation of LiveFoley against offline and online baselines: mean opinion scores for audio quality (MOS-Q, left) and audio-visual alignment (MOS-A, right). The dashed line separates offline from online systems. Protocol details are given in Appendix G .
Type
Visual encoder
τ (s)
FDPANNs ↓
FDPaSST ↓
KLPANNs ↓
DeSync ↓
Frame
SigLIP2
0.08
27.92
267.44
1.91
1.14
DINOv2
0.08
28.25
278.18
2.10
1.10
Segment
VideoMAE-V2
0.64
30.37
281.62
2.36
0.88
Synchformer (ours)
0.08
28.10
263.04
2.01
0.61
Table 2: Visual frontend comparison on VGGSound-Curated. Only the visual encoder changes; the V-AURA backbone, decoding, and training budget are identical. τ is the look-ahead of each encoder. Bold and underline mark the best and second-best scores.
s ( τ )
Model
FDPANNs ↓
FDPaSST ↓
KLPANNs ↓
DeSync ↓
IB ↑
1 (0.08)
Expert
17.07
140.22
1.78
0.47
0.33
Unified (ours)
16.94
145.02
1.90
0.47
0.32
w/o Synchronizer
17.31
147.76
1.88
0.50
0.32
2 (0.16)
Expert
16.36
139.88
1.83
0.45
0.32
Unified (ours)
16.88
143.92
1.86
0.46
0.32
w/o Synchronizer
16.58
146.44
1.90
0.49
0.32
Table 3: Ablation of the Unified Varhop Framework on VGGSound-Curated. At each hop size s , our unified model is compared with a single-hop expert and with a unified variant trained without the Synchronizer. τ is the frontend look-ahead.
LocDiT window
FDPANNs ↓
FDPaSST ↓
KLPANNs ↓
IS ↑
DeSync ↓
IB ↑
Independent LocDiT
21.08
154.62
1.94
12.01
0.47
0.31
Low-Rate LocDiT
18.90
180.52
2.01
10.38
0.54
0.30
History LocDiT
17.55
158.04
1.92
12.00
0.47
0.32
Future-Warmup LocDiT
16.88
143.92
1.86
12.13
0.46
0.32
Table 4: LocDiT ablation on VGGSound-Curated, comparing four denoising window designs under the same backbone. Bold and underline mark the best and second-best scores.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Domain
Clips
Hours
SoundAtlas (VGGSound)
in-the-wild
146K
401
SoundAtlas (AudioSet)
in-the-wild
340K
929
EPIC-Sounds
egocentric
49K
134
Ego4D-Sounds
egocentric
210K
578
Total
745K
2,042
Appendix
Table 5: Training data composition.
Figure 5: Training-free online adaptation of MMAudio (first three ticks shown). At each tick, the pretrained bidirectional flow-matching (FM) model is rerun over its full 8 s window, conditioned only on the video that has arrived. Committed audio chunks are pinned during denoising, the next chunk is emitted after the run, and the remaining chunks are discarded and regenerated at the next tick. Dashed arrows indicate that the same initial noise is carried across ticks.
Table 6: End-to-end latency breakdown on a single A800 GPU (default setting: History LocDiT, hop s=1 ). The end-to-end latency is measured with a real-time simulator that releases video frames at 25 fps.
LocDiT
s=1
s=2
s=4
History
0.08/0.00/ 156
0.16/0.00/236
0.32/0.00/396
Future-Warmup
0.08/0.08/236
0.16/0.08/316
0.32/0.08/476
Appendix
Table 7: Frontend look-ahead, backend look-ahead, and end-to-end latency for each hop size s , reported as frontend look-ahead (s) / backend look-ahead (s) / end-to-end latency (ms). Only History LocDiT at s=1 (156 ms, bold) is measured; the other latencies are obtained by adding the extra algorithmic look-ahead to this measurement, assuming the same per-patch compute.
Method
FDPANNs ↓
FDPaSST ↓
KLPANNs ↓
IS ↑
PQ ↑
DeSync ↓
IB ↑
WorldSonus [ 10 ] w/o text
16.32
197.68
2.54
8.73
6.11
0.67
0.26
WorldSonus [ 10 ] w/ text
9.96
193.75
1.51
9.89
6.19
0.76
0.26
LiveFoley-16k w/o text (ours)
9.61
90.54
2.47
11.36
6.44
0.47
0.29
LiveFoley-16k w/ text (ours)
7.30
82.51
1.58
11.88
6.36
0.50
0.30
LiveFoley-48k w/o text (ours)
10.16
129.48
2.45
10.95
6.39
0.48
0.28
LiveFoley-48k w/ text (ours)
7.34
122.86
1.61
11.21
6.38
0.49
0.29
Appendix
Table 8: Text-conditioning comparison on the VGGSound test set. For each of LiveFoley -16k and LiveFoley -48k, both rows come from the same checkpoint, trained on a mixture of text-conditioned and video-only samples and evaluated without and with per-clip text prompts. Their w/o-text scores therefore differ from those in Table 1 , whose LiveFoley model is trained on video only. All other evaluation settings follow Table 1 ; bold marks the best score in each column.