LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds
Authors: Yusheng Dai, Vladimir Iashin, Kangdi Wang, Yuxuan Jiang, Ilpo Viertola, Weiqiang Wang, Qiuhong Ke, Min Zhao, Jianfei Cai
Organizations: Monash University · ShengShu Technology · Tampere University · Alibaba Token Hub, Alibaba Group · Tsinghua University · Nanjing University
Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/
Figures & tables
Figure 1: (a) User study: the minimum audio delay at which participants notice the sound lagging behind the video, for action sounds, speech, and ambient sounds in free roaming (mean ± 95% CI). (b) Empirical study of streaming visual frontends on the V-AURA [ 54 ] backbone: the traditional chunked window (Figure 2 a) cuts latency by shrinking the window w at a sharp cost in audio-visual alignment, while our trailing sliding window (Figure 2 b) shortens the hop s and preserves alignment.
Figure 2: Streaming visual frontend. Each dot denotes the token at one window position. (a) Chunked window: shrinking the window w lowers latency but also narrows the receptive field, which degrades audio-visual alignment. (b) Trailing sliding window: anchoring the window tail to the latest frame reduces look-ahead to one hop while preserving the full receptive field w and hence the alignment. Black dots are emitted tokens at newly arrived frames. Gray and dashed dots correspond to earlier frames and first-frame padding, respectively; these frames provide context only within the encoder, and their tokens are discarded. (c) Unified Varhop Framework: a Synchronizer maps tokens from different hop sizes into a shared latent space, allowing a single backend to switch among hop sizes.
Figure 3: Continuous autoregressive audio generation backbone. Left: at each 12.5 Hz step, the causal LM fuses the current visual token vi and the LocEnc embedding ei−1 of the previous clean patch xi−10 into a hidden state hi , on which Independent LocDiT denoises the current patch xit in isolation. Right: History LocDiT (top) adds clean patches xi−20 and xi−10 as context without extra look-ahead. Future-Warmup LocDiT (bottom) jointly denoises xit and a warmup patch xi+1t , which is conditioned on hi and vi+1 and discarded at inference.
Type
Method
Frontend Look-ahead (s)
Distribution Matching
Audio Quality
Modality Alignment
FDPANNs ↓
FDPaSST ↓
KLPANNs ↓
IS ↑
PQ ↑
DeSync ↓
IB ↑
Offline
Frieren [ 57 ]
8.00
12.13
123.75
2.73
11.32
5.82
0.86
0.21
AudioX [ 51 ]
10.00
12.73
121.82
2.96
13.34
6.17
1.22
0.24
MMAudio-L [ 4 ]
8.00
8.48
74.16
2.18
11.62
6.18
0.47
0.31
Online
MMAudio (online)
0.64
9.82
175.92
2.11
10.11
5.93
0.71
0.29
V-AURA [ 54 ]
0.64
16.43
245.25
2.28
10.82
5.74
0.69
0.28
Table 1: System-level comparison on the VGGSound test set. Frontend look-ahead is the duration of video that must be buffered before the visual encoder can encode a new chunk or hop; offline models see the entire clip before generating. MMAudio (online) is our training-free online adaptation of MMAudio-L (Appendix D ). Bold and underline mark the best and second-best scores within each group (offline and online); gray entries are quoted from the original paper (no official checkpoint) and are not ranked. The frontend look-ahead of WorldSonus (0.10 s) equals three frames at 30 fps. LiveFoley uses History LocDiT as the default with s=1 for both the 16 kHz and 48 kHz variants.
Figure 4: Subjective evaluation of LiveFoley against offline and online baselines: mean opinion scores for audio quality (MOS-Q, left) and audio-visual alignment (MOS-A, right). The dashed line separates offline from online systems. Protocol details are given in Appendix G .
Type
Visual encoder
τ (s)
FDPANNs ↓
FDPaSST ↓
KLPANNs ↓
DeSync ↓
Frame
SigLIP2
0.08
27.92
267.44
1.91
1.14
DINOv2
0.08
28.25
278.18
2.10
1.10
Segment
VideoMAE-V2
0.64
30.37
281.62
2.36
0.88
Synchformer (ours)
0.08
28.10
263.04
2.01
0.61
Table 2: Visual frontend comparison on VGGSound-Curated. Only the visual encoder changes; the V-AURA backbone, decoding, and training budget are identical. τ is the look-ahead of each encoder. Bold and underline mark the best and second-best scores.
s ( τ )
Model
FDPANNs ↓
FDPaSST ↓
KLPANNs ↓
DeSync ↓
IB ↑
1 (0.08)
Expert
17.07
140.22
1.78
0.47
0.33
Unified (ours)
16.94
145.02
1.90
0.47
0.32
w/o Synchronizer
17.31
147.76
1.88
0.50
0.32
2 (0.16)
Expert
16.36
139.88
1.83
0.45
0.32
Unified (ours)
16.88
143.92
1.86
0.46
0.32
w/o Synchronizer
16.58
146.44
1.90
0.49
0.32
Table 3: Ablation of the Unified Varhop Framework on VGGSound-Curated. At each hop size s , our unified model is compared with a single-hop expert and with a unified variant trained without the Synchronizer. τ is the frontend look-ahead.
LocDiT window
FDPANNs ↓
FDPaSST ↓
KLPANNs ↓
IS ↑
DeSync ↓
IB ↑
Independent LocDiT
21.08
154.62
1.94
12.01
0.47
0.31
Low-Rate LocDiT
18.90
180.52
2.01
10.38
0.54
0.30
History LocDiT
17.55
158.04
1.92
12.00
0.47
0.32
Future-Warmup LocDiT
16.88
143.92
1.86
12.13
0.46
0.32
Table 4: LocDiT ablation on VGGSound-Curated, comparing four denoising window designs under the same backbone. Bold and underline mark the best and second-best scores.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Domain
Clips
Hours
SoundAtlas (VGGSound)
in-the-wild
146K
401
SoundAtlas (AudioSet)
in-the-wild
340K
929
EPIC-Sounds
egocentric
49K
134
Ego4D-Sounds
egocentric
210K
578
Total
745K
2,042
Appendix
Table 5: Training data composition.
Figure 5: Training-free online adaptation of MMAudio (first three ticks shown). At each tick, the pretrained bidirectional flow-matching (FM) model is rerun over its full 8 s window, conditioned only on the video that has arrived. Committed audio chunks are pinned during denoising, the next chunk is emitted after the run, and the remaining chunks are discarded and regenerated at the next tick. Dashed arrows indicate that the same initial noise is carried across ticks.
Table 6: End-to-end latency breakdown on a single A800 GPU (default setting: History LocDiT, hop s=1 ). The end-to-end latency is measured with a real-time simulator that releases video frames at 25 fps.
LocDiT
s=1
s=2
s=4
History
0.08/0.00/ 156
0.16/0.00/236
0.32/0.00/396
Future-Warmup
0.08/0.08/236
0.16/0.08/316
0.32/0.08/476
Appendix
Table 7: Frontend look-ahead, backend look-ahead, and end-to-end latency for each hop size s , reported as frontend look-ahead (s) / backend look-ahead (s) / end-to-end latency (ms). Only History LocDiT at s=1 (156 ms, bold) is measured; the other latencies are obtained by adding the extra algorithmic look-ahead to this measurement, assuming the same per-patch compute.
Method
FDPANNs ↓
FDPaSST ↓
KLPANNs ↓
IS ↑
PQ ↑
DeSync ↓
IB ↑
WorldSonus [ 10 ] w/o text
16.32
197.68
2.54
8.73
6.11
0.67
0.26
WorldSonus [ 10 ] w/ text
9.96
193.75
1.51
9.89
6.19
0.76
0.26
LiveFoley-16k w/o text (ours)
9.61
90.54
2.47
11.36
6.44
0.47
0.29
LiveFoley-16k w/ text (ours)
7.30
82.51
1.58
11.88
6.36
0.50
0.30
LiveFoley-48k w/o text (ours)
10.16
129.48
2.45
10.95
6.39
0.48
0.28
LiveFoley-48k w/ text (ours)
7.34
122.86
1.61
11.21
6.38
0.49
0.29
Appendix
Table 8: Text-conditioning comparison on the VGGSound test set. For each of LiveFoley -16k and LiveFoley -48k, both rows come from the same checkpoint, trained on a mixture of text-conditioned and video-only samples and evaluated without and with per-clip text prompts. Their w/o-text scores therefore differ from those in Table 1 , whose LiveFoley model is trained on video only. All other evaluation settings follow Table 1 ; bold marks the best score in each column.
Real-time text-driven joint audio-video avatar generation requires jointly synthesizing portrait video and speech with high fidelity and precise synchronization, yet existing audio-visual diffusion models remain too slow for interactive use and often degrade noticeably after aggressive acceleration. We present Hallo-Live, a streaming framework for joint audio-visual avatar generation that combines asynchronous dual-stream diffusion with human-centric preference-guided distillation. To reduce articulation lag in causal generation, we introduce Future-Expanding Attention, which allows each video block to access synchronous audio together with a short horizon of future phonetic cues. To mitigate the quality loss of few-step distillation, we further propose Human-Centric Preference-Guided DMD (HP-DMD), which reweights training samples using rewards from visual fidelity, speech naturalness, and audio-visual synchronization. On two NVIDIA H200 GPUs, Hallo-Live runs at 20.38 FPS with 0.94 seconds latency, yielding 16.0x higher throughput and 99.3x lower latency than the teacher model Ovi. Despite this speedup, it retains strong generation quality, reaching comparable VideoAlign overall score and Sync Confidence score while outperforming other accelerated baselines in the overall quality-efficiency trade-off. Qualitative results further show robust generalization across photorealistic, multi-speaker, and stylized scenarios. To the best of our knowledge, Hallo-Live is the first framework to combine streaming dual-stream diffusion with preference-guided distillation for real-time, text-driven audio-visual generation.
Chunyu Li, Jiaye Li, Ruiqiao Mei +4
Shanghai Innovation Institute · Fudan University · University of Science and Technology of China +2
Real-time long-form avatar audio-video generation requires causal, continuous synthesis while maintaining audiovisual synchronization and visual consistency. Adapting a pretrained bidirectional model to this setting presents two key dilemmas. First, autoregressively reusing generated blocks as context creates exposure bias, causing errors and visual drift to accumulate over long rollouts. Second, a global speech utterance does not indicates a causal generator which portion should be spoken next when only limited local audio-video context is available. We present Vorch-Streamer, a post-training framework that addresses these challenges and enables real-time long-form Text-to-Audio-Video (T2AV) streaming. We construct a synthetic corpus of 80K avatar clips spanning 12-21 seconds and first train a causal generator with mixed Teacher Forcing and Diffusion Forcing. We then apply long-horizon Self Forcing with DMD distillation, exposing the model to its own rollout distribution while preserving the quality of the pretrained bidirectional teacher. To explicitly control speech progression, an external language model predicts discrete 25-Hz speech-planning tokens, whose continuous features condition the audio diffusion branch and align each causal block with the content it should speak. With bounded causal context and four-step denoising, Vorch-Streamer jointly generates audio and video from text at 27.12 FPS, exceeding the 24-FPS real-time playback rate while maintaining competitive audio-lip synchronization and strong identity preservation over long-form generation.
Menglin Han, Yang Ding, Yulei Lu +6
Vorch Team · Tongji University · Harbin Institute of Technology, Shenzhen +1
Existing video-to-audio (V2A) generation methods predominantly rely on text prompts alongside visual information to synthesize audio. However, two critical bottlenecks persist: semantic granularity gaps in training data, such as conflating acoustically distinct sounds under coarse labels, and textual ambiguity in describing micro-acoustic features. These bottlenecks make it difficult to perform fine-grained sound synthesis using text-controlled modes. To address these limitations, we propose AC-Foley, an audio-conditioned V2A model that directly leverages reference audio to achieve precise and fine-grained control over generated sounds. This approach enables fine-grained sound synthesis, timbre transfer, zero-shot sound generation, and improved audio quality. By directly conditioning on audio signals, our approach bypasses the semantic ambiguities of text descriptions while enabling precise manipulation of acoustic attributes. Empirically, AC-Foley achieves state-of-the-art performance for Foley generation when conditioned on reference audio, while remaining competitive with state-of-the-art video-to-audio methods even without audio conditioning. Code and demo are available at: https://ff2416.github.io/AC-Foley-Page
Pengjun Fang, Yingqing He, Yazhou Xing +3
The Hong Kong University of Science and Technology · University of Central Florida