Current audio-driven video diffusion models can synthesize realistic talking-head videos but generally require dozens of diffusion denoising steps over multi-second video chunks, making them unsuitable for streaming applications such as voice assistants and virtual avatars, where low-latency and online causal generation are required properties. AURA generates video chunk by chunk with a rolling KV cache, requiring only the audio of the current chunk to start generating, rather than a multi-second audio look-ahead like bidirectional models. It comes in two sizes: AURA 1.3B streams 576p video in real time on a single GPU (20.7 fps for a 16 Hz playback rate), making it suited to interactive avatar applications, and AURA 5B streams 720p video at 9.7 fps. Our two-stage distillation pipeline first aligns the causal student with the teacher, by training it to match the teacher's trajectories with fewer diffusion steps and a chunk-wise sliding window causal attention mask, then applies Distribution Matching Distillation (DMD) with backward simulation to close the remaining quality gap. On standard talking-head benchmarks, AURA 1.3B runs 23x faster than its own bidirectional teacher and reaches a quality comparable to TalkVerse at a lower resolution, and AURA 5B matches its TalkVerse teacher at 720p while running 17x faster. To support reproducibility and further research on streaming avatar generation, we release the weights of both models along with a dataset of 30K synthetic videos and their precomputed teacher trajectories.
Figures & tables
Figure 1: AURA streams talking-head video chunk by chunk on a single GPU : AURA 5B at 720 p and 9.7 fps, AURA 1.3B at 576 p in real time ( 20.7 fps for 16 Hz playback). (Left) Two reference identities and a frame generated by each model at t=4 s, from the same reference image and audio. (Right) Quality–speed trade-off: FVD averaged over the Hallo3 test and CelebV-HQ test sets versus end-to-end time to generate 10 s of video on one H100 (log scale, lower is better). Circle areas are proportional to parameter counts, and the shaded band marks real-time generation. Video results can be found at kyutai.org/aura .
Figure 2: (Left) Sliding window chunk-wise causal attention mask used during training by the causal student. (Right) Per-chunk mixed denoising timestep training allows the use of a single shared KV cache across all denoising steps at inference.
Figure 3: Two-stage distillation pipeline . Stage 1 (top, ODE teacher forcing) : the frozen teacher generates a multi-step ODE trajectory from noise to clean latents, conditioned on a reference image, speech audio, and text prompt. A per-chunk mixed timestep state is sampled from this trajectory and fed to the causal student Gθ , which is trained to predict the corresponding clean latents under an MSE loss. Stage 2 (bottom, DMD with backward simulation) : the student generates a video in inference mode without gradients (backward simulation, left), and a per-chunk mixed-timestep state is sampled from its own denoising trajectory. The student is then re-run on this state with gradients enabled, and a DMD loss is computed by comparing the score predicted by a frozen teacher and an online fake score network (right). Gradients flow back through the gradient-enabled student pass only.
Method
Params
Res
Steps
DiT / VAE (fps) ↑
Total (fps) ↑
FID ↓
FVD ↓
JEDi ↓
CSIM ↑
Sync-C ↑
Sync-D ↓
WE ↓
Hallo3
14.5B
480p
50
0.24/3.2
0.22
42.8
183.4
0.180
0.74
5.06
9.9
0.0036
Wan-S2V
14B
720p
40
0.08/6.7
0.08
57.7
196.4
0.257
0.71
5.42
9.0
0.0032
TalkVerse
5B
720p
50
0.69/7.4
0.58
58.5
210.8
0.398
0.74
3.82
10.5
0.0019
Live Avatar
14B
480p
4
8.4/4.3
1.9
52.0
181.1
0.442
0.77
4.90
9.3
0.0039
AURA 5B (ours)
5B
720p
4
19.7/22.7
9.7
56.9
195.5
0.313
0.79
4.30
9.87
0.0032
AURA 1.3B (ours)
1.3B
480p
4
28.8/96.9
20.7
56.8
177.1
0.403
0.84
4.07
9.93
0.0036
Table 1: Main results on Hallo3 test . FVD and JEDi are averaged over the three temporal offsets ( @0 , @101 , @109 ); per-offset breakdowns are in the supplementary material (Tab. 8 ). WE : warping error. DiT / VAE : throughput of the diffusion transformer and VAE decoder alone, in frames per second on a single H 100 GPU. Total (fps): end-to-end throughput, including model offload and overhead. Bold : best; underlined : second best. AURA 1.3B outputs 480 p at 16 Hz (like Wan-S2V and Live Avatar), the other AURA and TalkVerse rows 25 Hz.
Method
Params
Res
Steps
DiT / VAE (fps) ↑
Total (fps) ↑
FID ↓
FVD ↓
JEDi ↓
CSIM ↑
Sync-C ↑
Sync-D ↓
WE ↓
Hallo3
14.5B
480p
50
0.24/3.2
0.22
39.1
202.5
0.243
0.71
6.23
9.4
0.0038
Wan-S2V
14B
720p
40
0.08/6.7
0.08
50.4
289.6
0.577
0.62
6.24
8.9
0.0061
TalkVerse
5B
720p
50
0.69/7.4
0.58
53.2
255.9
0.582
0.59
4.78
10.5
0.0059
Live Avatar
14B
480p
4
8.4/4.3
1.9
52.4
267.5
1.010
0.67
5.90
9.2
0.0057
AURA 5B (ours)
5B
720p
4
19.7/22.7
9.7
54.0
208.1
0.320
0.72
4.94
10.0
0.0043
AURA 1.3B (ours)
1.3B
480p
4
28.8/96.9
20.7
51.5
248.8
0.574
0.77
4.76
10.1
0.0046
Table 2: Main results on CelebV-HQ test . FVD and JEDi are averaged over the three temporal offsets ( @0 , @101 , @109 ); per-offset breakdowns are in supplementary material. Columns and naming conventions follow the same as in Table 1 .
Figure 4: Human preference study. Per-axis Bayesian Elo ( ↑ ) from 400 pairwise comparisons by 27 raters between AURA 5B, AURA 1.3B, TalkVerse and Hallo3 on the Hallo3 test clips; error bars are 95% CIs.
#Steps
FID ↓
FVD ↓
JEDi ↓
CSIM ↑
Sync-C ↑
Sync-D ↓
WE ↓
Sharp. ↑
DiT (fps) ↑
4
56.9
195.5
0.31
0.79
4.30
9.87
0.0032
50.4
19.7
3
55.4
198.8
0.33
0.77
4.39
9.77
0.0031
37.5
26.0
2
55.5
198.2
0.37
0.79
4.39
9.81
0.0031
36.5
38.5
1
58.6
229.2
0.66
0.65
4.26
9.71
0.0030
27.0
75.8
Table 3: Effect of the number of diffusion steps per chunk on quality and speed. DiT throughput is reported in frames per second on a single H 100 . FVD and JEDi are averaged over three temporal offsets ( @0 , @101 , @109 ). Other experiments use 4 steps. Best in bold .
Variant
FID ↓
FVD ↓
JEDi ↓
CSIM ↑
Sync-C ↑
Sync-D ↓
WE ↓
Sharp. ↑
Stage 1 only
68.3
264.8
0.58
0.55
3.55
10.07
0.0023
18.3
Stage 1 + Stage 2
56.9
195.5
0.31
0.79
4.30
9.87
0.0032
50.4
Table 4: Effect of Stage 2 on generation quality . Adding the DMD distillation stage substantially improves perceptual quality (FID, FVD, JEDi), identity preservation (CSIM), and lip-sync. Warp reaches lower values with Stage 1 only, but this is the same metric artifact on blurrier outputs as observed in Tab. 3 (see Sec. 4.4 ). Best value per column in bold .
Figure 5: Qualitative results from AURA on two diverse test sequences with different audios, generated by AURA 5B (top) and AURA 1.3B (bottom). Each row shows the reference identity image (left) and three frames sampled from the generated video at approximately t=1,3,5 s. The samples illustrate the diversity of inputs AURA handles, including varied lighting, head poses, and identities. Figure 1 shows both models on the same inputs; full videos, more results and comparisons with the baselines are on the project page .
Variant
FID ↓
FVD ↓
JEDi ↓
CSIM ↑
Sync-C ↑
WE ↓
Δ Col. ↓
DiT (fps) ↑
4 steps (default)
56.9
195.5
0.31
0.79
4.30
0.0032
0.4
19.7
5-step ( t=0 refresh)
57.7
196.7
0.33
0.78
4.26
0.0032
0.4
15.6
5-step ( t=50 refresh)
54.5
210.0
0.38
0.74
4.29
0.0038
3.1
15.6
5-step ( t=50 , used)
54.5
209.8
0.37
0.74
4.31
0.0038
3.1
15.6
Table 5: KV cache refresh strategy . Effect of an additional inference step to refresh it. Best value per column in bold .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Stage 1
Stage 2 (student)
Stage 2 (critic)
Training steps
120 K
≈833
≈4,167
Global batch size
256
256
256
Learning rate
1e-5
3.5e-7
3.5e-7
Optimizer
AdamW
AdamW
AdamW
Optimizer (β1,β2)
(0.9, 0.999)
(0.9, 0.999)
(0.9, 0.999)
Weight decay
0.01
0.01
0.01
Appendix
Table 6: AURA 5B training hyperparameters . Stage 1 trains the student via ODE teacher forcing; Stage 2 trains the student and critic jointly via DMD with backward simulation, alternating 5 critic updates per student update. All trainings use LoRA adapters with the optimizer settings below.
Teacher
Stage 1
Stage 2 (student)
Stage 2 (critic)
Data
23 K real Hallo3 clips
30 K teacher trajectories
92 K conditions
Training steps
60 K
40 K
≈1,083
≈5,417
Global batch size
256
256
256
256
Learning rate
1e-5
1e-5
2e-7
2e-7
Gradient clipping
—
—
0.5
Initialization
Wan2.1 T2V 1.3 B
Teacher
Stage 1 merged
Teacher
Appendix
Table 7: AURA 1.3B training hyperparameters. Same optimizer settings as Tab. 6 (AdamW, LoRA on attention, rank and α128 , bf16); 5 critic updates per student update in Stage 2.
Method
Params
Res
Steps
DiT / VAE (fps) ↑
Total (fps) ↑
FID ↓
FVD @ (0/101/109 ) ↓
JEDi @( 0/101/109 ) ↓
CSIM ↑
Sync-C ↑
Sync-D ↓
WE ↓
Hallo3
14.5B
480p
50
0.24/3.2
0.22
42.8
185.4 / 184.7 / 180.1
0.25 / 0.15 / 0.14
0.74
5.06
9.9
0.0036
Wan-S2V
14B
720p
40
0.08/6.7
0.08
57.7
186.0 / 203.1 / 200.2
0.22 / 0.28 / 0.27
0.71
5.42
9.0
0.0032
TalkVerse
5B
720p
50
0.69/7.4
0.58
58.5
222.6 / 212.1 / 197.5
0.40 / 0.43 / 0.37
0.74
3.82
10.5
0.0019
Live Avatar
14B
480p
4
8.4/4.3
1.9
52.0
179.8 / 183.8 / 179.7
0.30 / 0.51 / 0.52
0.77
4.90
9.3
0.0039
AURA 5B (ours)
5B
720p
4
19.7/22.7
9.7
56.9
198.1 / 188.2 / 200.2
0.51 / 0.22 / 0.21
0.79
4.30
9.87
0.0032
AURA 1.3B (ours)
1.3B
480p
4
28.8/96.9
20.7
56.8
172.3 / 179.9 / 179.2
0.45 / 0.35 / 0.41
0.84
4.07
9.93
0.0036
Appendix
Table 8: Per-offset breakdown on Hallo3 test , complementing the averaged version in the main paper (Tab. 1 ). FVD and JEDi are reported as triplets at temporal offsets at three different (temporal) frames: @0/@101/@109 . WE : warping error. DiT / VAE : throughput of the diffusion transformer and VAE decoder alone, in frames per second on a single H 100 GPU. Total (fps): end-to-end throughput, including model offload and overhead. Bold : best; underlined : second best (applied per offset within each triplet). AURA 1.3B outputs 480 p at 16 Hz (like Wan-S2V and Live Avatar), the other AURA and TalkVerse rows 25 Hz.
Method
Params
Res
Steps
fps ↑
FID ↓
FVD ( @0/@101/@109 ) ↓
JEDi ( @0/@101/@109 ) ↓
CSIM ↑
Sync-C/D
WE ↓
Hallo3
14.5 B
480 p
50
0.22
39.1
192.2 / 208.4 / 206.9
0.26 / 0.25 / 0.22
0.71
6.23 / 9.4
0.0038
Wan-S2V
14 B
720 p
40
0.08
50.4
294.3 / 288.7 / 285.9
0.66 / 0.52 / 0.55
0.62
6.24 / 8.9
0.0061
TalkVerse
5 B
720 p
50
0.58
53.2
233.2 / 268.9 / 265.5
0.33 / 0.66 / 0.75
0.59
4.78 / 10.5
0.0059
Live Avatar
14 B
480 p
4
1.90
52.4
237.2 / 281.4 / 283.9
0.48 / 1.26 / 1.29
0.67
5.90 / 9.2
0.0057
AURA 5B (ours)
5 B
720 p
4
9.7
54.0
182.0 / 219.4 / 223.0
0.33 / 0.27 / 0.36
0.72
4.94 / 10.0
0.0043
AURA 1.3B (ours)
1.3 B
480 p
4
20.7
51.5
243.1 / 251.0 / 252.2
0.57 / 0.53 / 0.62
0.77
4.76 / 10.1
0.0046
Appendix
Table 9: Per-offset breakdown on CelebV-HQ test , complementing the averaged version in the main paper. FVD and JEDi are reported as triplets at temporal offsets @0/@101/@109 . WE : warping error. Sync-C/D : Sync-C ( ↑ ) / Sync-D ( ↓ ). fps : end-to-end throughput on a single H100 GPU for a 10 s clip; per-component DiT and VAE values appear in the main paper. Bold : best; underlined : second best (per offset within each triplet).
Figure 6: Quality–speed trade-off across audio-driven avatar generation models . The horizontal axis reports end-to-end generation time for 10 s of video on a single H 100 GPU (log scale, lower is better). The vertical axis reports FVD ( left ) and JEDi ( right ), averaged over the Hallo3 and CelebV-HQ test sets (lower is better). Bubble area is proportional to the number of model parameters. AURA ( green ) achieves quality comparable to multi-step baselines at a fraction of their generation time.
Metric
0–10s
10–20s
20–30s
30–40s
40–50s
50–60s
Mean
AURA 5B
FVD ↓
182.1
206.0
202.8
198.3
202.2
195.2
197.8
CSIM ↑
0.764
0.735
0.720
0.710
0.657
0.689
0.713
Sync-C ↑
5.00
5.22
5.00
4.85
4.74
4.79
4.93
Warp ↓
3.50
3.44
3.45
3.43
3.99
3.53
3.56
AURA 1.3B
Appendix
Table 10: Long-form evaluation. AURA 5B and AURA 1.3B ( 4 steps, fixed KV cache) on the 100 Hallo3 test references driven by 60 s of speech; metrics per 10 s segment (FVD: mean of three windows per segment). Warping error in 10−3 .
Figure 7: One minute of streaming. (Top) Frames of one Hallo3 test clip at t=0,10,…,60 s for AURA 5B and AURA 1.3B, next to the reference image. (Bottom) FVD, CSIM and Sync-C per 10 s segment over the 100 references (Tab. 10 ).
Figure 8: RoPE re-indexing of the reference frame. (Top) just after a re-indexation, the reference (orange) sits 40 latent positions ahead of the generation front. (Middle) after 30 generated frames, the reference is 10 positions ahead — the minimum offset reached during training. (Bottom) the reference is shifted by +30 , restoring the maximum +40 offset. The offset thus cycles in [+10,+40] , staying within the training-time regime.
Variant
FID ↓
FVD ↓
JEDi ↓
CSIM ↑
Sync-C ↑
Warp ↓
DiT (fps) ↑
n=1 (default)
56.9
195.5
0.31
0.79
4.30
0.0032
20
n=2
56.8
193.2
0.37
0.78
4.20
0.0035
18
Appendix
Table 11: Rolling cache window size. Effect of the number n of retained past chunks on quality and speed, on Hallo3 test ( 10 s clips). Enlarging the window from n=1 to n=2 slightly improves FVD but degrades JEDi, CSIM, Sync-C and WE, while increasing cache size and reducing throughput. FVD and JEDi are averaged over the three temporal offsets ( @0 , @101 , @109 ). Best per column in bold .
Figure 9: Qualitative comparison on the Hallo3 test set . Despite stronger FID/FVD scores (Table 1 ), Hallo3 generations exhibit recurrent hand artifacts ( bottom ) that distribution-based metrics fail to penalize. AURA ( top ) produces cleaner outputs at a fraction of the generation time.
Figure 10: Qualitative comparison between AURA 5B and its TalkVerse teacher . Given the same reference image ( left ) and a 10 s audio input, AURA 5B ( bottom ) generates video of comparable visual quality to the teacher ( top ) while running approximately 17× faster end-to-end on a single H100 GPU. Full videos are on the project page .
Production-ready audio-driven avatar generation requires efficient inference without sacrificing fidelity or motion expressiveness. However, existing acceleration methods often compromise quality through restrictive architectural choices, such as causal attention and short temporal horizons, or by reducing model capacity and resolution. Without such compromises, we propose AptAvatar, a 14B-parameter long-form audio-driven avatar generation framework that delivers fast and expressive inference. For efficiency in production-level applications, AptAvatar addresses the extreme two-step generation challenge. To bridge the gap between the multi-step teacher model and the two-step student model, we introduce Endpoint-Anchored Distribution Distillation. It augments vanilla distribution matching with a dedicated Anchor Score Estimator trained on the trajectory-endpoint distribution defined from a frozen pretrained 4-step bridge generator. This provides an attainable endpoint-level anchor for the evolving two-step student. To improve long-horizon consistency, we further introduce Self-Generated History Replay, which reuses cached outputs from earlier generator checkpoints as history conditions during chunk-wise training. This approximates inference-time conditioning on self-generated histories without costly online rollouts, mitigating quality degradation from accumulated history errors. Extensive experiments demonstrate that AptAvatar generates vivid 720p long-form avatar videos with only 2 NFEs, achieving a 60x speedup while preserving visual fidelity and long-horizon identity. Code is available at https://github.com/TaoLiveAIGC/AptAvatar
Real-time talking avatar generation requires low latency and minute-level temporal stability. Autoregressive (AR) forcing enables streaming inference but suffers from exposure bias, which causes errors to accumulate and become irreversible over long rollouts. In contrast, full-sequence diffusion transformers mitigate drift but remain computationally prohibitive for real-time long-form synthesis. We present AvatarForcing, a one-step streaming diffusion framework that denoises a fixed local-future window with heterogeneous noise levels and emits one clean block per step under constant per-step cost. To stabilize unbounded streams, the method introduces dual-anchor temporal forcing: a style anchor that re-indexes RoPE to maintain a fixed relative position with respect to the active window and applies anchor-audio zero-padding, and a temporal anchor that reuses recently emitted clean blocks to ensure smooth transitions. Real-time one-step inference is enabled by two-stage streaming distillation with offline ODE backfill and distribution matching. Experiments on standard benchmarks and a new 400-video long-form benchmark show strong visual quality and lip synchronization at 34 ms/frame using a 1.3B-parameter student model for realtime streaming. Our page is available at: https://cuiliyuan121.github.io/AvatarForcing/
Liyuan Cui, Wentao Hu, Wenyuan Zhang +3
Zhejiang University · Kling Team, Kuaishou Technology · Beijing University of Posts and Telecommunications +1
Real-time long-form avatar audio-video generation requires causal, continuous synthesis while maintaining audiovisual synchronization and visual consistency. Adapting a pretrained bidirectional model to this setting presents two key dilemmas. First, autoregressively reusing generated blocks as context creates exposure bias, causing errors and visual drift to accumulate over long rollouts. Second, a global speech utterance does not indicates a causal generator which portion should be spoken next when only limited local audio-video context is available. We present Vorch-Streamer, a post-training framework that addresses these challenges and enables real-time long-form Text-to-Audio-Video (T2AV) streaming. We construct a synthetic corpus of 80K avatar clips spanning 12-21 seconds and first train a causal generator with mixed Teacher Forcing and Diffusion Forcing. We then apply long-horizon Self Forcing with DMD distillation, exposing the model to its own rollout distribution while preserving the quality of the pretrained bidirectional teacher. To explicitly control speech progression, an external language model predicts discrete 25-Hz speech-planning tokens, whose continuous features condition the audio diffusion branch and align each causal block with the content it should speak. With bounded causal context and four-step denoising, Vorch-Streamer jointly generates audio and video from text at 27.12 FPS, exceeding the 24-FPS real-time playback rate while maintaining competitive audio-lip synchronization and strong identity preservation over long-form generation.
Menglin Han, Yang Ding, Yulei Lu +6
Vorch Team · Tongji University · Harbin Institute of Technology, Shenzhen +1