Audio-driven streaming avatar generation requires real-time synthesis of speech-synchronized videos with dynamic and diverse motion. Self Forcing uses Distribution Matching Distillation (DMD) to distill bidirectional video diffusion models into causal, few-step generators for real-time streaming. However, DMD minimizes a reverse KL divergence, which is inherently mode-seeking: it causes the student to discard high-dynamic modes and collapse onto static outputs, compressing both dynamics and diversity of generated videos. We find that this collapse is region-heterogeneous: person regions involving pose and gesture variations suffer the largest diversity loss, the audio-driven mouth region shows a small loss, and the background remains nearly stable. Based on this observation, we propose Routed Forcing, which routes the distillation objective by semantic region and noise stage to improve dynamics and diversity while preserving visual quality. Specifically, (1) Where to Force: Semantic-Region Routing applies Data-Forcing Distillation (DFD), which supervises the student with real videos, to the person region where diversity collapse is most severe, while retaining DMD for the mouth and background to preserve lip synchronization and scene stability. (2) When to Force: Noise-Stage Routing activates DFD at high noise stages, where real video serves as effective supervision to inject diverse and dynamic motion patterns. At low noise stages, DMD is used to refine details, avoiding blur and artifacts from spatial differences between real video and student-generated video. Experiments show that Routed Forcing improves dynamics by up to 45% and diversity by 7-25% over Self Forcing, while preserving video quality and lip synchronization.
Figures & tables
Figure 1: DMD compresses dynamics and diversity, and the collapse is region-heterogeneous. We use OmniAvatar as teacher model and LiveTalk, distilled from OmniAvatar via DMD, as student model. (a) Given the same first frame and audio, the teacher produces diverse expressions and dynamic gestures across two seeds, while the student generates similar static trajectories. (b) Regional diversity (mean pairwise LPIPS across four seeds over 100 first-frame-and-audio pairs from SpeakerVid) reveals that the non-mouth person region suffers the largest drop, the mouth region shows a small drop, and the background remains nearly stable.
Figure 2: Routed Forcing routes distillation by semantic region and noise stage. The causal student generates streaming video from a first frame, audio, and text prompt. DMD queries both the fake score and real score on the student rollout video, while DFD queries the real score on real video with the same condition to inject diverse and dynamic motion patterns from real data. The router selects DFD only for the non-mouth person region at high noise stages and DMD elsewhere.
Method
Video Quality
Lip Sync
Dynamics ( ×100 )
Diversity ( ×100 )
FID ↓
FVD ↓
Sync-C ↑
Sync-D ↓
RAFT ↑
DINO-T ↑
CLIP ↑
DINO ↑
LPIPS ↑
L1 ↑
SpeakerVid
Self Forcing
27.17
320.91
4.03
10.97
5.62
1.07
3.07
1.37
4.96
2.14
DFD
28.78
325.33
3.92
11.10
5.47
1.11
3.08
1.42
4.67
1.84
DP-DMD
27.22
320.27
4.00
10.98
5.67
1.06
3.00
1.33
4.89
2.13
Reward Forcing
26.99
315.84
4.04
10.97
5.79
1.09
3.14
1.41
5.20
2.21
Table 1: Controlled comparison of training strategies. All methods share the same initialization checkpoint, training data and random seeds, differing only in the algorithm design.
Figure 3: Qualitative comparison of different training strategies. All methods are conditioned on the same first frame and audio, and each column shows frames at the same timestep. Routed Forcing produces richer dynamics (e.g., diverse gestures) than Self Forcing and better video quality (e.g., clearer hands) than DFD. More examples are provided in Figure 8 in the appendix.
Method
Size
Efficiency
Video Quality
Lip Sync
Dynamics ( ×100 )
FPS ↑
Latency (s) ↓
FID ↓
FVD ↓
Sync-C ↑
Sync-D ↓
RAFT ↑
DINO-T ↑
SpeakerVid
LiveTalk
1.3B
24.82
0.33
28.77
357.28
3.10
11.62
4.77
1.07
FantasyTalking
14B
0.35
232.06
33.98
364.56
2.44
12.11
3.99
0.57
AniPortrait
2.5B
0.38
211.77
33.49
570.41
3.22
11.24
4.88
1.18
OmniAvatar
1.3B
0.97
83.44
62.83
902.13
1.46
13.74
2.27
1.70
Table 2: Comparison with audio-driven avatar models of comparable scale.
Self Forcing
Region
Noise
Video Quality
Lip Sync
Dynamics ( ×100 )
Diversity ( ×100 )
FID ↓
FVD ↓
Sync-C ↑
Sync-D ↓
RAFT ↑
DINO-T ↑
CLIP ↑
DINO ↑
LPIPS ↑
L1 ↑
✓
27.17
320.91
4.03
10.97
5.62
1.07
3.07
1.37
4.96
2.14
✓
✓
26.97
316.81
4.04
10.96
6.27
1.22
3.22
1.51
5.16
2.00
✓
✓
26.61
311.85
4.01
11.02
7.50
1.22
3.56
1.60
5.92
2.22
✓
✓
✓
25.43
309.92
4.07
10.94
7.80
1.25
3.55
1.58
6.09
2.29
Table 3: Key component ablation. We isolate the contributions of region routing and noise routing (Top-50%). Region routing applies DFD only to the non-mouth person region. Noise routing activates DFD only at high noise stages.
Figure 4: Noise threshold ablation: training curves on SpeakerVid. Region routing is kept active throughout: all training variants are applied only to the non-mouth person region, with DMD retained for the mouth and background. The y-axis shows the mean normalized score across metrics.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Update
r
A
V
DMD, from θ0
0.1
0.052
0.052
DFD, fixed point
0.1
625.90
0.709
DFD, fixed point
10
625.99
0.0007
Appendix
Table 4: Motion and seed variation in the model of Appendix A.3 . The parameters are (D,ℓ,φ∗,w0)=(25,1,π/3,0.1) , and A and V follow Eq. 14 . Each dynamic video of the teacher has A=626 , and seeds placed on x+ and x− give V=0.75 .
Person coverage (%) ↑
Person IoU (%) ↑
Mouth coverage (%) ↑
Mouth IoU (%) ↑
94.49
89.73
43.61
32.74
Appendix
Table 5: Mask alignment between real video and student rollout video. Person masks are extracted via a segmentation model, and mouth masks via facial landmarks.
Mouth mask source
FID ↓
FVD ↓
Sync-C ↑
Sync-D ↓
Step time (s) ↓
Student rollout
25.43
309.92
4.07
10.94
63.73
Real video
25.49
307.91
4.03
10.99
61.11
Appendix
Table 6: Mouth mask source ablation. Comparison of training with mouth masks from the student rollout video versus ground-truth video.
Figure 5: Comparison of mouth masks between real video and student rollout video.
Figure 6: Diversity decreases during DMD training. Starting from the LiveTalk checkpoint, we train Self Forcing with DMD for 1,000 steps and measure diversity (mean pairwise LPIPS and L1 across four seeds over 100 SpeakerVid samples) at each checkpoint. Both metrics drop during training, confirming that DMD training itself drives diversity collapse.
Figure 7: Regional diversity measured by L1 . Mean pairwise L1 distance across different random seeds for teacher and student models, confirming the same trend as the LPIPS-based analysis in § 3.2 : the person region suffers the largest diversity drop ( −39.7% ), the mouth region shows a moderate drop ( −14.8% ), and the background remains stable ( −3.0% ).
Figure 8: Additional qualitative comparison of training strategies. Each column shows frames at the same timestep across methods. Routed Forcing produces more dynamic and diverse motion than Self Forcing while maintaining better visual quality than DFD.
Audio-driven avatar generation requires realistic lip-sync, expressive motion, and real-time streaming. Recent work achieves the latter via self-forcing with Distribution Matching Distillation (DMD), but this paradigm suffers from a critical failure that has not been systematically characterized: dynamic collapse, where the student model converges to a near-static optimum with high perceptual quality but severely suppressed temporal dynamics. We trace this to two causes: the reverse KL objective in DMD, which biases toward low-motion modes, and unanchored self-conditioning, which creates a feedback loop that amplifies collapse. This is especially harmful for avatars, where even subtle motion loss breaks lip-sync and expression. To address this, we propose DynaForcing, a training framework with three complementary strategies applied at different levels. Specifically, Hybrid Forcing anchors rollouts to ground-truth dynamics at the data level to break the feedback loop. Dynamics-Aware Reward Regularization introduces explicit motion rewards via the RL interpretation of DMD to counteract the reverse KL bias at the loss level. Reference Perturbation perturbs reference images to decouple identity from static details, forcing the model to rely on audio for motion at the conditioning level. We further introduce computation graph pruning and gradient replay, reducing the GPU footprint of self-forcing by over an order of magnitude. Experiments show that DynaForcing recovers dynamics to teacher-comparable levels (Dyn-Deg: 0.31 -> 0.73, Sync-C: 7.03 -> 7.68) while improving visual quality, resolving the quality-dynamics trade-off throughout training without early stopping.
Yubo Huang, Sirui Zhao, Xinchen Yao +5
University of Science and Technology of China · Nanjing University · Hefei University of Technology +1
Real-time talking avatar generation requires low latency and minute-level temporal stability. Autoregressive (AR) forcing enables streaming inference but suffers from exposure bias, which causes errors to accumulate and become irreversible over long rollouts. In contrast, full-sequence diffusion transformers mitigate drift but remain computationally prohibitive for real-time long-form synthesis. We present AvatarForcing, a one-step streaming diffusion framework that denoises a fixed local-future window with heterogeneous noise levels and emits one clean block per step under constant per-step cost. To stabilize unbounded streams, the method introduces dual-anchor temporal forcing: a style anchor that re-indexes RoPE to maintain a fixed relative position with respect to the active window and applies anchor-audio zero-padding, and a temporal anchor that reuses recently emitted clean blocks to ensure smooth transitions. Real-time one-step inference is enabled by two-stage streaming distillation with offline ODE backfill and distribution matching. Experiments on standard benchmarks and a new 400-video long-form benchmark show strong visual quality and lip synchronization at 34 ms/frame using a 1.3B-parameter student model for realtime streaming. Our page is available at: https://cuiliyuan121.github.io/AvatarForcing/
Liyuan Cui, Wentao Hu, Wenyuan Zhang +3
Zhejiang University · Kling Team, Kuaishou Technology · Beijing University of Posts and Telecommunications +1
Streaming talking-head generation produces each frame as its driving audio arrives, yet fidelity and efficiency have so far pulled in opposite directions: end-to-end methods condition a video diffusion model on audio directly and achieve high quality but only at large scale, while cheaper two-stage methods generate an intermediate motion representation and trail in fidelity. We argue the cost of the former lies in the target of fusion: the video latent is dominated by identity, appearance and background, none of which audio bears on, so coupling audio to every pixel blurs detail and wastes capacity. We instead fuse conditions in a low-dimensional identity-disentangled motion space, routing audio and motion captions by their temporal granularity, and generate motion latents with a small causal autoregressive transformer that a pretrained diffusion renderer turns into video. Conditions thus control video transitively, and high fidelity no longer requires a large backbone. Streaming this decomposition needs both models to be causal, and the exposure-bias problem could be solved by self-forcing given a bidirectional teacher. But there is no such teacher in motion space. Our decoupled self-forcing distillation resolves both models under one frozen teacher: conditioned on motion, it distills the renderer into a block-causal student; unconditionally, it scores rendered rollouts against real videos, supervising motion by the video it produces. This lifts the fidelity ceiling from the motion generator onto the stronger renderer. The two models run as parallel causal streams, reaching 15.4 FPS at 1.3 s latency with no quality degradation.