Where and When to Force: Routed Forcing for Streaming Avatars

Zihan Su; Siwen Lu; Junhao Zhuang; Zeyue Xue; Haoyang Huang; Guanghao Li; Xiaofeng Tan; Chun Yuan

cs.CVSep 25, 2026

Where and When to Force: Routed Forcing for Streaming Avatars

Authors: Zihan Su, Siwen Lu, Junhao Zhuang, Zeyue Xue, Haoyang Huang, Guanghao Li, Xiaofeng Tan, Chun Yuan,

Organizations: Tsinghua University · Joy Future Academy, JD · Southeast University

Abstract

Audio-driven streaming avatar generation requires real-time synthesis of speech-synchronized videos with dynamic and diverse motion. Self Forcing uses Distribution Matching Distillation (DMD) to distill bidirectional video diffusion models into causal, few-step generators for real-time streaming. However, DMD minimizes a reverse KL divergence, which is inherently mode-seeking: it causes the student to discard high-dynamic modes and collapse onto static outputs, compressing both dynamics and diversity of generated videos. We find that this collapse is region-heterogeneous: person regions involving pose and gesture variations suffer the largest diversity loss, the audio-driven mouth region shows a small loss, and the background remains nearly stable. Based on this observation, we propose Routed Forcing, which routes the distillation objective by semantic region and noise stage to improve dynamics and diversity while preserving visual quality. Specifically, (1) Where to Force: Semantic-Region Routing applies Data-Forcing Distillation (DFD), which supervises the student with real videos, to the person region where diversity collapse is most severe, while retaining DMD for the mouth and background to preserve lip synchronization and scene stability. (2) When to Force: Noise-Stage Routing activates DFD at high noise stages, where real video serves as effective supervision to inject diverse and dynamic motion patterns. At low noise stages, DMD is used to refine details, avoiding blur and artifacts from spatial differences between real video and student-generated video. Experiments show that Routed Forcing improves dynamics by up to 45% and diversity by 7-25% over Self Forcing, while preserving video quality and lip synchronization.

Figures & tables

Figure 1: DMD compresses dynamics and diversity, and the collapse is region-heterogeneous. We use OmniAvatar as teacher model and LiveTalk, distilled from OmniAvatar via DMD, as student model. (a) Given the same first frame and audio, the teacher produces diverse expressions and dynamic gestures across two seeds, while the student generates similar static trajectories. (b) Regional diversity (mean pairwise LPIPS across four seeds over 100 first-frame-and-audio pairs from SpeakerVid) reveals that the non-mouth person region suffers the largest drop, the mouth region shows a small drop, and the background remains nearly stable.

Figure 2: Routed Forcing routes distillation by semantic region and noise stage. The causal student generates streaming video from a first frame, audio, and text prompt. DMD queries both the fake score and real score on the student rollout video, while DFD queries the real score on real video with the same condition to inject diverse and dynamic motion patterns from real data. The router selects DFD only for the non-mouth person region at high noise stages and DMD elsewhere.

SpeakerVid
Method	Video Quality		Lip Sync		Dynamics ( $\times 100$ )		Diversity ( $\times 100$ )
Method	FID $\downarrow$	FVD $\downarrow$	Sync-C $\uparrow$	Sync-D $\downarrow$	RAFT $\uparrow$	DINO-T $\uparrow$	CLIP $\uparrow$	DINO $\uparrow$	LPIPS $\uparrow$	L1 $\uparrow$
Self Forcing	27.17	320.91	4.03	10.97	5.62	1.07	3.07	1.37	4.96	2.14
DFD	28.78	325.33	3.92	11.10	5.47	1.11	3.08	1.42	4.67	1.84
DP-DMD	27.22	320.27	4.00	10.98	5.67	1.06	3.00	1.33	4.89	2.13
Reward Forcing	26.99	315.84	4.04	10.97	5.79	1.09	3.14	1.41	5.20	2.21

Table 1: Controlled comparison of training strategies. All methods share the same initialization checkpoint, training data and random seeds, differing only in the algorithm design.

Figure 3: Qualitative comparison of different training strategies. All methods are conditioned on the same first frame and audio, and each column shows frames at the same timestep. Routed Forcing produces richer dynamics (e.g., diverse gestures) than Self Forcing and better video quality (e.g., clearer hands) than DFD. More examples are provided in Figure 8 in the appendix.

Method	Size	Efficiency		Video Quality		Lip Sync		Dynamics ( $\times 100$ )
Method	Size	FPS $\uparrow$	Latency (s) $\downarrow$	FID $\downarrow$	FVD $\downarrow$	Sync-C $\uparrow$	Sync-D $\downarrow$	RAFT $\uparrow$	DINO-T $\uparrow$
SpeakerVid
LiveTalk	1.3B	24.82	0.33	28.77	357.28	3.10	11.62	4.77	1.07
FantasyTalking	14B	0.35	232.06	33.98	364.56	2.44	12.11	3.99	0.57
AniPortrait	2.5B	0.38	211.77	33.49	570.41	3.22	11.24	4.88	1.18
OmniAvatar	1.3B	0.97	83.44	62.83	902.13	1.46	13.74	2.27	1.70

Table 2: Comparison with audio-driven avatar models of comparable scale.

Self Forcing	Region	Noise	Video Quality		Lip Sync		Dynamics ( $\times 100$ )		Diversity ( $\times 100$ )
Self Forcing	Region	Noise	FID $\downarrow$	FVD $\downarrow$	Sync-C $\uparrow$	Sync-D $\downarrow$	RAFT $\uparrow$	DINO-T $\uparrow$	CLIP $\uparrow$	DINO $\uparrow$	LPIPS $\uparrow$	L1 $\uparrow$
✓			27.17	320.91	4.03	10.97	5.62	1.07	3.07	1.37	4.96	2.14
✓	✓		26.97	316.81	4.04	10.96	6.27	1.22	3.22	1.51	5.16	2.00
✓		✓	26.61	311.85	4.01	11.02	7.50	1.22	3.56	1.60	5.92	2.22
✓	✓	✓	25.43	309.92	4.07	10.94	7.80	1.25	3.55	1.58	6.09	2.29

Table 3: Key component ablation. We isolate the contributions of region routing and noise routing (Top-50%). Region routing applies DFD only to the non-mouth person region. Noise routing activates DFD only at high noise stages.

Figure 4: Noise threshold ablation: training curves on SpeakerVid. Region routing is kept active throughout: all training variants are applied only to the non-mouth person region, with DMD retained for the mouth and background. The y-axis shows the mean normalized score across metrics.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Update	$r$	$\mathcal{A}$	$\mathcal{V}$
DMD, from $\theta_{0}$	0.1	0.052	0.052
DFD, fixed point	0.1	625.90	0.709
DFD, fixed point	10	625.99	0.0007

Appendix

Table 4: Motion and seed variation in the model of Appendix A.3 . The parameters are

(D,\ell,\varphi_{*},w_{0})=(25,1,\pi/3,0.1)

, and

\mathcal{A}

and

\mathcal{V}

follow Eq. 14 . Each dynamic video of the teacher has

\mathcal{A}=626

, and seeds placed on

x_{+}

and

x_{-}

give

\mathcal{V}=0.75

Person coverage (%) $\uparrow$	Person IoU (%) $\uparrow$	Mouth coverage (%) $\uparrow$	Mouth IoU (%) $\uparrow$
94.49	89.73	43.61	32.74

Appendix

Table 5: Mask alignment between real video and student rollout video. Person masks are extracted via a segmentation model, and mouth masks via facial landmarks.

Mouth mask source	FID $\downarrow$	FVD $\downarrow$	Sync-C $\uparrow$	Sync-D $\downarrow$	Step time (s) $\downarrow$
Student rollout	25.43	309.92	4.07	10.94	63.73
Real video	25.49	307.91	4.03	10.99	61.11

Appendix

Table 6: Mouth mask source ablation. Comparison of training with mouth masks from the student rollout video versus ground-truth video.

Figure 5: Comparison of mouth masks between real video and student rollout video.

Figure 6: Diversity decreases during DMD training. Starting from the LiveTalk checkpoint, we train Self Forcing with DMD for 1,000 steps and measure diversity (mean pairwise LPIPS and

L_{1}

across four seeds over 100 SpeakerVid samples) at each checkpoint. Both metrics drop during training, confirming that DMD training itself drives diversity collapse.

Figure 7: Regional diversity measured by

L_{1}

. Mean pairwise

L_{1}

distance across different random seeds for teacher and student models, confirming the same trend as the LPIPS-based analysis in § 3.2 : the person region suffers the largest diversity drop (

-39.7\%

), the mouth region shows a moderate drop (

-14.8\%

), and the background remains stable (

-3.0\%

Figure 8: Additional qualitative comparison of training strategies. Each column shows frames at the same timestep across methods. Routed Forcing produces more dynamic and diverse motion than Self Forcing while maintaining better visual quality than DFD.

Streaming Video Generation

Streaming video generation creates video incrementally as frames or chunks arrive, enabling low-latency interactive synthesis and editing.

See all 95 papers

DynaForcing: Overcoming Dynamic Collapse in Self-Forcing Distillation for Streaming Avatar Generation
Aug 18, 2026Yubo Huang, Sirui Zhao, Xinchen Yao +5Audio-Driven Facial Animation Self-Distillation
University of Science and Technology of China · Nanjing University · Hefei University of Technology +1
AvatarForcing: One-Step Streaming Talking Avatars via Local-Future Sliding-Window Denoising
Mar 15, 2026Liyuan Cui, Wentao Hu, Wenyuan Zhang +3Audio-Video Generation Video Diffusion Models
Zhejiang University · Kling Team, Kuaishou Technology · Beijing University of Posts and Telecommunications +1
Decoupled Self-Forcing Distillation for Streaming Talking Head Generation
Sep 9, 2026Yanru An, Ruiyan Wang, Wenwu Wei +7Audio-Video Generation Diffusion Model Distillation
Shanghai Jiao Tong University · Ant Group

Aug 18, 2026J/K move · Enter open · S save

DynaForcing: Overcoming Dynamic Collapse in Self-Forcing Distillation for Streaming Avatar Generation

Yubo Huang, Sirui Zhao, Xinchen Yao, Zhengye Zhang+4

University of Science and Technology of China · Nanjing University · Hefei University of Technology+1

Audio-Driven Facial Animation Self-Distillation Diffusion Model Distillation Real-Time Interactive Video Generation

Open paper AskPDF arXiv

Audio-driven avatar generation requires realistic lip-sync, expressive motion, and real-time streaming. Recent work achieves the latter via self-forcing with Distribution Matching Distillation (DMD), but this paradigm suffers from a critical failure that has not been systematically characterized: dynamic collapse, where the student model converges to a near-static optimum with high perceptual quality but severely suppressed temporal dynamics. We trace this to two causes: the reverse KL objective in DMD, which biases toward low-motion modes, and unanchored self-conditioning, which creates a feedback loop that amplifies collapse. This is especially harmful for avatars, where even subtle motion loss breaks lip-sync and expression. To address this, we propose DynaForcing, a training framework with three complementary strategies applied at different levels. Specifically, Hybrid Forcing anchors rollouts to ground-truth dynamics at the data level to break the feedback loop. Dynamics-Aware Reward Regularization introduces explicit motion rewards via the RL interpretation of DMD to counteract the reverse KL bias at the loss level. Reference Perturbation perturbs reference images to decouple identity from static details, forcing the model to rely on audio for motion at the conditioning level. We further introduce computation graph pruning and gradient replay, reducing the GPU footprint of self-forcing by over an order of magnitude. Experiments show that DynaForcing recovers dynamics to teacher-comparable levels (Dyn-Deg: 0.31 -> 0.73, Sync-C: 7.03 -> 7.68) while improving visual quality, resolving the quality-dynamics trade-off throughout training without early stopping.

Abstract

Streaming Video Generation

Explore similar work

DynaForcing: Overcoming Dynamic Collapse in Self-Forcing Distillation for Streaming Avatar Generation

AvatarForcing: One-Step Streaming Talking Avatars via Local-Future Sliding-Window Denoising

Decoupled Self-Forcing Distillation for Streaming Talking Head Generation