Beyond Distribution Matching: Self-Supervised Representation Forcing for Few-Step Video Generation

Junyi Chen; Haiyu Zhang; Zhoujie Fu; Tengfei Wang; Chunchao Guo

cs.CVOct 26, 2026

Beyond Distribution Matching: Self-Supervised Representation Forcing for Few-Step Video Generation

Authors: Junyi Chen, Haiyu Zhang, Zhoujie Fu, Tengfei Wang, Chunchao Guo

Organizations: Shanghai Jiao Tong University · Hunyuan, Tencent · Nanyang Technological University

Abstract

Few-step distillation is essential for deploying modern video diffusion models. Among existing approaches, Distribution Matching Distillation (DMD) has emerged as a highly promising paradigm by achieving strong output-space fidelity without adversarial training, while remaining flexible across various student architectures. We observe, however, that DMD fundamentally relies on an output-space objective, lacking explicit regularization on the model's internal representations. As the distillation step count aggressively shrinks, this output-only supervision becomes insufficient, causing the distilled student to degrade in both fine-grained spatial structures and coherent temporal dynamics. To address this limitation, we introduce Self Flow Forcing (SF $^2$ ), which incorporates a predictive representation loss into the DMD framework without altering its output objective. By providing the student and an Exponential Moving Average (EMA) teacher with heterogeneous noise levels, we establish an information asymmetry. This formulation forces the student to predict the teacher's internal representations from a more corrupted input, effectively integrating self-supervised representation alignment into distribution matching. Extensive experiments demonstrate that SF $^2$ effectively mitigates temporal flickering and consistently improves both spatial fidelity and motion quality. Furthermore, the proposed mechanism is architecture-agnostic - seamlessly supporting both bidirectional and causal models - and requires no external feature extractors, introducing minimal computational overhead.

Figures & tables

Figure 1: Overview of SF 2 . From the student’s denoising trajectory, the cleanest prediction drives the standard DMD loss $\mathcal{L}_{\text{DMD}}$ , while two intermediate states $(x_{t},x_{s})$ with $t>s$ feed the SF 2 branch: a spatial mask mixes them into an asymmetric, highly corrupted input for the student, while the EMA teacher sees only the cleaner $x_{s}$ . Hidden states from asymmetric layers are aligned via learnable projectors with a stop-gradient cosine loss $\mathcal{L}_{\text{rep}}$ . The teacher is updated by EMA and requires no external pre-trained features.

	Generator update ( $\times 1$ )				Critic update ( $\times 5$ )			Total
Component	Rollout	DMD	Ours	Backward	Rollout	MSE	Backward	Total
% of cycle	4.4	20.1	6.2	6.4	25.0	8.7	29.2	100.0

Table 1: Per-cycle training-time breakdown (% of one cycle). One cycle

=

1 generator update + 5 critic updates. The SF 2 branch is only

\sim

6\%

of the cycle.

Figure 2: Qualitative comparison on causal few-step T2V generation. Top row: multi-step bidirectional Wan2.1 teacher. Rows 2–5: few-step distilled causal students (CausVid, Self-Forcing, Causal-Forcing, SF 2 ). The four panels expose four facets of representation-level coherence that the rep loss is designed to compress into the student: articulated-motion plausibility (top-left), event irreversibility (top-right), object persistence (bottom-left), and scene-structural stability (bottom-right). SF 2 satisfies all four, whereas baselines fail in distinct ways on each.

Figure 3: Hyperparameter selection for asymmetric inputs. (a) Masking ratio $\rho_{\max}$ . The two paradigms peak at different optima — $\rho_{\max}{=}0.4$ for bidirectional and $\rho_{\max}{=}0.2$ for causal — which we adopt as defaults in the following experiments. (b) Mask sampling pattern. Space-only sampling (a single $H_{p}\times W_{p}$ mask broadcast across all frames) versus agnostic sampling (independent per-token masking) on VBench’s Quality Score, Dynamic Degree, and Background Consistency.

Model	Params	Throughput $\uparrow$	Latency $\downarrow$	Resolution	Total $\uparrow$	Quality $\uparrow$	Semantic $\uparrow$	Vision. $\uparrow$
Bidirectional Video Diffusion Models
LTX-Video [ 13 ]	1.9B	8.98	13.5	768 $\times$ 512	79.83	81.88	71.62	–6.218
Wan2.1 [ 41 ]	1.3B	0.78	103	848 $\times$ 480	83.37	84.30	79.65	5.275
Distilled Bidirectional Video Models
CausVid [ 50 ]	1.3B	16.2	4.22	848 $\times$ 480	83.19	84.08	79.63	5.310
Ours	1.3B	16.2	4.22	848 $\times$ 480	84.15	84.92	81.07	7.060

Table 2: Quantitative comparison with existing methods on VBench [ 17 ] and VisionReward [ 46 ] . Models are grouped by sampling regime — bidirectional multi-step, distilled bidirectional, autoregressive multi-step, and distilled autoregressive — and bold marks the best score within each comparable group. Throughput is in FPS and latency in second, both measured on a single H100. All distilled models (in both distilled groups) share the same Wan2.1-T2V-1.3B backbone.

Figure 4: Pairwise quantitative comparison. All compared methods share the same Wan2.1-T2V-1.3B backbone. Each bar shows the per-prompt preference of VisionReward [ 46 ] ; the dark-blue segment is the fraction of prompts on which SF 2 is preferred. SF 2 wins the majority of prompts against all four baselines, on both bidirectional (DMD, multi-step Wan2.1 base model) and causal (Causal-Forcing, Self-Forcing) sides.

Figure 5: Qualitative comparison on bidirectional few-step T2V generation. Both compared methods are distilled from the same HunyuanVideo-1.5 backbone. Each pair shows the DMD student (top) and SF 2 (bottom) on the same prompt and noise; a zoomed crop on the right of each panel highlights the rendered text. Across all four prompts, SF 2 produces sharp, legible glyphs where DMD collapses into blurred or scrambled characters, indicating that the rep loss strengthens the student’s spatial representation.

Figure 6: Quantitative evaluation on interactive video generation. We evaluate the adherence of the generated video to the input camera and action conditions. Compared to the WorldPlay baseline, SF 2 achieves notably higher translation, rotation, and action accuracy across training steps, demonstrating that a stronger internal representation leads to more precise state transitions in interactive environments.

Figure 7: Qualitative camera trajectory alignment. Reconstructed 3D point clouds and camera trajectories from the generated videos. The trajectory produced by SF 2 ( blue ) tightly tracks the input condition ( green ), whereas the WorldPlay baseline ( red ) exhibits severe drift. This visualizes how the stronger internal representation of SF 2 enforces strict adherence to complex control signals.

Model	Params	Throughput	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Dynamic $\uparrow$	Flickering $\uparrow$	Consistency $\uparrow$
DMD (Ours w/o $\mathcal{L}_{\text{rep}}$ )	8.3B	2.21	20.64	0.6982	0.2144	90.00	96.39	94.11
Ours	8.3B	2.21	21.24	0.7110	0.2096	90.00	96.86	94.65

Table 3: Probing representation quality via re-rendering and frame-level coherence. We report PSNR / SSIM / LPIPS against the generated frames, plus VBench Temporal Flickering and Background Consistency. SF 2 improves over vanilla DMD on every reconstruction and coherence metric at matched Dynamic Degree, despite using no 3D supervision during training.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Backbones
	Wan2.1-1.3B (Bid.)	HunyuanVideo-1.5 (Bid.)	Wan2.1-1.3B (Causal)
Student backbone	Wan2.1-T2V-1.3B	HunyuanVideo-1.5	Wan2.1-T2V-1.3B
Real-score critic (frozen)	Wan2.1-T2V-14B	HunyuanVideo-1.5	Wan2.1-T2V-14B
Fake-score critic (init.)	Wan2.1-T2V-1.3B	HunyuanVideo-1.5	Wan2.1-T2V-1.3B
Student init. ckpt	Wan2.1-T2V-1.3B	HunyuanVideo-1.5	Causal-Forcing ODE-regression
Backbone depth $N_{b}$	$30$	$54$	$30$

Appendix

Table 4: Full hyperparameters for the three SF 2 instantiations. The SF 2 branch is identical in spirit across settings; the only paradigm-dependent quantities are the masking ratio

\rho_{\max}

and the chunk size

F_{b}

(causal only). All bidirectional models use

K{=}4

denoising steps; the causal student also uses

K{=}4

unless otherwise noted.

Diffusion Model Distillation

Diffusion model distillation transfers a large model’s generative behavior to faster or smaller models that need fewer sampling steps.

See all 191 papers

Data-Forcing Distillation: Restoring Diversity and Fidelity in Few-Step Video Generation
Jun 16, 2026Siyi Chen, Shaowei Liu, Yixuan Jia +4Video Diffusion Models Diffusion Model Distillation
University of Michigan · NVIDIA · University of Illinois Urbana-Champaign
RepForcing: Representation-Space Initialization for Autoregressive Video Distillation
Oct 14, 2026Siming Fu, Zheming Fu, Lianrui Mu +3Diffusion Model Distillation Autoregressive Video Generation
Joy Future Academy · Zhejiang University
CoDMD: Copula-aware Distribution Matching Distillation for Fast Video Generation
Jun 20, 2026Wenhu Zhang, Kun Cheng, Changyuan Wang +7Video Diffusion Models Diffusion Model Distillation
1HKUST · 2Wan Team, Alibaba Group · 3THU +2

Jun 16, 2026J/K move · Enter open · S save

Data-Forcing Distillation: Restoring Diversity and Fidelity in Few-Step Video Generation

Siyi Chen, Shaowei Liu, Yixuan Jia, Zian Wang+3

University of Michigan · NVIDIA · University of Illinois Urbana-Champaign

Video Diffusion Models Diffusion Model Distillation Few-Step Diffusion Sampling Mode Collapse

Open paper AskPDF arXiv

Recent progress has shown promise in distilling multi-step video diffusion models into efficient few-step students. Among them, Distribution Matching Distillation (DMD) and its successor DMD2 achieved strong generation quality and fast convergence. However, due to the nature of the reverse Kullback--Leibler (KL) objective, these methods exhibit two persistent failure modes: a substantial drop in sample diversity, and visibly over-saturated outputs that deviate from real-video appearance. In this work, we propose Data-Forcing Distillation (DFD), a simple post-training framework that restores diversity and fidelity in DMD with only a single-line of code change. At its core is the teacher score discrepancy to guide the student toward the real-data distribution, pulling it to missing modes (mitigating mode collapse) and away from problematic modes absent in real data (avoiding over-saturation). We provide an in-depth theoretical analysis of our framework and validate our approach on text-to-video, image-to-video, and autoregressive video generation. With only 100--300 steps of finetuning, DFD effectively restores diversity and fidelity on both Wan2.1-1.3B and Cosmos-Predict2.5-2B model, resolving the over-saturation artifacts with significantly better video dynamics and appearance, and even outperforms the teacher model.

Abstract

Diffusion Model Distillation

Explore similar work

Data-Forcing Distillation: Restoring Diversity and Fidelity in Few-Step Video Generation

RepForcing: Representation-Space Initialization for Autoregressive Video Distillation

CoDMD: Copula-aware Distribution Matching Distillation for Fast Video Generation