Enhancing Autoregressive Video Generation via Representation Adversarial Distillation

Fangyu Lin; Xingtong Ge; Lunjie Zhu; Yi Zhang; Zhening Liu; Tianhang Wang; Mengfei Li; Yumeng Zhang

cs.CVSep 30, 2026

Enhancing Autoregressive Video Generation via Representation Adversarial Distillation

Authors: Fangyu Lin, Xingtong Ge, Lunjie Zhu, Yi Zhang, Zhening Liu, Tianhang Wang, Mengfei Li, Yumeng Zhang, +3 more

Organizations: The Hong Kong University of Science and Technology · Vivix Group Limited · Zhejiang University

Abstract

Few-step autoregressive video generation enables efficient streaming synthesis, but errors introduced in early temporal blocks are reused as context and can propagate through subsequent rollouts, leading to detail degradation, structural drift, and unstable motion. Existing distribution matching distillation (DMD) primarily aligns student and teacher distributions in diffusion latent space, but provides no direct supervision over the perceptual quality of decoded videos. We introduce Radian, a representation-space adversarial distillation framework that complements on-policy DMD with real-data adversarial supervision in the feature space defined by a frozen visual foundation model (VFM). During training, Radian sparsely decodes frames from autoregressive student rollouts, extracts multi-level visual representations, and applies lightweight discriminator heads to distinguish generated outputs from real video frames. The DMD objective anchors the student to the pretrained teacher, while the representation-space adversarial objective supplies complementary perceptual and semantic gradients that promote high-quality modes. These additional components are discarded after training, leaving the generator architecture and inference-time denoising budget unchanged. Experiments on Wan2.1-1.3B cover four-step chunk-wise, one-step frame-wise, and minute-long autoregressive generation. Our method achieves a VBench Total of 0.8444 and a VideoAlign Total of 0.8033 under four-step generation, and improves VBench-Long from 0.7805 to 0.8041 over Rolling Forcing while using fewer denoising steps. Controlled comparisons across image, video, and diffusion representations further indicate that the choice of representation spaces induces distinct adversarial signals, and external VFM gradients complement DMD more effectively than adversarial supervision derived from diffusion-internal features.

Figures & tables

Figure 1: Complementary Representation-Space Supervision. Radian complements DMD with perceptual gradients, guiding the student toward a better-matched data distribution.

Figure 2: Overview of the Three-stage Training Pipeline.

Figure 3: Architecture of Lightweight Discriminator Head.

Table 1: Quantitative results under four-step chunk-wise (a), one-step frame-wise (b), and long-horizon autoregressive generation settings (c). #Params counts only the inference-time generator. Darker green indicates a higher rank in each column.

Figure 4: Selected short-video generation results using four-step chunk-wise generation and one-step frame-wise generation. More qualitative results and full prompts are provided in Appendix B .

Feature source	Feature model	Total $\uparrow$	Quality $\uparrow$	Semantic $\uparrow$	Multi-Obj. $\uparrow$	Color $\uparrow$	Consist. $\uparrow$	Dynamic $\uparrow$	Imaging $\uparrow$	VQ $\uparrow$	MQ $\uparrow$	TA $\uparrow$	VA Total $\uparrow$
Image VFM	DINOv2	0.8444	0.8541	0.8054	0.8343	0.8722	0.2662	0.8056	0.7148	0.2134	0.2094	0.3805	0.8033
Image VFM	DINOv3	0.8401	0.8498	0.8012	0.8528	0.8729	0.2649	0.7685	0.7104	0.1462	0.2937	0.3607	0.8006
Image VFM	SigLIP2	0.8391	0.8475	0.8052	0.8384	0.8662	0.2680	0.6343	0.6992	0.1156	0.1320	0.3751	0.6226
Video VFM	V-JEPA 2.1	0.8360	0.8430	0.8078	0.8604	0.8627	0.2676	0.5991	0.7042	0.0498	0.1275	0.4119	0.5892
Video VFM	VideoMAE	0.8372	0.8436	0.8115	0.8417	0.8787	0.2676	0.5213	0.7026	0.1000	0.1452	0.3436	0.5888
Image + Video VFM	DINOv2 + VideoMAE	0.8434	0.8525	0.8070	0.8652	0.8545	0.2679	0.6593	0.7074	0.0831	0.0575	0.3079	0.4485

Table 2: Ablation on different feature models for adversarial feature supervision.

Table 7

Figure 5: Selected long-video generation results using four-step chunk-wise generation.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$c$ , $z$	Text condition and initial Gaussian generator input.
$G_{\theta}$ , $p_{\theta}^{\mathrm{AR}}$	Causal student and its autoregressive output distribution.
$\hat{z}^{(b)}$ , $\hat{z}^{(<b)}$	Generated latent block and its preceding context.
$x_{0}$ , $x_{t}$	Clean and diffusion-corrupted video latents.
$s_{r}$ , $s_{\phi}$	Frozen teacher score and trainable fake-score model.
$\mathcal{D}_{\mathrm{VAE}}$	Frozen latent-to-RGB decoder.

Appendix

Table 5: Principal notation.

Setting	Four-step short	One-step short	One-minute
Latent / RGB frames	21 / 81	21 / 81	240 / 957
Latent frames per block	3	1	3
Denoising schedule	$\{1000,750,500,250\}$	$\{1000\}$	$\{1000,750,500,250\}$
First-block exception	None	Four-step enhancement	None
Student inference CFG	Disabled	Disabled	Disabled
Evaluation seeds	15	15	15

Appendix

Table 6: Sampling settings at

832\times 480

, 16 FPS. Schedules are indices before scheduler warping.

Setting	Generator	Fake-score	Discriminator
Optimizer	AdamW	AdamW	AdamW
$(\beta_{1},\beta_{2})$	$(0,0.999)$	$(0,0.999)$	$(0,0.95)$
Weight decay	0.01	0.01	0
Stage II learning rate	$2\times 10^{-6}$	$4\times 10^{-7}$	$3\times 10^{-5}$
Stage III learning rate	$5\times 10^{-7}$	$4\times 10^{-7}$	$3\times 10^{-6}$
Gradient-norm clipping	10	10	10

Appendix

Table 7: Default chunk-wise optimization. Learning rates refer to trainable networks; the visual encoder remains frozen.

Backbone	Input	Feature levels	Channels	Readout / heads
DINOv2-S/14	$8\times 518^{2}$	$P+\{6,8,10,11\}$	$384$	CLS + dense
DINOv3-S/16	$8\times 512^{2}$	$P+\{2,5,8,11\}$	$384$	Norm. patch mean + dense
SigLIP2-B/16	$8\times 512^{2}$	$P+\{6,8,10,11\}$	$768\rightarrow 512$	Norm. patch mean + dense
V-JEPA 2.1-B (distilled)	Video clip	$P+\{6,8,10,11\}$	$768\rightarrow 512$	Spatiotemporal dense
VideoMAE	Video clip	$P+\{6,8,10,11\}$	$768\rightarrow 512$	Spatiotemporal dense
DINOv2 + VideoMAE	Images + clip	Image levels + video output	$384;\ 768\rightarrow 512$	Image dense + temporal

Appendix

Table 8: Representation designs with frozen backbones.

P

denotes the patch-input branch; channel arrows indicate trainable projections. Image inputs are written as frame count

\times

spatial resolution. Video inputs use 32 frames, short side 256, and RGB stride two. The mixed variant combines DINOv2-S/14 with VideoMAE; its video head uses spatial pooling.

Transformer layers	Block indices	Total heads
1	$\{11\}$	2
2	$\{6,11\}$	3
4	$\{6,8,10,11\}$	5
8	$\{2,4,6,7,8,9,10,11\}$	9

Appendix

Table 9: DINOv2-S feature-depth ablation. All variants retain a patch-input head and use the same 15 evaluation seeds.

Figure 6: Additional four-step chunk-wise comparisons. A retro-futuristic telephone conversation and piano performance, with Self Forcing, Causal Forcing, and Radian arranged from top to bottom. The full prompts appear below the corresponding cases.

Figure 7: Additional one-step frame-wise comparisons. A snowy campfire and piano performance, with Causal Forcing++, One Forcing, and Radian arranged from top to bottom. The sampled frames show changes in scene appearance and subject framing.

Figure 8: Additional long-video comparisons. From top to bottom: coffee in a cafe, panda surfing, drinking water in a European town, and a telephone conversation in the snow. Each case places Rolling Forcing above Radian and its prompt below. Six frames span each 957-frame sequence; the shared time labels are rounded to the nearest second.

Figure 9: Feature-space comparisons. Rows show DINOv2-S, DINOv3, SigLIP2, V-JEPA 2.1, VideoMAE, DINOv2 with VideoMAE, and DiT-GAN. The four cases depict a black-and-white corgi scene, a panda drinking coffee, glazing croissants, and a coastal lighthouse.

Figure 10: Encoder-size comparisons. DINOv2-S, B, and G are shown on a wolf beside a frozen stream, fireworks viewed from a bridge, a traditional waterwheel, and mixing colored liquids. Each case displays three frames with its prompt below.

Figure 11: Feature-layer comparisons. Rows use one, two, four, and eight DINOv2-S feature layers, respectively. The cases show a corgi in a park, an astronaut, a train on tracks, and clownfish among coral; Radian uses the four-layer configuration.

Explore similar work

Jun 2, 2026cs.CV

AAD-1: Asymmetric Adversarial Distillation for One-Step Autoregressive Video Generation

We present AAD-1, an Asymmetric Adversarial Distillation framework for One-step autoregressive image-to-video generation. State-of-the-art methods adopt adversarial distillation but suffer from motion collapse and training instability, resulting in static videos. AAD-1 addresses these challenges through two key designs in architecture and training strategy. Our key architectural insight is to break the symmetry between generator and discriminator. While the generator remains causal to preserve autoregressive sampling capability, the discriminator attends bidirectionally over the full spatiotemporal context and produces a single holistic realism score for the entire video sequence. This asymmetric design enables the discriminator to effectively detect global temporal failures and long-range drift that cause motion collapse in autoregressive generation. To stabilize training, we introduce a phased strategy that first uses distribution matching to bootstrap a stable one-step generator, providing a warm-up phase that brings the student distribution closer to the teacher before adversarial distillation begins. Extensive experiments on VBench demonstrate that AAD-1 achieves state-of-the-art performance in one-step autoregressive video generation.

Haobo Li, Yanhong Zeng, Yunhong Lu +6

AutoLab, SAI, SJTU · Ant Group · Department of Automation, Tsinghua University +2

Autoregressive Video Generation Dataset Distillation

Ask PDF

Sep 28, 2026cs.CV

From Scores to Samples: Elastic Forcing for Autoregressive Video Generation

Few-step autoregressive video generation commonly relies on Distribution Matching Distillation (DMD), requiring a bidirectional diffusion teacher and an online fake-score model. We instead learn the rollout distribution directly from reference videos, eliminating both score models during post-training. Our framework minimizes maximum mean discrepancy (MMD) in frozen self-supervised video representation spaces, using a hybrid Nyström--Monte Carlo estimator to balance approximation bias and sampling variance. Memory-efficient replay and gradient subsampling make this objective practical. Using the same architecture and initialization as Self-Forcing, our 1.3B model improves the VBench Total score from 83.80 to 84.64 while retaining 17 FPS. Removing auxiliary score models also enables 14B post-training on eight H200 GPUs. Beyond distillation, learning from reference videos enables the acquisition of new visual styles, semantic concepts, and spatial priors without a target-specific diffusion teacher.

Chi Zhang, Yueyi Liu, Shi Haoyang +5

College of AI, Tsinghua University · IAIR, Xi’an Jiaotong University · Xianghui Academy, Fudan University +1

Autoregressive Video Generation Distribution Matching Distillation Scores

Ask PDF

May 25, 2026cs.CV

On-Policy Adversarial Flow Distillation for Autoregressive Video Generation

Autoregressive video generators are attractive for streaming, long-horizon, and interactive applications, but distilling strong black-box teachers into causal students remains difficult. The student must learn under its own rollout distribution, whereas practical teachers may expose only prompt-conditioned completed videos and may differ in architecture, capacity, temporal design, and sampling schedule. This interface makes supervised fine-tuning off-policy, score-based distillation inapplicable, and direct adversarial imitation too sparse for denoising-time credit assignment. We propose Adversarial Flow Distillation (AFD), an on-policy framework for heterogeneous black-box video distillation. AFD queries the teacher and rolls out the current student on the same prompts, trains a prompt-paired Bradley-Terry discriminator to estimate clean-sample teacher-student discrepancy, and converts the resulting on-policy advantage into forward-process flow-matching updates on the student's own noised states. Thus, AFD provides dense velocity-field supervision while requiring no teacher scores, latents, denoising trajectories, step alignment, or reverse-chain reinforcement learning. Experiments across two causal AR student families show that AFD consistently improves motion- and physics-sensitive generation while preserving general video quality, and ablations validate the importance of adaptive on-policy feedback and forward-process credit assignment. The method requires only clean teacher videos and student rollouts, providing a practical route for distilling proprietary or heterogeneous video generators into efficient autoregressive students.

Yang Luo, Shengju Qian, Xiaohang Tang +4

National Univesity of Singapore · LIGHTSPEED · University College London

Autoregressive Video Generation Video Generation Dataset Distillation Teacher Autoregressive Model Credit Assignment

Ask PDF