Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models

Songchun Zhang; Zeyue Xue; Siming Fu; Jie Huang; Xianghao Kong; Y Ma; Haoyang Huang; Nan Duan

cs.CVMar 17, 2026

Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models

Authors: Songchun Zhang, Zeyue Xue, Siming Fu, Jie Huang, Xianghao Kong, Y Ma, Haoyang Huang, Nan Duan,

Organizations: HKUST · JD Explore Academy · HKU

Abstract

Distilled autoregressive (AR) video models enable efficient streaming generation but frequently misalign with human visual preferences. Existing reinforcement learning (RL) frameworks are not naturally suited to these architectures, typically requiring either expensive re-distillation or solver-coupled reverse-process optimization that introduces considerable memory and computational overhead. We present Astrolabe, an efficient online RL framework tailored for distilled AR models. To overcome existing bottlenecks, we introduce a forward-process RL formulation based on negative-aware fine-tuning. By contrasting positive and negative samples directly at inference endpoints, this approach establishes an implicit policy improvement direction without requiring reverse-process unrolling. To scale this alignment to long videos, we propose a streaming training scheme that generates sequences progressively via a rolling KV-cache, applying RL updates exclusively to local clip windows while conditioning on prior context to ensure long-range coherence. Finally, to mitigate reward hacking, we integrate a multi-reward objective stabilized by uncertainty-aware selective regularization and dynamic reference updates. Extensive experiments demonstrate that our method consistently enhances generation quality across multiple distilled AR video models, serving as a robust and scalable alignment solution.

Figures & tables

Figure 1: A s t r o l a b e efficiently aligns distilled streaming video models with human preferences without re-distillation, enhancing baselines (e.g., Causal Forcing [ 64 ] , LongLive [ 51 ] and Infinite-RoPE [ 53 ] ) by mitigating artifacts and improving temporal consistency. We demonstrate boosted perceptual quality across: (Top) single-prompt short, (Middle) single-prompt long, and (Bottom) multi-prompt long video generation.

Figure 2: Overview of Astrolabe. We propose a memory-efficient RL framework for distilled streaming video models. The method combines group-wise streaming rollout using a rolling KV cache for efficient group-wise sampling (see left), and clip-level forward-process RL for solver-agnostic optimization (see middle). To scale to long videos, we utilize Streaming Long Tuning with detached historical gradients. Furthermore, a multi-reward formulation paired with uncertainty-based selective regularization is employed to effectively mitigate reward hacking during training (see right). The pseudocode of the algorithm can be found in the supplementary materials.

Figure 3: Qualitative comparison under the short-video, single-prompt setting. We evaluate our framework (+Ours) against other baselines. Visual results confirm that our method generates videos with significantly sharper textures and superior motion coherence, aligning better with human preferences. More results can be found in supplementary material.

Diffusion Models
Method	Total $\uparrow$	Quality $\uparrow$	Semantic $\uparrow$	HPSv3 $\uparrow$	MQ $\uparrow$	Throughput $\uparrow$
LTX-Video [ 15 ]	80.00	82.30	70.79	8.32	1.34	8.98
Wan2.1 [ 42 ]	84.26	85.30	80.09	9.26	1.62	0.78
AR Models
SkyReels-V2 [ 7 ]	82.67	84.70	74.53	9.08	1.59	0.49
MAGI-1 [ 40 ]	79.18	82.04	67.74	7.95	1.52	0.19

Table 1: Quantitative results on VBench benchmarks. Integrating our approach into existing distilled models yields consistent improvements in motion quality, semantic alignment, and overall generation quality.

Figure 4: Qualitative results under the single-prompt long-video setting. Our framework (+Ours) effectively translates alignment optimizations from short videos to extended temporal horizons. Our approach delivers enhanced visual details and more stable throughout the sequence.

Figure 5: Performance improvements across different models. We evaluate our method on three models. The dashed grey lines indicate the baseline performance of the respective base models. The results demonstrate that our approach consistently improves both HPSv3 and MQ scores across all three models.

Method	Total $\uparrow$	Quality $\uparrow$	Semantic $\uparrow$	HPSv3 $\uparrow$	MQ $\uparrow$
SkyReels-V2 [ 7 ]	75.29	80.77	53.37	8.72	1.54
FramePack [ 56 ]	81.95	83.61	75.32	8.94	1.58
Self-Forcing [ 22 ]	81.59	83.82	72.70	9.12	1.61
$+$ Ours	82.03	84.36	72.71	10.38	1.72
LongLive [ 51 ]	83.52	85.44	75.82	9.21	1.48
$+$ Ours	84.07	86.12	75.87	10.67	1.64

Table 2: Quantitative results on VBench-Long benchmarks. Integrating our method consistently improves the performance of long video generation baselines across both video quality and human preference metrics.

Figure 6: Qualitative comparison of multi-prompt long-video generation. We compare the LongLive [ 51 ] with our method. The generated sequences exhibit noticeable improvements in visual aesthetics and fine details during complex narrative transitions.

Method	Quality Score $\uparrow$	Consistency Score $\uparrow$	Aesthetic Score $\uparrow$	CLIP Score $\uparrow$
Method	Quality Score $\uparrow$	Consistency Score $\uparrow$	Aesthetic Score $\uparrow$	0-10	10-20	20-30	30-40	40-50	50-60
SkyReels-V2 [ 7 ]	81.55	94.72	56.83	25.31	23.40	22.50	21.62	21.67	20.91
FramePack [ 56 ]	84.40	96.77	59.44	26.51	22.60	22.18	21.53	21.98	21.62
Self-Forcing [ 22 ]	83.94	95.74	58.45	26.24	24.87	23.46	21.92	22.05	21.07
+ Ours	84.72	95.98	59.62	26.42	24.75	23.95	22.40	21.85	21.50
LongLive [ 51 ]	84.28	96.05	59.89	26.63	25.77	24.65	23.99	24.52	24.11

Table 3: Quantitative evaluation on long video generation. We compare these overall metrics alongside CLIP Scores evaluated across 10-second intervals (0-60s).

Panel of Figure 7 — Figure 7: Ablation studies on reward formulation and interpolation strength. (a) Optimizing a single objective induces reward hacking and degrades other quality dimensions, whereas our aggregated formulation balances visual aesthetics and motion consistency. (b) The parameter $\beta$ controls the implicit contrast between positive and negative samples, with $\beta=1.0$ yielding the optimal trade-off for quality.

Config	HPSv3 $\uparrow$	MQ $\uparrow$	Mem $\downarrow$
Seq + Full BP	OOM	OOM	$>$ 140
Seq + Detach	10.21	1.72	96.4
Clip + Full BP	10.58	1.76	112.3
Clip+Detach	10.84	1.80	54.3

Table 4: Ablation studies on each component. (a) and (b) show the impact of the streaming strategy and selective KL regularization. (c) compares different reward combinations.

Panel of Figure 8 — Figure 8: Ablation study of stabilization techniques in Astrolabe. (a) Effect of the selective KL penalty. (b) Impact of the dynamic adaptive weight mechanism.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Module	Hyperparameter	Value
Model & Video Specs	Base Architecture	Causal Wan 2.1
Model & Video Specs	Video Resolution ( $H\times W$ )	$480\times 832$
LoRA Fine-Tuning	Rank ( $r$ )	256
	Scaling Factor ( $\alpha$ )	256
	Dropout Rate	0.0
	Gradient Checkpointing	True

Appendix

Table S1 : Comprehensive Hyperparameters for Astrolabe Training. We detail the configurations used across the model architecture, optimization, diffusion process, reinforcement learning, and streaming rollout.

Figure S1 : Qualitative comparison between Krea 14B (odd rows) and with our method (+Ours, even rows)

Figure S2 : Qualitative comparison on long video generation (30s) under the single-prompt setting (Set 1).

Figure S3 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 2).

Figure S4 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 3).

Figure S5 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 4).

Figure S6 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 5).

Figure S7 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 6).

Figure S8 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 7).

Figure S9 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 8).

Figure S10 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 9).

Figure S11 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 10).

Figure S12 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 11).

Figure S13 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 12).

Figure S14 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 13).

Figure S15 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 14).

Figure S16 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 15).

Figure S17 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 16).

Figure S18 : Additional ualitative comparison on long video generation (30s) under the single-prompt setting (Set 17).

Figure S19 : Additional ualitative comparison on long video generation (30s) under the single-prompt setting (Set 18).

Figure S20 : Additional ualitative comparison on long video generation (30s) under the single-prompt setting (Set 19).

Figure S21 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 20).

Figure S22 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 21).

Figure S23 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 22).

Figure S24 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 23).

Figure S25 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 24).

Figure S26 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 25).

Figure S27 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 26).

Figure S28 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 27).

Figure S29 : Additional qualitative comparison on long video generation (30s) under the single-prompt setting (Set 28).

RL for Video Generation

Reinforcement learning for video generation fine-tunes video models with reward feedback to improve quality, motion, controllability, and physical consistency.

See all 48 papers

Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation
May 5, 2026Bin Wu, Mengqi Huang, Shaojin Wu +4Video Diffusion Models Diffusion Model Distillation
University of Science and Technology of China · Independent Researcher
AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization
Mar 18, 2026Dailan He, Guanlin Feng, Xingtong Ge +5RL for Video Generation Streaming Video Generation
CUHK MMLab, China · Vivix Group Limited, China · HKUST, China +1
Rollout-Marginal Distillation for Long-Horizon Autoregressive Video Generation
Sep 29, 2026Chenjian Gao, Zhihao Hu, Jianqi Ma +3Autoregressive Diffusion Long-Horizon Video Generation
MMLab, The Chinese University of Hong Kong · Tencent AIPD

May 5, 2026J/K move · Enter open · S save

Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation

Bin Wu, Mengqi Huang, Shaojin Wu, Weinan Jia+3

University of Science and Technology of China · Independent Researcher

Video Diffusion Models Diffusion Model Distillation Streaming Video Generation Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Distillation-based acceleration has become foundational for making autoregressive streaming video diffusion models practical, with distribution matching distillation (DMD) as the de facto choice. Existing methods, however, train the student to match the teacher's output indiscriminately, treating every rollout, frame, and pixel as equally reliable supervision. We argue that this caps distilled quality, since it overlooks two complementary axes of variance in DMD supervision: Inter-Reliability across student rollouts whose supervision varies in reliability, and Intra-Perplexity across spatial regions and temporal frames that contribute unequally to where quality can still be improved. The objective thus conflates two questions under a uniform weight: whether to learn from each rollout, and where to concentrate optimization within it. To address this, we propose Stream-R1, a Reliability-Perplexity Aware Reward Distillation framework that adaptively reweights the distillation objective at both rollout and spatiotemporal-element levels through a single shared reward-guided mechanism. At the Inter-Reliability level, Stream-R1 rescales each rollout's loss by an exponential of a pretrained video reward score, so that rollouts with reliable supervision dominate optimization. At the Intra-Perplexity level, it back-propagates the same reward model to extract per-pixel gradient saliency, which is factored into spatial and temporal weights that concentrate optimization pressure on regions and frames where refinement yields the largest expected gain. An adaptive balancing mechanism prevents any single quality axis from dominating across visual quality, motion quality, and text alignment. Stream-R1 attains consistent improvements on all three dimensions over distillation baselines on standard streaming video generation benchmarks, without architectural modification or additional inference cost.

Abstract

RL for Video Generation

Explore similar work

Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation

AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization

Rollout-Marginal Distillation for Long-Horizon Autoregressive Video Generation