VideoAWR: Reinforcing Video Flow Models with Decoupled-Timestep Advantage-Weighted Rectification

Zheming Fu; Wenbin Ning; Ruizhe He; Wei Shang; Xiaoxiao Ma; Lianrui Mu; Chang Liu; Lei Wang

cs.CVOct 9, 2026

VideoAWR: Reinforcing Video Flow Models with Decoupled-Timestep Advantage-Weighted Rectification

Authors: Zheming Fu, Wenbin Ning, Ruizhe He, Wei Shang, Xiaoxiao Ma, Lianrui Mu, Chang Liu, Lei Wang, +1 more

Organizations: Beihang University · Zhongguancun Academy · Joy Future Academy · Zhejiang University · University of Science and Technology of China

Abstract

Regression-based RL for flow models mandates sampling without classifier-free guidance (CFG) to align the rectification target with the reference marginal velocity. Extending Advantage-Weighted Rectification (AWR), which dynamically scales this regression target based on reward, to video generation encounters a cold-start pathology: unguided rollouts collapse into low-fidelity regions where intra-group reward gaps remain wide but no longer track perceptual quality, rendering the optimization target arbitrary. Conversely, utilizing guided rollouts to restore fidelity, while keeping the regression target tied to the guided sampling distribution, provably triggers a geometric guidance blow-up under exponential-moving-average (EMA) updates. We propose VideoAWR, which resolves this tension by untying sampling from the optimization anchor. An EMA reference policy generates guided rollouts to restore reward separability, while the trainable policy is rectified against the unguided conditional anchor. We prove this decoupled objective is mathematically exact: the centered advantage cancels the anchor in expectation, filtering out the reward-neutral guidance offset and retaining only the reward-correlated covariance increment. Consequently, inference runs single-branch and CFG-free, yet inference-time CFG preserves its original semantics. Furthermore, we introduce Decoupled Timestep Rectification, supervising each rollout at an independently sampled timestep without bias to improve sample efficiency. Experiments on Wan2.1-1.3B and 14B demonstrate stable optimization and superior visual and motion quality in both CFG-free and CFG-enabled regimes. Code is available at \pdfstartlink attr{/Border [0 0 0]} user{/Subtype /Link /A << /S /URI /URI (https://github.com/Fuzm7/FlowAWR) >>}% \textcolor{pearDark}{VideoAWR}\pdfendlink.

Figures & tables

Figure 2: Decoupled Timestep Rectification and scaling to 14B. Left: DTR assigns each rollout in a prompt group its own independently drawn timestep on $[0,N]$ , so one group supervises the velocity field at several distinct points instead of a single shared timestep (Sec. 4.3 ). Right: HPSv3 reward of VideoAWR on Wan2.1-1.3B and Wan2.1-14B against training cost in GPU hours, for (a) training and (b) the held-out evaluation prompts. Both backbones improve under the same hyperparameters.

Table 1: Detailed comparison of General Video Quality using VBench metrics. We evaluate aesthetic quality, image quality, subject consistency, and object class. We reproduce the official VBench results. Best scores are in Bold .

Figure 4: Qualitative comparison of generated videos. Comparison across Aesthetic quality, Motion dynamics, and Prompt Following. Our VideoAWR achieves superior visual fidelity, temporal consistency, and prompt alignment compared to Flash-GRPO and Vanilla baselines.

Figure 5: Ablations on Wan2.1-1.3B with the HPSv3 reward, plotted against training cost in GPU hours. (a) Rollout and anchor. Ours is highest throughout; no-CFG sampling falls behind during cold start and does not close the gap, while anchor-on-CFG decreases monotonically and diverges before

300

GPU hours, as predicted by Proposition 1 . (b) Timestep scheme. Independent timesteps (DTR) versus a group-shared timestep (Flash), under our objective and under GRPO: ours exceeds GRPO at either scheme, and DTR helps within our objective (Ours

>

Ours-Flash) but hurts under GRPO (DTR-GRPO

<

Flash-GRPO). (c) Group size. At a matched GPU budget

G{=}8

is the best trade-off.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Task Metric			DrawBench Metric
	GenEval	OCR	PickScore	CLIPScore	HPSv2.1	Aesthetic	ImgRwd	PickScore
DiffusionNFT (CFG-free)	0.93	0.90	23.36	0.293	0.330	5.99	1.48	23.79
DiffusionNFT (with CFG)	0.84	0.69	20.99	0.291	0.273	5.16	1.03	22.03

Appendix

Table 2: CFG-free and CFG-enabled evaluation of the official DiffusionNFT checkpoint. All metrics are higher-is-better. Best values between the two inference modes are in bold .

Figure 6: CFG-free and CFG-enabled evaluation under different behavior samplers. We train Wan2.1-T2V-1.3B with HPSv3 while anchoring the rectification target on the unguided conditional field

v^{\mathrm{old}}

in both panels. (a) No-CFG behavior sampling. (b) CFG-guided behavior sampling. Each trained checkpoint is evaluated both CFG-free (red) and with CFG (blue). Guided behavior raises both curves and preserves a consistent inference-time CFG gain.

Figure 7: Qualitative comparison among VideoAWR, Flash-GRPO, and the Vanilla baseline fine-tuned with the HPSv3 reward on VBench prompts (Wan14B).

Explore similar work

FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification
Jun 29, 2026Zheming Fu, Ruizhe He, Wei Shang +5Flow Matching Generative Modeling
Beihang University · Joy Future Academy · Zhongguancun Academy +2
AdvantageFlow: Regularized Advantage-Weighted RL in Flow Models
May 25, 2026Branislav Kveton, Anup Rao, Subhojyoti Mukherjee +2Rectified Flow RL for Image Generation
Adobe Research
GraftOOD: Grafting Out-of-Distribution Exemplars into Advantage-Weighted Rectification for Video Reward Optimization
Oct 9, 2026Zheming Fu, Ruizhe He, Wenbin Ning +7RL for Video Generation RL for Generative Models
Beihang University · Zhongguancun Academy · Joy Future Academy +2

Jun 29, 2026J/K move · Enter open · S save

FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification

Zheming Fu, Ruizhe He, Wei Shang, Xiaoxiao Ma+4

Beihang University · Joy Future Academy · Zhongguancun Academy+2

Flow Matching Generative Modeling Flow-Based Generative Modeling RL for Generative Models

Open paper AskPDF arXiv

Aligning generative flow models on continuous spaces via online reinforcement learning is constrained by intractable trajectory likelihoods. Existing density-approximated policy gradient methods rely on stochastic SDE samplers to construct tractable transition kernels, which introduce training-inference inconsistencies and necessitates Classifier-Free Guidance (CFG). While implicit frameworks such as DiffusionNFT directly optimize forward-process velocity fields, its heuristic fixed-magnitude corrections prevent optimization strength from relative intra-group quality. We propose Flow Advantage-Weighted Rectification (FlowAWR), a paradigm that recasts continuous generative policy optimization as supervised regression toward a theoretically optimal velocity field. Starting from the optimal policy of a KL-constrained reward maximization, FlowAWR derives the optimal velocity field that admits a magnitude-aware, advantage-weighted rectification form, yielding SDE-free optimization and CFG-free generation. In comparative evaluations on SD3.5-Medium, FlowAWR achieves improved alignment performance alongside a 2

\times

to 5

\times

convergence acceleration over DiffusionNFT (e.g., reaching a 24.12 PickScore in 1.2k steps, versus 23.82 in 2.0k steps for DiffusionNFT and 23.50 in

>

4k steps for FlowGRPO). Under multi-reward constraints, FlowAWR sustains generation quality, satisfying structural rules while maintaining stable out-of-domain performance. Code at FlowAWR.

Abstract

Explore similar work

FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification

AdvantageFlow: Regularized Advantage-Weighted RL in Flow Models

GraftOOD: Grafting Out-of-Distribution Exemplars into Advantage-Weighted Rectification for Video Reward Optimization