Regression-based RL for flow models mandates sampling without classifier-free guidance (CFG) to align the rectification target with the reference marginal velocity. Extending Advantage-Weighted Rectification (AWR), which dynamically scales this regression target based on reward, to video generation encounters a cold-start pathology: unguided rollouts collapse into low-fidelity regions where intra-group reward gaps remain wide but no longer track perceptual quality, rendering the optimization target arbitrary. Conversely, utilizing guided rollouts to restore fidelity, while keeping the regression target tied to the guided sampling distribution, provably triggers a geometric guidance blow-up under exponential-moving-average (EMA) updates. We propose VideoAWR, which resolves this tension by untying sampling from the optimization anchor. An EMA reference policy generates guided rollouts to restore reward separability, while the trainable policy is rectified against the unguided conditional anchor. We prove this decoupled objective is mathematically exact: the centered advantage cancels the anchor in expectation, filtering out the reward-neutral guidance offset and retaining only the reward-correlated covariance increment. Consequently, inference runs single-branch and CFG-free, yet inference-time CFG preserves its original semantics. Furthermore, we introduce Decoupled Timestep Rectification, supervising each rollout at an independently sampled timestep without bias to improve sample efficiency. Experiments on Wan2.1-1.3B and 14B demonstrate stable optimization and superior visual and motion quality in both CFG-free and CFG-enabled regimes. Code is available at \pdfstartlink attr{/Border [0 0 0]} user{/Subtype /Link /A << /S /URI /URI (https://github.com/Fuzm7/FlowAWR) >>}% \textcolor{pearDark}{VideoAWR}\pdfendlink.
Figures & tables
Figure 2: Decoupled Timestep Rectification and scaling to 14B. Left: DTR assigns each rollout in a prompt group its own independently drawn timestep on [0,N] , so one group supervises the velocity field at several distinct points instead of a single shared timestep (Sec. 4.3 ). Right: HPSv3 reward of VideoAWR on Wan2.1-1.3B and Wan2.1-14B against training cost in GPU hours, for (a) training and (b) the held-out evaluation prompts. Both backbones improve under the same hyperparameters.
Table 1: Detailed comparison of General Video Quality using VBench metrics. We evaluate aesthetic quality, image quality, subject consistency, and object class. We reproduce the official VBench results. Best scores are in Bold .
Figure 4: Qualitative comparison of generated videos. Comparison across Aesthetic quality, Motion dynamics, and Prompt Following. Our VideoAWR achieves superior visual fidelity, temporal consistency, and prompt alignment compared to Flash-GRPO and Vanilla baselines.
Figure 5: Ablations on Wan2.1-1.3B with the HPSv3 reward, plotted against training cost in GPU hours. (a) Rollout and anchor. Ours is highest throughout; no-CFG sampling falls behind during cold start and does not close the gap, while anchor-on-CFG decreases monotonically and diverges before 300 GPU hours, as predicted by Proposition 1 . (b) Timestep scheme. Independent timesteps (DTR) versus a group-shared timestep (Flash), under our objective and under GRPO: ours exceeds GRPO at either scheme, and DTR helps within our objective (Ours > Ours-Flash) but hurts under GRPO (DTR-GRPO < Flash-GRPO). (c) Group size. At a matched GPU budget G=8 is the best trade-off.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Task Metric
DrawBench Metric
GenEval
OCR
PickScore
CLIPScore
HPSv2.1
Aesthetic
ImgRwd
PickScore
DiffusionNFT (CFG-free)
0.93
0.90
23.36
0.293
0.330
5.99
1.48
23.79
DiffusionNFT (with CFG)
0.84
0.69
20.99
0.291
0.273
5.16
1.03
22.03
Appendix
Table 2: CFG-free and CFG-enabled evaluation of the official DiffusionNFT checkpoint. All metrics are higher-is-better. Best values between the two inference modes are in bold .
Figure 6: CFG-free and CFG-enabled evaluation under different behavior samplers. We train Wan2.1-T2V-1.3B with HPSv3 while anchoring the rectification target on the unguided conditional field vold in both panels. (a) No-CFG behavior sampling. (b) CFG-guided behavior sampling. Each trained checkpoint is evaluated both CFG-free (red) and with CFG (blue). Guided behavior raises both curves and preserves a consistent inference-time CFG gain.
Figure 7: Qualitative comparison among VideoAWR, Flash-GRPO, and the Vanilla baseline fine-tuned with the HPSv3 reward on VBench prompts (Wan14B).
Aligning generative flow models on continuous spaces via online reinforcement learning is constrained by intractable trajectory likelihoods. Existing density-approximated policy gradient methods rely on stochastic SDE samplers to construct tractable transition kernels, which introduce training-inference inconsistencies and necessitates Classifier-Free Guidance (CFG). While implicit frameworks such as DiffusionNFT directly optimize forward-process velocity fields, its heuristic fixed-magnitude corrections prevent optimization strength from relative intra-group quality. We propose Flow Advantage-Weighted Rectification (FlowAWR), a paradigm that recasts continuous generative policy optimization as supervised regression toward a theoretically optimal velocity field. Starting from the optimal policy of a KL-constrained reward maximization, FlowAWR derives the optimal velocity field that admits a magnitude-aware, advantage-weighted rectification form, yielding SDE-free optimization and CFG-free generation. In comparative evaluations on SD3.5-Medium, FlowAWR achieves improved alignment performance alongside a 2× to 5× convergence acceleration over DiffusionNFT (e.g., reaching a 24.12 PickScore in 1.2k steps, versus 23.82 in 2.0k steps for DiffusionNFT and 23.50 in >4k steps for FlowGRPO). Under multi-reward constraints, FlowAWR sustains generation quality, satisfying structural rules while maintaining stable out-of-domain performance. Code at FlowAWR.
Zheming Fu, Ruizhe He, Wei Shang +5
Beihang University · Joy Future Academy · Zhongguancun Academy +2
We present AdvantageFlow, a forward-process reinforcement learning (RL) algorithm for rectified flow models. The algorithm minimizes an advantage-weighted prediction loss, which maximizes reward, regularized by the rollout policy, which convexifies the objective and makes its optimization stable. Our objective can be viewed as fitting a local reward-improving target distribution. The rollout regularization arises as a variance reduction step. We evaluate AdvantageFlow empirically on text-to-image generation with Stable Diffusion 3.5 Medium and FLUX.1, and compare it to both forward- and reverse-process RL algorithms.
Video reward optimization can increase proxy rewards while reducing video dynamics, a mismatch we identify as motion collapse, including under a video-level motion-quality reward. We propose GraftOOD, which replaces part of the online rollouts in selected prompt groups with out-of-distribution video exemplars. A shared velocity-regression objective combines group-relative reward rectification with sequence-level exemplar supervision, requiring neither exemplar generation trajectories nor an additional supervised-loss coefficient. We characterize its pointwise optimal field as a source-posterior-weighted combination of the two source-specific target fields. We construct GraftSet-Video, containing 9,588 format-matched exemplars for 2,397 prompts, to supply external supervision. On Wan2.1-1.3B, GraftOOD raises Dynamic Degree from 4.86 to 44.41 relative to reward-only FlowAWR, while HPSv3 decreases from 8.10 to 7.32 and VisionReward remains close. At the main setting, GraftOOD leads nine of ten metrics among three exemplar-augmented methods. Ablations of exemplar frequency and within-group count reveal dynamics--reward trade-offs rather than uniform gains from increasing exemplar usage. These results support external exemplar supervision as a complement to scalar reward feedback, without equating higher measured dynamics with coherent, visually plausible motion in the generated videos. Code is available at GraftOOD.
Zheming Fu, Ruizhe He, Wenbin Ning +7
Beihang University · Zhongguancun Academy · Joy Future Academy +2