GraftOOD: Grafting Out-of-Distribution Exemplars into Advantage-Weighted Rectification for Video Reward Optimization

Zheming Fu; Ruizhe He; Wenbin Ning; Xiaoxiao Ma; Xuan Yang; Lianrui Mu; Haoyang Huang; Chang Liu

cs.CVOct 9, 2026

GraftOOD: Grafting Out-of-Distribution Exemplars into Advantage-Weighted Rectification for Video Reward Optimization

Authors: Zheming Fu, Ruizhe He, Wenbin Ning, Xiaoxiao Ma, Xuan Yang, Lianrui Mu, Haoyang Huang, Chang Liu, +2 more

Organizations: Beihang University · Zhongguancun Academy · Joy Future Academy · Zhejiang University · University of Science and Technology of China

Abstract

Video reward optimization can increase proxy rewards while reducing video dynamics, a mismatch we identify as motion collapse, including under a video-level motion-quality reward. We propose GraftOOD, which replaces part of the online rollouts in selected prompt groups with out-of-distribution video exemplars. A shared velocity-regression objective combines group-relative reward rectification with sequence-level exemplar supervision, requiring neither exemplar generation trajectories nor an additional supervised-loss coefficient. We characterize its pointwise optimal field as a source-posterior-weighted combination of the two source-specific target fields. We construct GraftSet-Video, containing 9,588 format-matched exemplars for 2,397 prompts, to supply external supervision. On Wan2.1-1.3B, GraftOOD raises Dynamic Degree from 4.86 to 44.41 relative to reward-only FlowAWR, while HPSv3 decreases from 8.10 to 7.32 and VisionReward remains close. At the main setting, GraftOOD leads nine of ten metrics among three exemplar-augmented methods. Ablations of exemplar frequency and within-group count reveal dynamics--reward trade-offs rather than uniform gains from increasing exemplar usage. These results support external exemplar supervision as a complement to scalar reward feedback, without equating higher measured dynamics with coherent, visually plausible motion in the generated videos. Code is available at GraftOOD.

Figures & tables

Figure 1: Reward-only optimization versus GraftOOD. Same-prompt sequences from reward-only optimization (left) and GraftOOD (right), with frames in temporal order.

Figure 2: GraftOOD motivation and overview. Top: exemplar-guided correction $w_{s}(v^{\mathrm{ood}}-v_{\mathrm{old}}^{\star})$ and motion examples. Bottom: (a) HPSv3, (b) Dynamic Degree, and (c) VisionReward during reward optimization. Adding RAFT to HPSv3 mitigates but does not prevent declining dynamics.

Figure 3: GraftSet-Video composition and scores. Source proportions among 9,588 exemplars (left) and mean HPSv3 by source, ranked in descending order (right). Colors match across panels.

Method	AQ	IQ	SC	OC	DD	VR	VQ	MQ	TA	HPSv3
Wan2.1 T2V 1.3B	65.63	66.79	94.58	88.70	48.61	1.26	-0.59	-0.34	-0.66	4.70
Flash-GRPO	67.46	68.10	95.85	90.36	28.67	1.65	-0.69	-0.43	-0.59	5.11
DiffusionNFT	68.30	71.50	97.49	90.9	14.24	5.20	-0.09	0.21	-0.51	7.23
FlowAWR	69.18	71.01	98.48	92.53	4.86	5.11	0.06	0.47	-0.65	8.10
FlashOOD	65.87	67.91	94.68	88.29	58.26	1.65	-0.62	-0.47	-0.50	5.31
NFTOOD	66.10	70.52	96.99	87.91	41.12	3.90	-0.45	-0.25	-0.47	6.55

Table 1: HPSv3 optimization of Wan2.1-1.3B with CFG-enabled evaluation. OOD methods use

\rho=0.167

and

n_{\mathrm{ood}}=4

. All metrics are higher-is-better (definitions in Appendix B.2 ). Bold marks column bests among FlashOOD, NFTOOD, and GraftOOD only.

Figure 4: OOD prompt-ratio sweep. (a) HPSv3, (b) DD, and (c) VisionReward with

n_{\mathrm{ood}}=4

\rho>0

. At

\rho=0

, FlashOOD, NFTOOD, and GraftOOD use Flash-GRPO, DiffusionNFT, and FlowAWR, respectively, without introducing external exemplars.

Figure 5: Qualitative comparison. Each prompt block shows Vanilla, FlashOOD, NFTOOD, and GraftOOD from top to bottom, with three frames per method in temporal order.

Figure 6: GraftOOD exemplar-count ablation. (a) HPSv3, (b) DD, and (c) VisionReward versus

\rho

for

n_{\mathrm{ood}}\in\{1,2,4\}

(

G=8

). Curves share one FlowAWR baseline at

\rho=0

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

System	Videos
wan2_2	2,164
vidu_q2	1,163
seedance_1_5	1,144
kling_2_5	1,098
kling3	915
seedance2	911

Appendix

Table 2: Exemplars by source.

Panel of Figure 7 — Figure 7: Motion-quality reward and video dynamics. VideoAlign Motion Quality reward (left) and Dynamic Degree (right) during the same reward-optimization run.

Method	AQ	IQ	SC	OC	DD	VR	VQ	MQ	TA	HPSv3
FlashOOD	65.87	67.91	94.68	88.29	58.26	1.65	-0.62	-0.47	-0.50	5.31
NFTOOD	66.10	70.52	96.99	87.91	41.12	3.90	-0.45	-0.25	-0.47	6.55
GraftOOD	67.21	71.26	98.20	90.10	44.41	5.08	-0.25	-0.07	-0.45	7.32

Appendix

Table 3: OOD comparison at

\rho=0.167

n_{\mathrm{ood}}=4

. Bold marks column bests.

Method	AQ	IQ	SC	OC	DD	VR	VQ	MQ	TA	HPSv3
FlashOOD	65.78	67.17	95.05	88.21	51.60	1.87	-0.52	-0.38	-0.49	5.62
NFTOOD	66.41	69.11	96.95	89.85	43.12	4.04	-0.38	-0.21	-0.46	6.61
GraftOOD	66.34	71.39	97.04	90.19	45.72	4.76	-0.32	-0.14	-0.50	6.94

Appendix

Table 4: OOD comparison at

\rho=0.33

n_{\mathrm{ood}}=4

. Bold marks column bests.

Method	AQ	IQ	SC	OC	DD	VR	VQ	MQ	TA	HPSv3
FlashOOD	66.10	68.40	94.72	87.10	55.13	1.96	-0.63	-0.49	-0.51	5.68
NFTOOD	66.27	69.01	95.77	89.88	45.36	3.52	-0.37	-0.14	-0.46	6.21
GraftOOD	66.03	69.21	96.75	88.92	48.68	3.97	-0.43	-0.25	-0.39	6.68

Appendix

Table 5: OOD comparison at

\rho=0.5

n_{\mathrm{ood}}=4

. Bold marks column bests.

$\rho$	AQ	IQ	SC	OC	DD	VR	VQ	MQ	TA	HPSv3
0	69.18	71.01	98.48	92.53	4.86	5.11	0.06	0.47	-0.65	8.10
0.167	67.21	71.26	98.20	90.10	44.41	5.08	-0.25	-0.07	-0.45	7.32
0.33	66.34	71.39	97.04	90.19	45.72	4.76	-0.32	-0.14	-0.50	6.94
0.5	66.03	69.21	96.75	88.92	48.68	3.97	-0.43	-0.25	-0.39	6.68

Appendix

Table 6: GraftOOD ratio sweep with

n_{\mathrm{ood}}=4

(

G=8

). The

\rho=0

row repeats FlowAWR; positive rows repeat Tables 3 – 5 . Bold marks column bests among positive ratios only.

$\rho$	AQ	IQ	SC	OC	DD	VR	VQ	MQ	TA	HPSv3
0	69.18	71.01	98.48	92.53	4.86	5.11	0.06	0.47	-0.65	8.10
0.167	66.59	71.29	98.18	90.59	31.41	5.34	-0.03	0.15	-0.53	7.6
0.33	67.57	71.08	98.25	88.26	40.79	4.89	-0.17	0.09	-0.5	7.06
0.5	66.25	69.43	97.17	89.71	45.72	3.9	-0.47	-0.28	-0.47	6.54

Appendix

Table 7: GraftOOD ratio sweep with

n_{\mathrm{ood}}=2

(

G=8

). The

\rho=0

row is the shared FlowAWR baseline. Bold marks column bests among positive ratios only.

$\rho$	AQ	IQ	SC	OC	DD	VR	VQ	MQ	TA	HPSv3
0	69.18	71.01	98.48	92.53	4.86	5.11	0.06	0.47	-0.65	8.10
0.167	68.21	72.10	98.17	88.38	30.58	5.52	-0.32	-0.13	-0.39	7.74
0.33	67.43	71.23	98.27	90.66	33.93	5.09	-0.13	0.05	-0.57	7.36
0.5	66.81	69.31	98.01	91.48	46.76	4.69	-0.38	-0.17	-0.43	7.13

Appendix

Table 8: GraftOOD ratio sweep with

n_{\mathrm{ood}}=1

(

G=8

). The

\rho=0

row is the shared FlowAWR baseline. Bold marks column bests among positive ratios only.

Figure 8: Additional qualitative comparisons. Each block shows Vanilla, FlashOOD, NFTOOD, and GraftOOD from top to bottom, with three frames in temporal order. Full prompts appear in Appendix D.2 .

Explore similar work

VideoAWR: Reinforcing Video Flow Models with Decoupled-Timestep Advantage-Weighted Rectification
Oct 9, 2026Zheming Fu, Wenbin Ning, Ruizhe He +6RL for Video Generation Flow-Based Generative Modeling
Beihang University · Zhongguancun Academy · Joy Future Academy +2
Manifold-Aware Exploration for Reinforcement Learning in Video Generation
Mar 23, 2026Mingzhe Zheng, Weijie Kong, Yue Wu +9RL for Video Generation RL Exploration
HKUST · Tencent Hunyuan
Wan-R1: Verifiable-Reinforcement Learning for Video Reasoning
Mar 29, 2026Ming Liu, Yunbei Zhang, Shilong Liu +2Video Reasoning Reinforcement Learning with Verifiable Rewards
Iowa State University, Ames, IA 50011, USA · Tulane University, New Orleans, LA 70118, USA · Princeton University, Princeton, NJ 08544, USA

Oct 9, 2026J/K move · Enter open · S save

VideoAWR: Reinforcing Video Flow Models with Decoupled-Timestep Advantage-Weighted Rectification

Zheming Fu, Wenbin Ning, Ruizhe He, Wei Shang+5

Beihang University · Zhongguancun Academy · Joy Future Academy+2

RL for Video Generation Flow-Based Generative Modeling Classifier-Free Guidance

Open paper AskPDF arXiv

Regression-based RL for flow models mandates sampling without classifier-free guidance (CFG) to align the rectification target with the reference marginal velocity. Extending Advantage-Weighted Rectification (AWR), which dynamically scales this regression target based on reward, to video generation encounters a cold-start pathology: unguided rollouts collapse into low-fidelity regions where intra-group reward gaps remain wide but no longer track perceptual quality, rendering the optimization target arbitrary. Conversely, utilizing guided rollouts to restore fidelity, while keeping the regression target tied to the guided sampling distribution, provably triggers a geometric guidance blow-up under exponential-moving-average (EMA) updates. We propose VideoAWR, which resolves this tension by untying sampling from the optimization anchor. An EMA reference policy generates guided rollouts to restore reward separability, while the trainable policy is rectified against the unguided conditional anchor. We prove this decoupled objective is mathematically exact: the centered advantage cancels the anchor in expectation, filtering out the reward-neutral guidance offset and retaining only the reward-correlated covariance increment. Consequently, inference runs single-branch and CFG-free, yet inference-time CFG preserves its original semantics. Furthermore, we introduce Decoupled Timestep Rectification, supervising each rollout at an independently sampled timestep without bias to improve sample efficiency. Experiments on Wan2.1-1.3B and 14B demonstrate stable optimization and superior visual and motion quality in both CFG-free and CFG-enabled regimes. Code is available at \pdfstartlink attr{/Border [0 0 0]} user{/Subtype /Link /A << /S /URI /URI (https://github.com/Fuzm7/FlowAWR) >>}% \textcolor{pearDark}{VideoAWR}\pdfendlink.

Abstract

Explore similar work

VideoAWR: Reinforcing Video Flow Models with Decoupled-Timestep Advantage-Weighted Rectification

Manifold-Aware Exploration for Reinforcement Learning in Video Generation

Wan-R1: Verifiable-Reinforcement Learning for Video Reasoning