Manifold-Aware Exploration for Reinforcement Learning in Video Generation

Mingzhe Zheng; Weijie Kong; Yue Wu; Dengyang Jiang; Yue Ma; Xuanhua He; Bin Lin; Kaixiong Gong

cs.CVMar 23, 2026

Manifold-Aware Exploration for Reinforcement Learning in Video Generation

Authors: Mingzhe Zheng, Weijie Kong, Yue Wu, Dengyang Jiang, Yue Ma, Xuanhua He, Bin Lin, Kaixiong Gong, +4 more

Organizations: HKUST · Tencent Hunyuan

Abstract

Group Relative Policy Optimization (GRPO) methods for video generation like FlowGRPO remain far less reliable than their counterparts for language models and images. This gap arises because video generation has a complex solution space, and the ODE-to-SDE conversion used for exploration can inject excess noise, lowering rollout quality and making reward estimates less reliable, which destabilizes post-training alignment. To address this problem, we view the pre-trained model as defining a valid video data manifold and formulate the core problem as constraining exploration within the vicinity of this manifold, ensuring that rollout quality is preserved and reward estimates remain reliable. We propose SAGE-GRPO (Stable Alignment via Exploration), which applies constraints at both micro and macro levels. At the micro level, we derive a precise manifold-aware SDE with a logarithmic curvature correction and introduce a gradient norm equalizer to stabilize sampling and updates across timesteps. At the macro level, we use a dual trust region with a periodic moving anchor and stepwise constraints so that the trust region tracks checkpoints that are closer to the manifold and limits long-horizon drift. We evaluate SAGE-GRPO on HunyuanVideo1.5 using the original VideoAlign as the reward model and observe consistent gains over previous methods in VQ, MQ, TA, and visual metrics (CLIPScore, PickScore), demonstrating superior performance in both reward maximization and overall video quality. The code and visual gallery are available at https://dungeonmassster.github.io/SAGE-GRPO-Page/.

Figures & tables

Figure 2: Geometric interpretation of noise injection strategies. Conventional linear SDEs (red) inject exploration noise using first-order approximations, ignoring signal decay curvature and causing off-manifold drift that results in temporal jitter and artifacts. Our Manifold-Aware SDE (blue) uses a logarithmic correction term so that exploration noise is concentrated closer to the flow trajectory and the video manifold, reducing off-manifold drift.

Method	Standard Deviation $\Sigma_{t}^{1/2}$
DanceGRPO	$\eta\sqrt{\sigma_{t}-\sigma_{t+1}}$
FlowGRPO	$\eta\sqrt{\frac{\sigma_{t}}{1-\sigma_{t}}(\sigma_{t}-\sigma_{t+1})}$
Ours (Precise)	$\eta\sqrt{\left[-(\sigma_{t}-\sigma_{t+1})+\log\left(\frac{1-\sigma_{t+1}}{1-\sigma_{t}}\right)\right]}$

Table 1: Comparison of SDE noise injection strategies used in video GRPO.

Panel of Figure 3 — Figure 3: Temporal gradient balancing ablation across SDE formulations. Overall VideoAlign reward curves comparing runs with and without the Gradient Norm Equalizer. Without balancing, low-noise timesteps dominate optimization, leading to unstable or plateaued rewards. With balancing, reward curves become smoother with consistent improvement, and gradient scale variation is reduced from more than one order of magnitude to within a small constant factor.

Figure 4: Empirical gradient norm imbalance across noise levels. Observed norms (blue) decrease rapidly as $\sigma$ increases and match the predicted relationship (red) $\|\nabla\log\pi\|\propto 1/\Sigma_{t}^{1/2}$ , leading to vanishing gradients at high noise ( $\sigma\to 1$ ) and exploding gradients at low noise ( $\sigma\to 0$ ).

Figure 5: The SAGE-GRPO Framework. Our method resolves the stability-plasticity dilemma with three coupled components: (Left) a manifold-aware SDE that keeps exploration noise tangent to the video manifold, (Middle) a Temporal Gradient Equalizer that balances optimization across timesteps, and (Right) a Dual Trust Region that combines moving anchors and step-wise KL constraints for long-term stable alignment.

Figure 6: Qualitative comparison against baselines. Three prompts illustrate our core gains: (Top) Reduced temporal jitter while preserving accurate visual contents; (Middle) Enhanced alignment and photorealism under occlusion and lighting changes; (Bottom) Stronger semantic alignment with consistent prompt matching across frames.

Method	Configuration	VideoAlign Metrics				Visual Metrics
Method	Configuration	Overall	VQ	MQ	TA	CLIPScore	PickScore
HunyuanVideo 1.5 (Original)	-	0.0654	-0.7539	-0.5870	1.4063	0.5409	0.7397
Setting A: Averaged Rewards ( $w_{vq}=1.0,w_{mq}=1.0,w_{ta}=1.0$ )
DanceGRPO	w/o KL	0.2768	-0.7589	-0.3852	1.4209	0.5386	0.7378
DanceGRPO	w/ Fixed KL	0.0979	-0.8077	-0.5091	1.4147	0.5403	0.7355
FlowGRPO	w/o KL	0.2733	-0.7151	-0.5286	1.5170	0.5443	0.7394

Table 2: Main Comparison on Video Generation Benchmarks. Comparison of SAGE-GRPO with baselines under two reward settings. The first row reports the original HunyuanVideo 1.5 performance. For each method, we report results without KL regularization ( w/o KL ) and with their Fixed KL constraints ( w/ Fixed KL ). For SAGE-GRPO, we demonstrate the w/ Dual Moving KL mechanism. Bold , underline , and gray indicate the best, second best, and third best results, computed across both settings (A+B).

Panel of Figure 7 — Figure 7: KL weight ablation on VideoAlign rewards. Comparison of three KL weight schedules: fixed $10^{-5}$ (green), two-stage $10^{-7}\rightarrow 10^{-5}$ (red), and two-stage $10^{-7}\rightarrow 10^{-6}$ (yellow). The two-stage schedule $10^{-7}\rightarrow 10^{-5}$ achieves the strongest and most consistent gains across VQ, MQ, and TA, supporting gradually increasing $\lambda_{KL}$ to tighten the trust region (Appendix A.6 ).

SAGE-GRPO vs.	Visual Quality	Motion Quality	Semantic Alignment
DanceGRPO	85.9%	75.8%	79.2%
FlowGRPO	83.8%	79.2%	71.9%
CPS	80.2%	70.8%	67.9%

Table 3: User Preference Study. Win rates of SAGE-GRPO against baselines. Results indicate a strong human preference for our method, especially in Motion Quality, confirming that automatic metrics align with perceptual quality.

Panel of Figure 8 — Figure 8: KL strategy ablation. (a) Dual Moving KL achieves the highest and most stable reward, supporting the position-velocity control interpretation (Equation ( 14 )). (b) Moving KL attains high exploration in early training steps but the exploration level drops in later stages. Dual Moving KL maintains a higher and more stable exploration level throughout training.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Step-wise std comparison: our precise SDE vs. FlowGRPO. (a) When both use FlowGRPO’s $\sigma$ schedule, our integral-based formulation yields near-zero std at the first step due to equal endpoints. (b) When both are clamped at $(1-\sigma)\geq 3\times 10^{-3}$ , FlowGRPO explodes at step 1, while ours remains stable. (c) Under default implementations, ours maintains lower std across most steps. This supports that we remove ineffective high-frequency noise and explore more precisely along the manifold (Section 3.2.1 ).

Figure 10: Qualitative comparison highlighting emotional alignment. Two prompts illustrate SAGE-GRPO’s ability to better align with emotional descriptions: (Top) A teenage boy in a coffee shop, where SAGE-GRPO captures the ”calm, contemplative expression” and gentle motion of lowering the mug, while baselines show neutral expressions and abrupt movements. (Bottom) An older chef in a kitchen, where SAGE-GRPO consistently renders the ”lines of fatigue” and ”somber mood” through deep side-lighting shadows, while baselines fail to convey the intended emotional depth. Our manifold-aware exploration enables precise alignment with subtle emotional and action cues.

Figure 11: KL strategy ablation: qualitative comparison (Case 1). Visual comparison across different KL strategies (no KL, Fixed KL, Step-wise KL, Moving KL, Dual Moving KL) on a prompt describing a fatigued soldier. Dual Moving KL produces more realistic facial details, better dirt and grime rendering, and maintains temporal consistency across frames compared to other variants.

Figure 12: KL strategy ablation: qualitative comparison (Case 2). Additional visual comparison demonstrating how different KL strategies affect generation quality. Dual Moving KL consistently achieves better photorealism and alignment with prompt descriptions compared to alternatives, validating the position-velocity control mechanism discussed in Section 3.2.2 .

Explore similar work

Rethinking Reward Signals in Video GRPO: When Scores Become Targets
Nov 24, 2025Rui Li, Yuanzhi Liang, Ziqi Ni +3Reward Hacking RL for Video Generation
University of Science and Technology of China, Hefei, China · Institute of Artificial Intelligence (TeleAI), China Telecom, Shanghai, China · Southeast University, Nanjing, China
Token-Level Video Reinforcement Learning
Oct 1, 2026Yifan Wang, Gordon Guocheng Qian, Yanyu Li +2Video Diffusion Models RL for Video Generation
Northeastern University
KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration
May 14, 2026Ruicheng Zhang, Kaixi Cong, Jun Zhou +5RL for Video Generation Long Video Generation
Tsinghua University · Video Rebirth · HKUST

Nov 24, 2025J/K move · Enter open · S save

Rethinking Reward Signals in Video GRPO: When Scores Become Targets

Rui Li, Yuanzhi Liang, Ziqi Ni, Haibin Huang+2

University of Science and Technology of China, Hefei, China · Institute of Artificial Intelligence (TeleAI), China Telecom, Shanghai, China · Southeast University, Nanjing, China

Reward Hacking RL for Video Generation Group Relative Policy Optimization Multimodal Reward Modeling Video Generation

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) enables stable and preference-oriented updates via group-wise comparisons for post-training video generation. However, GRPO directly optimizes reward-induced advantages. Under sustained optimization, the reward score can lose fidelity as a proxy for true video quality, consistent with the phenomenon described by Goodhart's Law. This leads to two recurring issues: (i) shortcut-driven optimization under composite objectives and (ii) reward saturation within prompt groups. To address these issues, we introduce TaRoS, a Target-Robust Reward Signaling framework for Video generation GRPO. TaRoS leverages component level performance assessment together with intra-group sparsity to organize multi-aspect rewards towards optimization objectives. In addition, it adaptively downweights components that exhibit saturation, thereby preserving effective optimization directions and mitigating redundancy. This maintains meaningful optimization directions and preserves within-group ranking separation, thereby preventing reward hacking and leading to more reliable policy updates. Extensive experiments show consistent improvements in visual fidelity, motion coherence, and text-video alignment over strong baselines.

Abstract

Explore similar work

Rethinking Reward Signals in Video GRPO: When Scores Become Targets

Token-Level Video Reinforcement Learning

KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration