Group Relative Policy Optimization (GRPO) methods for video generation like FlowGRPO remain far less reliable than their counterparts for language models and images. This gap arises because video generation has a complex solution space, and the ODE-to-SDE conversion used for exploration can inject excess noise, lowering rollout quality and making reward estimates less reliable, which destabilizes post-training alignment. To address this problem, we view the pre-trained model as defining a valid video data manifold and formulate the core problem as constraining exploration within the vicinity of this manifold, ensuring that rollout quality is preserved and reward estimates remain reliable. We propose SAGE-GRPO (Stable Alignment via Exploration), which applies constraints at both micro and macro levels. At the micro level, we derive a precise manifold-aware SDE with a logarithmic curvature correction and introduce a gradient norm equalizer to stabilize sampling and updates across timesteps. At the macro level, we use a dual trust region with a periodic moving anchor and stepwise constraints so that the trust region tracks checkpoints that are closer to the manifold and limits long-horizon drift. We evaluate SAGE-GRPO on HunyuanVideo1.5 using the original VideoAlign as the reward model and observe consistent gains over previous methods in VQ, MQ, TA, and visual metrics (CLIPScore, PickScore), demonstrating superior performance in both reward maximization and overall video quality. The code and visual gallery are available at https://dungeonmassster.github.io/SAGE-GRPO-Page/.
Figures & tables
Figure 2: Geometric interpretation of noise injection strategies. Conventional linear SDEs (red) inject exploration noise using first-order approximations, ignoring signal decay curvature and causing off-manifold drift that results in temporal jitter and artifacts. Our Manifold-Aware SDE (blue) uses a logarithmic correction term so that exploration noise is concentrated closer to the flow trajectory and the video manifold, reducing off-manifold drift.
Method
Standard Deviation Σt1/2
DanceGRPO
ησt−σt+1
FlowGRPO
η1−σtσt(σt−σt+1)
Ours (Precise)
η[−(σt−σt+1)+log(1−σt1−σt+1)]
Table 1: Comparison of SDE noise injection strategies used in video GRPO.
Figure 3: Temporal gradient balancing ablation across SDE formulations. Overall VideoAlign reward curves comparing runs with and without the Gradient Norm Equalizer. Without balancing, low-noise timesteps dominate optimization, leading to unstable or plateaued rewards. With balancing, reward curves become smoother with consistent improvement, and gradient scale variation is reduced from more than one order of magnitude to within a small constant factor.
Figure 4: Empirical gradient norm imbalance across noise levels. Observed norms (blue) decrease rapidly as σ increases and match the predicted relationship (red) ∥∇logπ∥∝1/Σt1/2 , leading to vanishing gradients at high noise ( σ→1 ) and exploding gradients at low noise ( σ→0 ).
Figure 5: The SAGE-GRPO Framework. Our method resolves the stability-plasticity dilemma with three coupled components: (Left) a manifold-aware SDE that keeps exploration noise tangent to the video manifold, (Middle) a Temporal Gradient Equalizer that balances optimization across timesteps, and (Right) a Dual Trust Region that combines moving anchors and step-wise KL constraints for long-term stable alignment.
Figure 6: Qualitative comparison against baselines. Three prompts illustrate our core gains: (Top) Reduced temporal jitter while preserving accurate visual contents; (Middle) Enhanced alignment and photorealism under occlusion and lighting changes; (Bottom) Stronger semantic alignment with consistent prompt matching across frames.
Table 2: Main Comparison on Video Generation Benchmarks. Comparison of SAGE-GRPO with baselines under two reward settings. The first row reports the original HunyuanVideo 1.5 performance. For each method, we report results without KL regularization ( w/o KL ) and with their Fixed KL constraints ( w/ Fixed KL ). For SAGE-GRPO, we demonstrate the w/ Dual Moving KL mechanism. Bold , underline , and gray indicate the best, second best, and third best results, computed across both settings (A+B).
Figure 7: KL weight ablation on VideoAlign rewards. Comparison of three KL weight schedules: fixed 10−5 (green), two-stage 10−7→10−5 (red), and two-stage 10−7→10−6 (yellow). The two-stage schedule 10−7→10−5 achieves the strongest and most consistent gains across VQ, MQ, and TA, supporting gradually increasing λKL to tighten the trust region (Appendix A.6 ).
SAGE-GRPO vs.
Visual Quality
Motion Quality
Semantic Alignment
DanceGRPO
85.9%
75.8%
79.2%
FlowGRPO
83.8%
79.2%
71.9%
CPS
80.2%
70.8%
67.9%
Table 3: User Preference Study. Win rates of SAGE-GRPO against baselines. Results indicate a strong human preference for our method, especially in Motion Quality, confirming that automatic metrics align with perceptual quality.
Figure 8: KL strategy ablation. (a) Dual Moving KL achieves the highest and most stable reward, supporting the position-velocity control interpretation (Equation ( 14 )). (b) Moving KL attains high exploration in early training steps but the exploration level drops in later stages. Dual Moving KL maintains a higher and more stable exploration level throughout training.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Step-wise std comparison: our precise SDE vs. FlowGRPO. (a) When both use FlowGRPO’s σ schedule, our integral-based formulation yields near-zero std at the first step due to equal endpoints. (b) When both are clamped at (1−σ)≥3×10−3 , FlowGRPO explodes at step 1, while ours remains stable. (c) Under default implementations, ours maintains lower std across most steps. This supports that we remove ineffective high-frequency noise and explore more precisely along the manifold (Section 3.2.1 ).
Figure 10: Qualitative comparison highlighting emotional alignment. Two prompts illustrate SAGE-GRPO’s ability to better align with emotional descriptions: (Top) A teenage boy in a coffee shop, where SAGE-GRPO captures the ”calm, contemplative expression” and gentle motion of lowering the mug, while baselines show neutral expressions and abrupt movements. (Bottom) An older chef in a kitchen, where SAGE-GRPO consistently renders the ”lines of fatigue” and ”somber mood” through deep side-lighting shadows, while baselines fail to convey the intended emotional depth. Our manifold-aware exploration enables precise alignment with subtle emotional and action cues.
Figure 11: KL strategy ablation: qualitative comparison (Case 1). Visual comparison across different KL strategies (no KL, Fixed KL, Step-wise KL, Moving KL, Dual Moving KL) on a prompt describing a fatigued soldier. Dual Moving KL produces more realistic facial details, better dirt and grime rendering, and maintains temporal consistency across frames compared to other variants.
Figure 12: KL strategy ablation: qualitative comparison (Case 2). Additional visual comparison demonstrating how different KL strategies affect generation quality. Dual Moving KL consistently achieves better photorealism and alignment with prompt descriptions compared to alternatives, validating the position-velocity control mechanism discussed in Section 3.2.2 .
Group Relative Policy Optimization (GRPO) enables stable and preference-oriented updates via group-wise comparisons for post-training video generation. However, GRPO directly optimizes reward-induced advantages. Under sustained optimization, the reward score can lose fidelity as a proxy for true video quality, consistent with the phenomenon described by Goodhart's Law. This leads to two recurring issues: (i) shortcut-driven optimization under composite objectives and (ii) reward saturation within prompt groups. To address these issues, we introduce TaRoS, a Target-Robust Reward Signaling framework for Video generation GRPO. TaRoS leverages component level performance assessment together with intra-group sparsity to organize multi-aspect rewards towards optimization objectives. In addition, it adaptively downweights components that exhibit saturation, thereby preserving effective optimization directions and mitigating redundancy. This maintains meaningful optimization directions and preserves within-group ranking separation, thereby preventing reward hacking and leading to more reliable policy updates. Extensive experiments show consistent improvements in visual fidelity, motion coherence, and text-video alignment over strong baselines.
Rui Li, Yuanzhi Liang, Ziqi Ni +3
University of Science and Technology of China, Hefei, China · Institute of Artificial Intelligence (TeleAI), China Telecom, Shanghai, China · Southeast University, Nanjing, China
Reinforcement learning (RL) for video generation usually assigns one scalar reward to an entire sampled video. Yet a video is not uniformly flawed: some visual tokens may already satisfy the prompt, whereas others require correction. A scalar reward cannot localize errors, causing optimization to perturb satisfactory tokens while under-targeting the tokens that actually need to change. We introduce Token-Level Video Reinforcement Learning, TVRL, a framework that derives token-level credit from the reward being optimized. Our key insight is that the answer likelihood of a frozen vision-language model provides both signals: its outputs contribute to the video-level reward, while magnitudes of its video-input gradients reveal which generated video tokens most affect that score. We instantiate TVRL in Group Relative Policy Optimization by averaging prompt-derived question rewards into one group-relative advantage and using detached, question-conditioned token-credit maps to reweight dense denoising-transition log-probabilities inside the clipped policy ratio. On VBench-2.0, TVRL achieves an Overall score of 57.69, outperforming the base model by 3.60 points. TVRL also improves matched GRPO baselines across three SDE samplers (SAGE, Flow, and Dance) by 2.68--3.15 points and across four reward models (VideoAlign, VideoScore2, UnifiedReward2, and Qwen3.5-9B) by 1.33--3.15 points.
Aligning streaming autoregressive (AR) video generators with human preferences is challenging. Existing reinforcement learning methods predominantly rely on noise-based exploration and SDE-based surrogate policies that are mismatched to the deterministic ODE dynamics of distilled AR models, and tend to perturb low-level appearance rather than the high-level semantic storyline progression critical for long-horizon coherence. To address these limitations, we present KVPO, an ODE-native online Group Relative Policy Optimization (GRPO) framework for aligning streaming video generators. For diversity exploration, KVPO introduces a causal-semantic exploration paradigm that relocates the source of variation from stochastic noise to the historical KV cache. By stochastically routing historical KV entries, it constructs semantically diverse generation branches that remain strictly on the data manifold. For policy modeling, KVPO introduces a velocity-field surrogate policy based on Trajectory Velocity Energy (TVE), which quantifies branch likelihood in flow-matching velocity space and yields a reward-weighted contrastive objective fully consistent with the native ODE formulation. Experiments on multiple distilled AR video generators demonstrate consistent gains in visual quality, motion quality, and text-video alignment across both single-prompt short-video and multi-prompt long-video settings.