cs.LGFeb 21, 2026
SaveAdvances in GRPO for Generation Models: A Survey
Organizations: SJTU · THU · CUHK
Abstract
Large-scale flow matching models have achieved strong performance across generative tasks such as text-to-image, video, 3D, and speech synthesis. However, aligning their outputs with human preferences and task-specific objectives remains challenging. Flow-GRPO extends Group Relative Policy Optimization (GRPO) to generation models, enabling stable reinforcement learning alignment for generative systems. Since its introduction, Flow-GRPO has triggered rapid research growth, spanning methodological refinements and diverse application domains. This survey provides a comprehensive review of Flow-GRPO and its subsequent developments. We organize existing work along two primary dimensions. First, we analyze methodological advances beyond the original framework, including reward signal design, credit assignment, sampling efficiency, diversity preservation, reward hacking mitigation, and reward model construction. Second, we examine extensions of GRPO-based alignment across generative paradigms and modalities, including text-to-image, video generation, image editing, speech and audio, 3D modeling, embodied vision-language-action systems, unified multimodal models, autoregressive and masked diffusion models, and restoration tasks. By synthesizing theoretical insights and practical adaptations, this survey highlights Flow-GRPO as a general alignment framework for modern generative models and outlines key open challenges for scalable and robust reinforcement-based generation.
Figures & tables
| Category | Method | Core Innovation | Key Contribution / Result |
| Step-Level Reward Densification | DenseGRPO Deng et al. (2026) | ODE-predicted clean sample for step-level reward gain + reward-aware adaptive stochasticity | PickScore 23.1 (vs. 22.5); GenEval 0.74 (vs. 0.71); Improved reward signal granularity |
| SuperFlow Chen et al. (2025f) | Variance-aware dynamic group sizing + continuous-time step-level advantage | 1.7%–16.0% improvement over Flow-GRPO; 5.4%–56.3% training cost | |
| Value Function Anchoring | VGPO Shao et al. () | Process-aware value estimation + absolute-value-enhanced group normalization | SOTA on PickScore, HPS v2.1, GenEval; Stabilized advantage estimation |
| Process-Level Reward Guidance | Euphonium Zhong et al. (2026) | Injecting into SDE drift + Dual-Reward GRPO | 1.66 faster convergence on T2V; Unifies Flow-GRPO and DanceGRPO as special cases |
Table 1: Representative methods addressing sparse reward in Flow-GRPO. We summarize step-level reward densification, variance-aware sampling, value anchoring, and process-level gradient guidance strategies.
| Strategy Type | Method | Core Mechanism | Key Benefit / Result |
| Tree / Branch Search | TreeGRPO Ding and Ye (2025) | Reconstructs denoising as a search tree; Compares sibling rewards to assign step credit: | 2.4 training speedup with maintained alignment quality |
| BranchGRPO Li et al. (2025g) | Flexible on-demand branching at arbitrary timesteps; Depth-wise advantage estimation; Branches adapt to reward variance | +16% alignment over DanceGRPO; 55% training time reduction; Hybrid variant 4.7 speedup | |
| Dynamic-TreeRPO Fu et al. (2025) | Dynamic noise intensity across tree levels (larger early, smaller late) + LayerTuning-RL (update only selected layers) | +4.9% (HPS v2.1), +5.91% (PickScore), +8.66% (ImageReward); 50% efficiency gain | |
| Grouping / Decomposition | Multi-GRPO Lyu et al. (2025b) | Temporal grouping (MCTS-like branching across segments) + reward grouping for multi-objective: | Reduces interference from reward scale/variance; Improves multi-objective stability |
| Chunk-GRPO Luo et al. (2025b) | Groups consecutive steps into coherent chunks of length ; Chunk-level optimization with shared advantage | Lower-variance credit estimation while preserving temporal hierarchy | |
| Noise Causality & Temporal Awareness | G 2 RPO Zhou et al. (2025c) | Singular stochastic sampling: inject noise only at one random timestep (others ODE); multi-granularity advantage integration across scales: | Strengthens causal link between a specific step’s stochastic decision and terminal reward |
Table 2: Credit assignment strategies for Flow-GRPO: from trajectory-level terminal rewards to structured step-/block-level attribution.
| Acceleration Type | Method | Core Mechanism | Speedup / Key Result |
| Entropy / Temporal Pruning | E-GRPO Zhang et al. (2026b) | Identifies high-entropy timesteps via ; Merges low-entropy steps; Group-normalized advantages within merged segments | Reduced SDE computation; Improved signal-to-noise ratio |
| MixGRPO Li et al. (2025a) | Sliding ODE–SDE window ; SDE only inside window; ODE elsewhere | MixGRPO-Flash: 71% time reduction with maintained alignment | |
| Sampling Distribution Optimization | Smart-GRPO Yu et al. (2025a) | Iteratively reweights initial noise distribution based on reward feedback | More efficient exploration; Avoids low-reward regions |
| Pro-GRPO Ge et al. (2025) | Expand-and-Prune: Increase , diversity pruning, multi-step OVF | Mitigates reward clustering; Higher signal variance utilization | |
| Forward / Matching-Based RL | DiffusionNFT Zheng et al. (2025b) | Online RL on forward noising process; Contrastive positive/negative flow matching | 25 speedup; GenEval 0.24 0.98 in 1k steps |
| AWM Xue et al. (2025b) | Advantage-weighted flow matching: | 24 speedup; No SDE rollouts; Simplified training |
Table 3: Sampling efficiency and training acceleration strategies for Flow-GRPO. These methods reduce rollout cost via entropy-aware pruning, sliding windows, noise distribution optimization, forward-process training, distribution-level matching, and preference-based learning.
| Category | Method | Core Idea | Key Results |
| Distribution-Level Regularization | DiverseGRPO Liu et al. (2025b) | Spectral clustering in CLIP space; Exploration reward ; Early-step stronger KL to prevent structural collapse | +13%–18% Vendi diversity without quality drop |
| DRIFT Liu et al. (2026) | Reward-concentrated sampling; Stochastic prompt perturbation; Potential-based shaping | +9–43% diversity at equal alignment; Pareto-dominant trade-off | |
| Orthogonal Perturbation | OSCAR Wu et al. (2025c) | Inject stochastic noise orthogonal to generation flow in latent space; Monotonic increase of distribution volume proxy | Training-free diversity boost; No alignment degradation |
| Bias Direction Decoupling | D 2 -Align Chen et al. (2025b) | Identify bias direction in reward embedding space; Remove preference bias via lightweight correction module; Introduces DivGenBench benchmark | Significant mitigation of preference mode collapse |
| Identity Diversity | DisCo Borse et al. (2026) | RL with compositional reward; Pairwise facial similarity penalty across subjects | 98.6% Unique Face Accuracy (vs. <50% baseline) |
Table 4: Mode collapse mitigation and diversity preservation strategies in GRPO / Flow-GRPO alignment. These methods address reward-induced preference collapse via distribution-level regularization, orthogonal perturbation, reward reshaping, directional decoupling, and identity-aware diversity optimization.
| Mitigation Strategy | Method | Core Idea | Key Result / Contribution |
| Clipping Stabilization | GRPO-Guard Wang et al. (2025e) | Ratio normalization ; Gradient reweighting for asymmetric importance shift | Stabilizes PPO clipping; Prevents ratio collapse |
| Uncertainty & Diversity Regularization | GARDO He et al. (2025b) | Penalize high-uncertainty reward outputs; EMA reference update; Diversity-amplified reward | Reduces blind-spot exploitation; Improves stability |
| ConsistentRFT Tan et al. (2026b) | Multi-granularity rollout (DGR); Consistent policy gradient optimization (CPGO) | -49% perceptual hallucination; -38% semantic hallucination | |
| Data Anchoring | DDRL Ye et al. (2025a) | Forward KL regularization | Industrial-scale validation; Prevents forgetting data modes |
| Reward Model Analysis & Robustification | ArtifactReward Hong et al. (2026) | Typology of hacking (over-stylization vs. deformation); Artifact-aware auxiliary reward | First systematic categorization of T2I reward hacking |
| SoliReward Lian et al. (2025) | Single-item binary labeling; Cross-prompt pairing; Hierarchical progressive query attention | Robust video reward modeling under noisy labels |
Table 5: Reward hacking mitigation strategies in GRPO / Flow-GRPO alignment. These methods address reward exploitation via regulated clipping, uncertainty-aware penalization, data anchoring, sampler redesign, artifact detection, reward model robustness, and offline correction.
| Stochasticity Regime | Method | Core Idea | Key Contribution |
| Pure ODE | Neighbor GRPO He et al. (2025a) | Deterministic ODE sampling; Diversity from perturbed initial noise ; Softmax distance proxy for policy gradient | Efficient; No noise artifacts; ODE-compatible high-order solvers |
| Few-Step Annealed Noise | TAFS-GRPO Yue et al. (2026) | Adaptive temporal noise injection with annealed temperature ; Step-aware advantage weighting | Efficient RL alignment for 1–4 step generation |
| Structured Stochastic Flows | GLASS Flows Holderrieth et al. (2025) | Transition kernel modeled via auxiliary flow matching; Structured stochasticity instead of Gaussian noise; Feynman-Kac steering | Improved inference-time scaling; Coherent exploration |
| Diamond Maps Holderrieth et al. (2026) | Amortized stochastic flow map ; Distilled from GLASS Flows | Single-step stochastic sampling; Reward-adaptive inference | |
| Theoretical Analysis | SDE–ODE Reward Gap Sheng et al. (2025) | Reward gap bounds under gDDIM; High-stochasticity training improves ODE inference; Optimal stochasticity level characterization | First formal theory of SDE–ODE reward gap |
Table 6: ODE–SDE sampling spectrum in Flow-GRPO alignment. Methods explore different stochasticity regimes ranging from pure ODE (zero noise) to structured SDE and amortized stochastic flow maps, with corresponding theoretical insights.
| Category | Model | Core Design | Key Contribution |
| Lightweight / Task-Specific | HuDA Ashutosh et al. (2026) | Human detector confidence + temporal prompt alignment | 73% human win rate vs. Wan 2.1 baseline |
| RealGen Ye et al. (2025b) | Detector-guided artifact reward (reverse-use of synthetic image detector) | Label-free photorealism alignment | |
| Generative Reward Paradigm | RewardDance Wu et al. (2025b) | Large-scale generative reward (up to 26B); Natural language reasoning before scoring | Strong reward hacking resistance |
| OneReward Gong et al. (2025) | Unified VLM generative reward for generation/editing/SR | Multi-task winner–loser discrimination | |
| Structured / Granular Feedback | VR-Thinker Wang et al. (2025f) | CoT-style structured visual reasoning before scoring | 80.5% accuracy on video preference benchmarks |
| REACT Wang et al. (2026g) | Frame-level structural distortion evaluation (SFT+GRPO) | Fine-grained video feedback |
Table 7: Reward model designs for GRPO-aligned generative models. Methods range from lightweight task-specific reward signals to large-scale generative reward models with reasoning capability.
| Category | Method | Core Innovation | Key Contribution / Result |
| Reasoning-Augmented Alignment | PromptRL Wang et al. (2026a) | Trainable LM as prompt optimizer within RL loop | End-to-end co-optimization; 0.97 GenEval; 50% fewer rollouts |
| Think-Then-Generate Kou et al. (2026) | Two-stage reasoning + generation | Dual-GRPO optimization of reasoner and generator | |
| ThinkGen Jiao et al. (2025) | Think-driven structured visual planning | SepGRPO alternating optimization for stability | |
| Unified Thinker Zhou et al. (2026) | Fully decoupled reasoning engine | Task-agnostic reasoning across T2I/editing/SR | |
| CoF-T2I Tong et al. (2026) | Chain-of-Frame visual reasoning | Progressive scene construction via intermediate frames | |
| AgentComp Zarei et al. (2025) | Agent-based compositional data construction | Tool-use-based preference supervision |
Table 8: Representative extensions of Flow-GRPO in text-to-image generation. We categorize existing works into reasoning augmentation, multi-objective alignment, structural controllability, distribution preservation, adversarial rewards, test-time scaling, and advanced preference modeling.
| Category | Method | Core Innovation | Key Contribution / Result |
| Core T2V / I2V Methods | TAGRPO Wang et al. (2026c) | Memory-bank contrastive I2V alignment | Progressive lower-bound quality improvement |
| PhysRVG Zhang et al. (2026a) | Physics-engine-based reward | Verifiable rigid-body trajectory consistency | |
| Diffusion-DRF Wang et al. (2026e) | Frozen VLM critic (e.g., GPT-4V) | Training-free natural-language reward backprop | |
| Self-Paced GRPO Li et al. (2025b) | Competence-aware curriculum reward | Stable progressive temporal alignment | |
| PG-DPO Xu et al. (2025b) | Adaptive rejection scaling | Stabilizes high-dimensional DPO optimization | |
| DenseDPO Deng et al. (2026) | Segment-level preference labeling | Motion improvement with 1/3 annotation data |
Table 9: Representative extensions of Flow-GRPO in video generation. We categorize existing works into core T2V/I2V alignment, identity preservation, motion control, editing, video reasoning, frontier large-scale systems, and reward modeling.
| Category | Method | Core Innovation | Key Contribution / Result |
| Reasoning-Driven Editing | ThinkRL-Edit Li et al. (2026a) | CoT-based semantic analysis + binary checklist reward | Improves correctness–preservation balance |
| Re-Align He et al. (2026) | In-Context CoT (IC-CoT) alignment | Example-guided reasoning during inference | |
| JarvisEvo Lin et al. (2025) | Editor–Evaluator alternating optimization (SEPO) | +18.95% preservation; +44.96% pixel fidelity | |
| Skywork UniPic 2.0 Wei et al. (2025) | Progressive Dual-Task Reinforcement (PDTR) | Joint image understanding + editing; Surpasses larger models | |
| Dense Reward & Attention Control | CogniEdit Li et al. (2025d) | Dense GRPO + Dynamic Token Focus Relocation | Region-aware attention for edit vs. preserve |
| EditScore Luo et al. (2025a) | Structured editing reward evaluation | Interpretable multi-criterion scoring |
Table 10: Representative GRPO-based extensions in image editing. Recent works emphasize reasoning-driven editing, dense reward shaping, motion- and geometry-aware control, scalable data construction, and paradigm-agnostic RL alignment.
| Category | Method | Core Innovation | Key Contribution / Result |
| Speech Enhancement | FlowSE-GRPO Wang et al. (2026b) | Online GRPO for flow-matching speech enhancement with multi-metric reward balancing | Joint optimization of PESQ, STOI, and SI-SDR within unified RL framework |
| Music & Singing Generation | YingMusic-Singer Zheng et al. (2025a) | Multi-objective GRPO for zero-shot singing voice synthesis | Style-consistent singing from lyrics + melody without target-speaker training data |
| YingMusic-SVC Chen et al. (2025c) | GRPO-based zero-shot singing voice conversion with singing-specific inductive biases | Timbre conversion preserving lyrics and melody; Reward models formant and vibrato structure | |
Table 11: Representative extensions of GRPO alignment in speech and audio generation. These works demonstrate the modality-agnostic nature of GRPO in continuous-time signal modeling, multi-metric optimization, and style-preserving generation.
| Category | Method | Core Innovation | Key Contribution / Result |
| 3D Generation | Hi-GRPO / AR3D-R1 Tang et al. (2025b) | Hierarchical coarse-to-fine RL optimization for text-to-3D | Progressive global shape local texture refinement; Improved multi-view consistency |
| Nabla-R2D3 Liu et al. (2025d) | Cross-dimensional reward transfer (2D 3D) | Multi-view rendering + 2D reward averaging for 3D alignment | |
| VIST3A Go et al. (2025) | Video-to-3D model stitching | Multi-view consistent video frames enable coherent 3D reconstruction | |
| TRIM Yin and Liu (2025) | 3D Gaussian Splatting acceleration | Efficient 3D diffusion with preserved visual fidelity | |
| Scientific Applications | Open Materials Generation Hoellmer and Martiniani (2026) | RL-guided crystal structure search | Thermodynamic stability + symmetry-aware reward via DFT energy |
| Elign / FED-GRPO Li et al. (2026b) | Force-Energy Disentangled GRPO for molecular modeling | Separate rewards for force accuracy and energy smoothness |
Table 12: Extensions of GRPO alignment in 3D generation and scientific foundation models. These works demonstrate hierarchical optimization, cross-dimensional reward transfer, efficient 3D representations, and physically grounded reinforcement objectives.
| Category | Method | Core Innovation | Key Contribution / Result |
| Spatially-Grounded Flow Policies | SA-VLA Pan et al. (2026) | Spatially-Aware Flow Matching with 3D cues (depth, normals) | SCAN exploration strategy for efficient action-space exploration |
| RL Chen et al. (2025g) | Flow-Noise vs. Flow-SDE RL integration | Stability–exploration trade-off analysis in flow-based VLA | |
| PolicyFlow Yang et al. (2026) | Continuous normalizing flow (CNF) policies + Brownian regularizer | Structured stochastic exploration under PPO optimization | |
| Structured Policy Optimization | FPO Lyu et al. (2025a) | Structure-aware credit assignment | Hierarchical reward decomposition + Q-ensemble estimation |
| Flow Matching Policy Gradients McAllister et al. (2025) | Theoretical PPO compatibility proof | Validates flow matching as RL policy parameterization | |
| Simulator-Based & World Models | ProphRL Zhang et al. (2025b) | Action-to-video world model for few-shot simulation | 5–17% sim gain; 24–30% real-robot improvement via FA-GRPO |
Table 13: Extensions of Flow-GRPO style in Vision-Language-Action (VLA) and embodied AI. These works address continuous high-dimensional action spaces, real-time control, simulator-based learning, structured credit assignment, and socially aware policy optimization.
| Category | Method | Core Innovation | Key Contribution / Result |
| Unified RL Optimization | UAE Yan et al. (2025) | Auto-encoder unification (encode=understanding, decode=generation) + Unified-GRPO | Joint optimization of VQA and preference rewards; Bidirectional enhancement |
| UniRL-Zero Wang et al. (2025a) | Single RL algorithm for language + diffusion experts | Eliminates modality-decoupled training; Unified policy optimization | |
| Unified T2I (Weakness-Targeted PT) Chen et al. (2026b) | Weakness-targeted post-training reinforcement | Automatic strengthening of underperforming prompt categories | |
| Hybrid / Parallel Architectures | BLIP3o-NEXT Chen et al. (2025e) | AR (planning) + Diffusion (generation) hybrid framework | Open-source multimodal frontier with RL-enhanced generation |
| ParaUni Tan et al. (2025) | Hierarchical parallel VLM interaction + LDAM gradient balancing | Cross-layer bidirectional information coupling | |
| MMaDA-Parallel Tian et al. (2025) | Parallel multimodal diffusion with cross-attention; ParaRL training | Simultaneous denoising of text and image streams |
Table 14: Representative unified multimodal models aligned with GRPO-style optimization. We categorize existing works into unified RL optimization, hybrid/parallel architectures, reasoning-augmented generation, efficiency-oriented structural design, and auxiliary constraint / system-level scaling.
| Category | Method | Core Innovation | Key Contribution / Result |
| Autoregressive Image Generation | AR-GRPO Yuan et al. (2025) | Token-level GRPO for AR image generation | Structured reward optimization over discrete visual tokens |
| STAGE Ma et al. (2025b) | Similarity-aware KL/advantage reweighting | Adaptive KL scaling + entropy reward to prevent collapse | |
| VAR RL Done Right Sun et al. (2026) | VMR + PANW + Mask Propagation | Token-level return estimation + position-sensitive normalization | |
| GCPO Zhang et al. (2025a) | Critical-token selective optimization | RL applied to top 30% important tokens only | |
| Layout-Conditioned AR T2I Zheng et al. (2025c) | Layout-aware masking + spatial rewards | Improved spatial controllability in AR generation | |
| AR-Drag Zhao et al. (2025a) | RL-enhanced autoregressive trajectory control | Extends AR alignment to video/motion generation |
Table 15: Extensions of GRPO alignment to autoregressive (AR), masked diffusion, and discrete flow matching models. These works explore token-level credit assignment, adaptive regularization, masking policy optimization, search-based decoding, and distribution-level reinforcement learning.
| Category | Method | Core Innovation | Key Contribution / Result |
| Blind Restoration | LRPO Bin et al. (2025) | Online RL for blind face restoration with composite reward | Identity + perceptual + degradation removal reward; likelihood regularization |
| Diffusion Restoration via RL Xu et al. (2025c) | MLLM-based IQA reward + RL/SFT hybrid training | Difficulty-adaptive switching between SFT and RL | |
| Hierarchical Restoration | Pref-Restore Yao et al. (2026) | Two-stage coarse-to-fine restoration framework | Stage-specific RL preference constraints within diffusion loop |
| FinPercep-RM Liu et al. (2025g) | Spatially-aware degradation maps + co-evolutionary curriculum | Region-weighted rewards for heavily corrupted areas | |
| Efficiency & Hybrid Enhancement | FlowLUT Hu et al. (2025) | LUT-based global adjustment + flow matching refinement | Fast global color/contrast correction with local detail enhancement |
| ReasonX Dirik et al. (2025) | MLLM perceptual reward for intrinsic image decomposition | 9–25% WHDR reduction via language-guided supervision |
Table 16: Extensions of GRPO-based alignment in image restoration and super-resolution. These works address blind degradation, hierarchical restoration, spatially adaptive weighting, language-guided quality assessment, and hybrid efficiency-oriented enhancement architectures.