Advances in GRPO for Generation Models: A Survey

Zexiang Liu; Xianglong He; Yangguang Li

cs.LGFeb 21, 2026

Advances in GRPO for Generation Models: A Survey

Authors: Zexiang Liu, Xianglong He, Yangguang Li

Organizations: SJTU · THU · CUHK

Abstract

Large-scale flow matching models have achieved strong performance across generative tasks such as text-to-image, video, 3D, and speech synthesis. However, aligning their outputs with human preferences and task-specific objectives remains challenging. Flow-GRPO extends Group Relative Policy Optimization (GRPO) to generation models, enabling stable reinforcement learning alignment for generative systems. Since its introduction, Flow-GRPO has triggered rapid research growth, spanning methodological refinements and diverse application domains. This survey provides a comprehensive review of Flow-GRPO and its subsequent developments. We organize existing work along two primary dimensions. First, we analyze methodological advances beyond the original framework, including reward signal design, credit assignment, sampling efficiency, diversity preservation, reward hacking mitigation, and reward model construction. Second, we examine extensions of GRPO-based alignment across generative paradigms and modalities, including text-to-image, video generation, image editing, speech and audio, 3D modeling, embodied vision-language-action systems, unified multimodal models, autoregressive and masked diffusion models, and restoration tasks. By synthesizing theoretical insights and practical adaptations, this survey highlights Flow-GRPO as a general alignment framework for modern generative models and outlines key open challenges for scalable and robust reinforcement-based generation.

Figures & tables

Category	Method	Core Innovation	Key Contribution / Result
Step-Level Reward Densification	DenseGRPO Deng et al. (2026)	ODE-predicted clean sample $\hat{x}_{1}^{(t)}$ for step-level reward gain $\Delta r_{t}$ + reward-aware adaptive stochasticity	PickScore 23.1 (vs. 22.5); GenEval 0.74 (vs. 0.71); Improved reward signal granularity
	SuperFlow Chen et al. (2025f)	Variance-aware dynamic group sizing + continuous-time step-level advantage	1.7%–16.0% improvement over Flow-GRPO; 5.4%–56.3% training cost
Value Function Anchoring	VGPO Shao et al. ()	Process-aware value estimation $V_{\phi}(x_{t},t)$ + absolute-value-enhanced group normalization	SOTA on PickScore, HPS v2.1, GenEval; Stabilized advantage estimation
Process-Level Reward Guidance	Euphonium Zhong et al. (2026)	Injecting $\nabla_{x_{t}}R_{t}(x_{t})$ into SDE drift + Dual-Reward GRPO	1.66 $\times$ faster convergence on T2V; Unifies Flow-GRPO and DanceGRPO as special cases

Table 1: Representative methods addressing sparse reward in Flow-GRPO. We summarize step-level reward densification, variance-aware sampling, value anchoring, and process-level gradient guidance strategies.

Strategy Type	Method	Core Mechanism	Key Benefit / Result
Tree / Branch Search	TreeGRPO Ding and Ye (2025)	Reconstructs denoising as a search tree; Compares sibling rewards to assign step credit: $\hat{A}_{t}^{\text{tree}}=\frac{\bar{r}_{\text{children}}(t)-\bar{r}_{\text{siblings}}(t)}{\mathrm{std}(r_{\text{siblings}}(t))}$	2.4 $\times$ training speedup with maintained alignment quality
	BranchGRPO Li et al. (2025g)	Flexible on-demand branching at arbitrary timesteps; Depth-wise advantage estimation; Branches adapt to reward variance	+16% alignment over DanceGRPO; 55% training time reduction; Hybrid variant 4.7 $\times$ speedup
	Dynamic-TreeRPO Fu et al. (2025)	Dynamic noise intensity across tree levels (larger early, smaller late) + LayerTuning-RL (update only selected layers)	+4.9% (HPS v2.1), +5.91% (PickScore), +8.66% (ImageReward); $\sim$ 50% efficiency gain
Grouping / Decomposition	Multi-GRPO Lyu et al. (2025b)	Temporal grouping (MCTS-like branching across segments) + reward grouping for multi-objective: $\hat{A}_{i}=\sum_{k=1}^{K}w_{k}\hat{A}_{i}^{(k)}$	Reduces interference from reward scale/variance; Improves multi-objective stability
	Chunk-GRPO Luo et al. (2025b)	Groups consecutive steps into coherent chunks of length $k$ ; Chunk-level optimization with shared advantage	Lower-variance credit estimation while preserving temporal hierarchy
Noise Causality & Temporal Awareness	G 2 RPO Zhou et al. (2025c)	Singular stochastic sampling: inject noise only at one random timestep $t^{*}$ (others ODE); multi-granularity advantage integration across scales: $\hat{A}_{i}^{\text{multi}}=\sum_{s}\alpha_{s}\hat{A}_{i}^{(s)}$	Strengthens causal link between a specific step’s stochastic decision and terminal reward

Table 2: Credit assignment strategies for Flow-GRPO: from trajectory-level terminal rewards to structured step-/block-level attribution.

Acceleration Type	Method	Core Mechanism	Speedup / Key Result
Entropy / Temporal Pruning	E-GRPO Zhang et al. (2026b)	Identifies high-entropy timesteps via $H(t)$ ; Merges low-entropy steps; Group-normalized advantages within merged segments	Reduced SDE computation; Improved signal-to-noise ratio
	MixGRPO Li et al. (2025a)	Sliding ODE–SDE window $[t_{w},t_{w}+W]$ ; SDE only inside window; ODE elsewhere	MixGRPO-Flash: 71% time reduction with maintained alignment
Sampling Distribution Optimization	Smart-GRPO Yu et al. (2025a)	Iteratively reweights initial noise distribution based on reward feedback	More efficient exploration; Avoids low-reward regions
	Pro-GRPO Ge et al. (2025)	Expand-and-Prune: Increase $G^{\prime}$ , diversity pruning, multi-step OVF	Mitigates reward clustering; Higher signal variance utilization
Forward / Matching-Based RL	DiffusionNFT Zheng et al. (2025b)	Online RL on forward noising process; Contrastive positive/negative flow matching	25 $\times$ speedup; GenEval 0.24 $\rightarrow$ 0.98 in 1k steps
	AWM Xue et al. (2025b)	Advantage-weighted flow matching: $\mathcal{L}_{AWM}=\hat{A}_{i}\\|v_{\theta}-\text{target}\\|^{2}$	24 $\times$ speedup; No SDE rollouts; Simplified training

Table 3: Sampling efficiency and training acceleration strategies for Flow-GRPO. These methods reduce rollout cost via entropy-aware pruning, sliding windows, noise distribution optimization, forward-process training, distribution-level matching, and preference-based learning.

Category	Method	Core Idea	Key Results
Distribution-Level Regularization	DiverseGRPO Liu et al. (2025b)	Spectral clustering in CLIP space; Exploration reward $r_{i}^{\text{div}}=r_{i}+\lambda/\|C_{i}\|$ ; Early-step stronger KL to prevent structural collapse	+13%–18% Vendi diversity without quality drop
	DRIFT Liu et al. (2026)	Reward-concentrated sampling; Stochastic prompt perturbation; Potential-based shaping $\Phi(s_{t+1})-\gamma\Phi(s_{t})$	+9–43% diversity at equal alignment; Pareto-dominant trade-off
Orthogonal Perturbation	OSCAR Wu et al. (2025c)	Inject stochastic noise orthogonal to generation flow in latent space; Monotonic increase of distribution volume proxy	Training-free diversity boost; No alignment degradation
Bias Direction Decoupling	D 2 -Align Chen et al. (2025b)	Identify bias direction in reward embedding space; Remove preference bias via lightweight correction module; Introduces DivGenBench benchmark	Significant mitigation of preference mode collapse
Identity Diversity	DisCo Borse et al. (2026)	RL with compositional reward; Pairwise facial similarity penalty across subjects	98.6% Unique Face Accuracy (vs. <50% baseline)

Table 4: Mode collapse mitigation and diversity preservation strategies in GRPO / Flow-GRPO alignment. These methods address reward-induced preference collapse via distribution-level regularization, orthogonal perturbation, reward reshaping, directional decoupling, and identity-aware diversity optimization.

Mitigation Strategy	Method	Core Idea	Key Result / Contribution
Clipping Stabilization	GRPO-Guard Wang et al. (2025e)	Ratio normalization $\tilde{\rho}_{i}=\rho_{i}/\bar{\rho}$ ; Gradient reweighting for asymmetric importance shift	Stabilizes PPO clipping; Prevents ratio collapse
Uncertainty & Diversity Regularization	GARDO He et al. (2025b)	Penalize high-uncertainty reward outputs; EMA reference update; Diversity-amplified reward	Reduces blind-spot exploitation; Improves stability
	ConsistentRFT Tan et al. (2026b)	Multi-granularity rollout (DGR); Consistent policy gradient optimization (CPGO)	-49% perceptual hallucination; -38% semantic hallucination
Data Anchoring	DDRL Ye et al. (2025a)	Forward KL regularization $D_{\text{KL}}(p_{\text{data}}\\|\pi_{\theta})$	Industrial-scale validation; Prevents forgetting data modes
Reward Model Analysis & Robustification	ArtifactReward Hong et al. (2026)	Typology of hacking (over-stylization vs. deformation); Artifact-aware auxiliary reward	First systematic categorization of T2I reward hacking
	SoliReward Lian et al. (2025)	Single-item binary labeling; Cross-prompt pairing; Hierarchical progressive query attention	Robust video reward modeling under noisy labels

Table 5: Reward hacking mitigation strategies in GRPO / Flow-GRPO alignment. These methods address reward exploitation via regulated clipping, uncertainty-aware penalization, data anchoring, sampler redesign, artifact detection, reward model robustness, and offline correction.

Stochasticity Regime	Method	Core Idea	Key Contribution
Pure ODE	Neighbor GRPO He et al. (2025a)	Deterministic ODE sampling; Diversity from perturbed initial noise $x_{0}^{(i)}=x_{0}+\delta_{i}$ ; Softmax distance proxy for policy gradient	Efficient; No noise artifacts; ODE-compatible high-order solvers
Few-Step Annealed Noise	TAFS-GRPO Yue et al. (2026)	Adaptive temporal noise injection with annealed temperature $\tau(k)$ ; Step-aware advantage weighting	Efficient RL alignment for 1–4 step generation
Structured Stochastic Flows	GLASS Flows Holderrieth et al. (2025)	Transition kernel $p(x_{t+\Delta t}\|x_{t})$ modeled via auxiliary flow matching; Structured stochasticity instead of Gaussian noise; Feynman-Kac steering	Improved inference-time scaling; Coherent exploration
	Diamond Maps Holderrieth et al. (2026)	Amortized stochastic flow map $(x_{0},z)\mapsto x_{1}$ ; Distilled from GLASS Flows	Single-step stochastic sampling; Reward-adaptive inference
Theoretical Analysis	SDE–ODE Reward Gap Sheng et al. (2025)	Reward gap bounds under gDDIM; High-stochasticity training improves ODE inference; Optimal stochasticity level characterization	First formal theory of SDE–ODE reward gap

Table 6: ODE–SDE sampling spectrum in Flow-GRPO alignment. Methods explore different stochasticity regimes ranging from pure ODE (zero noise) to structured SDE and amortized stochastic flow maps, with corresponding theoretical insights.

Category	Model	Core Design	Key Contribution
Lightweight / Task-Specific	HuDA Ashutosh et al. (2026)	Human detector confidence + temporal prompt alignment	73% human win rate vs. Wan 2.1 baseline
	RealGen Ye et al. (2025b)	Detector-guided artifact reward (reverse-use of synthetic image detector)	Label-free photorealism alignment
Generative Reward Paradigm	RewardDance Wu et al. (2025b)	Large-scale generative reward (up to 26B); Natural language reasoning before scoring	Strong reward hacking resistance
	OneReward Gong et al. (2025)	Unified VLM generative reward for generation/editing/SR	Multi-task winner–loser discrimination
Structured / Granular Feedback	VR-Thinker Wang et al. (2025f)	CoT-style structured visual reasoning before scoring	80.5% accuracy on video preference benchmarks
	REACT Wang et al. (2026g)	Frame-level structural distortion evaluation (SFT+GRPO)	Fine-grained video feedback

Table 7: Reward model designs for GRPO-aligned generative models. Methods range from lightweight task-specific reward signals to large-scale generative reward models with reasoning capability.

Category	Method	Core Innovation	Key Contribution / Result
Reasoning-Augmented Alignment	PromptRL Wang et al. (2026a)	Trainable LM as prompt optimizer within RL loop	End-to-end co-optimization; 0.97 GenEval; 50% fewer rollouts
	Think-Then-Generate Kou et al. (2026)	Two-stage reasoning + generation	Dual-GRPO optimization of reasoner and generator
	ThinkGen Jiao et al. (2025)	Think-driven structured visual planning	SepGRPO alternating optimization for stability
	Unified Thinker Zhou et al. (2026)	Fully decoupled reasoning engine	Task-agnostic reasoning across T2I/editing/SR
	CoF-T2I Tong et al. (2026)	Chain-of-Frame visual reasoning	Progressive scene construction via intermediate frames
	AgentComp Zarei et al. (2025)	Agent-based compositional data construction	Tool-use-based preference supervision

Table 8: Representative extensions of Flow-GRPO in text-to-image generation. We categorize existing works into reasoning augmentation, multi-objective alignment, structural controllability, distribution preservation, adversarial rewards, test-time scaling, and advanced preference modeling.

Category	Method	Core Innovation	Key Contribution / Result
Core T2V / I2V Methods	TAGRPO Wang et al. (2026c)	Memory-bank contrastive I2V alignment	Progressive lower-bound quality improvement
	PhysRVG Zhang et al. (2026a)	Physics-engine-based reward	Verifiable rigid-body trajectory consistency
	Diffusion-DRF Wang et al. (2026e)	Frozen VLM critic (e.g., GPT-4V)	Training-free natural-language reward backprop
	Self-Paced GRPO Li et al. (2025b)	Competence-aware curriculum reward	Stable progressive temporal alignment
	PG-DPO Xu et al. (2025b)	Adaptive rejection scaling	Stabilizes high-dimensional DPO optimization
	DenseDPO Deng et al. (2026)	Segment-level preference labeling	Motion improvement with 1/3 annotation data

Table 9: Representative extensions of Flow-GRPO in video generation. We categorize existing works into core T2V/I2V alignment, identity preservation, motion control, editing, video reasoning, frontier large-scale systems, and reward modeling.

Category	Method	Core Innovation	Key Contribution / Result
Reasoning-Driven Editing	ThinkRL-Edit Li et al. (2026a)	CoT-based semantic analysis + binary checklist reward	Improves correctness–preservation balance
	Re-Align He et al. (2026)	In-Context CoT (IC-CoT) alignment	Example-guided reasoning during inference
	JarvisEvo Lin et al. (2025)	Editor–Evaluator alternating optimization (SEPO)	+18.95% preservation; +44.96% pixel fidelity
	Skywork UniPic 2.0 Wei et al. (2025)	Progressive Dual-Task Reinforcement (PDTR)	Joint image understanding + editing; Surpasses larger models
Dense Reward & Attention Control	CogniEdit Li et al. (2025d)	Dense GRPO + Dynamic Token Focus Relocation	Region-aware attention for edit vs. preserve
	EditScore Luo et al. (2025a)	Structured editing reward evaluation	Interpretable multi-criterion scoring

Table 10: Representative GRPO-based extensions in image editing. Recent works emphasize reasoning-driven editing, dense reward shaping, motion- and geometry-aware control, scalable data construction, and paradigm-agnostic RL alignment.

Category	Method	Core Innovation	Key Contribution / Result
Speech Enhancement	FlowSE-GRPO Wang et al. (2026b)	Online GRPO for flow-matching speech enhancement with multi-metric reward balancing	Joint optimization of PESQ, STOI, and SI-SDR within unified RL framework
Music & Singing Generation	YingMusic-Singer Zheng et al. (2025a)	Multi-objective GRPO for zero-shot singing voice synthesis	Style-consistent singing from lyrics + melody without target-speaker training data
	YingMusic-SVC Chen et al. (2025c)	GRPO-based zero-shot singing voice conversion with singing-specific inductive biases	Timbre conversion preserving lyrics and melody; Reward models formant and vibrato structure

Table 11: Representative extensions of GRPO alignment in speech and audio generation. These works demonstrate the modality-agnostic nature of GRPO in continuous-time signal modeling, multi-metric optimization, and style-preserving generation.

Category	Method	Core Innovation	Key Contribution / Result
3D Generation	Hi-GRPO / AR3D-R1 Tang et al. (2025b)	Hierarchical coarse-to-fine RL optimization for text-to-3D	Progressive global shape $\rightarrow$ local texture refinement; Improved multi-view consistency
	Nabla-R2D3 Liu et al. (2025d)	Cross-dimensional reward transfer (2D $\rightarrow$ 3D)	Multi-view rendering + 2D reward averaging for 3D alignment
	VIST3A Go et al. (2025)	Video-to-3D model stitching	Multi-view consistent video frames enable coherent 3D reconstruction
	TRIM Yin and Liu (2025)	3D Gaussian Splatting acceleration	Efficient 3D diffusion with preserved visual fidelity
Scientific Applications	Open Materials Generation Hoellmer and Martiniani (2026)	RL-guided crystal structure search	Thermodynamic stability + symmetry-aware reward via DFT energy
	Elign / FED-GRPO Li et al. (2026b)	Force-Energy Disentangled GRPO for molecular modeling	Separate rewards for force accuracy and energy smoothness

Table 12: Extensions of GRPO alignment in 3D generation and scientific foundation models. These works demonstrate hierarchical optimization, cross-dimensional reward transfer, efficient 3D representations, and physically grounded reinforcement objectives.

Category	Method	Core Innovation	Key Contribution / Result
Spatially-Grounded Flow Policies	SA-VLA Pan et al. (2026)	Spatially-Aware Flow Matching with 3D cues (depth, normals)	SCAN exploration strategy for efficient action-space exploration
	$\pi$ RL Chen et al. (2025g)	Flow-Noise vs. Flow-SDE RL integration	Stability–exploration trade-off analysis in flow-based VLA
	PolicyFlow Yang et al. (2026)	Continuous normalizing flow (CNF) policies + Brownian regularizer	Structured stochastic exploration under PPO optimization
Structured Policy Optimization	FPO Lyu et al. (2025a)	Structure-aware credit assignment	Hierarchical reward decomposition + Q-ensemble estimation
	Flow Matching Policy Gradients McAllister et al. (2025)	Theoretical PPO compatibility proof	Validates flow matching as RL policy parameterization
Simulator-Based & World Models	ProphRL Zhang et al. (2025b)	Action-to-video world model for few-shot simulation	5–17% sim gain; 24–30% real-robot improvement via FA-GRPO

Table 13: Extensions of Flow-GRPO style in Vision-Language-Action (VLA) and embodied AI. These works address continuous high-dimensional action spaces, real-time control, simulator-based learning, structured credit assignment, and socially aware policy optimization.

Category	Method	Core Innovation	Key Contribution / Result
Unified RL Optimization	UAE Yan et al. (2025)	Auto-encoder unification (encode=understanding, decode=generation) + Unified-GRPO	Joint optimization of VQA and preference rewards; Bidirectional enhancement
	UniRL-Zero Wang et al. (2025a)	Single RL algorithm for language + diffusion experts	Eliminates modality-decoupled training; Unified policy optimization
	Unified T2I (Weakness-Targeted PT) Chen et al. (2026b)	Weakness-targeted post-training reinforcement	Automatic strengthening of underperforming prompt categories
Hybrid / Parallel Architectures	BLIP3o-NEXT Chen et al. (2025e)	AR (planning) + Diffusion (generation) hybrid framework	Open-source multimodal frontier with RL-enhanced generation
	ParaUni Tan et al. (2025)	Hierarchical parallel VLM interaction + LDAM gradient balancing	Cross-layer bidirectional information coupling
	MMaDA-Parallel Tian et al. (2025)	Parallel multimodal diffusion with cross-attention; ParaRL training	Simultaneous denoising of text and image streams

Table 14: Representative unified multimodal models aligned with GRPO-style optimization. We categorize existing works into unified RL optimization, hybrid/parallel architectures, reasoning-augmented generation, efficiency-oriented structural design, and auxiliary constraint / system-level scaling.

Category	Method	Core Innovation	Key Contribution / Result
Autoregressive Image Generation	AR-GRPO Yuan et al. (2025)	Token-level GRPO for AR image generation	Structured reward optimization over discrete visual tokens
	STAGE Ma et al. (2025b)	Similarity-aware KL/advantage reweighting	Adaptive KL scaling + entropy reward to prevent collapse
	VAR RL Done Right Sun et al. (2026)	VMR + PANW + Mask Propagation	Token-level return estimation + position-sensitive normalization
	GCPO Zhang et al. (2025a)	Critical-token selective optimization	RL applied to top 30% important tokens only
	Layout-Conditioned AR T2I Zheng et al. (2025c)	Layout-aware masking + spatial rewards	Improved spatial controllability in AR generation
	AR-Drag Zhao et al. (2025a)	RL-enhanced autoregressive trajectory control	Extends AR alignment to video/motion generation

Table 15: Extensions of GRPO alignment to autoregressive (AR), masked diffusion, and discrete flow matching models. These works explore token-level credit assignment, adaptive regularization, masking policy optimization, search-based decoding, and distribution-level reinforcement learning.

Category	Method	Core Innovation	Key Contribution / Result
Blind Restoration	LRPO Bin et al. (2025)	Online RL for blind face restoration with composite reward	Identity + perceptual + degradation removal reward; likelihood regularization
	Diffusion Restoration via RL Xu et al. (2025c)	MLLM-based IQA reward + RL/SFT hybrid training	Difficulty-adaptive switching between SFT and RL
Hierarchical Restoration	Pref-Restore Yao et al. (2026)	Two-stage coarse-to-fine restoration framework	Stage-specific RL preference constraints within diffusion loop
	FinPercep-RM Liu et al. (2025g)	Spatially-aware degradation maps + co-evolutionary curriculum	Region-weighted rewards for heavily corrupted areas
Efficiency & Hybrid Enhancement	FlowLUT Hu et al. (2025)	LUT-based global adjustment + flow matching refinement	Fast global color/contrast correction with local detail enhancement
	ReasonX Dirik et al. (2025)	MLLM perceptual reward for intrinsic image decomposition	9–25% WHDR reduction via language-guided supervision

Table 16: Extensions of GRPO-based alignment in image restoration and super-resolution. These works address blind degradation, hierarchical restoration, spatially adaptive weighting, language-guided quality assessment, and hybrid efficiency-oriented enhancement architectures.

Explore similar work

Apr 5, 2026

OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models

Post training via GRPO has demonstrated remarkable effectiveness in improving the generation quality of flow-matching models. However, GRPO suffers from inherently low sample efficiency due to its on-policy training paradigm. To address this limitation, we present OP-GRPO, the first Off-Policy GRPO framework tailored for flow-matching models. First, we actively select high-quality trajectories and adaptively incorporate them into a replay buffer for reuse in subsequent training iterations. Second, to mitigate the distribution shift introduced by off-policy samples, we propose a sequence-level importance sampling correction that preserves the integrity of GRPO's clipping mechanism while ensuring stable policy updates. Third, we theoretically and empirically show that late denoising steps yield ill-conditioned off-policy ratios, and mitigate this by truncating trajectories at late steps. Across image and video generation benchmarks, OP-GRPO achieves comparable or superior performance to Flow-GRPO with only 34.2% of the training steps on average, yielding substantial gains in training efficiency while maintaining generation quality.

Liyu Zhang, Kehan Li, Tingrui Han +4

College of Control Science and Engineering, Zhejiang University · Central Research Institute, Huawei

RL for Generative Models Off-Policy RL Flow Matching Image Generation Video Generation Experience Replay

Ask PDF

Feb 6, 2026

Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO

Deploying GRPO on Flow Matching models has proven effective for text-to-image generation. However, existing paradigms typically propagate an outcome-based reward to all preceding denoising steps without distinguishing the local effect of each step. Moreover, current group-wise ranking mainly compares trajectories at matched timesteps and ignores within-trajectory dependencies, where certain early denoising actions can affect later states via delayed, implicit interactions. We propose TurningPoint-GRPO (TP-GRPO), a GRPO framework that alleviates step-wise reward sparsity and explicitly models long-term effects within the denoising trajectory. TP-GRPO makes two key innovations: (i) it replaces outcome-based rewards with step-level incremental rewards, providing a dense, step-aware learning signal that better isolates each denoising action's "pure" effect, and (ii) it identifies turning points-steps that flip the local reward trend and make subsequent reward evolution consistent with the overall trajectory trend-and assigns these actions an aggregated long-term reward to capture their delayed impact. Turning points are detected solely via sign changes in incremental rewards, making TP-GRPO efficient and hyperparameter-free. Extensive experiments also demonstrate that TP-GRPO exploits reward signals more effectively and consistently improves generation. Demo code is available at https://github.com/YunzeTong/TurningPoint-GRPO.

Yunze Tong, Mushui Liu, Canyu Zhao +7

1Zhejiang University, Hangzhou, China · 2Taobao & Tmall Group of Alibaba, Hangzhou, China · 3Tsinghua University, Beijing, China

Flow Matching Temporal Credit Assignment RL for Image Generation Flow-Based Generative Modeling RL for Generative Models

Ask PDF

Mar 13, 2026

From Sparse to Dense: Multi-View GRPO for Flow Models via Augmented Condition Space

Group Relative Policy Optimization (GRPO) has emerged as a powerful framework for preference alignment in text-to-image (T2I) flow models. However, we observe that the standard paradigm where evaluating a group of generated samples against a single condition suffers from insufficient exploration of inter-sample relationships, constraining both alignment efficacy and performance ceilings. To address this sparse single-view evaluation scheme, we propose Multi-View GRPO (MV-GRPO), a novel approach that enhances relationship exploration by augmenting the condition space to create a dense multi-view reward mapping. Specifically, for a group of samples generated from one prompt, MV-GRPO leverages a flexible Condition Enhancer to generate semantically adjacent yet diverse captions. These captions enable multi-view advantage re-estimation, capturing diverse semantic attributes and providing richer optimization signals. By deriving the probability distribution of the original samples conditioned on these new captions, we can incorporate them into the training process without costly sample regeneration. Extensive experiments demonstrate that MV-GRPO achieves superior alignment performance over state-of-the-art methods.

Jiazi Bu, Pengyang Ling, Yujie Zhou +8

Shanghai Jiao Tong University · S-Lab, Nanyang Technological University · Shanghai AI Laboratory +7

RL for Image Generation Group Relative Policy Optimization Diffusion Model Alignment Flow-Based Generative Modeling

Ask PDF