From Sparse to Dense: Multi-View GRPO for Flow Models via Augmented Condition Space
Authors: Jiazi Bu, Pengyang Ling, Yujie Zhou, Yibin Wang, Yuhang Zang, Tianyi Wei, Xiaohang Zhan, Jiaqi Wang, +3 more
Organizations: Shanghai Jiao Tong University · S-Lab, Nanyang Technological University · Shanghai AI Laboratory · University of Science and Technology of China · Fudan University · Shanghai Innovation Institute · Adobe Research · Stanford University · The Chinese University of Hong Kong · CPII under InnoHK
Group Relative Policy Optimization (GRPO) has emerged as a powerful framework for preference alignment in text-to-image (T2I) flow models. However, we observe that the standard paradigm where evaluating a group of generated samples against a single condition suffers from insufficient exploration of inter-sample relationships, constraining both alignment efficacy and performance ceilings. To address this sparse single-view evaluation scheme, we propose Multi-View GRPO (MV-GRPO), a novel approach that enhances relationship exploration by augmenting the condition space to create a dense multi-view reward mapping. Specifically, for a group of samples generated from one prompt, MV-GRPO leverages a flexible Condition Enhancer to generate semantically adjacent yet diverse captions. These captions enable multi-view advantage re-estimation, capturing diverse semantic attributes and providing richer optimization signals. By deriving the probability distribution of the original samples conditioned on these new captions, we can incorporate them into the training process without costly sample regeneration. Extensive experiments demonstrate that MV-GRPO achieves superior alignment performance over state-of-the-art methods.
Figures & tables
Figure 1: Gallery of MV-GRPO. Our MV-GRPO substantially elevates the generation quality of flow models (Flux.1-dev in this figure), particularly in terms of fine-grained details and photorealism. Prompts are listed in the supplementary material.
Figure 2: Reward Evaluation in GRPO Training . (a) Standard flow-based GRPO methods evaluate generated samples under the single original condition, resulting in sparse reward mapping and insufficient inter-sample relationship exploration. (b) Our MV-GRPO leverages an augmented set of conditions to facilitate a dense multi-view mapping, fostering a comprehensive exploration of relationship among samples.
Figure 3: Reward Ranking Varies with Conditions . Reward rankings of SDE samples across multiple semantically similar yet different conditions exhibit large variations, indicating that relying on a single condition for advantage estimation is inadequate.
Figure 4: Overview of MV-GRPO . MV-GRPO leverages a flexible Condition Enhancer module (a pretrained VLM or LLM) to generate diverse augmented conditions for dense multi-view reward signals, facilitating comprehensive advantage estimation.
Figure 5
Figure 5: Distribution of Probability Drift at Different SDE Steps. Most condition pairs exhibit a drift near zero, demonstrating that the SDE transition probability is effectively preserved when substituting the original with augmented conditions.
Figure 6: Reward Curves during Training. Our MV-GRPO outperforms baselines in both convergence speed and performance ceiling under various training settings.
Reward Model
Method
HPS-v3
UR-v2-A
UR-v2-C
UR-v2-S
HPS-v2
CLIP
IR
UR-v1
/
Flux.1-dev [ 14 ]
0.133
3.250
3.634
3.223
0.305
0.389
1.042
3.604
HPS-v3
Flow-GRPO [ 19 ]
0.147
3.235
3.670
3.236
0.326
0.380
1.149
3.548
DanceGRPO [ 43 ]
0.145
3.238
3.663
3.224
0.322
0.381
1.165
3.585
TempFlow-GRPO [ 11 ]
0.150
3.240
3.683
3.299
0.332
0.376
1.184
3.559
DiffusionNFT [ 46 ]
0.149
3.244
3.676
3.259
0.330
0.364
1.161
3.551
MV-GRPO (LLM)
0.153
3.232
3.692
3.313
0.336
0.373
1.176
3.557
Table 1: Quantitative comparison of different methods. The best results are in bold , while the second-best result is underlined . UR-v2-A, UR-v2-C, and UR-v2-S denote the Alignment, Coherence, and Style dimensions of UnifiedReward-v2, respectively.
Figure 7: Qualitative Comparisons with Baselines on HPS-v3.
Figure 8: Qualitative Comparisons with Baselines on UnifiedReward-v2.
Table 11
Figure 9: Augmented Conditions. MV-GRPO generates diverse augmented conditions by leveraging variations among SDE samples and multi-view descriptive prompts.
Components
Values & Choices
HPS-v3
UR-v2-A
UR-v2-C
UR-v2-S
HPS-v2
CLIP
IR
UR-v1
(a) Condition Number K
w/o VK
0.149
3.238
3.668
3.269
0.332
0.378
1.152
3.553
K=3
0.152
3.235
3.684
3.293
0.335
0.377
1.170
3.567
K=6
0.154
3.233
3.696
3.326
0.338
0.372
1.178
3.573
K=12
0.155
3.229
3.701
3.320
0.340
0.370
1.193
3.569
(b) Condition Diversity
w/o SDE query
0.153
3.233
3.692
3.304
0.336
0.374
1.183
3.560
w/o PVLM
0.152
3.237
3.687
3.295
0.334
0.376
1.174
3.564
Table 4: Ablation experiments on MV-GRPO components and hyperparameters.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Parameter
Value
Random seed
42
Learning rate
2×10−6
Train batch size
1
Weight decay
1×10−4
Warmup steps
0
Mixed precision
bfloat16
Dataloader workers
4
Max grad norm
1.0
Eta
0.7
Sampler seed
1223627
Group size
12
Scheduler shift
3
Appendix
Table 5: Hyperparameter settings in our experiments.
Figure 10: VLM Prompt Template . It integrates a descriptive instruction from PVLM , along with an image and its original text prompt, to construct a complete VLM prompt for querying the VLM Condition Enhancer to obtain augmented conditions.
ID
Instructions
1
Focus on the main subjects and key attributes.
2
Focus on actions, motion, or interactions.
3
Focus on scene layout and spatial relationships.
4
Focus on lighting conditions and shadows.
5
Focus on color palette and contrast.
6
Focus on camera angle, lens feel, and depth of field.
Appendix
Table 6: List of instructions in the set PVLM used to query the VLM. Each instruction guides the model to focus on a specific visual dimension when generating conditions.
Figure 11: LLM Prompt Template . This template incorporates a specific operation from PLLM and a memory buffer of prior outputs, guiding the LLM Condition Enhancer to refine the original text prompt into diverse augmented conditions.
ID
Instructions
1
ADD: Expand the prompt by enriching existing elements with descriptive details (<= 20 tokens), strictly avoiding the introduction of new subjects.
2
DELETE: Remove some non-essential details (<= 20 tokens) to produce a concise caption, but the output must remain a valid description of image.
3
PARAPHRASE: Swap words/phrases for close synonyms (adjectives -> near-synonyms; nouns -> another common term for the same thing).
Appendix
Table 7: List of operations in the set PLLM used to query the LLM. Each operation directs the model to modify the input prompt while maintaining semantic consistency.
Figure 12: Illustration of Equivalent SDE Noise. (a) Standard flow SDE generates samples based on the transition mean anchored to a single condition c . (b) When substituting c with an augmented condition c′ , the transition mean shifts accordingly. To reach the original samples, an equivalent noise term ϵSDE′ is implicitly required. The μθ and μθ′ in the figure are shorthand for μθ(xt,c) and μθ(xt,c′) , respectively.
Figure 13: Additional Comparison Results on HPS-v3. (1/2)
Figure 14: Additional Comparison Results on HPS-v3. (2/2)
Figure 15: Additional Comparison Results on UnifiedReward-v2. (1/2)
Figure 16: Additional Comparison Results on UnifiedReward-v2. (2/2)
Figure 17: Additional Visual Samples of MV-GRPO. (1/4)
Figure 18: Additional Visual Samples of MV-GRPO. (2/4)
Figure 19: Additional Visual Samples of MV-GRPO. (3/4)
Figure 20: Additional Visual Samples of MV-GRPO. (4/4)
Figure 21: Results using same prompts and different seeds. (HPS-v3)
Figure 22: Results using same prompts and different seeds. (UR-v2)
Figure
Text Prompt
Figure. 1
A sleek silver supercar is parked on a wet city street.
A massive and brightly colored spacecraft in a deserted landscape, depicted in retro 1960s sci-fi art.
Close-up view of ancient Greek ruins set against a colourful, starry night sky creating a mystical atmosphere.
An ornate, lantern-lit courtyard with carved balconies and tiled floors.
A beautiful woman in a white gown wears gold jewels, warm outdoor light.
Close-up on a flickering neon installation in a dark futuristic nightclub. The text "Multi-View GRPO" is rendered in intertwining blue and magenta neon tubes, casting colored light onto chrome pipes and exposed wires on a brick wall. Cyberpunk aesthetic, shallow depth of field.
Appendix
Table 8: T2I prompts used in this paper (1/2). Prompts for each figure are listed sequentially, following the order from left to right and top to bottom.
Figure
Text Prompt
Figure. 18
American cowboy with a scruffy appearance in a retrofuturistic style, inspired by the animations of Studio Ghibli.
Undead army on riding beasts with symbols and music.
A Monet portrait.
a man walking alone down the street in a velvet jacket.
a shiny metallic renaissance steampunk robot in the style of Jan van Eyck.
A key visual of a young female swat officer with a neon futuristic gas mask in a cyberpunk setting.
Appendix
Table 9: T2I prompts used in this paper (2/2). Prompts for each figure are listed sequentially, following the order from left to right and top to bottom.
Group Relative Policy Optimization (GRPO) has demonstrated remarkable success in aligning text-to-image (T2I) flow models with human preferences. However, we have identified that the learning loop of current flow-based GRPO is fundamentally decoupled from the learner's current capability, suffering from critical blind spots at both prompt selection and advantage estimation: (i) Existing methods sample prompts randomly, overlooking the substantial impact of data selection on reinforcement learning (RL) efficacy--a factor proven crucial in GRPO for large language models; (ii) They evaluate sample quality solely relying on intra-group statistics, lacking a global perspective to accurately measure true policy improvement. To address these issues, we propose Adaptive GRPO (AdaGRPO), a novel capability-aware RL algorithm tailored for flow models. Specifically, AdaGRPO consists of two principal components: (i) Online Curriculum Filtering Strategy: Dynamically tracks the model's proficiency and adaptively selects prompts that best match its current learning boundary; (ii) Cross-Level Advantage Fusion: Synergistically integrates fine-grained intra-group advantages with macro-level global advantages, providing a comprehensive and unbiased policy evaluation. As a lightweight, plug-and-play module, AdaGRPO can be seamlessly integrated with existing frameworks such as Flow-GRPO, DanceGRPO, and Flow-CPS. Extensive experiments demonstrate that AdaGRPO consistently drives performance gains while significantly stabilizes GRPO training for flow models.
Jiazi Bu, Pengyang Ling, Yujie Zhou +8
1Shanghai Jiao Tong University · 2S-Lab, Nanyang Technological University · 3Shanghai AI Laboratory +7
Deploying GRPO on Flow Matching models has proven effective for text-to-image generation. However, existing paradigms typically propagate an outcome-based reward to all preceding denoising steps without distinguishing the local effect of each step. Moreover, current group-wise ranking mainly compares trajectories at matched timesteps and ignores within-trajectory dependencies, where certain early denoising actions can affect later states via delayed, implicit interactions. We propose TurningPoint-GRPO (TP-GRPO), a GRPO framework that alleviates step-wise reward sparsity and explicitly models long-term effects within the denoising trajectory. TP-GRPO makes two key innovations: (i) it replaces outcome-based rewards with step-level incremental rewards, providing a dense, step-aware learning signal that better isolates each denoising action's "pure" effect, and (ii) it identifies turning points-steps that flip the local reward trend and make subsequent reward evolution consistent with the overall trajectory trend-and assigns these actions an aggregated long-term reward to capture their delayed impact. Turning points are detected solely via sign changes in incremental rewards, making TP-GRPO efficient and hyperparameter-free. Extensive experiments also demonstrate that TP-GRPO exploits reward signals more effectively and consistently improves generation. Demo code is available at https://github.com/YunzeTong/TurningPoint-GRPO.
Yunze Tong, Mushui Liu, Canyu Zhao +7
1Zhejiang University, Hangzhou, China · 2Taobao & Tmall Group of Alibaba, Hangzhou, China · 3Tsinghua University, Beijing, China
Large-scale flow matching models have achieved strong performance across generative tasks such as text-to-image, video, 3D, and speech synthesis. However, aligning their outputs with human preferences and task-specific objectives remains challenging. Flow-GRPO extends Group Relative Policy Optimization (GRPO) to generation models, enabling stable reinforcement learning alignment for generative systems. Since its introduction, Flow-GRPO has triggered rapid research growth, spanning methodological refinements and diverse application domains. This survey provides a comprehensive review of Flow-GRPO and its subsequent developments. We organize existing work along two primary dimensions. First, we analyze methodological advances beyond the original framework, including reward signal design, credit assignment, sampling efficiency, diversity preservation, reward hacking mitigation, and reward model construction. Second, we examine extensions of GRPO-based alignment across generative paradigms and modalities, including text-to-image, video generation, image editing, speech and audio, 3D modeling, embodied vision-language-action systems, unified multimodal models, autoregressive and masked diffusion models, and restoration tasks. By synthesizing theoretical insights and practical adaptations, this survey highlights Flow-GRPO as a general alignment framework for modern generative models and outlines key open challenges for scalable and robust reinforcement-based generation.