UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation

Jie Liu; Zilyu Ye; Linxiao Yuan; Shenhan Zhu; Yu Gao; Jie Wu; Kunchang Li; Xionghui Wang

cs.CVMar 24, 2026

UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation

Authors: Jie Liu, Zilyu Ye, Linxiao Yuan, Shenhan Zhu, Yu Gao, Jie Wu, Kunchang Li, Xionghui Wang, +3 more

Organizations: The Chinese University of Hong Kong · ByteDance Seed

Abstract

Unified models capable of interleaved generation have emerged as a promising paradigm, with the community increasingly converging on autoregressive modeling for text and flow matching for image generation. To advance this direction, we propose a unified reinforcement learning framework tailored for interleaved generation. We validate our approach on its fundamental unit: a single round of reasoning-driven image generation, where the model first expands the user prompt through reasoning, followed by image synthesis. Formulating this multimodal generation process as a Markov Decision Process with sparse terminal rewards, we introduce UniGRPO to jointly optimize text and image generation policies using GRPO. Adopting a minimalist methodology to avoid over-design, we leverage established training recipes for both modalities by seamlessly integrating standard GRPO for reasoning and FlowGRPO for visual synthesis. To ensure scalability to multi-round interleaved generation, we introduce two critical modifications to the original FlowGRPO: (1) eliminating classifier-free guidance to maintain linear, unbranched rollouts, which is essential for scaling to complex scenarios involving multi-turn interactions and multi-condition generation (e.g., editing); and (2) replacing the standard latent KL penalty with an MSE penalty directly on the velocity fields, providing a more robust and direct regularization signal to mitigate reward hacking effectively. Our experiments demonstrate that this unified training recipe significantly enhances image generation quality through reasoning, providing a robust and scalable baseline for the future post-training of fully interleaved models.

Figures & tables

Figure 1: Overview of UniGRPO. By formulating interleaved generation as a joint MDP, this illustration demonstrates how UniGRPO jointly optimizes discrete language actions ( $y_{k}$ ) in the LLM’s next-token prediction, and continuous visual actions ( $x_{t_{k}-\Delta t}$ ) in flow matching. Both policies are updated using group-relative advantages derived from sparse terminal rewards.

Model / Method	Thinking	TA Score	GenEval
Bagel	$\times$	0.6810	0.78
Bagel	$\checkmark$	0.7132	0.79
SFT	$\times$	0.7486	0.83
SFT	$\checkmark$	0.7769	0.82
ReFL	$\times$	0.7786	0.85
ReFL	$\checkmark$	0.8120	0.84

Table 1: Main results on TA and GenEval. All RL methods start from the Bagel checkpoint after SFT. “Thinking” denotes whether the method explicitly generates intermediate reasoning tokens.

-

indicates training collapse.

Figure 3: Training and Validation reward curves of UniGRPO on the Finetuned Bagel base model at a resolution of 1024. The x-axis represents the gradient update steps.

Figure 4: Ablation Study on CFG. Removing CFG during training yields comparable or superior performance, showing that CFG is unnecessary for RL-based training. Note that CFG is applied at evaluation for all settings. Furthermore, these results are not directly comparable to the curves in Figure 3 , as this ablation uses the original Bagel as the base model at a resolution of 512.

Figure 5: Ablation Study on Regularization Strategies. From left to right: training reward, validation reward, and images generated under three different regularization strategies. Without regularization , the validation reward drops after an initial increase, leading to unnatural, oversaturated textures in the generated images. For KL divergence on the latents , the significant drop in training reward indicates that a sufficiently large KL coefficient has been used, yet grid-like artifacts still emerge as early as step 250, prompting us to terminate this run early. In contrast, directly applying MSE regularization on the velocity field ensures stable training dynamics and produces high-fidelity images with realistic textures.

Figure 6: Reasoning and visual outputs of UniGRPO. Our joint RL optimization produces task-oriented reasoning that guides the synthesis policy toward faithful, photorealistic image generation.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model / Method	Thinking	Overall	Single Obj.	Two Obj.	Counting	Colors	Position	Attr. Binding
Bagel	$\times$	0.78	0.98	0.96	0.78	0.84	0.52	0.58
Bagel	$\checkmark$	0.79	0.99	0.92	0.77	0.88	0.56	0.62
SFT	$\times$	0.83	0.99	0.95	0.83	0.89	0.58	0.75
SFT	$\checkmark$	0.82	0.98	0.93	0.63	0.91	0.68	0.79
ReFL	$\times$	0.85	1.00	0.97	0.86	0.92	0.57	0.81
ReFL	$\checkmark$	0.84	0.99	0.96	0.63	0.94	0.70	0.82

Appendix

Table 2: Main quantitative results on the GenEval benchmark. All RL methods use the Bagel checkpoint after SFT. “Thinking” denotes whether the method explicitly generates intermediate reasoning tokens.

\times

indicates training collapse.

Figure 7: Reasoning trace and generated image of the baseline Bagel model.

Figure 8: Reasoning trace and generation result after SFT. During SFT, we standardized the original <think> format to match prevalent LLM conventions. Although the model produces extensive reasoning steps, the translation from textual thoughts to fine-grained visual details remains suboptimal.

Model Configuration
Training Timesteps	25
CFG Scale	1
Timestep Shift	3
Image Resolution	1024
Evaluation Timesteps	50
Training

Appendix

Table 3: Hyperparameters for UniGRPO.

Explore similar work

Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization
Mar 10, 2026Ming Nie, Chunwei Wang, Jianhua Han +2Multimodal Generation Unified Multimodal Models
School of Data Science, Fudan University · Noah’s Ark Lab, Huawei · Yinwang Intelligent Technology Co., Ltd.
Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process
Jul 4, 2026Zican Hu, Xuyang Hu, Yiming Liu +10Unified Multimodal Models Multimodal Reasoning
Nanjing University · Shanghai AI Laboratory · Tsinghua University +5
Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning
Apr 6, 2026Lei Zhang, Junjiao Tian, Zhipeng Fan +9Image Generation T2I Generation
Meta Superintelligence Labs · University of California, San Diego · Worcester Polytechnic Institute +1

Mar 10, 2026J/K move · Enter open · S save

Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization

Ming Nie, Chunwei Wang, Jianhua Han, Hang Xu+1

School of Data Science, Fudan University · Noah’s Ark Lab, Huawei · Yinwang Intelligent Technology Co., Ltd.

Multimodal Generation Unified Multimodal Models RL for Generative Models Group Relative Policy Optimization

Open paper AskPDF arXiv

Unified vision-language models have made significant progress in multimodal understanding and generation, yet they largely fall short in producing multimodal interleaved outputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual reasoning. In this work, we propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models, without relying on large-scale multimodal interleaved datasets. We begin with a warm-up stage using a hybrid dataset comprising curated interleaved sequences and limited data for multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities. To further refine interleaved generation, we propose a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting. Our approach jointly models text and image generation within a single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text alignment, and structural fidelity. Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex multimodal tasks. Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.

Abstract

Explore similar work

Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization

Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning