UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation

Jie Liu; Zilyu Ye; Linxiao Yuan; Shenhan Zhu; Yu Gao; Jie Wu; Kunchang Li; Xionghui Wang

cs.CVMar 24, 2026

UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation

Authors: Jie Liu, Zilyu Ye, Linxiao Yuan, Shenhan Zhu, Yu Gao, Jie Wu, Kunchang Li, Xionghui Wang, +3 more

Organizations: The Chinese University of Hong Kong · ByteDance Seed

Abstract

Unified models capable of interleaved generation have emerged as a promising paradigm, with the community increasingly converging on autoregressive modeling for text and flow matching for image generation. To advance this direction, we propose a unified reinforcement learning framework tailored for interleaved generation. We validate our approach on its fundamental unit: a single round of reasoning-driven image generation, where the model first expands the user prompt through reasoning, followed by image synthesis. Formulating this multimodal generation process as a Markov Decision Process with sparse terminal rewards, we introduce UniGRPO to jointly optimize text and image generation policies using GRPO. Adopting a minimalist methodology to avoid over-design, we leverage established training recipes for both modalities by seamlessly integrating standard GRPO for reasoning and FlowGRPO for visual synthesis. To ensure scalability to multi-round interleaved generation, we introduce two critical modifications to the original FlowGRPO: (1) eliminating classifier-free guidance to maintain linear, unbranched rollouts, which is essential for scaling to complex scenarios involving multi-turn interactions and multi-condition generation (e.g., editing); and (2) replacing the standard latent KL penalty with an MSE penalty directly on the velocity fields, providing a more robust and direct regularization signal to mitigate reward hacking effectively. Our experiments demonstrate that this unified training recipe significantly enhances image generation quality through reasoning, providing a robust and scalable baseline for the future post-training of fully interleaved models.

Figures & tables

Figure 1: Overview of UniGRPO. By formulating interleaved generation as a joint MDP, this illustration demonstrates how UniGRPO jointly optimizes discrete language actions ( $y_{k}$ ) in the LLM’s next-token prediction, and continuous visual actions ( $x_{t_{k}-\Delta t}$ ) in flow matching. Both policies are updated using group-relative advantages derived from sparse terminal rewards.

Model / Method	Thinking	TA Score	GenEval
Bagel	$\times$	0.6810	0.78
Bagel	$\checkmark$	0.7132	0.79
SFT	$\times$	0.7486	0.83
SFT	$\checkmark$	0.7769	0.82
ReFL	$\times$	0.7786	0.85
ReFL	$\checkmark$	0.8120	0.84

Table 1: Main results on TA and GenEval. All RL methods start from the Bagel checkpoint after SFT. “Thinking” denotes whether the method explicitly generates intermediate reasoning tokens.

-

indicates training collapse.

Figure 3: Training and Validation reward curves of UniGRPO on the Finetuned Bagel base model at a resolution of 1024. The x-axis represents the gradient update steps.

Figure 4: Ablation Study on CFG. Removing CFG during training yields comparable or superior performance, showing that CFG is unnecessary for RL-based training. Note that CFG is applied at evaluation for all settings. Furthermore, these results are not directly comparable to the curves in Figure 3 , as this ablation uses the original Bagel as the base model at a resolution of 512.

Figure 5: Ablation Study on Regularization Strategies. From left to right: training reward, validation reward, and images generated under three different regularization strategies. Without regularization , the validation reward drops after an initial increase, leading to unnatural, oversaturated textures in the generated images. For KL divergence on the latents , the significant drop in training reward indicates that a sufficiently large KL coefficient has been used, yet grid-like artifacts still emerge as early as step 250, prompting us to terminate this run early. In contrast, directly applying MSE regularization on the velocity field ensures stable training dynamics and produces high-fidelity images with realistic textures.

Figure 6: Reasoning and visual outputs of UniGRPO. Our joint RL optimization produces task-oriented reasoning that guides the synthesis policy toward faithful, photorealistic image generation.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model / Method	Thinking	Overall	Single Obj.	Two Obj.	Counting	Colors	Position	Attr. Binding
Bagel	$\times$	0.78	0.98	0.96	0.78	0.84	0.52	0.58
Bagel	$\checkmark$	0.79	0.99	0.92	0.77	0.88	0.56	0.62
SFT	$\times$	0.83	0.99	0.95	0.83	0.89	0.58	0.75
SFT	$\checkmark$	0.82	0.98	0.93	0.63	0.91	0.68	0.79
ReFL	$\times$	0.85	1.00	0.97	0.86	0.92	0.57	0.81
ReFL	$\checkmark$	0.84	0.99	0.96	0.63	0.94	0.70	0.82

Appendix

Table 2: Main quantitative results on the GenEval benchmark. All RL methods use the Bagel checkpoint after SFT. “Thinking” denotes whether the method explicitly generates intermediate reasoning tokens.

\times

indicates training collapse.

Figure 7: Reasoning trace and generated image of the baseline Bagel model.

Figure 8: Reasoning trace and generation result after SFT. During SFT, we standardized the original <think> format to match prevalent LLM conventions. Although the model produces extensive reasoning steps, the translation from textual thoughts to fine-grained visual details remains suboptimal.

Model Configuration
Training Timesteps	25
CFG Scale	1
Timestep Shift	3
Image Resolution	1024
Evaluation Timesteps	50
Training

Appendix

Table 3: Hyperparameters for UniGRPO.

Explore similar work

Mar 10, 2026

Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization

Unified vision-language models have made significant progress in multimodal understanding and generation, yet they largely fall short in producing multimodal interleaved outputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual reasoning. In this work, we propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models, without relying on large-scale multimodal interleaved datasets. We begin with a warm-up stage using a hybrid dataset comprising curated interleaved sequences and limited data for multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities. To further refine interleaved generation, we propose a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting. Our approach jointly models text and image generation within a single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text alignment, and structural fidelity. Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex multimodal tasks. Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.

Ming Nie, Chunwei Wang, Jianhua Han +2

School of Data Science, Fudan University · Noah’s Ark Lab, Huawei · Yinwang Intelligent Technology Co., Ltd.

Multimodal Generation Unified Multimodal Models RL for Generative Models Group Relative Policy Optimization

Ask PDF

Jul 4, 2026

Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

Unified multi-modal models (UMMs) have shown promising interleaved text-image reasoning capabilities, yet effectively optimizing such multi-turn generation via reinforcement learning (RL) remains an open challenge. Existing approaches apply RL exclusively to text steps, relegating image generation to supervised surrogates, preventing policy gradients from propagating through the full interleaved trajectory across heterogeneous modalities. This leaves the potential of RL for UMMs largely untapped. In the paper, we introduce BRAID (Bridging inteRleAved multI-modal reasoning as a unified Decision process), a simple framework that casts multi-turn text-image-text reasoning as a unified Markov decision process (MDP), enabling joint optimization of textual and visual generation via a single, principled RL objective. BRAID computes a shared trajectory-level advantage and propagates it coherently into both text tokens and image denoising paths, each optimized through its modality-native policy gradient mechanism. To further address long-horizon credit assignment, BRAID employs a vision-language model (VLM) judge that scores each intermediate image on its reasoning utility, supplying dense turn-level feedback to sharpen learning at critical visual branches. Experiments on spatial reasoning and visual perception benchmarks show that BRAID consistently outperforms various baselines, confirming that a unified MDP formulation with vision-thinking guidance is essential for effective multi-modal reasoning.

Zican Hu, Xuyang Hu, Yiming Liu +10

Nanjing University · Shanghai AI Laboratory · Tsinghua University +5

Unified Multimodal Models Multimodal Reasoning Multimodal Optimization Multimodal Generation Multimodal Foundation Models RL for VLMs+1

Ask PDF

Apr 6, 2026

Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning

Humans paint images incrementally: they plan a global layout, sketch a coarse draft, inspect, and refine details, and most importantly, each step is grounded in the evolving visual states. However, can unified multimodal models trained on text-image interleaved datasets also imagine the chain of intermediate states? In this paper, we introduce process-driven image generation, a multi-step paradigm that decomposes synthesis into an interleaved reasoning trajectory of thoughts and actions. Rather than generating images in a single step, our approach unfolds across multiple iterations, each consisting of 4 stages: textual planning, visual drafting, textual reflection, and visual refinement. The textual reasoning explicitly conditions how the visual state should evolve, while the generated visual intermediate in turn constrains and grounds the next round of textual reasoning. A core challenge of process-driven generation stems from the ambiguity of intermediate states: how can models evaluate each partially-complete image? We address this through dense, step-wise supervision that maintains two complementary constraints: for the visual intermediate states, we enforce the spatial and semantic consistency; for the textual intermediate states, we preserve the prior visual knowledge while enabling the model to identify and correct prompt-violating elements. This makes the generation process explicit, interpretable, and directly supervisable. To validate proposed method, we conduct experiments under various text-to-image generation benchmarks.

Lei Zhang, Junjiao Tian, Zhipeng Fan +9

Meta Superintelligence Labs · University of California, San Diego · Worcester Polytechnic Institute +1

Image Generation T2I Generation Multimodal Reasoning Unified Multimodal Models

Ask PDF