LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation

Shentong Mo; Sukmin Yun

cs.CVMar 29, 2026

LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation

Authors: Shentong Mo, Sukmin Yun

Organizations: Department of Machine Learning, CMU, USA · Department of Machine Learning, MBZUAI, UAE · Department of Artificial Intelligence, Hanyang University ERICA, South Korea

Abstract

Unified multimodal pretraining has emerged as a promising paradigm for jointly modeling language and vision within a single foundation model. However, existing approaches largely rely on implicit or indirect alignment signals and remain suboptimal for simultaneously supporting multimodal understanding and generation, particularly in settings that require fine-grained language-visual reasoning and controllable generation. In this work, we propose LVRPO, a language-visual reinforcement-based preference optimization framework that explicitly aligns language and visual representations using Group Relative Policy Optimization (GRPO). Instead of introducing additional alignment losses at the representation level, LVRPO directly optimizes multimodal model behaviors through preference-driven reinforcement signals, encouraging consistent and semantically grounded interactions between language and vision across both understanding and generation tasks. This formulation enables effective alignment without requiring auxiliary encoders or handcrafted cross-modal objectives, and naturally extends to diverse multimodal capabilities. Empirically, LVRPO consistently outperforms strong unified-pretraining baselines on a broad suite of benchmarks spanning multimodal understanding, generation, and reasoning.

Figures & tables

Figure 1: Illustration of the proposed LVRPO framework for unified multimodal understanding and generation. The framework proceeds in three stages: (1) Group Sampling: For a given multimodal prompt $q$ , the unified MoT backbone (initialized from BAGEL) samples a group of $G$ independent outputs $\{o_{1},\dots,o_{G}\}$ , which can include text reasoning, visual tokens, or image edits. (2) Behavioral Reward Estimation: Each output is evaluated by a multi-dimensional reward system. We utilize a frozen SigLIP 2 referee to provide dense semantic grounding signals ( $r_{sem}$ ), alongside rule-based verifiers for instruction following ( $r_{ins}$ ) and knowledge consistency ( $r_{kn}$ ). (3) Group Relative Optimization: Instead of using a separate critic network, LVRPO employs Group Relative Policy Optimization to compute the advantage $\hat{A}_{i}$ for each sample by normalizing its reward against the mean and standard deviation of the group. This behavioral feedback $\nabla_{\theta}\mathcal{J}_{\mathrm{LVRPO}}$ is backpropagated through the shared attention layers to jointly optimize the reasoning and generative experts, enforcing cross-modal consistency without the need for auxiliary representation-level alignment losses.

Type	Model	Single Obj.	Two Obj.	Counting	Colors	Position	Color Attri.	Overall $\uparrow$
Gen. Only	PixArt- $\alpha$ ( Chen et al., 2024a )	0.98	0.50	0.44	0.80	0.08	0.07	0.48
	SDv $2.1$ ( Rombach et al., 2022 )	0.98	0.51	0.44	0.85	0.07	0.17	0.50
	DALL-E $2$ ( Ramesh et al., 2022 )	0.94	0.66	0.49	0.77	0.10	0.19	0.52
	Emu $3$ -Gen ( Wang et al., 2024c )	0.98	0.71	0.34	0.81	0.17	0.21	0.54
	SDXL ( Podell et al., 2024 )	0.98	0.74	0.39	0.85	0.15	0.23	0.55
	DALL-E $3$ ( Betker et al., 2023 )	0.96	0.87	0.47	0.83	0.43	0.45	0.67

Table 1: Comparison of text-to-image generation on GenEval benchmark.

Type	Model	Cultural	Time	Space	Biology	Physics	Chemistry	Overall $\uparrow$
Gen. Only	SDv1.5 ( Rombach et al., 2022 )	0.34	0.35	0.32	0.28	0.29	0.21	0.32
	SDXL ( Podell et al., 2024 )	0.43	0.48	0.47	0.44	0.45	0.27	0.43
	SD3.5-large ( Esser et al., 2024 )	0.44	0.50	0.58	0.44	0.52	0.31	0.46
	PixArt-Alpha ( Chen et al., 2024a )	0.45	0.50	0.48	0.49	0.56	0.34	0.47
	playground-v2.5 ( Li et al., 2024 )	0.49	0.58	0.55	0.43	0.48	0.33	0.49
	FLUX.1-dev ( Labs, 2024 )	0.48	0.58	0.62	0.42	0.51	0.35	0.50

Table 2: Comparison of world knowledge reasoning on WISE.

Type	Model	GEdit-Bench-EN (Full set) $\uparrow$			GEdit-Bench-CN (Full set) $\uparrow$
Type	Model	G_SC	G_PQ	G_O	G_SC	G_PQ	G_O
Private	Gemini 2.0 ( Gemini2, 2025 )	6.73	6.61	6.32	5.43	6.78	5.36
Private	GPT-4o ( OpenAI, 2025 )	7.85	7.62	7.53	7.67	7.56	7.30
Open-source	Instruct-Pix2Pix ( Brooks et al., 2023 )	3.58	5.49	3.68	-	-	-
	MagicBrush ( Zhang et al., 2023 )	4.68	5.66	4.52	-	-	-
	AnyEdit ( Yu et al., 2024a )	3.18	5.82	3.21	-	-	-

Table 3: Comparison on GEdit-Bench.

Type	Model	Score $\uparrow$
Private	GPT-4o ( OpenAI, 2025 )	78.9
Private	Gemini 2.0 ( Gemini2, 2025 )	57.6
Open-source	Step1X-Edit ( Liu et al., 2025a )	14.9
	BAGEL ( Deng et al., 2025 )	55.3
	LVRPO (ours)	58.7

Table 4: Comparison on IntelligentBench.

Type	Model	# LLM Params	MME-P $\uparrow$	MME-S $\uparrow$	MMBench $\uparrow$	MMMU $\uparrow$	MM-Vet $\uparrow$	MathVista $\uparrow$	MMVP $\uparrow$
Und. Only	InternVL2 ( Chen et al., 2024c )	1.8B	1440	1877	73.2	34.3	44.6	46.4	35.3
	InternVL2.5 ( Chen et al., 2024b )	1.8B	-	2138	74.7	43.6	60.8	51.3	-
	Qwen2-VL ( Wang et al., 2024b )	1.5B	-	1872	74.9	41.1	49.5	43.0	-
	Qwen2.5-VL ( Bai et al., 2025 )	3B	-	2157	79.1	53.1	61.8	62.3	-
	BLIP-3 ( Xue et al., 2024 )	4B	-	-	76.8	41.1	-	39.6	-
	LLava-OV ( Li et al., 2025 )	7B	1580	-	80.8	48.8	57.5	63.2	-

Table 5: Comparison with state-of-the-arts on viusal understanding benchmarks.

Figure 2: Reward convergence and policy stability during the GRPO phase. Left: Evolution of individual reward components ( $r_{sem}$ , $r_{ins}$ , $r_{kn}$ ) showing consistent upward trajectories and fast semantic convergence. Right: Policy stability analysis showing stable KL-divergence as the advantage signal increases.

Group Size $G$	GenEval	WISE	Training Time (per step)
2	0.84	0.74	0.8s
4	0.87	0.79	1.3s
8	0.91	0.85	2.1s
16	0.92	0.86	4.4s

Table 6: Impact of Group Size

G

on GenEval and WISE scores.

Figure 3: Analysis of the Understanding-Generation trade-off. While optimizing only for generative rewards leads to a decline in reasoning (MMMU), the full LVRPO objective demonstrates positive interference, where behavioral alignment in generation improves discriminative visual perception (MMVP).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimizer	AdamW
Learning Rate	$5\times 10^{-6}$
Weight Decay	$0.05$
Global Batch Size	256
Group Size $G$	8
KL Coefficient $\beta$	$0.01$

Appendix

Table 7: Hyperparameters for LVRPO Training.

Figure 4: Ablation Study: Impact of Freezing the Reward Model. We compare the training dynamics of LVRPO with a Frozen SigLIP 2 referee (blue) versus a Trainable Reward Model (red). While the trainable reward model allows the total reward to skyrocket (Left), a visual inspection of the outputs at step 2000 (Right) reveals ”reward hacking,” where the model generates high-frequency adversarial noise that exploits the drifting encoder. In contrast, the frozen baseline maintains semantic integrity, proving that a stable metric anchor is essential for the convergence of the GRPO objective.

Figure 5: Scaling behaviors of LVRPO across model sizes. We track the GenEval score improvement over training steps. The 1.3B model (green) saturates early, struggling to internalize complex spatial logic rewards ( $r_{ins}$ ) due to limited capacity. The 7B model (blue) shows steady, robust improvement. The 13B model (red) demonstrates accelerated convergence , reaching state-of-the-art performance (0.92) in less than 2,000 steps, less than 40% of the training time required by the 7B model, validating the efficiency of GRPO on larger unified backbones.

Figure 6: Qualitative comparison on challenging prompts. Left (Spatial Reasoning): ”A cat under a table, with a red ball on top of the table.” LVRPO correctly places objects, while baselines often merge them. Right (Attribute Binding): ”A green fire truck.” Baselines revert to red due to training priors; LVRPO follows the instruction.

Vision-Language Alignment

Vision-language alignment connects visual and textual representations so models can match, interpret, and ground language in images and video.

See all 361 papers

Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization
Mar 10, 2026Ming Nie, Chunwei Wang, Jianhua Han +2Multimodal Generation Unified Multimodal Models
School of Data Science, Fudan University · Noah’s Ark Lab, Huawei · Yinwang Intelligent Technology Co., Ltd.
PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping
Jun 7, 2026Qiming Li, Tianlun Li, Xiaolong Cheng +5Token-Level Credit Assignment VLM Reasoning
Amap CV Lab, Alibaba Group · Peking University
Structured Role-Aware Policy Optimization for Multimodal Reasoning
May 8, 2026Bingqing Jiang, Difan ZouToken-Level Credit Assignment Multimodal Reasoning
School of Computing & Data Science, The University of Hong Kong · School of Computing & Data Science and Institute of Data Science, The University of Hong Kong

Mar 10, 2026J/K move · Enter open · S save

Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization

Ming Nie, Chunwei Wang, Jianhua Han, Hang Xu+1

School of Data Science, Fudan University · Noah’s Ark Lab, Huawei · Yinwang Intelligent Technology Co., Ltd.

Multimodal Generation Unified Multimodal Models RL for Generative Models Group Relative Policy Optimization

Open paper AskPDF arXiv

Unified vision-language models have made significant progress in multimodal understanding and generation, yet they largely fall short in producing multimodal interleaved outputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual reasoning. In this work, we propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models, without relying on large-scale multimodal interleaved datasets. We begin with a warm-up stage using a hybrid dataset comprising curated interleaved sequences and limited data for multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities. To further refine interleaved generation, we propose a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting. Our approach jointly models text and image generation within a single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text alignment, and structural fidelity. Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex multimodal tasks. Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.

Abstract

Vision-Language Alignment

Explore similar work

Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization

PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping

Structured Role-Aware Policy Optimization for Multimodal Reasoning