Unified vision-language models have made significant progress in multimodal understanding and generation, yet they largely fall short in producing multimodal interleaved outputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual reasoning. In this work, we propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models, without relying on large-scale multimodal interleaved datasets. We begin with a warm-up stage using a hybrid dataset comprising curated interleaved sequences and limited data for multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities. To further refine interleaved generation, we propose a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting. Our approach jointly models text and image generation within a single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text alignment, and structural fidelity. Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex multimodal tasks. Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.
Figures & tables
Figure 1: Overview of our reinforcement fine-tuning framework. Multimodal tokens are autoregressively generated and decoded into completions, with token probabilities used to compute KL divergence along a single trajectory. Hybrid rewards are assigned to each completion, and token-level group relative advantages are calculated to guide policy optimization along with KL regularization.
MMIE
Method
Params
Situational analysis
Project-based learning
Multi-step reasoning
AVG
MiniGPT-5 Zheng et al. (2023)
7B
47.63
55.12
42.17
50.92
EMU-2 Sun et al. (2024)
37B
39.65
46.12
50.75
45.33
GILL Koh et al. (2023)
7B
46.72
57.57
39.33
51.58
Anole Chern et al. (2024)
7B
48.95
59.05
51.72
55.22
Ours (QLoRA)
7B
53.12
60.34
53.28
57.27
Table 1: Comparison with existing methods on MMIE. The proposed method significantly improves the generation quality of unified models, producing interleaved text–image outputs that are both coherent and aligned with instructions.
InterleavedBench
Method
Params
Text quality
Perceptual quality
Image coherence
TIC
Helpfulness
AVG
MiniGPT-5 Zheng et al. (2023)
7B
1.22
2.45
1.62
2.03
1.77
1.82
EMU-2 Sun et al. (2024)
37B
1.26
2.28
1.89
1.34
1.64
1.68
GILL Koh et al. (2023)
7B
0.75
3.21
2.25
1.53
1.48
1.84
Ours (QLoRA)
7B
2.33
3.39
3.01
2.75
2.85
2.87
Ours
7B
2.86
3.58
3.25
3.02
2.94
3.13
Table 2: Comparison with existing methods on InterleavedBench, where our method demonstrates superior performance across multiple evaluation metrics.
Warm-up
GRPO
MMIE
InterleavedBench
✗
✗
-
0.51
✓
✗
53.31
1.97
✓
✓
59.50
3.13
Table 3: Effects of warm-up stage and GRPO. The warm-up stage enables to generate interleaved multimodal outputs. Building on the warm-up initialization, the proposed GRPO-based policy optimization substantially improves the quality of interleaved generations.
Hybrid rewards
rf
rt
rv
Process reward
MMIE
InterleavedBench
✓
✗
✗
✗
53.56
2.05
✓
✓
✗
✗
54.62
2.30
✓
✓
✓
✗
57.83
2.79
✓
✓
✓
✓
59.50
3.13
Table 4: Hybrid reward components ablation. Beginning with the format reward rf , we gradually integrate the textual, visual and process-level rewards. The results demonstrate that the inclusion of complementary reward signals progressively enhances interleaved generation quality.
Understanding
Generation
Method
ILO-Uni.
Params
MME-P Liang et al. (2024)
MMvet Yu et al. (2024)
SEEDBench-img Li et al. (2024)
POPE Li et al. (2023)
#Train images
Image res.
Score
VILA-U Wu et al. (2024b)
✗
7B
1401.8
33.5
59.0
85.8
15M
384
0.42
SEED-X Ge et al. (2024)
✗
17B
1435.7
-
-
84.2
158M
-
0.49
TokenFlow-XL Qu et al. (2024)
✗
13B
1545.9
40.7
68.7
86.8
60M
384
0.55
Janus-Pro Chen et al. (2025b)
✗
7B
1516.7
45.1
70.1
78.9
72M
384
0.80
Chameleon Team (2024)
✓
7B
153.1
8.3
30.5
19.4
1.4B
512
0.39
Table 5: Comparison with existing methods on visual understanding and generation benchmarks. ILO-Uni. denotes “Unified models with interleaved text-image output”. Our approach maintains comparable performance to baseline models.
KL
G
rv
MMIE
InterleavedBench
✗
2
ImageReward Xu et al. (2023)
53.04
1.72
✓
2
ImageReward
55.14
2.27
✓
4
ImageReward
59.50
3.13
✓
4
CLIP-score
56.94
2.58
Table 6: Aspects of GRPO hyper-parameter implementation. We first analyze the effect of the KL-penalty term by duplicating its application during training. Results show that maintaining a KL constraint stabilizes optimization and mitigates policy drift from the initialization, underscoring its critical role. We then ablate the number of generations G to evaluate its impact on learning dynamics, and finally, we compare different formulations of the visual reward function.
Figure 2: Visualizations of multimodal interleaved generation. Qualitative examples illustrate the model’s capacity to produce coherent interleaved outputs, smoothly transitioning between text and image modalities within a unified generation process.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Illustration of data preparation process during the warm-up stage.
Figure 4: Failure cases analysis about hallucinated visual content.
Harbin Institute of Technology, Harbin, China · Huawei Noah’s Ark Lab, Shenzhen, China · Zhengzhou Advanced Research Institute of Harbin Institute of Technology, Zhengzhou, China +1