Unified vision-language models have made significant progress in multimodal understanding and generation, yet they largely fall short in producing multimodal interleaved outputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual reasoning. In this work, we propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models, without relying on large-scale multimodal interleaved datasets. We begin with a warm-up stage using a hybrid dataset comprising curated interleaved sequences and limited data for multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities. To further refine interleaved generation, we propose a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting. Our approach jointly models text and image generation within a single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text alignment, and structural fidelity. Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex multimodal tasks. Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.
Figures & tables
Figure 1: Overview of our reinforcement fine-tuning framework. Multimodal tokens are autoregressively generated and decoded into completions, with token probabilities used to compute KL divergence along a single trajectory. Hybrid rewards are assigned to each completion, and token-level group relative advantages are calculated to guide policy optimization along with KL regularization.
MMIE
Method
Params
Situational analysis
Project-based learning
Multi-step reasoning
AVG
MiniGPT-5 Zheng et al. (2023)
7B
47.63
55.12
42.17
50.92
EMU-2 Sun et al. (2024)
37B
39.65
46.12
50.75
45.33
GILL Koh et al. (2023)
7B
46.72
57.57
39.33
51.58
Anole Chern et al. (2024)
7B
48.95
59.05
51.72
55.22
Ours (QLoRA)
7B
53.12
60.34
53.28
57.27
Table 1: Comparison with existing methods on MMIE. The proposed method significantly improves the generation quality of unified models, producing interleaved text–image outputs that are both coherent and aligned with instructions.
InterleavedBench
Method
Params
Text quality
Perceptual quality
Image coherence
TIC
Helpfulness
AVG
MiniGPT-5 Zheng et al. (2023)
7B
1.22
2.45
1.62
2.03
1.77
1.82
EMU-2 Sun et al. (2024)
37B
1.26
2.28
1.89
1.34
1.64
1.68
GILL Koh et al. (2023)
7B
0.75
3.21
2.25
1.53
1.48
1.84
Ours (QLoRA)
7B
2.33
3.39
3.01
2.75
2.85
2.87
Ours
7B
2.86
3.58
3.25
3.02
2.94
3.13
Table 2: Comparison with existing methods on InterleavedBench, where our method demonstrates superior performance across multiple evaluation metrics.
Warm-up
GRPO
MMIE
InterleavedBench
✗
✗
-
0.51
✓
✗
53.31
1.97
✓
✓
59.50
3.13
Table 3: Effects of warm-up stage and GRPO. The warm-up stage enables to generate interleaved multimodal outputs. Building on the warm-up initialization, the proposed GRPO-based policy optimization substantially improves the quality of interleaved generations.
Hybrid rewards
rf
rt
rv
Process reward
MMIE
InterleavedBench
✓
✗
✗
✗
53.56
2.05
✓
✓
✗
✗
54.62
2.30
✓
✓
✓
✗
57.83
2.79
✓
✓
✓
✓
59.50
3.13
Table 4: Hybrid reward components ablation. Beginning with the format reward rf , we gradually integrate the textual, visual and process-level rewards. The results demonstrate that the inclusion of complementary reward signals progressively enhances interleaved generation quality.
Understanding
Generation
Method
ILO-Uni.
Params
MME-P Liang et al. (2024)
MMvet Yu et al. (2024)
SEEDBench-img Li et al. (2024)
POPE Li et al. (2023)
#Train images
Image res.
Score
VILA-U Wu et al. (2024b)
✗
7B
1401.8
33.5
59.0
85.8
15M
384
0.42
SEED-X Ge et al. (2024)
✗
17B
1435.7
-
-
84.2
158M
-
0.49
TokenFlow-XL Qu et al. (2024)
✗
13B
1545.9
40.7
68.7
86.8
60M
384
0.55
Janus-Pro Chen et al. (2025b)
✗
7B
1516.7
45.1
70.1
78.9
72M
384
0.80
Chameleon Team (2024)
✓
7B
153.1
8.3
30.5
19.4
1.4B
512
0.39
Table 5: Comparison with existing methods on visual understanding and generation benchmarks. ILO-Uni. denotes “Unified models with interleaved text-image output”. Our approach maintains comparable performance to baseline models.
KL
G
rv
MMIE
InterleavedBench
✗
2
ImageReward Xu et al. (2023)
53.04
1.72
✓
2
ImageReward
55.14
2.27
✓
4
ImageReward
59.50
3.13
✓
4
CLIP-score
56.94
2.58
Table 6: Aspects of GRPO hyper-parameter implementation. We first analyze the effect of the KL-penalty term by duplicating its application during training. Results show that maintaining a KL constraint stabilizes optimization and mitigates policy drift from the initialization, underscoring its critical role. We then ablate the number of generations G to evaluate its impact on learning dynamics, and finally, we compare different formulations of the visual reward function.
Figure 2: Visualizations of multimodal interleaved generation. Qualitative examples illustrate the model’s capacity to produce coherent interleaved outputs, smoothly transitioning between text and image modalities within a unified generation process.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Illustration of data preparation process during the warm-up stage.
Figure 4: Failure cases analysis about hallucinated visual content.
Unified models capable of interleaved generation have emerged as a promising paradigm, with the community increasingly converging on autoregressive modeling for text and flow matching for image generation. To advance this direction, we propose a unified reinforcement learning framework tailored for interleaved generation. We validate our approach on its fundamental unit: a single round of reasoning-driven image generation, where the model first expands the user prompt through reasoning, followed by image synthesis. Formulating this multimodal generation process as a Markov Decision Process with sparse terminal rewards, we introduce UniGRPO to jointly optimize text and image generation policies using GRPO. Adopting a minimalist methodology to avoid over-design, we leverage established training recipes for both modalities by seamlessly integrating standard GRPO for reasoning and FlowGRPO for visual synthesis. To ensure scalability to multi-round interleaved generation, we introduce two critical modifications to the original FlowGRPO: (1) eliminating classifier-free guidance to maintain linear, unbranched rollouts, which is essential for scaling to complex scenarios involving multi-turn interactions and multi-condition generation (e.g., editing); and (2) replacing the standard latent KL penalty with an MSE penalty directly on the velocity fields, providing a more robust and direct regularization signal to mitigate reward hacking effectively. Our experiments demonstrate that this unified training recipe significantly enhances image generation quality through reasoning, providing a robust and scalable baseline for the future post-training of fully interleaved models.
Jie Liu, Zilyu Ye, Linxiao Yuan +8
The Chinese University of Hong Kong · ByteDance Seed
The advancement of generative AI models capable of producing text and image marks a critical step forward in the realm of multimodal intelligence, particularly for tasks involving the interleaving of both modalities. To advance this intelligence to the next stage, it is crucial for models to autonomously generate free-form interleaved text-image sequences. In this paper, we introduce ILLUME-X, an advanced unified multimodal paradigm that enables high-quality, free-form interleaved text-image generation by improving multimodal data efficiency and stabilizing the multimodal training process. ILLUME-X comprises three key components: (i) an expanded training data pipeline optimized for interleaved text-image generation, (ii) a progressive training strategy with self-adaptive objectives for free-length multimodal token sequences, and (iii) an objective and comprehensive evaluation method ILScore for interleaved text-image sequences. Notably, our ILLUME-X outperforms previous unified models across multiple interleaved text-image generation tasks like style transfer, image decomposition and storytelling.
Chonghuinan Wang, Zhikai Chen, Chunwei Wang +9
Harbin Institute of Technology, Harbin, China · Huawei Noah’s Ark Lab, Shenzhen, China · Zhengzhou Advanced Research Institute of Harbin Institute of Technology, Zhengzhou, China +1
Recent unified models have made unprecedented progress in both understanding and generation. However, while most of them accept multi-modal inputs, they typically produce only single-modality outputs. This challenge of producing interleaved content is mainly due to training data scarcity and the difficulty of modeling long-range cross-modal context. To address this issue, we decompose interleaved generation into textual planning and visual consistency modeling, and introduce a framework consisting of a planner and a visualizer. The planner produces dense textual descriptions for visual content, while the visualizer synthesizes images accordingly. Under this guidance, we construct large-scale textual-proxy interleaved data (where visual content is represented in text) to train the planner, and curate reference-guided image data to train the visualizer. These designs give rise to Wan-Weaver, which exhibits emergent interleaved generation ability with long-range textual coherence and visual consistency. Meanwhile, the integration of diverse understanding and generation data into planner training enables Wan-Weaver to achieve robust task reasoning and generation proficiency. To assess the model's capability in interleaved generation, we further construct a benchmark that spans a wide range of use cases across multiple dimensions. Extensive experiments demonstrate that, even without access to any real interleaved data, Wan-Weaver achieves superior performance over existing methods.