Self-correction Optimization for Interleaved Multimodal Generation

Xin You; Zhiwei Ning; Zukai Chen; Minghui Zhang; Xuanke Shi; Hanxiao Zhang; Jingsong Liu; Jie Yang

cs.CVOct 7, 2026

Self-correction Optimization for Interleaved Multimodal Generation

Authors: Xin You, Zhiwei Ning, Zukai Chen, Minghui Zhang, Xuanke Shi, Hanxiao Zhang, Jingsong Liu, Jie Yang, +2 more

Organizations: Shanghai Jiao Tong University · SenseTime Research · Technical University of Munich

Abstract

Multimodal large language models (MLLMs) have made significant progress in visual understanding and generation. However, generating interleaved image--text content remains challenging, as it requires tightly integrated multimodal understanding and generation capabilities. Although existing MLLMs provide promising solutions, most rely on additional training with augmented data, which is computationally expensive and remains limited in preserving visual subjects, temporal consistency, and physical plausibility. In this work, we propose self-correction optimization (SCO), an effective training-free method for consistent interleaved generation. SCO treats the classifier-free guidance update as a reference and performs minimal self-correction under two complementary constraints, including new-event and state-preserving constraints. Specifically, the new-event constraint promotes temporal consistency across image--text sequences, while the state-preserving constraint maintains the coherence of visual subjects throughout subsequent generation steps. Experiments on challenging interleaved multimodal generation benchmarks demonstrate significant improvements in temporal coherence and visual-subject preservation. Furthermore, SCO can be extended to video generation and improves the modeling of physically grounded processes, including robot manipulation and long-horizon handcrafting.

Figures & tables

Figure 1: The mechanism of self-correction optimization, which consists of new-event and state-preserving constraints for the new-event realization and visual subject consistency.

Figure 2: Qualitative results on various interleaved generation tasks when evaluating the efficacy of SCO, including iterative image editing, novel view synthesis, and embodied-AI scenarios.

Type	Method	FID $\downarrow$	CLIP-I $\uparrow$	CLIP-T $\uparrow$	GPT-J1 $\uparrow$	$C_{s}$ $\uparrow$	$C_{b}$ $\uparrow$	GPT-J2 $\uparrow$
Modular Model	MiniGPT-5 Zheng et al. (2023)	91.3	0.151	0.269	0.327	0.13	0.20	0.243
	MiniGPT-5 + SCO	88.5	0.170	0.285	0.365	0.16	0.22	0.279
	Show-o2 Xie et al. (2026)	83.6	0.146	0.270	0.337	0.12	0.17	0.248
	Show-o2 + SCO	79.7	0.169	0.281	0.349	0.13	0.19	0.264
	MM-Interleaved Tian et al. (2024)	76.2	0.201	0.329	0.397	0.18	0.22	0.351
	MM-Interleaved + SCO	73.6	0.227	0.337	0.419	0.20	0.23	0.384

Table 1: Quantitative results on the ISG-Bench dataset when plugging SCO into various open-source models, including modular and unified models.

Type	Method	FID $\downarrow$	CLIP-I $\uparrow$	CLIP-T $\uparrow$	GPT-J1 $\uparrow$	$C_{s}$ $\uparrow$	$C_{b}$ $\uparrow$	GPT-J2 $\uparrow$
Modular Model	MiniGPT-5 Zheng et al. (2023)	90.4	0.149	0.257	0.445	0.18	0.23	0.344
	MiniGPT-5 + SCO	89.9	0.138	0.247	0.423	0.17	0.21	0.319
	Show-o2 Xie et al. (2026)	88.5	0.173	0.280	0.471	0.19	0.26	0.385
	Show-o2 + SCO	84.0	0.179	0.291	0.496	0.19	0.28	0.401
	MM-Interleaved Tian et al. (2024)	94.1	0.140	0.269	0.432	0.15	0.19	0.327
	MM-Interleaved + SCO	81.3	0.164	0.293	0.450	0.17	0.22	0.369

Table 2: Quantitative results on the Opening dataset when plugging SCO into various open-source models, including modular and unified models.

Figure 3: Sensitivity analysis for hyper-parameters of SCO ( $\alpha_{0}$ , $s_{state}$ , $M_{state}$ , $M_{new}$ , and $\tau_{end}$ ).

	MM-Interleaved					SenseNova-U1
Setting	FID $\downarrow$	CLIP-I $\uparrow$	GPT-J1 $\uparrow$	$C_{\mathrm{s}}$ $\uparrow$	GPT-J2 $\uparrow$	FID $\downarrow$	CLIP-I $\uparrow$	GPT-J1 $\uparrow$	$C_{\mathrm{s}}$ $\uparrow$	GPT-J2 $\uparrow$
Baseline	94.1	0.140	0.432	0.15	0.327	52.1	0.385	0.677	0.36	0.591
$+a_{\mathrm{state}}$	85.6	0.149	0.445	0.18	0.320	51.1	0.393	0.689	0.39	0.620
$+a_{\mathrm{new}}$	89.4	0.153	0.440	0.13	0.362	50.6	0.397	0.686	0.34	0.563
$+\mathrm{Both}$	81.3	0.164	0.450	0.17	0.369	49.7	0.409	0.695	0.38	0.624

Table 4: Ablation analysis of two key constraints. MM-interleaved and SenseNova-U1 are respectively adopted as the baseline model on the Opening and ISG-Bench datasets.

Figure 5: Qualitative results when applied to robotic arm manipulation and handcrafting scenarios.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Qualitative results on the effectiveness of SCO when applied to embodiment and handcrafting fields.

Figure 7: The relationship between the new-event constraint/state-preserving constraint and how to rewrite text prompts.

Figure 8: Qualitative results on the interleaved multimodal generation for the storytelling task.

Figure 9: Qualitative results on the task of iterative image editing.

Figure 10: Qualitative results on the interleaved multimodal generation for the sequential object removal task.

Explore similar work

Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation
Jun 29, 2026Chonghuinan Wang, Zhikai Chen, Chunwei Wang +9Unified Multimodal Models Multimodal Model Evaluation
Harbin Institute of Technology, Harbin, China · Huawei Noah’s Ark Lab, Shenzhen, China · Zhengzhou Advanced Research Institute of Harbin Institute of Technology, Zhengzhou, China +1
Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generation
May 12, 2026Yabo Zhang, Kunchang Li, Dewei Zhou +2Unified Multimodal Models Compositional Image Generation
ByteDance Seed
InterleaveThinker: Reinforcing Agentic Interleaved Generation
Jun 11, 2026Dian Zheng, Harry Lee, Manyuan Zhang +4Agentic Image Generation Multimodal Generation
1CUHK MMLab · 2Meituan · 3CUHK IMIXR

Jun 29, 2026, cs.CVJ/K move · Enter open · S save

Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation

Chonghuinan Wang, Zhikai Chen, Chunwei Wang, Yecong Wan+8

Harbin Institute of Technology, Harbin, China · Huawei Noah’s Ark Lab, Shenzhen, China · Zhengzhou Advanced Research Institute of Harbin Institute of Technology, Zhengzhou, China+1

Unified Multimodal Models Multimodal Model Evaluation Multimodal Learning

Open paper AskPDF arXiv

The advancement of generative AI models capable of producing text and image marks a critical step forward in the realm of multimodal intelligence, particularly for tasks involving the interleaving of both modalities. To advance this intelligence to the next stage, it is crucial for models to autonomously generate free-form interleaved text-image sequences. In this paper, we introduce ILLUME-X, an advanced unified multimodal paradigm that enables high-quality, free-form interleaved text-image generation by improving multimodal data efficiency and stabilizing the multimodal training process. ILLUME-X comprises three key components: (i) an expanded training data pipeline optimized for interleaved text-image generation, (ii) a progressive training strategy with self-adaptive objectives for free-length multimodal token sequences, and (iii) an objective and comprehensive evaluation method ILScore for interleaved text-image sequences. Notably, our ILLUME-X outperforms previous unified models across multiple interleaved text-image generation tasks like style transfer, image decomposition and storytelling.

Abstract

Explore similar work

Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation

Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generation

InterleaveThinker: Reinforcing Agentic Interleaved Generation