Unified multimodal models (UMMs) have opened up new possibilities for generating rich interleaved text-image content. However, they generally lack native support for coordinating the two modalities, and translating coarse user requests into coherent multimodal compositions also requires explicit planning. We propose UniWeaver, a structured, model-agnostic harness framework that integrates multimodal planning, generation, and visual feedback to produce coherent interleaved outputs. To train UMMs for this task, we construct UniWeaver-10K, a dataset of 10,000 tasks across 20 task families, paired with execution trajectories guided by UniWeaver. We also introduce UniWeaver-Bench, a task-oriented benchmark systematically assessing text coverage, writing quality, visual coverage, and image-text alignment. Experiments show that UniWeaver achieves state-of-the-art performance and improves interleaved text-image generation across multiple UMM backbones.
Figures & tables
Figure 1: Paradigms of interleaved text-image generation. (a) Existing UMMs generate text and images sequentially. (b) UniWeaver structures multimodal planning, generation, and local visual verification.
Figure 2: Generated results of the proposed UniWeaver. Our harness framework enables coherent interleaved text-image generation for UMMs across diverse real-world tasks, producing task-relevant visuals paired with informative and actionable text.
Figure 3: Overview of UniWeaver. ① UniWeaver introduces a structured workflow for interleaved text-image generation, with execution trajectories providing supervision for fine-tuning (Section 3.1 ). ② UniWeaver-10K Dataset provides 10,000 goal-only tasks paired with structured planning trajectories across 20 task families (Section 3.2 ). ③ UniWeaver-Bench systematically evaluates final interleaved outputs on 120 held-out tasks (Section 3.3 ).
Figure 4: Statistics of UniWeaver-10K and UniWeaver-Bench. (a) Task distribution across 20 diverse families, with balanced coverage in both datasets. (b) Distribution of training sample lengths in tokens. (c) Vocabulary characteristics of the training corpus, visualized as a word cloud.
Model
Procedures & Instructions
Comparison & Recommendation
Observation & Description
Explanation & Storytelling
Overall
TC
WQ
VC
ITA
Total
TC
WQ
VC
ITA
Total
TC
WQ
VC
ITA
Total
TC
WQ
VC
ITA
Total
TC
WQ
VC
ITA
Total
BAGEL
14.8
7.3
48.0
14.1
25.6
28.3
14.2
35.1
14.1
27.8
26.1
4.9
45.9
15.0
29.8
16.3
6.8
52.2
13.0
27.6
21.4
8.3
45.3
14.0
27.7
OmniGen2
12.3
13.8
47.9
21.9
25.9
19.6
14.2
40.4
24.7
26.8
16.0
2.5
44.6
23.3
25.4
23.2
5.4
53.3
40.4
33.7
17.8
9.0
46.5
27.6
27.9
TokenFlow
30.2
24.6
65.4
40.3
43.0
32.1
8.8
55.8
25.8
37.4
31.3
6.7
60.4
27.2
38.6
28.1
10.4
60.3
29.4
37.7
30.4
12.6
60.5
30.7
39.2
ILLUME+
23.3
27.1
51.7
47.2
36.0
35.2
10.8
55.0
35.0
39.7
34.2
12.9
57.1
26.5
39.3
27.1
5.8
66.7
25.8
38.7
30.0
14.2
57.6
33.6
38.4
Show-o2
10.0
43.4
21.5
9.2
17.3
15.6
36.6
36.7
12.5
24.8
20.1
30.2
38.4
13.2
26.8
19.3
32.5
41.7
21.8
28.7
16.2
35.7
34.6
14.2
24.4
Table 1: Quantitative Comparison on the 120-task UniWeaver-Bench. Each major category contains 5 task families covering 30 tasks. The best result is bold and the second best underlined . Red values indicate performance gains over the corresponding UMM baseline. Only gate-passing tasks are averaged, and a criterion whose judge medians span more than one tier scores 0 . Each family’s score is multiplied by rf , where rf is its gate pass rate.
Figure 5: Qualitative comparison on two representative tasks. UniWeaver produces more coherent, actionable instructions with task-relevant visuals, while baselines exhibit repetitive content, missing details, or ineffective visual support. Zoom in for better view.
Ablation
Total
Δ
UniWeaver (w/ BAGEL backbone)
58.89
—
P1 : Task Scope
w/o Task Goal
53.01
− 5.88
w/o Per-step Content Goal
55.56
− 3.33
P2 : Text and Image Planning
w/o Shared Visual Style
48.21
− 10.68
Table 2: Ablation studies of UniWeaver. All variants are re-run on the same 120 tasks with the same evaluation setup. Note that Δ denotes the performance gap measured against the full method.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Input
Output
P1
User request
Title, task endpoint, and unit-level visual subjects and content goals
P2
Request and P1
Shared visual style, reader-facing prose, and image prompts
P3
Request and planned image prompts
Reference bank (0–2 images) and unit-specific reference assignments
P4
Planned image prompts
Observable visual checks for each unit
Review
Generated image and checklist
Per-criterion PASS / FAIL with feedback
Appendix
Table 3: UniWeaver organizes multimodal generation through four structured planning stages and visual verification. The table summarizes the inputs and outputs of each stage and the image reviewer.
Dimension
# Criteria
Evaluation Evidence
TC
4–5
User request, hidden coverage requirements, and final text
WQ
2
User request and final text
VC
2–3
User request, hidden visual requirements, and generated images
ITA
one per unit
User request, final text, and corresponding images
Appendix
Table 4: UniWeaver-Bench evaluates final outputs using a task-specific rubric across four dimensions. The table specifies the number of criteria and evaluation evidence for each dimension.
Figure 6: Additional qualitative comparison on the architectural design task. UniWeaver produces coherent, task-specific descriptions and corresponding visuals covering the entrance, rehearsal room, rooftop garden, and their integration, while baseline UMMs exhibit repetitive content or incomplete task coverage.
Setting
Value
LoRA (rank / alpha / dropout)
32 / 64 / 0.05
Optimizer
AdamW, β=(0.9,0.95) , ϵ=10−8 , weight decay 0.01
Learning rate
5×10−5
Schedule
200 -step warm-up, cosine decay
Training steps
40,000
Gradient clipping / seed
1.0 / 42
Appendix
Table 5: UniWeaver-SFT adapts BAGEL’s understanding decoder using LoRA. The table summarizes the key training and optimization settings.
Ablation
Modification
P1 : Task Scope
w/o Task Goal
Removes the endpoint field from P1
w/o Per-step Content Goal
Removes unit-level content goals while retaining visual subjects
P2 : Text and Image Planning
w/o Shared Visual Style
Removes the shared visual style from P2
w/ Merged Image Prompt and Content
Uses one field for both reader-facing prose and image-generation instructions
Appendix
Table 6: Seven ablation variants isolate individual components of UniWeaver’s planning workflow. The table specifies the modifications to P1 , P2 , and P3 .
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.
Recent unified models have made unprecedented progress in both understanding and generation. However, while most of them accept multi-modal inputs, they typically produce only single-modality outputs. This challenge of producing interleaved content is mainly due to training data scarcity and the difficulty of modeling long-range cross-modal context. To address this issue, we decompose interleaved generation into textual planning and visual consistency modeling, and introduce a framework consisting of a planner and a visualizer. The planner produces dense textual descriptions for visual content, while the visualizer synthesizes images accordingly. Under this guidance, we construct large-scale textual-proxy interleaved data (where visual content is represented in text) to train the planner, and curate reference-guided image data to train the visualizer. These designs give rise to Wan-Weaver, which exhibits emergent interleaved generation ability with long-range textual coherence and visual consistency. Meanwhile, the integration of diverse understanding and generation data into planner training enables Wan-Weaver to achieve robust task reasoning and generation proficiency. To assess the model's capability in interleaved generation, we further construct a benchmark that spans a wide range of use cases across multiple dimensions. Extensive experiments demonstrate that, even without access to any real interleaved data, Wan-Weaver achieves superior performance over existing methods.
The advancement of generative AI models capable of producing text and image marks a critical step forward in the realm of multimodal intelligence, particularly for tasks involving the interleaving of both modalities. To advance this intelligence to the next stage, it is crucial for models to autonomously generate free-form interleaved text-image sequences. In this paper, we introduce ILLUME-X, an advanced unified multimodal paradigm that enables high-quality, free-form interleaved text-image generation by improving multimodal data efficiency and stabilizing the multimodal training process. ILLUME-X comprises three key components: (i) an expanded training data pipeline optimized for interleaved text-image generation, (ii) a progressive training strategy with self-adaptive objectives for free-length multimodal token sequences, and (iii) an objective and comprehensive evaluation method ILScore for interleaved text-image sequences. Notably, our ILLUME-X outperforms previous unified models across multiple interleaved text-image generation tasks like style transfer, image decomposition and storytelling.
Chonghuinan Wang, Zhikai Chen, Chunwei Wang +9
Harbin Institute of Technology, Harbin, China · Huawei Noah’s Ark Lab, Shenzhen, China · Zhengzhou Advanced Research Institute of Harbin Institute of Technology, Zhengzhou, China +1
Humans routinely consume multimodal content, where textual and visual information is naturally interleaved. Although Unified Multimodal Models (UMMs) can jointly generate text and images, multimodal interleaved generation remains largely underexplored. Progress in this area is hindered by two key limitations: (1) the lack of well-defined interleaved generation tasks and corresponding training data, and (2) the absence of fine-grained evaluation protocols that can reliably diagnose model performance across different capability dimensions. To this end, we present IntReal, comprising two complementary components: IntReal-100K, a real-world dataset of multimodal interleaved samples spanning four major categories, and the IntReal Evaluation Suite, which evaluates model performance across seven complementary dimensions using specialized expert models, enabling coarse-to-fine diagnosis. Extensive experiments validate our evaluation suite through low inter-metric redundancy (max pairwise absolute Spearman's ρ≤0.42 ), high agreement with human judgments (83.3% accuracy), and strong consistency across different expert evaluators (Spearman's ρ>0.9 ), supporting the complementarity and reliability of our metrics. By benchmarking four families of UMMs, we show that IntReal Evaluation Suite reveals distinct strengths and weaknesses across models. Post-training experiments demonstrate the utility of IntReal-100K for improving interleaved generation and further show that training unimodal generation can also benefit interleaved generation. These findings suggest that interleaved generation can benefit from modality-specific optimization, rather than requiring joint text-image training at every stage.
Shaohan Yu, Jie Qin, Wende Tan +4
PRLab, Nanjing University · Meituan · Imperial College London +1