Unified multimodal models (UMMs) have opened up new possibilities for generating rich interleaved text-image content. However, they generally lack native support for coordinating the two modalities, and translating coarse user requests into coherent multimodal compositions also requires explicit planning. We propose UniWeaver, a structured, model-agnostic harness framework that integrates multimodal planning, generation, and visual feedback to produce coherent interleaved outputs. To train UMMs for this task, we construct UniWeaver-10K, a dataset of 10,000 tasks across 20 task families, paired with execution trajectories guided by UniWeaver. We also introduce UniWeaver-Bench, a task-oriented benchmark systematically assessing text coverage, writing quality, visual coverage, and image-text alignment. Experiments show that UniWeaver achieves state-of-the-art performance and improves interleaved text-image generation across multiple UMM backbones.
Figures & tables
Figure 1: Paradigms of interleaved text-image generation. (a) Existing UMMs generate text and images sequentially. (b) UniWeaver structures multimodal planning, generation, and local visual verification.
Figure 2: Generated results of the proposed UniWeaver. Our harness framework enables coherent interleaved text-image generation for UMMs across diverse real-world tasks, producing task-relevant visuals paired with informative and actionable text.
Figure 3: Overview of UniWeaver. ① UniWeaver introduces a structured workflow for interleaved text-image generation, with execution trajectories providing supervision for fine-tuning (Section 3.1 ). ② UniWeaver-10K Dataset provides 10,000 goal-only tasks paired with structured planning trajectories across 20 task families (Section 3.2 ). ③ UniWeaver-Bench systematically evaluates final interleaved outputs on 120 held-out tasks (Section 3.3 ).
Figure 4: Statistics of UniWeaver-10K and UniWeaver-Bench. (a) Task distribution across 20 diverse families, with balanced coverage in both datasets. (b) Distribution of training sample lengths in tokens. (c) Vocabulary characteristics of the training corpus, visualized as a word cloud.
Model
Procedures & Instructions
Comparison & Recommendation
Observation & Description
Explanation & Storytelling
Overall
TC
WQ
VC
ITA
Total
TC
WQ
VC
ITA
Total
TC
WQ
VC
ITA
Total
TC
WQ
VC
ITA
Total
TC
WQ
VC
ITA
Total
BAGEL
14.8
7.3
48.0
14.1
25.6
28.3
14.2
35.1
14.1
27.8
26.1
4.9
45.9
15.0
29.8
16.3
6.8
52.2
13.0
27.6
21.4
8.3
45.3
14.0
27.7
OmniGen2
12.3
13.8
47.9
21.9
25.9
19.6
14.2
40.4
24.7
26.8
16.0
2.5
44.6
23.3
25.4
23.2
5.4
53.3
40.4
33.7
17.8
9.0
46.5
27.6
27.9
TokenFlow
30.2
24.6
65.4
40.3
43.0
32.1
8.8
55.8
25.8
37.4
31.3
6.7
60.4
27.2
38.6
28.1
10.4
60.3
29.4
37.7
30.4
12.6
60.5
30.7
39.2
ILLUME+
23.3
27.1
51.7
47.2
36.0
35.2
10.8
55.0
35.0
39.7
34.2
12.9
57.1
26.5
39.3
27.1
5.8
66.7
25.8
38.7
30.0
14.2
57.6
33.6
38.4
Show-o2
10.0
43.4
21.5
9.2
17.3
15.6
36.6
36.7
12.5
24.8
20.1
30.2
38.4
13.2
26.8
19.3
32.5
41.7
21.8
28.7
16.2
35.7
34.6
14.2
24.4
Table 1: Quantitative Comparison on the 120-task UniWeaver-Bench. Each major category contains 5 task families covering 30 tasks. The best result is bold and the second best underlined . Red values indicate performance gains over the corresponding UMM baseline. Only gate-passing tasks are averaged, and a criterion whose judge medians span more than one tier scores 0 . Each family’s score is multiplied by rf , where rf is its gate pass rate.
Figure 5: Qualitative comparison on two representative tasks. UniWeaver produces more coherent, actionable instructions with task-relevant visuals, while baselines exhibit repetitive content, missing details, or ineffective visual support. Zoom in for better view.
Ablation
Total
Δ
UniWeaver (w/ BAGEL backbone)
58.89
—
P1 : Task Scope
w/o Task Goal
53.01
− 5.88
w/o Per-step Content Goal
55.56
− 3.33
P2 : Text and Image Planning
w/o Shared Visual Style
48.21
− 10.68
Table 2: Ablation studies of UniWeaver. All variants are re-run on the same 120 tasks with the same evaluation setup. Note that Δ denotes the performance gap measured against the full method.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Input
Output
P1
User request
Title, task endpoint, and unit-level visual subjects and content goals
P2
Request and P1
Shared visual style, reader-facing prose, and image prompts
P3
Request and planned image prompts
Reference bank (0–2 images) and unit-specific reference assignments
P4
Planned image prompts
Observable visual checks for each unit
Review
Generated image and checklist
Per-criterion PASS / FAIL with feedback
Appendix
Table 3: UniWeaver organizes multimodal generation through four structured planning stages and visual verification. The table summarizes the inputs and outputs of each stage and the image reviewer.
Dimension
# Criteria
Evaluation Evidence
TC
4–5
User request, hidden coverage requirements, and final text
WQ
2
User request and final text
VC
2–3
User request, hidden visual requirements, and generated images
ITA
one per unit
User request, final text, and corresponding images
Appendix
Table 4: UniWeaver-Bench evaluates final outputs using a task-specific rubric across four dimensions. The table specifies the number of criteria and evaluation evidence for each dimension.
Figure 6: Additional qualitative comparison on the architectural design task. UniWeaver produces coherent, task-specific descriptions and corresponding visuals covering the entrance, rehearsal room, rooftop garden, and their integration, while baseline UMMs exhibit repetitive content or incomplete task coverage.
Setting
Value
LoRA (rank / alpha / dropout)
32 / 64 / 0.05
Optimizer
AdamW, β=(0.9,0.95) , ϵ=10−8 , weight decay 0.01
Learning rate
5×10−5
Schedule
200 -step warm-up, cosine decay
Training steps
40,000
Gradient clipping / seed
1.0 / 42
Appendix
Table 5: UniWeaver-SFT adapts BAGEL’s understanding decoder using LoRA. The table summarizes the key training and optimization settings.
Ablation
Modification
P1 : Task Scope
w/o Task Goal
Removes the endpoint field from P1
w/o Per-step Content Goal
Removes unit-level content goals while retaining visual subjects
P2 : Text and Image Planning
w/o Shared Visual Style
Removes the shared visual style from P2
w/ Merged Image Prompt and Content
Uses one field for both reader-facing prose and image-generation instructions
Appendix
Table 6: Seven ablation variants isolate individual components of UniWeaver’s planning workflow. The table specifies the modifications to P1 , P2 , and P3 .
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.
Harbin Institute of Technology, Harbin, China · Huawei Noah’s Ark Lab, Shenzhen, China · Zhengzhou Advanced Research Institute of Harbin Institute of Technology, Zhengzhou, China +1