Humans routinely consume multimodal content, where textual and visual information is naturally interleaved. Although Unified Multimodal Models (UMMs) can jointly generate text and images, multimodal interleaved generation remains largely underexplored. Progress in this area is hindered by two key limitations: (1) the lack of well-defined interleaved generation tasks and corresponding training data, and (2) the absence of fine-grained evaluation protocols that can reliably diagnose model performance across different capability dimensions. To this end, we present IntReal, comprising two complementary components: IntReal-100K, a real-world dataset of multimodal interleaved samples spanning four major categories, and the IntReal Evaluation Suite, which evaluates model performance across seven complementary dimensions using specialized expert models, enabling coarse-to-fine diagnosis. Extensive experiments validate our evaluation suite through low inter-metric redundancy (max pairwise absolute Spearman's ρ≤0.42 ), high agreement with human judgments (83.3% accuracy), and strong consistency across different expert evaluators (Spearman's ρ>0.9 ), supporting the complementarity and reliability of our metrics. By benchmarking four families of UMMs, we show that IntReal Evaluation Suite reveals distinct strengths and weaknesses across models. Post-training experiments demonstrate the utility of IntReal-100K for improving interleaved generation and further show that training unimodal generation can also benefit interleaved generation. These findings suggest that interleaved generation can benefit from modality-specific optimization, rather than requiring joint text-image training at every stage.
Figures & tables
Figure 1: Overview of IntReal. IntReal-100K contains 100,000 multimodal interleaved samples with 538,452 images, curated from 7 real-world sources and organized into 4 categories. The IntReal Evaluation Suite performs coarse-to-fine diagnosis via 7 fine-grained dimensions.
Figure 2: Overview of the IntReal Evaluation Suite. We select 600 high-quality samples from IntReal-100K to construct this suite. Each sample is evaluated at two granularities: holistic evaluation generates the complete interleaved sequence at once, while fine-grained evaluation generates the next image conditioned on the preceding interleaved context to enable controllable diagnosis.
Model
Holistic Evaluation
Fine-grained Evaluation
Mean
Text Richness
Text Logicality
Image Aesthetic
Image Diversity
Quantity Compliance
Context Fidelity
Subject Consistency
Emu3.5
14.02
75.17
81.20
75.51
39.70
65.60
61.46
58.95
U1-A3B-MoT
24.67
69.17
80.70
88.70
72.50
86.81
63.45
69.43
Bagel +SFT
17.50
63.50
76.82
91.35
62.83
74.19
60.61
63.83
Bagel +SFT+RL
16.67
62.50
81.94
90.31
62.50
76.04
65.43
65.06
U1-8B-MoT
21.83
67.33
79.60
88.36
95.33
87.29
63.38
71.88
Table 3: IntReal evaluation result per dimension . We evaluate four UMM families across seven dimensions. All scores are scaled to the range of [0,100] , with higher values indicating better performance. The Δ vs. base rows report relative changes after post-training with respect to the corresponding base model, where green and red indicate improvements and degradations, respectively. Gray rows denote the closed-source API system, and Δ vs. OSS Best reports its relative difference with respect to the best open-source result in each dimension.
Figure 4Figure 5
Figure 7: Case study on SenseNova-U1-8B-MoT. SFT produces more detailed text but degrades visual details (distorted brush), while RL preserves the text improvements and restores image quality.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Source
# Samples
# Images
Images / Sample
# Text Tokens
Text Tokens / Sample
Planning
Wikivoyage
21,055
56,527
2.68
21.5M
1,021.13
Wikitravel
5,055
12,254
2.42
5.2M
1,020.53
Knowledge
Wikipedia
21,138
100,198
4.74
32.2M
1,522.34
Tutorial
Xiachufang
34,771
213,818
6.15
52.3M
1,504.77
Story
Curious George
7,298
63,088
8.64
14.2M
1,941.43
Rabbids Invasion
3,949
34,284
8.68
7.8M
1,965.36
Appendix
Table 4: Statistics of IntReal-100K across different data sources. We report the number of samples, number of images, average number of images per sample, total number of text tokens, and average number of text tokens per sample. Text tokens are computed using the Qwen2.5 tokenizer.
Figure 8: Calibration case for Text Richness. A low-scoring sample from Bagel+SFT , where the response fails to provide sufficiently specific insect characteristics and offers only limited detail when distinguishing the target insect from similar species, leading to a low Text Richness score.
Figure 9: Calibration case for Text Logicality. A low-scoring sample from U1-8B-MoT , where spatial and factual inconsistencies in the description lead to a low Text Logicality score.
Figure 10: Calibration case for Image Aesthetic. A low-scoring sample from U1-8B-MoT-SFT , where image quality degrades over steps, with later images exhibiting excessively strong saturation and contrast and appearing less photorealistic, leading to a low Image Aesthetic score.
Figure 11: Calibration case for Image Diversity. A low-scoring sample from Emu3.5 , where Images 1&2, 3&4, and 5&6 are excessively similar to each other, resulting in limited visual variation and a low Image Diversity score.
Figure 12: Calibration case for Quantity Compliance. A low-scoring sample from Bagel+SFT+RL , where the prompt requests eight images but the response generates only six, leading to a low Quantity Compliance score.
Figure 13: Calibration case for Context Fidelity. A low-scoring sample from U1-8B-MoT+SFT+RL , where the generated images fail to faithfully follow the described cooking process: the bread is insufficiently toasted in the first image, while the second image omits the preparation and spreading of the caramelized onions, leading to a low Context Fidelity score.
Figure 14: Calibration case for Subject Consistency. A low-scoring sample from U1-A3B-MoT , where the main subjects are not consistently preserved across images. For example, the green dinosaur differs across Images 1, 4, and 5, while the teddy bear changes color between Images 3 and 6. Since the reference sequence maintains a stable subject appearance throughout, such internal inconsistency in the generated images necessarily manifests as low similarity to the reference at the corresponding positions, leading to a low Subject Consistency score.
Figure 15
Hyperparameter
Bagel
SenseNova-U1
Initialization
BAGEL-7B-MoT
SenseNova-U1-8B-MoT-Interleaved
Training steps
6,700
7,000
Optimizer
AdamW
AdamW
Learning rate
2×10−5
2×10−5
LR schedule
Cosine
Cosine
Warmup steps
670
700
Appendix
Table 5: SFT configurations for Bagel and SenseNova-U1. We report the main optimization, batching, loss, and distributed training hyperparameters used in our post-training experiments.
Hyperparameter
Bagel
SenseNova-U1
Training resolution
1024
1024
Rollout denoising steps
10
20
Stochastic scheduler
Flow-CPS
Flow-SDE
Noise level
0.7
0.4
Candidate stochastic step indices
{1,2,3}
{1,2,3}
Selected stochastic steps / rollout
1
1
Appendix
Table 6: RL configurations for Bagel and SenseNova-U1. We report the main rollout, stochastic sampling, reward, and evaluation configurations used in our reinforcement learning experiments.
Model
Holistic Evaluation
Fine-grained Evaluation
Mean
Text Richness
Text Logicality
Image Aesthetic
Image Diversity
Quantity Compliance
Context Fidelity
Subject Consistency
Emu3.5
13.34
69.93
81.20
75.51
39.70
71.06
61.46
58.89
U1-A3B-MoT
29.50
69.00
80.70
88.70
72.50
89.11
63.45
70.42
Bagel +SFT
19.67
54.33
76.82
91.35
62.83
78.71
60.61
63.48
Bagel +SFT+RL
20.67
50.33
81.94
90.31
62.50
80.76
65.43
64.56
U1-8B-MoT
25.67
58.83
79.60
88.36
95.33
89.88
63.38
71.58
Appendix
Table 7: IntReal evaluation results with an alternative evaluator. We report results following the protocol in table 3 , using Qwen3.6-35B-A3B in place of Gemma4 as the evaluator.
Figure 17: IntReal evaluation results per category for individual models. We report the score of each metric for each category separately for each model.
Figure 18: Qualitative example from the Planning category generated by Nanobanana2. The response interleaves text and images to introduce the geography, history, and landmarks of Mayfair and Marylebone.
Figure 19: Qualitative example from the Knowledge category generated by Emu3.5. The response interleaves text and images to describe the appearance, reproductive behavior, and development of the copper demoiselle damselfly.
Figure 20: Qualitative example from the Tutorial category generated by U1-8B-MoT+SFT+RL. The response interleaves text and images to explain how to roast cherry tomatoes with garlic and rosemary and serve them with bread.
Figure 21: Qualitative example from the Story category generated by Bagel+SFT+RL. The response interleaves text and images to narrate a group of dinosaurs exploring and playing together in a lush jungle.
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.
In real-world multimodal applications, systems usually need to comprehend arbitrarily combined and interleaved multimodal inputs from users, while also generating outputs in any interleaved multimedia form. This capability defines the goal of any-to-any interleaved multimodal learning under a unified paradigm of understanding and generation, posing new challenges and opportunities for advancing Multimodal Large Language Models (MLLMs). To foster and benchmark this capability, this paper introduces the UniM benchmark, the first Unified Any-to-Any Interleaved Multimodal dataset. UniM contains 31K high-quality instances across 30 domains and 7 representative modalities: text, image, audio, video, document, code, and 3D, each requiring multiple intertwined reasoning and generation capabilities. We further introduce the UniM Evaluation Suite, which assesses models along three dimensions: Semantic Correctness & Generation Quality, Response Structure Integrity, and Interleaved Coherence. In addition, we propose UniMA, an agentic baseline model equipped with traceable reasoning for structured interleaved generation. Comprehensive experiments demonstrate the difficulty of UniM and highlight key challenges and directions for advancing unified any-to-any multimodal intelligence. The project page is https://any2any-mllm.github.io/unim.
Recent unified models have made unprecedented progress in both understanding and generation. However, while most of them accept multi-modal inputs, they typically produce only single-modality outputs. This challenge of producing interleaved content is mainly due to training data scarcity and the difficulty of modeling long-range cross-modal context. To address this issue, we decompose interleaved generation into textual planning and visual consistency modeling, and introduce a framework consisting of a planner and a visualizer. The planner produces dense textual descriptions for visual content, while the visualizer synthesizes images accordingly. Under this guidance, we construct large-scale textual-proxy interleaved data (where visual content is represented in text) to train the planner, and curate reference-guided image data to train the visualizer. These designs give rise to Wan-Weaver, which exhibits emergent interleaved generation ability with long-range textual coherence and visual consistency. Meanwhile, the integration of diverse understanding and generation data into planner training enables Wan-Weaver to achieve robust task reasoning and generation proficiency. To assess the model's capability in interleaved generation, we further construct a benchmark that spans a wide range of use cases across multiple dimensions. Extensive experiments demonstrate that, even without access to any real interleaved data, Wan-Weaver achieves superior performance over existing methods.
Automatically evaluating multimodal generation presents a significant challenge, as automated metrics often struggle to align with human evaluation, especially for complex tasks that involve multiple modalities. We present MMMG, the first benchmark to bring the verifiable-task paradigm to multimodal generation, spanning 4 modality combinations (image, audio, interleaved text and image, interleaved text and audio). As few multimodal outputs can be checked by programs alone, MMMG targets tasks that are either verifiable or near-verifiable: by providing references, and constraining model judges with explicit rubrics. We keep tasks challenging for generation models while enabling reliable automatic evaluation through a combination of models and programs. MMMG encompasses 55 tasks (including 31 newly developed ones), each with a carefully designed evaluation pipeline, and 1288 instructions to systematically assess reasoning, controllability, and other key capabilities of multimodal generation models. Extensive validation demonstrates that MMMG is highly aligned with human judgment, achieving an average agreement of 94.4%. Benchmarking results on 29 models reveal that even though the state-of-the-art model, GPT Image, achieves 70.7% accuracy for image generation, it falls short on interleaved generation. Furthermore, results suggest considerable improvement space in audio generation, highlighting an important future direction.