In real-world multimodal applications, systems usually need to comprehend arbitrarily combined and interleaved multimodal inputs from users, while also generating outputs in any interleaved multimedia form. This capability defines the goal of any-to-any interleaved multimodal learning under a unified paradigm of understanding and generation, posing new challenges and opportunities for advancing Multimodal Large Language Models (MLLMs). To foster and benchmark this capability, this paper introduces the UniM benchmark, the first Unified Any-to-Any Interleaved Multimodal dataset. UniM contains 31K high-quality instances across 30 domains and 7 representative modalities: text, image, audio, video, document, code, and 3D, each requiring multiple intertwined reasoning and generation capabilities. We further introduce the UniM Evaluation Suite, which assesses models along three dimensions: Semantic Correctness & Generation Quality, Response Structure Integrity, and Interleaved Coherence. In addition, we propose UniMA, an agentic baseline model equipped with traceable reasoning for structured interleaved generation. Comprehensive experiments demonstrate the difficulty of UniM and highlight key challenges and directions for advancing unified any-to-any multimodal intelligence. The project page is https://any2any-mllm.github.io/unim.
Figures & tables
Figure 1: Illustration of the any-to-any interleaved multimodal paradigm with different real-world application scenarios. Solving any-to-any interleaved multimodal learning requires complex and combined capabilities.
Benchmarks
Domains
Num.
Inter. Comb.
Cap. per Instance
Eval. Metric
Difficulty Tax.
Any-to
to-Any
Modalities
ITLVD-BENCH [ 34 ]
10
815
2
Single
5
✗
✗
✗
OpenING [ 79 ]
8
5,400
4
Single
7
✗
✗
✗
ISG-Bench [ 6 ]
8
1,150
3
Single
4
✗
✗
✗
CoMM [ 8 ]
3
/
4
Single
3
✗
✗
✗
MMIE [ 67 ]
10
20,103
3
Single
7
✗
✗
✗
UniM (Ours)
30
31,026
41
Multiple
13
✓
✓
✓
Table 1: Comparison with existing interleaved multimodal benchmarks. Inter. Comb.: Interleaved combinations of modalities. Cap. per Instance: Capability per instance. Difficulty Tax.: Difficulty taxonomy.
Modality
Image
Audio
Video
Document
Code
3D
Num.
22,936 (73.9%)
24,963 (80.5%)
2,336 (7.5%)
3,858 (12.4%)
807 (2.6%)
420 (1.4%)
Field
Difficulty
NS
SS
GA
Easy
Medium
Hard
Num.
10,624 (34.2%)
11,574 (37.3%)
8,828 (28.5%)
10,678 (34.4%)
14,088 (45.4%)
6,260 (20.2%)
Table 2: General statistics of UniM dataset.
Figure 2: Distribution of different difficulty levels.
Figure 3: Illustration of the UniM evaluation suite. ① refers to the calculation process of the StS and LeS (§ 4.2 ). ② represents the calculation process of the ICS in Eq. ( 2 ) . ③ refers to the calculation process of the SQCS; please refer to Eq. ( 1 ) .
Figure 4: Overview of the UniMA architecture.
Figure 5: Results for rationality verification of SQCS and ICS.
Figure 6: Results for rationality verification of StS and LeS.
Model
Natural Science
Social Science
General Area
StS abs
LeS abs
StS rel
LeS rel
StS abs
LeS abs
StS rel
LeS rel
StS abs
LeS abs
StS rel
LeS rel
AnyGPT [ 76 ]
12.9
27.8
12.2
21.4
14.9
16.6
14.5
16.2
12.5
16.4
9.8
13.6
NExT-GPT [ 64 ]
2.0
2.2
1.2
1.3
1.3
1.7
1.2
1.5
2.2
2.5
1.4
1.7
MIO [ 59 ]
1.3
1.9
0.9
1.3
4.1
5.2
4.0
5.1
3.3
3.8
2.4
2.9
UniMA
50.8
62.2
50.8
62.2
58.1
72.9
58.1
72.9
71.3
84.3
71.3
84.3
Table 3: Assessment results on Response Structure Integrity.
Table 4: Performance on Semantic Correctness & Generation Quality and Supporting Rate ( τ ).
Field
Models
HC
SH
ICS abs
ICS rel
Natural Science
AnyGPT [ 76 ]
39.9
46.3
41.8
38.5
NExT-GPT [ 64 ]
23.5
26.1
24.9
16.3
MIO [ 59 ]
49.4
63.7
52.1
31.8
UniMA
68.4
71.9
69.1
69.1
Social Science
AnyGPT [ 76 ]
31.3
35.3
32.1
29.2
NExT-GPT [ 64 ]
24.5
27.1
21.4
19.0
Table 5: Evaluation results on Interleaved Coherence.
Figure 7: Results across 10 capabilities on UniM . C1: Perceptual Understanding, C2: Spatial Understanding, C3: Temporal Understanding, C4: Semantic Generation, C5: Content Editing, C6: Creative Expression, C7: Reasoning Capability, C8: Emotional Analysis, C9: Structural Analysis, and C10: Planning Capability. Refer to Appendix § C.4 for details.
Figure 8: Comparison of model responses in the cross-modal painting instruction task. NExT-GPT [ 64 ] outputs only generic text without image generation, lacking procedural structure. MIO [ 59 ] produces a single sketch image but omits key intermediate steps. UniMA generates a coherent step-by-step text–image workflow aligned with the video, covering the entire painting process.
Figure 9: Performance comparison of MLLMs under different difficulty levels across evaluation metrics. Each curve illustrates the performance trend of a model as task complexity increases, where higher values indicate better performance.
Video captioning; Video localization; Visual question answering; Video editing; Text-to-video generation; Audio-to-video generation; Viewpoint-transformation video synthesis; …
Table 12: The number of multimodal files in UniM .
Natural Science
Social Science
General Area
ID
Name
Easy
Medium
Hard
ID
Name
Easy
Medium
Hard
ID
Name
Easy
Medium
Hard
#1
Math
806
777
48
#11
Finance
739
1694
135
#21
Sports
57
313
486
#2
Physics
40
591
374
#12
Law
256
634
121
#22
News
200
456
350
#3
Chemistry
312
617
113
#13
Linguistics
570
207
53
#23
Culture
128
170
100
#4
Biology
545
333
263
#14
Psychology
398
300
397
#24
Entertainment
301
188
299
#5
Compute Science
72
868
218
#15
Education
165
463
72
#25
Food
256
624
296
Appendix
Table 13: Detailed statistics of UniM .
Modality
Caption Tool
Text
Original format retained
Image
GPT-5 mini
Video
Qwen3-Omni
Audio
Qwen3-Omni
Document
GPT-5 mini
Code
Original format retained
Appendix
Table 14: Caption methods for each modality.
Grade
Semantic Correctness Description
5
a) Completely equivalent; b) All key facts correct/covered; c) No contradictions; d) Units, ranges, and relational constraints remain consistent (paraphrasing, reordering, and minor rounding are allowable).
4
a) Almost equivalent; b) Most of key facts correct/covered; c) No major contradiction; d) Only minor omissions/ambiguity that do not affect the main conclusion.
3
a) Partially correct; b) Roughly half key facts correct; c) No major contradiction but noticeable; d) Notable omissions or minor misinterpretations, but the main conclusion is not fully overturned.
2
a) Low correctness; b) Less than half covered; c) Important errors/contradictions/confusions (numbers/entities); d) The core conclusion drifts, but still loosely on topic.
1
a) Almost incorrect; b) Nearly irrelevant; c) Mostly contradictory, or hallucinated; d) Mostly wrong/missing, or non-answers.
Appendix
Table 15: Five-level rating criteria for Semantic Correctness .
Grade
Text Quality Description
5
a) Content is rich, self-consistent, and detailed; no major omissions or vague generalities; stands alone as a coherent text; b) Structure is clear and well-organized; transitions are smooth (e.g., “firstly…then…therefore…”); reasoning shows causal or hierarchical logic; c) Language is natural and grammatically flawless; diverse sentence structures; no syntactic errors; d) Entirely in one language; any foreign words appear only as necessary terminology.
4
a) Content is generally complete with sufficient detail but slightly shallow or missing minor points; b) Structure is good, with only mild jumps or awkward transitions that don’t affect comprehension; c) Language is fluent with few minor grammatical or collocation issues; d) Mostly consistent language, with rare short foreign terms that do not disrupt flow.
3
a) Content covers the main idea but lacks depth or specific details; b) Organization somewhat weak; order or topic shifts slightly; meaning still clear overall; c) Several grammatical or spelling mistakes; simple or repetitive sentence patterns; d) Minor language switching between sentences or paragraphs, noticeable but not confusing.
2
a) Content is shallow, missing key details or explanations; very low information density; b) Poor structure; sentences disjointed; reader must infer connections; c) Frequent grammar errors; awkward or broken phrasing; readability low; d) Frequent in-sentence language mixing that affects readability.
1
a) Content is empty or meaningless; repetitive or irrelevant phrases; conveys no clear information; b) No logical order; severe contradictions; text barely comprehensible; c) Major grammatical breakdowns; unnatural or non-human syntax; d) Chaotic multilingual mixing (e.g., Chinese + English + Spanish, random spelling noise)
Appendix
Table 16: Five-level rating criteria for text in Generation Quality .
Grade
Code Quality Description
5
a) High-quality, professional-grade code with virtually no noticeable defects; b) Logically rigorous, structurally well-organized, and highly robust; c) Fully adheres to modern software engineering best practices, easy to maintain, and directly reusable.
4
a) High overall quality with a reasonable structure and clear logic; b) Contains only minor issues that do not affect usability; c) Aligns with most software engineering best practices and is easy to maintain.
3
a) Basically usable, with acceptable overall correctness but clear deficiencies; b) Contains multiple areas for improvement that may affect maintainability or reliability; c) The overall quality is functional but remains below desirable standards.
2
a) Contains multiple evident issues, with overall quality falling below acceptable standards; b) Barely runnable or only partially functional, requiring substantial repairs; c) Exhibits major deficiencies in logic, structure, performance, or security.
1
a) Contains numerous critical defects, resulting in extremely low overall quality; b) Exhibits chaotic logic, unclear structure, and significant risks; c) Largely non-reusable and unmaintainable, requiring a complete rewrite.
Appendix
Table 17: Five-level rating criteria for code in Generation Quality .
Grade
Document Quality Description
5
a) All titles, labels, and column names are clear and unambiguous; b) The table structure is complete, with well-organized logical partitions and clear hierarchy; c) Units, naming conventions, capitalization, and punctuation are fully consistent; d) Numerical presentation is standardized, including uniform decimal places, units, and precision, with proper alignment; e) The surrounding context provides sufficient information for the table to be understood independently without additional explanation.
4
a) Most titles and column names are accurate, with only minor ambiguities; b) The table structure is generally reasonable, though some misalignment or slightly unclear grouping is present; c) Overall consistency is good, with only small variations in decimal places or units; d) Data presentation is mostly standardized, with mild irregularities that do not hinder usability; e) The table can largely be understood independently, though some contextual information is slightly lacking.
3
a) Contains noticeably ambiguous titles or labels whose meanings require contextual inference; b) Table structure is relatively disorganized, with unclear grouping, poorly defined relationships between columns; c) Multiple inconsistencies in formatting, naming, or units, reducing overall usability; d) Numerical presentation is irregular, with large variations in decimal places and missing or mixed units; e) Lacks sufficient contextual explanation, making the table difficult to interpret independently.
2
a) Titles or column names are missing or severely ambiguous, with clear potential for misinterpretation; b) The table structure is incomplete, exhibiting column misalignment, unclear row logic, or missing grouping; c) Formatting is highly inconsistent, with disordered or entirely missing units; d) Numerical presentation shows prominent issues, including irregular precision, inconsistent units, and potentially misleading values; e) The table is largely uninterpretable on its own and requires substantial supplementary explanation.
1
a) Titles or column names are extensively incorrect, missing, or unrecognizable; b) The table structure is entirely broken, with row–column relationships indistinguishable; c) No formatting consistency is preserved, with content unordered and misaligned; d) Numerical values are meaningless due to severe unit confusion, row mismatches, or erroneous combinations; e) The table is completely unusable on its own and requires re-OCR or full reconstruction from the original source.
Appendix
Table 18: Five-level rating criteria for document in Generation Quality .
Grade
Holistic Coherence Description
5
a) Output highly matches the input, multimodal references accurate and specific; b) Different modality information complements each other without noticeable contradictions; c) Logic is rigorous, structure is reasonable, overall semantic/logical order is clear; d) Interleaved multimodal tags naturally ordered, reading/understanding experience smooth.
4
a) Output mostly matches the input, references generally reasonable; b) Different modality information mostly complementary, minor omissions or redundancy; c) Overall logic coherent, local repetitions or jumps minor, not affecting understanding; d) Interleaved tags mostly ordered, minor adjustments do not affect understanding.
3
a) Output has general relation to input, references are vague or partially unclear; b) Some modality blocks repeated or missing, minor contradictions exist; c) Local logic jumps or slight contradictions, requires extra reasoning; d) Interleaved tags partially disordered, local understanding may be difficult.
2
a) Output lowly matches the input, most references vague or incorrect; b) Cross-modal information repetitive or conflicting; c) Logic obviously chaotic, many contradictions; d) Interleaved tags order clearly disordered, understanding difficult.
1
a) Output is almost irrelevant to input, references missing or wrong; b) Different modalities barely complement, major contradictions; c) Logic completely collapsed, frequent contradictions; d) Interleaved tags order completely disordered, hard to understand.
Appendix
Table 19: Five-level rating criteria for Holistic Coherence .
Grade
Stylistic Harmony Description
5
a) Style highly consistent, cross-modal narration uniform, tone and sentence structures without deviation; b) Terminology fully consistent, key concepts, tags, and naming completely aligned; c) Expression and visual style fully aligned, wording, sentence structures, and rhetorical/visual style coordinated, overall experience smooth.
4
a) Style mostly consistent, minor deviations; b) Terminology generally consistent, key concepts, tags, and naming mostly aligned; c) Expression and visual style mostly aligned, minor deviations, overall experience unaffected.
3
a) Style partially consistent, noticeable differences in narration; b) Terminology partially mixed, key concepts, tags, naming sometimes inconsistent; c) Expression and visual style partially aligned, wording, sentence structures, or rhetorical/visual style obviously deviate, reading/watching experience affected.
2
a) Style inconsistent, cross-modal narration uncoordinated; b) Terminology not uniform, key concepts, tags, naming frequently mixed; c) Expression and visual style not aligned, wording, sentence structures, or rhetorical/visual style conflicting, understanding affected.
1
a) Style completely inconsistent, cross-modal narration chaotic; b) Terminology completely wrong, key concepts, tags, naming incorrect or inconsistent; c) Expression and visual style completely chaotic, wording, sentence structures, or rhetorical/visual style extremely uncoordinated, almost impossible to understand.
Appendix
Table 20: Five-level rating criteria for Stylistic Harmony .
Modality
Caption Tool
Image
GPT-5 mini
Video
Qwen3-Omni
Audio
Qwen3-Omni
Document
GPT-5 mini
Code
Original format retained
3D
Three-view rendering → GPT-5 mini
Appendix
Table 21: Caption tools for each modality.
Metric
Range
Dimension
τ↑
[0, 1]
/
SC ↑
[0, 1]
Semantic Correctness & Generation Quality
GQ ↑
[0, 1]
Semantic Correctness & Generation Quality
SQCS abs ↑
[0, 1]
Semantic Correctness & Generation Quality
SQCS rel ↑
[0, 1]
Semantic Correctness & Generation Quality
StS abs ↑
[0, 1]
Response Structure Integrity
Appendix
Table 22: Metrics list of UniM evaluation suite.
Model
Multi-I Input
Multi-I Output
Any-M Input
Any-M Output
Any-I Input
Any-I Output
In-Modality
Out-Modality
AnyGPT
✗
✗
✗
✗
✗
✗
NExT-GPT
✗
✗
✗
✗
✗
✗
MIO
✓
✓
✗
✗
✗
✗
UniMA
✓
✓
✓
✓
✓
✓
Appendix
Table 23: Comparison between baseline models and UniMA on multimodal flexibility. In / Out -Modality: supported input / output modality. Multi-I Input / Output: multiple items of the same modalities in the input / output. Any-M Input / Output: any modality combinations in the input / output. Any-I Input / Output: any number of items of the same modality in the input / output.
Model
Mode
Temperature
Top_p
Top_k
Max_Tokens
GPT-5 mini
/
1
1
/
16,384
Qwen3-Omni
Non-Thinking
0.7
0.8
20
8,192
Qwen3-VL
Instruct
0.7
0.8
20
4,096
PointLLM
Eval
0.2
0.9
50
1,024
Appendix
Table 24: Detailed hyperparameter settings for UniMA .
Figure 15: Parameters selection experiments results of SQCS.
Figure 16: Parameters selection experiments results of ICS.
Figure 17: QA pairs construction prompt example.
Figure 18: Prompt template for Semantic Correctness .
Figure 19: Prompt template for text in Generation Quality .
Figure 20: Prompt template for code in Generation Quality .
Figure 21: Prompt template for document in Generation Quality .
Unified multimodal models capable of both understanding and generation have achieved remarkable strides. However, despite their unified designs, existing evaluations typically assess understanding and generation capabilities in isolation, overlooking the synergy between comprehension and generation. To bridge this gap, we introduce Unison, a comprehensive benchmark comprising 2,169 high-quality unified task samples, designed to evaluate joint understanding and generation in unified multimodal models. Unison offers three key strengths: 1) Comprehensive Dimensions: Unison encompasses internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement to enable holistic evaluation. 2) Diagnostic Evaluation: it provides both unified and decoupled tracks for understanding and generation, allowing fine-grained attribution of failure modes and quantitative analysis of the gains from unified modeling. 3) Human Alignment: we also introduce Unison-Judge, an evaluation model well aligned with human judgments to ensure reliable assessment. Based on systematic evaluations of state-of-the-art models on Unison, we uncover critical limitations in current unified multimodal systems and highlight promising directions for future research. Codes, Unison and Unison-Judge are publicly available at https://github.com/FudanCVL/Unison.
Jinyu Liu, Xincheng Shuai, Henghui Ding +1
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China
Automatically evaluating multimodal generation presents a significant challenge, as automated metrics often struggle to align with human evaluation, especially for complex tasks that involve multiple modalities. We present MMMG, the first benchmark to bring the verifiable-task paradigm to multimodal generation, spanning 4 modality combinations (image, audio, interleaved text and image, interleaved text and audio). As few multimodal outputs can be checked by programs alone, MMMG targets tasks that are either verifiable or near-verifiable: by providing references, and constraining model judges with explicit rubrics. We keep tasks challenging for generation models while enabling reliable automatic evaluation through a combination of models and programs. MMMG encompasses 55 tasks (including 31 newly developed ones), each with a carefully designed evaluation pipeline, and 1288 instructions to systematically assess reasoning, controllability, and other key capabilities of multimodal generation models. Extensive validation demonstrates that MMMG is highly aligned with human judgment, achieving an average agreement of 94.4%. Benchmarking results on 29 models reveal that even though the state-of-the-art model, GPT Image, achieves 70.7% accuracy for image generation, it falls short on interleaved generation. Furthermore, results suggest considerable improvement space in audio generation, highlighting an important future direction.
In recent years, unified multimodal models (UMMs) have emerged to support both understanding and generation within a single framework. Mastering dynamic, multi-turn interleaved image-text dialogues is a crucial task for UMMs in real-world applications. However, existing benchmarks fail to evaluate this important task, as they are often limited to single-turn or static settings, and typically overlook exposure bias in multi-turn interactions. To bridge this gap, we propose IMUG-Bench, a comprehensive benchmark for multi-turn interleaved image-text dialogue of UMMs that jointly evaluates their understanding and generation capabilities. Our IMUG-Bench comprises three classes: Static Spatial, Temporal Causal, and Hybrid, covering 3,113 samples and 12,034 interaction turns. It also includes dynamic understanding questions, thereby supporting evaluation that better reflects real-world multi-turn interaction scenarios. Large-scale experiments on IMUG-Bench systematically evaluate mainstream open-source and closed-source UMMs, revealing their capability boundaries and failure modes, and uncovering pronounced exposure bias on the generation side in multi-turn interactions. We further explore several test-time scaling strategies, including Chain-of-Thought, Self-Verification, and Best-of-N Sampling, which effectively improve generation accuracy and mitigate exposure bias in generation tasks. These findings provide insights into enhancing the robustness and multi-turn interaction capability of future UMMs.
Lingyi Meng, Zecong Tang, Haoran Li +12
Zhejiang University · Institute of Automation, Chinese Academy of Sciences · The University of Hong Kong +1