UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark

Yanlin Li; Minghui Guo; Kaiwen Zhang; Shize Zhang; Yiran Zhao; Haodong Li; Congyue Zhou; Weijie Zheng

cs.CVMar 5, 2026

UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark

Authors: Yanlin Li, Minghui Guo, Kaiwen Zhang, Shize Zhang, Yiran Zhao, Haodong Li, Congyue Zhou, Weijie Zheng,

Organizations: NUS · SCUT · NTU · NJU ·

Abstract

In real-world multimodal applications, systems usually need to comprehend arbitrarily combined and interleaved multimodal inputs from users, while also generating outputs in any interleaved multimedia form. This capability defines the goal of any-to-any interleaved multimodal learning under a unified paradigm of understanding and generation, posing new challenges and opportunities for advancing Multimodal Large Language Models (MLLMs). To foster and benchmark this capability, this paper introduces the UniM benchmark, the first Unified Any-to-Any Interleaved Multimodal dataset. UniM contains 31K high-quality instances across 30 domains and 7 representative modalities: text, image, audio, video, document, code, and 3D, each requiring multiple intertwined reasoning and generation capabilities. We further introduce the UniM Evaluation Suite, which assesses models along three dimensions: Semantic Correctness & Generation Quality, Response Structure Integrity, and Interleaved Coherence. In addition, we propose UniMA, an agentic baseline model equipped with traceable reasoning for structured interleaved generation. Comprehensive experiments demonstrate the difficulty of UniM and highlight key challenges and directions for advancing unified any-to-any multimodal intelligence. The project page is https://any2any-mllm.github.io/unim.

Figures & tables

Figure 1: Illustration of the any-to-any interleaved multimodal paradigm with different real-world application scenarios. Solving any-to-any interleaved multimodal learning requires complex and combined capabilities.

Benchmarks	Domains	Num.	Inter. Comb.	Cap. per Instance	Eval. Metric	Difficulty Tax.	Any-to	to-Any	Modalities
ITLVD-BENCH [ 34 ]	10	815	2	Single	5	✗	✗	✗
OpenING [ 79 ]	8	5,400	4	Single	7	✗	✗	✗
ISG-Bench [ 6 ]	8	1,150	3	Single	4	✗	✗	✗
CoMM [ 8 ]	3	/	4	Single	3	✗	✗	✗
MMIE [ 67 ]	10	20,103	3	Single	7	✗	✗	✗
UniM (Ours)	30	31,026	41	Multiple	13	✓	✓	✓

Table 1: Comparison with existing interleaved multimodal benchmarks. Inter. Comb.: Interleaved combinations of modalities. Cap. per Instance: Capability per instance. Difficulty Tax.: Difficulty taxonomy.

	Modality
	Image	Audio	Video	Document	Code	3D
Num.	22,936 (73.9%)	24,963 (80.5%)	2,336 (7.5%)	3,858 (12.4%)	807 (2.6%)	420 (1.4%)
	Field			Difficulty
	NS	SS	GA	Easy	Medium	Hard
Num.	10,624 (34.2%)	11,574 (37.3%)	8,828 (28.5%)	10,678 (34.4%)	14,088 (45.4%)	6,260 (20.2%)

Table 2: General statistics of UniM dataset.

Figure 2: Distribution of different difficulty levels.

Figure 3: Illustration of the UniM evaluation suite. ① refers to the calculation process of the StS and LeS (§ 4.2 ). ② represents the calculation process of the ICS in Eq. ( 2 ) . ③ refers to the calculation process of the SQCS; please refer to Eq. ( 1 ) .

Figure 4: Overview of the UniMA architecture.

Panel of Figure 5 — Figure 5: Results for rationality verification of SQCS and ICS.

Figure 6: Results for rationality verification of StS and LeS.

Model	Natural Science				Social Science				General Area
Model	StS abs	LeS abs	StS rel	LeS rel	StS abs	LeS abs	StS rel	LeS rel	StS abs	LeS abs	StS rel	LeS rel
AnyGPT [ 76 ]	12.9	27.8	12.2	21.4	14.9	16.6	14.5	16.2	12.5	16.4	9.8	13.6
NExT-GPT [ 64 ]	2.0	2.2	1.2	1.3	1.3	1.7	1.2	1.5	2.2	2.5	1.4	1.7
MIO [ 59 ]	1.3	1.9	0.9	1.3	4.1	5.2	4.0	5.1	3.3	3.8	2.4	2.9
UniMA	50.8	62.2	50.8	62.2	58.1	72.9	58.1	72.9	71.3	84.3	71.3	84.3

Table 3: Assessment results on Response Structure Integrity.

Panel of Table 4 — Table 4: Performance on Semantic Correctness & Generation Quality and Supporting Rate ( $\tau$ ).

Field	Models	HC	SH	ICS abs	ICS rel
Natural Science	AnyGPT [ 76 ]	39.9	46.3	41.8	38.5
	NExT-GPT [ 64 ]	23.5	26.1	24.9	16.3
	MIO [ 59 ]	49.4	63.7	52.1	31.8
	UniMA	68.4	71.9	69.1	69.1
Social Science	AnyGPT [ 76 ]	31.3	35.3	32.1	29.2
	NExT-GPT [ 64 ]	24.5	27.1	21.4	19.0

Table 5: Evaluation results on Interleaved Coherence.

Figure 7: Results across 10 capabilities on UniM . C1: Perceptual Understanding, C2: Spatial Understanding, C3: Temporal Understanding, C4: Semantic Generation, C5: Content Editing, C6: Creative Expression, C7: Reasoning Capability, C8: Emotional Analysis, C9: Structural Analysis, and C10: Planning Capability. Refer to Appendix § C.4 for details.

Figure 8: Comparison of model responses in the cross-modal painting instruction task. NExT-GPT [ 64 ] outputs only generic text without image generation, lacking procedural structure. MIO [ 59 ] produces a single sketch image but omits key intermediate steps. UniMA generates a coherent step-by-step text–image workflow aligned with the video, covering the entire painting process.

Figure 9: Performance comparison of MLLMs under different difficulty levels across evaluation metrics. Each curve illustrates the performance trend of a model as task complexity increases, where higher values indicate better performance.

	SQCS	ICS	StS	LeS
UniMA	85.1	63.4	52.7	82.6
w/o TER	72.9 (-12.2)	56.6 (-6.8)	16.4 (-36.3)	21.8 (-60.8)
w/o TCDC	78.4 (-6.7)	57.7 (-5.7)	46.2 (-6.5)	82.1 (-0.5)
w/o Ver.	72.9 (-12.2)	54.7 (-8.7)	38.3 (-14.4)	66.8 (-15.8)

Table 6: Ablation results on UniMA .

Appendix figures & tables64 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Overview of dataset construction.

Modality	Public Dataset
Image	FoodieQA [ 31 ] , MuSLR [ 72 ] , TaxaBench-8k [ 48 ] , MSEarth [ 77 ] , LAB-Bench [ 21 ] , Chempile [ 40 ] OmniCorpus [ 29 ] , OmniEarth-Bench [ 55 ] , MMRB [ 11 ] , StateControl-Bench [ 66 ] iMaterialist [ 18 ] , DeepFashion [ 35 ]
Audio	Panosent [ 37 ] , CMI-Bench [ 38 ] , ESC [ 46 ]
Video	Video-MME [ 15 ] , MuMa-ToM [ 49 ] , MovieChat [ 50 ] , Social-IQ [ 75 ] , PhysLab [ 47 ] , EgoTextVQA [ 80 ] , HowTo100M [ 39 ]
Document	MedTrinity-25M [ 70 ] , CaseSumm [ 19 ] , FDABench [ 60 ] , Synfintabs [ 4 ]
Code	MMCode [ 28 ] , ScreenSpot-Pro [ 27 ] , CodeEditorBench [ 16 ]
3D	MetaFood3D [ 9 ] , General-Bench [ 14 ]

Appendix

Table 7: Public dataset sources.

Modality	File Format
Image	PNG, JPG, JPEG
Audio	WAV, MP3
Video	MP4, WEBM
Document	PDF, DOC, CSV, TXT, PNG, JPG
Code	Markdown Block
3D	JPG+MTL+OBJ, OFF, PLY

Appendix

Table 8: Multimodal files formats.

Input

A+D

I+A

V+A

I+V+A

I+D

A+I

I+A

Output

I+A

V+A

A+V

Input

I+A

I+D

V+I

I+A

V+A

Output

I+D

V+A

Input

I+A+D

I+V+A

I+A+D

D+A

I+A

A+V+D

Output

I+A

V+A

Appendix

Table 9: Interleaved combinations in UniM . I: Image, V: Video, A: Audio, D: Document, C: Code.

Panel of Figure 12 — Figure 12: Definition of hard level.

Panel of Figure 13 — Figure 13: Definition of medium level.

Panel of Figure 14 — Figure 14: Definition of easy level.

ID	Capability	Definition
C1	Perceptual Understanding	Identifying and interpreting content, entities, attributes, and semantic relations from multimodal inputs.
C2	Spatial Understanding	Comprehending spatial relationships, geometric structures, 3D scene character- istics, and physical layouts.
C3	Temporal Understanding	Understanding event or action sequences, temporal dynamics, and time-depend- ent dependencies.
C4	Semantic Generation	Generating multimodal content that is semantically coherent, contextually appro- priate, and stylistically consistent.
C5	Content Editing	Modifying, refining, or transforming existing content, including stylistic adjust- ments and targeted edits.
C6	Creative Expression	Producing content with creativity, expressiveness, and aesthetic value.

Appendix

Table 10: Definitions of capabilities.

Modality	Task Type
Text	Text summarization; Text generation; Machine translation; Sentiment analysis; Stance detection; Named entity recognition; Text style transfer; …
Image	Image captioning; Image generation; Semantic segmentation; Image editing; Visual localization; Visual question answering; Image style transfer; …
Audio	Audio captioning; Audio generation; Audio timbre analysis; Audio sentiment analysis; Audio content analysis; …
Video	Video captioning; Video localization; Visual question answering; Video editing; Text-to-video generation; Audio-to-video generation; Viewpoint-transformation video synthesis; …
Document	Document OCR; Document generation; Document description; Document summarization; Document question answering; …
Code	Code generation; Code testing; Code debugging; Code translation; Code understanding; Code question answering; Algorithm implementation; …

Appendix

Table 11: Definitions of task types.

Image	Audio	Video	Document	Code	3D
68,651	54,329	2,559	4,728	1,207	820

Appendix

Table 12: The number of multimodal files in UniM .

Natural Science					Social Science					General Area
ID	Name	Easy	Medium	Hard	ID	Name	Easy	Medium	Hard	ID	Name	Easy	Medium	Hard
#1	Math	806	777	48	#11	Finance	739	1694	135	#21	Sports	57	313	486
#2	Physics	40	591	374	#12	Law	256	634	121	#22	News	200	456	350
#3	Chemistry	312	617	113	#13	Linguistics	570	207	53	#23	Culture	128	170	100
#4	Biology	545	333	263	#14	Psychology	398	300	397	#24	Entertainment	301	188	299
#5	Compute Science	72	868	218	#15	Education	165	463	72	#25	Food	256	624	296

Appendix

Table 13: Detailed statistics of UniM .

Modality	Caption Tool
Text	Original format retained
Image	GPT-5 mini
Video	Qwen3-Omni
Audio	Qwen3-Omni
Document	GPT-5 mini
Code	Original format retained

Appendix

Table 14: Caption methods for each modality.

Grade	Semantic Correctness Description
5	a) Completely equivalent; b) All key facts correct/covered; c) No contradictions; d) Units, ranges, and relational constraints remain consistent (paraphrasing, reordering, and minor rounding are allowable).
4	a) Almost equivalent; b) Most of key facts correct/covered; c) No major contradiction; d) Only minor omissions/ambiguity that do not affect the main conclusion.
3	a) Partially correct; b) Roughly half key facts correct; c) No major contradiction but noticeable; d) Notable omissions or minor misinterpretations, but the main conclusion is not fully overturned.
2	a) Low correctness; b) Less than half covered; c) Important errors/contradictions/confusions (numbers/entities); d) The core conclusion drifts, but still loosely on topic.
1	a) Almost incorrect; b) Nearly irrelevant; c) Mostly contradictory, or hallucinated; d) Mostly wrong/missing, or non-answers.

Appendix

Table 15: Five-level rating criteria for Semantic Correctness .

Grade	Text Quality Description
5	a) Content is rich, self-consistent, and detailed; no major omissions or vague generalities; stands alone as a coherent text; b) Structure is clear and well-organized; transitions are smooth (e.g., “firstly…then…therefore…”); reasoning shows causal or hierarchical logic; c) Language is natural and grammatically flawless; diverse sentence structures; no syntactic errors; d) Entirely in one language; any foreign words appear only as necessary terminology.
4	a) Content is generally complete with sufficient detail but slightly shallow or missing minor points; b) Structure is good, with only mild jumps or awkward transitions that don’t affect comprehension; c) Language is fluent with few minor grammatical or collocation issues; d) Mostly consistent language, with rare short foreign terms that do not disrupt flow.
3	a) Content covers the main idea but lacks depth or specific details; b) Organization somewhat weak; order or topic shifts slightly; meaning still clear overall; c) Several grammatical or spelling mistakes; simple or repetitive sentence patterns; d) Minor language switching between sentences or paragraphs, noticeable but not confusing.
2	a) Content is shallow, missing key details or explanations; very low information density; b) Poor structure; sentences disjointed; reader must infer connections; c) Frequent grammar errors; awkward or broken phrasing; readability low; d) Frequent in-sentence language mixing that affects readability.
1	a) Content is empty or meaningless; repetitive or irrelevant phrases; conveys no clear information; b) No logical order; severe contradictions; text barely comprehensible; c) Major grammatical breakdowns; unnatural or non-human syntax; d) Chaotic multilingual mixing (e.g., Chinese + English + Spanish, random spelling noise)

Appendix

Table 16: Five-level rating criteria for text in Generation Quality .

Grade	Code Quality Description
5	a) High-quality, professional-grade code with virtually no noticeable defects; b) Logically rigorous, structurally well-organized, and highly robust; c) Fully adheres to modern software engineering best practices, easy to maintain, and directly reusable.
4	a) High overall quality with a reasonable structure and clear logic; b) Contains only minor issues that do not affect usability; c) Aligns with most software engineering best practices and is easy to maintain.
3	a) Basically usable, with acceptable overall correctness but clear deficiencies; b) Contains multiple areas for improvement that may affect maintainability or reliability; c) The overall quality is functional but remains below desirable standards.
2	a) Contains multiple evident issues, with overall quality falling below acceptable standards; b) Barely runnable or only partially functional, requiring substantial repairs; c) Exhibits major deficiencies in logic, structure, performance, or security.
1	a) Contains numerous critical defects, resulting in extremely low overall quality; b) Exhibits chaotic logic, unclear structure, and significant risks; c) Largely non-reusable and unmaintainable, requiring a complete rewrite.

Appendix

Table 17: Five-level rating criteria for code in Generation Quality .

Grade	Document Quality Description
5	a) All titles, labels, and column names are clear and unambiguous; b) The table structure is complete, with well-organized logical partitions and clear hierarchy; c) Units, naming conventions, capitalization, and punctuation are fully consistent; d) Numerical presentation is standardized, including uniform decimal places, units, and precision, with proper alignment; e) The surrounding context provides sufficient information for the table to be understood independently without additional explanation.
4	a) Most titles and column names are accurate, with only minor ambiguities; b) The table structure is generally reasonable, though some misalignment or slightly unclear grouping is present; c) Overall consistency is good, with only small variations in decimal places or units; d) Data presentation is mostly standardized, with mild irregularities that do not hinder usability; e) The table can largely be understood independently, though some contextual information is slightly lacking.
3	a) Contains noticeably ambiguous titles or labels whose meanings require contextual inference; b) Table structure is relatively disorganized, with unclear grouping, poorly defined relationships between columns; c) Multiple inconsistencies in formatting, naming, or units, reducing overall usability; d) Numerical presentation is irregular, with large variations in decimal places and missing or mixed units; e) Lacks sufficient contextual explanation, making the table difficult to interpret independently.
2	a) Titles or column names are missing or severely ambiguous, with clear potential for misinterpretation; b) The table structure is incomplete, exhibiting column misalignment, unclear row logic, or missing grouping; c) Formatting is highly inconsistent, with disordered or entirely missing units; d) Numerical presentation shows prominent issues, including irregular precision, inconsistent units, and potentially misleading values; e) The table is largely uninterpretable on its own and requires substantial supplementary explanation.
1	a) Titles or column names are extensively incorrect, missing, or unrecognizable; b) The table structure is entirely broken, with row–column relationships indistinguishable; c) No formatting consistency is preserved, with content unordered and misaligned; d) Numerical values are meaningless due to severe unit confusion, row mismatches, or erroneous combinations; e) The table is completely unusable on its own and requires re-OCR or full reconstruction from the original source.

Appendix

Table 18: Five-level rating criteria for document in Generation Quality .

Grade	Holistic Coherence Description
5	a) Output highly matches the input, multimodal references accurate and specific; b) Different modality information complements each other without noticeable contradictions; c) Logic is rigorous, structure is reasonable, overall semantic/logical order is clear; d) Interleaved multimodal tags naturally ordered, reading/understanding experience smooth.
4	a) Output mostly matches the input, references generally reasonable; b) Different modality information mostly complementary, minor omissions or redundancy; c) Overall logic coherent, local repetitions or jumps minor, not affecting understanding; d) Interleaved tags mostly ordered, minor adjustments do not affect understanding.
3	a) Output has general relation to input, references are vague or partially unclear; b) Some modality blocks repeated or missing, minor contradictions exist; c) Local logic jumps or slight contradictions, requires extra reasoning; d) Interleaved tags partially disordered, local understanding may be difficult.
2	a) Output lowly matches the input, most references vague or incorrect; b) Cross-modal information repetitive or conflicting; c) Logic obviously chaotic, many contradictions; d) Interleaved tags order clearly disordered, understanding difficult.
1	a) Output is almost irrelevant to input, references missing or wrong; b) Different modalities barely complement, major contradictions; c) Logic completely collapsed, frequent contradictions; d) Interleaved tags order completely disordered, hard to understand.

Appendix

Table 19: Five-level rating criteria for Holistic Coherence .

Grade	Stylistic Harmony Description
5	a) Style highly consistent, cross-modal narration uniform, tone and sentence structures without deviation; b) Terminology fully consistent, key concepts, tags, and naming completely aligned; c) Expression and visual style fully aligned, wording, sentence structures, and rhetorical/visual style coordinated, overall experience smooth.
4	a) Style mostly consistent, minor deviations; b) Terminology generally consistent, key concepts, tags, and naming mostly aligned; c) Expression and visual style mostly aligned, minor deviations, overall experience unaffected.
3	a) Style partially consistent, noticeable differences in narration; b) Terminology partially mixed, key concepts, tags, naming sometimes inconsistent; c) Expression and visual style partially aligned, wording, sentence structures, or rhetorical/visual style obviously deviate, reading/watching experience affected.
2	a) Style inconsistent, cross-modal narration uncoordinated; b) Terminology not uniform, key concepts, tags, naming frequently mixed; c) Expression and visual style not aligned, wording, sentence structures, or rhetorical/visual style conflicting, understanding affected.
1	a) Style completely inconsistent, cross-modal narration chaotic; b) Terminology completely wrong, key concepts, tags, naming incorrect or inconsistent; c) Expression and visual style completely chaotic, wording, sentence structures, or rhetorical/visual style extremely uncoordinated, almost impossible to understand.

Appendix

Table 20: Five-level rating criteria for Stylistic Harmony .

Modality	Caption Tool
Image	GPT-5 mini
Video	Qwen3-Omni
Audio	Qwen3-Omni
Document	GPT-5 mini
Code	Original format retained
3D	Three-view rendering $\rightarrow$ GPT-5 mini

Appendix

Table 21: Caption tools for each modality.

Metric	Range	Dimension
$\tau\uparrow$	[0, 1]	/
SC $\uparrow$	[0, 1]	Semantic Correctness & Generation Quality
GQ $\uparrow$	[0, 1]	Semantic Correctness & Generation Quality
SQCS abs $\uparrow$	[0, 1]	Semantic Correctness & Generation Quality
SQCS rel $\uparrow$	[0, 1]	Semantic Correctness & Generation Quality
StS abs $\uparrow$	[0, 1]	Response Structure Integrity

Appendix

Table 22: Metrics list of UniM evaluation suite.

Model	Multi-I Input	Multi-I Output	Any-M Input	Any-M Output	Any-I Input	Any-I Output	In-Modality	Out-Modality
AnyGPT	✗	✗	✗	✗	✗	✗
NExT-GPT	✗	✗	✗	✗	✗	✗
MIO	✓	✓	✗	✗	✗	✗
UniMA	✓	✓	✓	✓	✓	✓

Appendix

Table 23: Comparison between baseline models and UniMA on multimodal flexibility. In / Out -Modality: supported input / output modality. Multi-I Input / Output: multiple items of the same modalities in the input / output. Any-M Input / Output: any modality combinations in the input / output. Any-I Input / Output: any number of items of the same modality in the input / output.

Model	Mode	Temperature	Top_p	Top_k	Max_Tokens
GPT-5 mini	/	1	1	/	16,384
Qwen3-Omni	Non-Thinking	0.7	0.8	20	8,192
Qwen3-VL	Instruct	0.7	0.8	20	4,096
PointLLM	Eval	0.2	0.9	50	1,024

Appendix

Table 24: Detailed hyperparameter settings for UniMA .

Figure 15: Parameters selection experiments results of SQCS.

Figure 16: Parameters selection experiments results of ICS.

Figure 17: QA pairs construction prompt example.

Figure 18: Prompt template for Semantic Correctness .

Figure 19: Prompt template for text in Generation Quality .

Figure 20: Prompt template for code in Generation Quality .

Figure 21: Prompt template for document in Generation Quality .

Figure 22: Interleaved coherence evaluation prompt.

Figure 23: The screenshot of human evaluation tool we developed.

Figure 24: The screenshot of systematic verification tool we developed.

Figure 25: The screenshot of multi-reviewer quality evaluation tool we developed.

Figure 42: Prompt template for step 1 in TER module.

Figure 43: Prompt template for step 2 in TER module.

Figure 44: Prompt template for step 3 in TER module.

Figure 45: Prompt template for step 4 in TER module.

Figure 46: Prompt template for generating module.

Semantic Correctness & Generation Quality
	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10
	#11	#12	#13	#14	#15	#16	#17	#18	#19	#20
	#21	#22	#23	#24	#25	#26	#27	#28	#29	#30
AnyGPT [ 76 ]
SC	13.3	4.4	16.7	8.9	13.8	6.2	13.7	16.0	24.9	19.6
	16.9	18.9	33.4	6.5	23.3	51.9	5.4	17.4	4.8	2.0

Appendix

Table 25: Detailed experiment results on Semantic Correctness & Generation Quality .

Semantic Correctness & Generation Quality
	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10
	#11	#12	#13	#14	#15	#16	#17	#18	#19	#20
	#21	#22	#23	#24	#25	#26	#27	#28	#29	#30
AnyGPT [ 76 ]
$\tau$	99.9	79.6	98.8	98.0	95.0	99.4	47.4	92.5	95.3	98.3
	98.4	98.3	98.2	91.0	95.7	97.0	99.2	70.2	99.3	100

Appendix

Table 26: Detailed experiment results on Semantic Correctness & Generation Quality .

Response Structure Integrity
	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10
	#11	#12	#13	#14	#15	#16	#17	#18	#19	#20
	#21	#22	#23	#24	#25	#26	#27	#28	#29	#30
AnyGPT [ 76 ]
StS abs	3.8	14.2	25.1	18.7	9.3	0	0	23.4	22.9	11.3
	0.8	1.3	81.5	0.8	2.5	49.7	6.8	0.4	3.8	0.3

Appendix

Table 27: Detailed experiment results on Response Structure Integrity .

Response Structure Integrity
	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10
	#11	#12	#13	#14	#15	#16	#17	#18	#19	#20
	#21	#22	#23	#24	#25	#26	#27	#28	#29	#30
AnyGPT [ 76 ]
StS rel	3.8	11.3	24.8	18.4	8.8	0	0	21.7	21.8	11.1
	0.7	1.2	80.0	0.7	2.3	48.2	6.8	0.3	3.8	0.3

Appendix

Table 28: Detailed experiment results on Response Structure Integrity .

Interleaved Coherence
	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10
	#11	#12	#13	#14	#15	#16	#17	#18	#19	#20
	#21	#22	#23	#24	#25	#26	#27	#28	#29	#30
AnyGPT [ 76 ]
HC	47.2	44.5	37.5	23.9	34.5	33.4	38.5	40.8	54.6	44.2
	65.3	36.1	58.4	8.7	35.8	60.3	12.0	28.4	5.9	2.0

Appendix

Table 29: Detailed experiment results on Interleaved Coherence .

Interleaved Coherence
	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10
	#11	#12	#13	#14	#15	#16	#17	#18	#19	#20
	#21	#22	#23	#24	#25	#26	#27	#28	#29	#30
AnyGPT [ 76 ]
ICS abs	49.4	46.4	39.4	24.2	35.7	34.5	39.6	40.6	56.8	45.7
	67.6	36.8	58.8	8.9	37.2	62.0	12.1	29.7	5.9	2.0

Appendix

Table 30: Detailed experiment results on Interleaved Coherence .

Panel of Figure 47 — Figure 47: SQCS performance across baseline models, difficulty levels and domains.

Panel of Figure 48 — Figure 48: StS performance across baseline models, difficulty levels and domains.

Panel of Figure 49 — Figure 49: LeS performance across baseline models, difficulty levels and domains.

Figure 61: Case illustrating the responses produced by AnyGPT, NExT-GPT, MIO and UniMA to T+I+A+D as input.

Figure 62: Case illustrating the responses produced by AnyGPT, NExT-GPT, MIO and UniMA to T+I as input.

Figure 63: Case illustrating the responses produced by AnyGPT, NExT-GPT, MIO and UniMA to T+D as input.

Multimodal Learning

Multimodal learning studies how models combine and reason over different data types, such as images, text, audio, signals, and structured records.

See all 338 papers

Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation
Jun 25, 2026Jinyu Liu, Xincheng Shuai, Henghui Ding +1Unified Multimodal Models Multimodal Foundation Models
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China
MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation
May 23, 2025Jihan Yao, Yushi Hu, Wenyuan Wang +10Multimodal Generation Multimodal Model Evaluation
University of Washington · Allen Institute for AI
IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation
Jun 8, 2026Lingyi Meng, Zecong Tang, Haoran Li +12Unified Multimodal Models Multimodal Generation
Zhejiang University · Institute of Automation, Chinese Academy of Sciences · The University of Hong Kong +1

Jun 25, 2026J/K move · Enter open · S save

Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

Jinyu Liu, Xincheng Shuai, Henghui Ding, Yu-Gang Jiang

Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China

Unified Multimodal Models Multimodal Foundation Models Multimodal Model Evaluation Multimodal Learning Multimodal Understanding

Open paper AskPDF arXiv

Unified multimodal models capable of both understanding and generation have achieved remarkable strides. However, despite their unified designs, existing evaluations typically assess understanding and generation capabilities in isolation, overlooking the synergy between comprehension and generation. To bridge this gap, we introduce Unison, a comprehensive benchmark comprising 2,169 high-quality unified task samples, designed to evaluate joint understanding and generation in unified multimodal models. Unison offers three key strengths: 1) Comprehensive Dimensions: Unison encompasses internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement to enable holistic evaluation. 2) Diagnostic Evaluation: it provides both unified and decoupled tracks for understanding and generation, allowing fine-grained attribution of failure modes and quantitative analysis of the gains from unified modeling. 3) Human Alignment: we also introduce Unison-Judge, an evaluation model well aligned with human judgments to ensure reliable assessment. Based on systematic evaluations of state-of-the-art models on Unison, we uncover critical limitations in current unified multimodal systems and highlight promising directions for future research. Codes, Unison and Unison-Judge are publicly available at https://github.com/FudanCVL/Unison.

Abstract

Multimodal Learning

Explore similar work

Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation

IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation