TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training

Yinyi Luo; Wenwen Wang; Hayes Bai; Hongyu Zhu; Hao Chen; Pan He; Marios Savvides; Sharon Li

cs.AIApr 12, 2026

TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training

Authors: Yinyi Luo, Wenwen Wang, Hayes Bai, Hongyu Zhu, Hao Chen, Pan He, Marios Savvides, Sharon Li,

Organizations: Carnegie Mellon University · William & Mary · Auburn University · University of Wisconsin-Madison

Abstract

Recent advances in unified multimodal models (UMMs) have led to a proliferation of architectures capable of understanding, generating, and editing across visual and textual modalities. However, developing a unified framework for UMMs remains challenging due to the diversity of model architectures and the heterogeneity of training paradigms and implementation details. In this paper, we present TorchUMM, the first unified codebase for comprehensive evaluation, analysis, and post-training across diverse UMM backbones, tasks, and datasets. TorchUMM supports a broad spectrum of models covering a wide range of scales and design paradigms. Our benchmark encompasses three core task dimensions: multimodal understanding, generation, and editing, and integrates both established and novel datasets to evaluate perception, reasoning, compositionality, and instruction-following abilities. By providing a unified interface and standardized evaluation protocols, TorchUMM enables fair and reproducible comparisons across heterogeneous models and fosters deeper insights into their strengths and limitations, facilitating the development of more capable unified multimodal systems. Code is available at: https://github.com/AIFrontierLab/TorchUMM.

Figures & tables

Model	Venue	Architecture	DPG Bench	GenEval	WISE
Show-o (1.3B)	ICLR’25	AR+diffusion	78.74	65.06	0.304
Emu3 (8B)	arxiv2409	AR	80.31	45.76	0.337
Janus (1.3B)	arxiv2410	AR	73.53	40.04	0.222
JanusFlow (1.3B)	CVPR’25	AR + diffusion	72.03	49.99	0.296
TokenFlow (7B)	CVPR’25	AR	71.29	52.21	0.306
Janus-Pro (7B)	arxiv2501	AR	83.73	78.92	0.381

Table 1: Text-to-image generation results. DPG Bench and GenEval are reported as overall accuracy (%); WISE reports the WiScore (0–1) evaluated by Qwen2.5-VL-72B-Instruct.

Model	MME (P)	MME (C)	MMMU	MMBench(EN)	MMBench(CN)	MM-Vet	MathVista	MMStar
Bagel (14B)	1691.5	695.4	0.519	0.8265	0.8094	65.9	71.6	63.20
Janus-Pro (7B)	1547.9	293.2	0.407	0.6764	0.6494	33.7	42.8	42.00
JanusFlow (1.3B)	1305.6	251.1	0.290	0.6416	0.6031	31.8	34.8	39.19
Janus (1.3B)	1221.4	264.3	0.273	0.2769	0.3719	27.0	26.6	26.67
Show-o2 (7B)	1619.8	387.5	0.479	0.7725	0.7668	47.1	51.5	54.53
Show-o2 (1.5B)	1413.3	291.8	0.371	0.6509	0.6104	46.1	37.9	43.87

Table 2: Multimodal understanding results. MME reports perception (P) and cognition (C) scores.

	GEdit-EN			GEdit-CN			ImgEdit
Model	SC	PQ	O	SC	PQ	O	ST	MT	UGE
DeepGen	7.44	7.54	7.33	7.41	7.59	7.36	4.07	4.37	4.81
Bagel	6.68	7.04	6.35	6.83	7.06	6.52	3.71	4.45	4.18
OmniGen2	6.49	7.18	6.27	6.25	7.18	6.03	3.88	3.27	4.06
Emu3.5	7.64	7.48	7.56	7.62	7.50	7.56	4.24	4.89	4.88
Ovis-U1	7.28	7.46	7.13	6.78	7.51	6.72	3.97	2.82	4.48

Table 3: Image editing results. SC = Semantic Correctness, PQ = Perceptual Quality, O = Overall (geometric mean), ST = SingleTurn, MT = MultiTurn, all on a 0–10 scale. Evaluated by Qwen2.5-VL-72B via VIEScore.

Model	Unified-Bench					RealUnified (GEU)					UEval
Model	Clip	Dinov2	Dinov3	Longclip	Overall	MC	CN	AF	MR	Total	UEval
Bagel (14B)	0.8947	0.7877	0.7240	0.9321	0.8346	0.30	0.39	0.52	0.34	0.39	30.9
Janus-Pro (7B)	0.8874	0.7753	0.7232	0.9312	0.8293	0.20	0.26	0.31	0.26	0.26	20.6
Janus (1.3B)	0.8465	0.6396	0.5691	0.8982	0.7384	0.04	0.24	0.17	0.20	0.16	–
Janus-flow (1.3B)	0.8290	0.5598	0.4775	0.8734	0.6849	0.24	0.20	0.26	0.22	0.23	–
Show-o2 (7B)	0.8281	0.5859	0.4994	0.8679	0.6953	0.26	0.30	0.38	0.31	0.31	29.1

Table 4: Results on unification benchmarks.

Variant	DPG	GenEval	WISE	UEval
Bagel (base)	84.11	78.81	0.399	30.9
+ RecA	85.20	83.05	0.423	31.0
+ UniCot	83.52	77.91	0.404	31.8
+ SFT	83.02	78.03	0.227	31.4
+ IRG	81.82	72.06	0.384	9.1
+ UniGame	65.77	85.80	0.403	31.0

Table 5: Post-training results for generation.

Variant	MME (P)	MME (C)	MMMU	MMBench(EN)	MMBench(CN)	MM-Vet	Mathvista
Bagel (base)	1691.5	695.4	0.519	0.8265	0.8094	65.9	71.6
+ RecA	1689.1	695.4	0.523	0.8265	0.8094	66.1	51.6
+ UniCot	1690.7	678.2	0.531	0.8312	0.8099	64.5	73.0
+ SFT	1680.7	678.9	0.526	0.8265	0.8088	61.2	73.1
+ IRG	1647.5	650.4	0.480	0.6047	0.5735	40.7	68.0
+ UniGame	1692.1	695.4	0.524	0.8275	0.8094	60.7	72.2

Table 6: Post-training results on understanding benchmark.

Variant	GEdit-EN(I/F)	GEdit-CN(I/F)	ImgEdit(S)	ImgEdit(M)	ImgEdit(U)
Bagel (base)	6.38/6.35	6.68/6.52	3.71	4.45	4.18
+ RecA	6.89/6.80	6.87/6.75	3.89	4.28	4.15
+ UniCot	7.04/6.92	6.90/6.81	3.77	4.22	4.34
+ SFT	6.62/6.49	6.71/6.54	3.73	4.48	4.12
+ IRG	6.52/6.44	6.51/6.41	3.79	3.89	4.54
+ UniGame	6.48/6.48	6.55/6.38	3.72	4.46	4.31

Table 7: Post-training on editing benchmarks. For GEdit-EN and GEdit-CN, (I/F) denotes Intersection and Full scores, respectively. ImgEdit results are reported as S/M/U (single-turn, multi-turn, UGE).

Panel of Figure 2 — Figure 2: Final-layer UEval latent spaces. CMAM measures how much closer a matched text–image pair is than mismatched pairs; larger values indicate a smaller effective modality gap. OmniGen2, although least unified by design, has the strongest cross-modal alignment.

Panel of Figure 3 — Figure 3: Reasoning-trace behavior on MathVista. From left to right: answer accuracy against median visible trace length; the full trace-length distribution; and teacher-forced entropy trajectories for the three open-source UMMs. OmniGen2, the least architecturally unified model among the three, most reliably follows the CoT instruction, produces long explicit traces, and achieves the best open-source accuracy. MMaDA and Show-o2, which apply more aggressive sharing across modalities and tasks, frequently collapse to short responses and fail to obtain comparable test-time scaling.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

model	single_object	two_object	counting	colors	position	color_attr	overall
bagel(w/o think)	99.38	94.19	78.75	87.77	51	61.75	78.81
blip3o	98.12	93.18	73.44	86.17	72.75	64.5	81.36
show_o2(7B)	97.81	71.46	48.75	78.46	20	42.75	59.87
show_o2(1.5B)	96.88	64.39	46.88	76.06	16.75	32	55.49
Janus_pro	97.81	86.62	57.5	89.36	76	66.25	78.92
Janus	85.62	37.63	18.75	53.46	17.5	27.25	40.04

Appendix

Table 8: Geneval sub-score.

model	culture	time	space	biology	physics	chemistry	overall
bagel(w/o think)	0.3883	0.4386	0.4714	0.362	0.4205	0.294	0.3989
blip3o	0.4028	0.4186	0.5259	0.4025	0.4255	0.3	0.4138
show_o2(7B)	0.3641	0.3497	0.4519	0.3455	0.369	0.239	0.3595
show_o2(1.5B)	0.3111	0.3563	0.4357	0.315	0.384	0.231	0.3349
show_o	0.2865	0.3225	0.4132	0.275	0.33	0.198	0.3037
Janus_pro	0.3616	0.3853	0.4789	0.3605	0.4745	0.2485	0.3811

Appendix

Table 9: Wise generation sub-score.

Model	Overall	Multi-choice	Free-form
Bagel	71.60%	80.19%	61.52%
Show-o2 (7B)	51.50%	63.52%	37.39%
Show-o2 (1.5B)	37.90%	53.33%	19.78%
Show-o	29%	44.07%	11.30%
Emu3	44.90%	57.59%	30.00%
Janus-Pro	42.80%	51.30%	32.83%

Appendix

Table 10: Mathvista sub-score.

Model	Art & Design	Business	Science	Health & Medicine	Humanities & Social Sci	Tech & Engineering	Overall
Bagel	0.583	0.433	0.433	0.587	0.725	0.438	0.519
Janus-Pro	0.508	0.38	0.347	0.413	0.5	0.352	0.407
Janus	0.325	0.18	0.267	0.273	0.3	0.3	0.273
Janus-Flow	0.392	0.233	0.233	0.333	0.358	0.243	0.29
Emu3	0.342	0.26	0.267	0.34	0.392	0.31	0.314
OmniGen2	0.558	0.387	0.34	0.533	0.7	0.352	0.46

Appendix

Table 11: MMMU sub-score.

Model	bg_change	color	material	motion	ps_human	style	subj-add	subj-rm	subj-repl	text	tone	Avg
Bagel SC	7.45	7.075	6.975	5.4	5.271	6.233	7.283	7.175	7.167	6.838	6.6	6.679
Bagel PQ	7.35	7.075	6.725	7.175	7.043	5.833	7.7	7.211	7.283	7.535	6.55	7.044
Bagel O	7.264	6.653	6.448	5.128	5.046	5.803	7.214	6.743	6.861	6.368	6.304	6.348
DeepGen SC	7.775	7.9	7.625	7.675	7.114	5.983	7.967	7.596	7.833	7.242	7.175	7.444
DeepGen PQ	7.825	7.325	7.025	7.875	7.657	6.867	7.8	7.667	7.817	7.626	7.4	7.535
DeepGen O	7.78	7.44	7.286	7.745	7.2	6.287	7.869	7.409	7.737	6.836	7.052	7.331

Appendix

Table 12: GEdit-EN-Overall sub-score.

Model	bg_change	color	material	motion	ps_human	style	subj-add	subj-rm	subj-repl	text	tone	Avg
Bagel SC	7.241	7.353	7.071	4.909	5.61	6.146	7.342	7.19	7.174	6.864	7.08	6.726
Bagel PQ	7.172	6.882	6.714	7.227	7.195	5.833	7.658	7.238	7.217	7.519	6.64	7.027
Bagel O	7.021	6.872	6.493	4.662	5.401	5.741	7.264	6.784	6.813	6.391	6.789	6.384
DeepGen SC	7.862	8.059	7.607	7.682	7.537	5.771	7.974	7.881	7.891	7.136	7.32	7.52
DeepGen PQ	7.828	7.176	7.143	8	7.805	6.812	7.711	7.857	7.87	7.642	7.68	7.593
DeepGen O	7.839	7.548	7.351	7.807	7.531	6.134	7.823	7.753	7.782	6.768	7.323	7.423

Appendix

Table 13: GEdit-EN-Intersection sub-score.

Model	RealUnify(UEG)
Model	WK	CR	LR	MR	SR	CTI	Total
Bagel(14B)	0.38	0.31	0.12	0.21	0.19	0.08	0.22
Janus-Pro(7B)	0.26	0.36	0.07	0.08	0.14	0.02	0.16
Janus(1.5B)	0.09	0.05	0.00	0.08	0.11	0.00	0.06
Janus-flow(1.5B)	0.14	0.17	0.05	0.15	0.17	0.02	0.12
Omnigen2(7B)	0.36	0.25	0.12	0.23	0.21	0.18	0.23

Appendix

Table 14: RealUnify UEG-score.

Model	und+edit		und+edit		und+edit		und+edit		und+gen			und+gen
	Jig. I	Jig. T	Maze I	Maze T	Slid. I	Slid. T	Geo I	Geo T	Sci. R.	Sci. T	Sci. I	C. T	C. S	C. P
bagel(w/o think)	0.66	0.553	0.004	0.101	0	0.05	0.05	0.143	0.592	0.522	0.185	0.115	0.375	0.275
Janus_pro	-	-	-	-	-	-	-	-	29.3	25.5	0	1.5	3.7	3.4

Appendix

Table 15: Uni-mmmu sub-score.

Panel of Figure 4 — Figure 4: Representative UEval cases across models with different degrees of unification. The first row requires procedural decomposition, where the model must align textual guidance with a progressive visual sequence. The second row requires technical understanding, structured layout, arrows, and text rendering in a scientific diagram. Across both cases, only OmniGen2 consistently attempts the requested structure, while Show-o2 tends to produce more visually plausible but semantically under-specified outputs and MMaDA more often collapses on both semantic grounding and visual organization.

Panel of Figure 5 — Figure 5: Query-variation analysis under two backbone–model pairings. In each row, the left panel shows consistency among the generated responses in a shared embedding space, while the right panel shows consistency of the corresponding response representations across layers. The first row compares the relatively low-unification pair OmniGen2 and Qwen2.5-VL-3B-Instruct; the second row compares the more aggressively unified Show-o2 and its Qwen2.5-VL-7B-Instruct reference. Read row-wise, the figure asks how strongly unification training has altered the inherited backbone behavior.

Panel of Figure 6 — Figure 6: Final-layer UEval latent spaces under different unification regimes. Each point is a pooled representation from the model’s own multimodal trace; blue circles denote image spans and orange squares denote text spans. We report Cross-Modal Alignment Margin (CMAM) in each panel. Despite being the least unified by architectural design, OmniGen2 exhibits the strongest text–image alignment and the smallest effective modality gap. MMaDA and Show-o2, which impose more aggressive cross-modal and cross-task sharing, show substantially weaker paired alignment.

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 287 papers

Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation
Jun 25, 2026Jinyu Liu, Xincheng Shuai, Henghui Ding +1Unified Multimodal Models Multimodal Foundation Models
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China
FlexUMM: Flexible Representation Learning for Unified Multimodal Models
Oct 15, 2026Yinyi Luo, Haoyue Bai, Yi Ge +3Task Interference Unified Multimodal Models
Carnegie Mellon University · William & Mary · University of Maryland
A Model-Internal Protocol for Assessing Multimodal Models as Integrated Systems
Aug 12, 2026Hao Zhang, Jiaxin Qi, Zhijiang Tang +1Unified Multimodal Models Multimodal Reasoning
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Computer Network Information Center, Chinese Academy of Sciences

Jun 25, 2026J/K move · Enter open · S save

Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

Jinyu Liu, Xincheng Shuai, Henghui Ding, Yu-Gang Jiang

Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China

Unified Multimodal Models Multimodal Foundation Models Multimodal Model Evaluation Multimodal Learning Multimodal Understanding

Open paper AskPDF arXiv

Unified multimodal models capable of both understanding and generation have achieved remarkable strides. However, despite their unified designs, existing evaluations typically assess understanding and generation capabilities in isolation, overlooking the synergy between comprehension and generation. To bridge this gap, we introduce Unison, a comprehensive benchmark comprising 2,169 high-quality unified task samples, designed to evaluate joint understanding and generation in unified multimodal models. Unison offers three key strengths: 1) Comprehensive Dimensions: Unison encompasses internal consistency, understanding-guided generation, generation-guided understanding, and mutual enhancement to enable holistic evaluation. 2) Diagnostic Evaluation: it provides both unified and decoupled tracks for understanding and generation, allowing fine-grained attribution of failure modes and quantitative analysis of the gains from unified modeling. 3) Human Alignment: we also introduce Unison-Judge, an evaluation model well aligned with human judgments to ensure reliable assessment. Based on systematic evaluations of state-of-the-art models on Unison, we uncover critical limitations in current unified multimodal systems and highlight promising directions for future research. Codes, Unison and Unison-Judge are publicly available at https://github.com/FudanCVL/Unison.

Abstract

Unified Multimodal Models

Explore similar work

Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

FlexUMM: Flexible Representation Learning for Unified Multimodal Models

A Model-Internal Protocol for Assessing Multimodal Models as Integrated Systems