Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities?
Authors: Hongbo Jiang, Jie Li, Yunhang Shen, Pingyang Dai, Xing Sun, Haoyu Cao, Liujuan Cao
Organizations: Tencent Youtu Lab, Shanghai, China · Xiamen University, Media Analytics and Computing Lab, Department of Artificial Intelligence, School of Informatics, Xiamen, China.
Unified Multimodal Large Language Models (U-MLLMs) integrate understanding and generation within a single architecture. However, existing evaluations typically assess these capabilities separately, overlooking semantic equivalence, i.e., the ability to manifest consistent reasoning results regardless of the output modality. In this work, we investigate whether current U-MLLMs satisfy this premise. We observe that while models demonstrate robust textual reasoning, they fail to maintain semantic equivalence when required to render the same results in the image modality. To rigorously diagnose this discrepancy, we introduce VGUBench, a framework to decouple reasoning logic from generation fidelity. VGUBench comprises three diagnostic tasks: (1)Textual Generative Understanding, establishing a baseline for reasoning accuracy in textual response; (2)Visual Generative Understanding, evaluating the ability to generate visual responses that represent the correct answer; and (3)a Visual Rendering control task, which assesses the ability to directly render explicit visual descriptions into images without complex reasoning. Our evaluation reveals a significant disparity: despite strong performance in textual understanding and visual rendering, U-MLLMs exhibit a marked performance collapse when required to generate visual answers to questions. Furthermore, we find a negligible correlation between visual answering performance and basic rendering quality. These results suggest that the failure stems not from insufficient generation fidelity, but from a breakdown in cross-modal semantic alignment. We provide diagnostic insights to address this challenge in future Unified Generation and Understanding Models.
Figures & tables
Figure 1: Provide U-MLLM with the same question and have it generate answers in text using its understanding capability and answer in image using its generation capability. Ultimately, U-MLLMs exhibited semantic non-equivalence across both output modalities. The images in the “Visual Answer” columns of this figure are all generated by AI models.
Figure 2: The pipeline of VGUBench construction. First, we randomly sampled and merged the existing 9 text-only question-answer benchmarks to obtain the TGU task test set. Subsequently, we automatically rendered the response texts from the TGU data into a unified format to generate the VGU task test set. Finally, we selected the response texts from TGU, removed duplicates, and applied the same rendering method. We then removed the question texts and used the response texts as input for the Render test set.
Figure 3: The relationship among the three tasks. In the figure, varying shades of orange represent the three distinct tasks, while dark blue denotes the test data. Black arrows originating from a task module indicate that the data serves as the ground-truth label for evaluation, whereas arrows pointing into a module signify the data as a task input. The VGU is formally similar to the TGU and Render tasks, but it is by no means a simple sum of the two.
Method
Legibility
Completeness
Correctness
Avg
VGU
TGU
Render
VGU
TGU
Render
VGU
TGU
Render
VGU
TGU
Render
Bagel
61.81
99.70
60.33
0.43
77.61
11.91
0.17
76.99
11.31
20.80
84.76
27.85
Emu3
35.82
96.04
42.59
0.17
47.53
2.83
0.43
46.37
2.80
12.14
63.31
16.07
BLIP3o
48.19
99.32
69.17
1.21
83.94
0.15
1.22
83.39
0.15
16.87
88.88
23.16
Janus-Pro
37.18
99.39
38.95
0.20
57.49
0.06
0.20
56.00
0.06
12.53
70.96
13.02
JanusFlow
40.22
97.13
38.59
0.33
50.58
0.00
0.33
49.38
0.00
13.63
65.70
12.86
Table 1: Performance comparison of different models across readability, completeness, correctness, and average metrics for the VGU, TGU, and Render tasks on the VGUBench test dataset. “ Avg ” denotes the average final score of a given model across three evaluation dimensions. In the table, bold, underlined, and italicized values represent the best, second-best, and third-best results for each column, respectively.
Method
Pearson
Spearmanr
r
pp
ρ
ps
Bagel
0.0347
0.1063
0.0368
0.0873
Emu3
0.0714
0.0009
-0.0686
0.0014
BLIP3o
-0.0137
0.5238
0.0156
0.4685
Janus-Pro
-0.0116
0.59
-0.0016
0.9394
JanusFlow
-0.018
0.9319
0.0112
0.6035
Table 2: Analysis of the relationship between Render and VGU tasks using Pearson and Spearman correlation coefficients.
Figure 4: Some inference cases of VGU task. All of this VGU inference sample is “Completeness < 2” and “Correctness > 2”. Despite the high scores achieved on the Correctness metric, the inference performance in VGU remains suboptimal. This deficiency is effectively captured and reflected by the lower scores in Completeness.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Inference Results of Different U-MLLMs on VGUBench. The images in the “VGU” and “Render” columns of this figure are all generated by AI models.
Figure 6: Trend of conditional expectation of VGU values across Render score bins. The red LOWESS curve illustrates how the average VGU value changes with Render scores, highlighting a non-linear relationship that global correlation metrics may not capture.
Unified multimodal models have recently demonstrated strong generative capabilities, yet whether and when generation improves understanding remains unclear. Existing benchmarks lack a systematic exploration of the specific tasks where generation facilitates understanding. To this end, we introduce UniG2U-Bench, a comprehensive benchmark categorizing generation-to-understanding (G2U) evaluation into 7 regimes and 30 subtasks, requiring varying degrees of implicit or explicit visual transformations. Extensive evaluation of over 30 models reveals three core findings: 1) Unified models generally underperform their base Vision-Language Models (VLMs), and Generate-then-Answer (GtA) inference typically degrades performance relative to direct inference. 2) Consistent enhancements emerge in spatial intelligence, visual illusions, or multi-round reasoning subtasks, where enhanced spatial and shape perception, as well as multi-step intermediate image states, prove beneficial. 3) Tasks with similar reasoning structures and models sharing architectures exhibit correlated behaviors, suggesting that generation-understanding coupling induces class-consistent inductive biases over tasks, pretraining data, and model architectures. These findings highlight the necessity for more diverse training data and novel paradigms to fully unlock the potential of unified multimodal modeling.
Zimo Wen, Boxiu Li, Wanbo Zhang +11
Microsoft Research Asia · Shanghai Jiao Tong University · Fudan University +2
Unified multimodal models (UMMs) aim to integrate multimodal understanding and generation within a unified architecture, yet it remains unclear to what extent textual and visual modalities are aligned. To investigate this question, we use reasoning-guided image generation as a diagnostic task, where models produce textual reasoning first and then generate images. We introduce UReason, a benchmark for evaluating reasoning-to-generation alignment in this paradigm, consisting of 2,000 human-curated and human-verified instances spanning five reasoning-intensive tasks: Code, Arithmetic, Spatial, Attribute, and Text. To enable controlled analysis, we develop an evaluation framework that compares direct generation, reasoning-guided generation, and decontextualized generation, which conditions only on the refined prompt extracted from reasoning. Across eight widely used open-source UMMs, while we find that reasoning-guided generation yields improvements over direct generation, somewhat surprisingly, decontextualized generation consistently outperforms reasoning-guided generation by a large margin. Our further analyses suggest that the intended visual semantics in textual reasoning are not reliably reflected in the generated images, despite their unified design and training. Overall, UReason serves as a practical litmus test for reasoning-to-generation alignment and provides a challenging benchmark for developing next-generation, more tightly aligned UMMs.
Cheng Yang, Chufan Shi, Bo Shui +7
University of California San Diego · University of Southern California · University of Illinois Urbana-Champaign +1
As Large Vision-Language Models increasingly aim to integrate visual generation and understanding within a single parameter space, evaluating such structural unification in a cohesive manner remains a critical challenge. Current evaluation protocols predominantly treat generative and discriminative capabilities as separate tasks, leaving a gap in system-level evaluation for unified multimodal models (UMMs). In this work, we propose Self-Generative-Understanding (SGU), a novel, annotation-free evaluation framework that probes the integrated capabilities of unified models through a semantic closed-loop challenge. Without requiring new annotations, SGU leverages the dual understanding-and-generation abilities of UMMs by asking them to first perceive an image and produce a textual description, subsequently reconstruct a visual context based on that description, and finally perform reasoning over the self-generated output. This pipeline provides a zero-cost testbed that yields an integrated performance score specifically tailored for evaluating UMMs as unified systems. Extensive experiments show that even high-performing UMMs often struggle to reason over their own generated contexts, revealing limitations that are not captured by separate evaluations of understanding or generation alone. Our work provides a complementary holistic evaluation framework and offers a foundation for benchmarking the development of next-generation unified multimodal models.
Hao Zhang, Jiaxin Qi, Zhijiang Tang +1
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · Computer Network Information Center, Chinese Academy of Sciences