cs.CLFeb 2, 2026

Quantifying the Gap between Understanding and Generation within Unified Multimodal Models

Authors: Chenlong Wang, Yuhang Chen, Zhihan Hu, Dongping Chen, Wenhu Chen, Sarah Wiegreffe, Tianyi Zhou

Organizations: Independent Researcher · University of Maryland · University of Waterloo · MBZUAI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent advances in unified multimodal models (UMM) have demonstrated remarkable progress in both understanding and generation tasks. However, whether these two capabilities are genuinely aligned and integrated within a single model remains unclear. To investigate this question, we introduce GapEval, a bidirectional benchmark designed to quantify the gap between understanding and generation capabilities, and quantitatively measure the cognitive coherence of the two "unified" directions. Each question can be answered in both modalities (image and text), enabling a symmetric evaluation of a model's bidirectional inference capability and cross-modal consistency. Experiments reveal a persistent gap between the two directions across a wide range of UMMs with different architectures, suggesting that current models achieve only surface-level unification rather than deep cognitive convergence of the two. To further explore the underlying mechanism, we conduct an empirical study from the perspective of knowledge manipulation to illustrate the underlying limitations. Our findings indicate that knowledge within UMMs often remains disjoint. The capability emergence and knowledge across modalities are unsynchronized, paving the way for further exploration.

Figures & tables

BenchmarkSizeCategoryAnnotationUnd. TaskGen. TaskFeatures
WK.RS.VP.IFWKRSSYN.BI.GQ.
MMMU [ 67 ]11,550I2THuman✓✓✗✗✗✗✗✗✗
MMBench [ 28 ]3,217I2TMixed✓✓✓✗✗✗✗✗✗
GenEval [ 14 ]553T2IMixed✗✗✗✓✗✗✗✗✗
DPG-Bench [ 18 ]1,065T2I(M)LLM✗✗✗✓✗✗✗✗✗
T2I-CoReBench [ 25 ]1,080T2I(M)LLM✗✗✗✓✓✓✗✗✗
Table 1: Comparison of benchmarks adapted to UMMs. I2T : Image-to-Text. T2I : Text-to-Image. WK. : World Knowledge, RS : Reasoning, VP. : Visual Perception, IF. : Instruction Following. SYN. : Evaluate the synergy effect within UMMs. BI. : Bidirectional formulation. GQ : Gap Quantification between understanding and generation.
Figure 2 : Illustrative examples from GapEval . The generated texts and images are shown with the ground truth (texts, images, or both). GPT Family denotes GPT5-mini (for text generation), and GPT-Image-1 (for image generation).
ModelWorld KnowledgeNumerical PerceptionInstruction FollowingReasoningGap ↓
Succ.Und.Gen.Gap↓Succ.Und.Gen.Gap↓Succ.Und.Gen.Gap↓Succ.Und.Gen.Gap↓
Open-source UMM
Bagel52.2488.1758.5056.672.4017.008.4084.8746.3859.1575.7457.382.4935.675.3887.1471.52
OneCAT57.1488.2662.9233.142.759.009.7562.3323.6750.0041.2237.510.6620.661.5685.3654.73
UniWorld-V187.8493.2892.7412.601.0010.603.6084.6828.5162.1346.6070.461.1136.781.4486.1163.47
UniPic239.2979.6041.1665.231.7515.007.5082.3751.8666.7558.7745.697.8634.183.9382.6168.97
Table 2: Evaluation results across 9 UMMs and 6 non-UMMs on GapEval . Succ. : Success Score, denotes correctly answering with both image and text. Und. : Understanding Score, denoting the accuracy of answering with texts. Gen. : Generation Score, denoting the accuracy of answering with images. Fail denotes failure to answer in neither text nor image. Gap denotes the gap between understanding and generation. Bold & Underline : bests & second bests.
Figure 3 : Gap score heatmap over understanding and generation performance. (Und., Gen., Gap Score) data points are plotted on the heatmap, reflecting the relation between three dimensions. The trend curve demonstrates the model performance distribution.
Figure 4 : Training data case gallery.
Figure 5 : Performance increases over the training steps on the knowledge edit task. The figure also shows the output of models from different training stages. The knowledge entity involved here is (Microwave Oven->Rice Cooker).
Figure 6 : Model performance under different training strategies. Training on one side does not affect the other side.
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Judge ModelBagelOneCATUniWorld-V1UniPic2
Gemini3-Flash72.9660.6766.6473.39
GPT5-mini71.5254.7363.4768.97
Judge ModelShow-o2OmniGen2Gemini2.5-F-IGPT-Image-1
Gemini3-Flash72.3388.8762.6547.47
GPT5-mini67.8389.7462.9150.61
Appendix
Table 3: Gap Score Results from Different Judge Models.
Figure 7 : Correlation between Gap Score and Performance on GIR-Bench.
Prompt for Understanding on World Knowledge Task.
[Image]
[Reference_Image]
Here is the question: [Question]
Here is the answer: [Answer]
Please judge the correctness of the answer. You should follow the following rules:
1. It includes the core information present in the [Answer]
Appendix
Table 4 : Prompt for Understanding on World Knowledge Task.
Prompt for Generating on World Knowledge Task.
[Image]
[Reference_Image]
Here is the question: [Question]
Here is the answer: [Answer]
Please judge the correctness of the answer. You should follow the following rules:
1. Compare the generated image [Image] with the reference image [Reference_Image] and the caption [Question] , and decide whether the image should be judged as pass (score 1) or fail (score 0).
Appendix
Table 5 : Prompt for Generating on World Knowledge Task.
Prompt for Understanding on Reasoning Task.
[Image]
[Reference_Image]
Here is the question: [Question]
Here is the answer: [Answer]
Please judge the correctness of the answer. You should follow the following rules:
1. You are given a reasoning problem [Question] , an authoritative reference_answer (the expected result or phenomenon), and a model-generated answer [Answer] . Your goal is to determine whether the final outcome/result expressed in [Answer] matches the reference_answer.
Appendix
Table 6 : Prompt for Understanding on Reasoning Task.
Prompt for Generating on Reasoning Task.
[Image]
[Reference_Image]
Here is the question: [Question]
Here is the answer: [Answer]
Please judge the correctness of the answer. You should follow the following rules:
1. Compare [Image] with [Reference_Image] and [Answer] and check whether all required answer-relevant elements are present: the main result, key objects, and core information needed to visually answer the physics question posed by the problem. Major answer-relevant objects must not be missing or clearly misrepresented.
Appendix
Table 7 : Prompt for Generating on Reasoning Task.
Prompt for Understanding on Numerical Perception Task.
[Image]
[Reference_Image]
Here is the question: [Question]
Here is the answer: [Answer]
Please judge the correctness of the answer. You should follow the following rules:
1. Use the JSON task in [Question] (its "objects" and "number" fields) as the authoritative specification of which object types and exact counts are required. The final confirmed result stated in [Answer] must include only those specified object types, with counts that exactly match the JSON, and must not introduce any extra or non-specified objects.
Appendix
Table 8 : Prompt for Understanding on Numerical Perception Task.
Prompt for Generating on Numerical Perception Task.
[Image]
[Reference_Image]
Here is the question: [Question]
Here is the answer: [Answer]
Please judge the correctness of the answer. You should follow the following rules:
1. Use the JSON specification in [Question] (its "objects" list and corresponding "number" fields) as the exact target: [Image] passes (score = 1) only if every specified object type appears with exactly the required quantity and class, regardless of other non-target real objects that may be present.
Appendix
Table 9 : Prompt for Generating on Numerical Perception Task.
Prompt for Understanding on Instruction Following Task.
[Image]
[Reference_Image]
Here is the question: [Question]
Here is the answer: [Answer]
Please judge the correctness of the answer. You should follow the following rules:
1. From [Question] , understand the rule that modifies the image scenario and the reference text that concisely describes the expected result or core feature after this rule is applied. Identify the core aspect, feature, or outcome that must appear once the rule is in effect.
Appendix
Table 10 : Prompt for Understanding on Instruction Following Task.
Prompt for Generating on Instruction Following Task.
[Image]
[Reference_Image]
Here is the question: [Question]
Here is the answer: [Answer]
Please judge the correctness of the answer. You should follow the following rules:
1. From [Question] , fully understand the rule’s intent and logic, and how it is supposed to modify the original image (objects, features, or arrangements in the original scenario). Identify the core effect or result that must appear after the rule is applied.
Appendix
Table 11 : Prompt for Generating on Instruction Following Task.
Prompt for evalution of und task in edit and inject knoledge.
[Image]
Here is the question: [Question]
Here is the answer: [Answer]
Please judge the correctness of the answer. You should follow the following rules:
1. Ensure the subject described in the [Answer] matches the subject in the ground_truth (whether it’s an animal, object, person, etc.).
2. If the output_text and ground_truth both describe the basic features, position, state, or other relevant characteristics of the subject consistently, it is considered correct.
Appendix
Table 12 : Prompt for evaluation of und task in edit and inject knowledge.
Prompt for evaluation of gen task in edit knowledge.
[Image]
Here is the question: [Question]
Here is the answer: [Answer]
Please judge the correctness of the answer. You should follow the following rules:
1. Ensure the subject depicted in the [Image] is the same as the subject in the ground_truth (whether it’s an animal, object, person, etc.).
2. If the [Image] clearly depicts the same main subject as the ground_truth, even if there are variations in its state, expression, angle, or other minor details, it is considered correct.
Appendix
Table 13 : Prompt for evaluation of gen task in edit knowledge.
Figure 8 : Case of Instruction Following
Figure 9 : Case of World Knowledge
Figure 10 : Case of Reasoning

Explore similar work

CardsList

  1. Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation

    Jun 25, 2026Jinyu Liu, Xincheng Shuai, Henghui Ding +1Multimodal ModelSynergy

    Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China

  2. Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

    Sep 1, 2026Penghao Wu, Haiwen Diao, Weichen Fan +3Data Generation

    1S-Lab, Nanyang Technological University · 2SenseTime Research

  3. Transferability Between Understanding and Generation in Unified Multimodal Models

    Jul 5, 2026Jiwon Kang, Heeji Yoon, Jaewoo Jung +5

    KAIST AI, South Korea · Blynx, South Korea · Trillionlabs, South Korea