Organizations: Shanghai AI Laboratory · CUHK MMLab · Fudan University · Shanghai Jiao Tong University · Tsinghua University · University of Science and Technology of China · South China University of Technology · Nanjing University · Xiamen University
Unified multimodal models (UMMs) that integrate understanding, reasoning, generation, and editing face inherent trade-offs between maintaining strong semantic comprehension and acquiring powerful generation capabilities. In this report, we present InternVL-U, a lightweight 4B-parameter UMM that democratizes these capabilities within a unified framework. Guided by the principles of unified contextual modeling and modality-specific modular design with decoupled visual representations, InternVL-U integrates a state-of-the-art Multimodal Large Language Model (MLLM) with a specialized MMDiT-based visual generation head. To further bridge the gap between aesthetic generation and high-level intelligence, we construct a comprehensive data synthesis pipeline targeting high-semantic-density tasks, such as text rendering and scientific reasoning, under a reasoning-centric paradigm that leverages Chain-of-Thought (CoT) to better align abstract user intent with fine-grained visual generation details. Extensive experiments demonstrate that InternVL-U achieves a superior performance - efficiency balance. Despite using only 4B parameters, it consistently outperforms unified baseline models with over 3x larger scales such as BAGEL (14B) on various generation and editing tasks, while retaining strong multimodal understanding and reasoning capabilities.
Figures & tables
Figure 1: Showcases of InternVL-U for general text-to-image generation (top) and image editing (bottom). InternVL-U supports high-fidelity image generation and editing at any resolution.
Figure 2: Showcases of InternVL-U for spatial-centric, perception, science-centric, humor-centric, and reasoning-centric text-to-image generation or editing tasks. InternVL-U demonstrates such core multimodal capabilities across various visual domains.
Figure 3: The architectural design of InternVL-U . The framework highlights three design principles: (1) unified contextual modeling supporting modality-adaptive generation targets, (2) structural efficiency via a unified backbone with modality-specific modular design, and (3) decoupled visual representations for understanding and generation tasks. Und. and Gen. denote Understanding and Generation, respectively.
Figure 4: Architecture of the Visual Generation Head . (a) Overview of the head with dual-stream MMDiT blocks. (b) Detailed structure of the Dual-Stream Attention Block and Dual-Stream FFN Block. (c) Illustration of the Unified MSRoPE (Multi-Scale Rotary Positional Embeddings) applied to VAE image latents and multimodal context embeddings.
Table 1: Overview of collected open-source datasets.
Figure 5: Examples of general data synthesized by our pipeline. The synthesized data features varied textual annotations and covers diverse visual domains, including portraits, posters, natural scenes, etc .
Figure 6: Overview of our general data synthesis pipeline. First, the preprocessing stage applies filtering, expansion, and deduplication to construct a high-quality source pool. Building upon this, two parallel branches are deployed to generate text-to-image pairs and instruction-guided editing data, respectively.
Figure 7: Three types of text-centric data synthesized by our pipeline. The first type overlays semantically relevant text onto natural images using a masked background image. The second type renders text on solid-color backgrounds, focusing on clean and aesthetically pleasing layouts. The third type performs text editing within existing images, such as modifying text on license plates, mobile interfaces, signboards, and similar surfaces.
Figure 8: Text rendering data construction pipeline. For synthetic text rendering, we prepare mask images, font colors, font styles, and adaptive layout options. During the rendering process, these attributes are randomly sampled, and the text is rendered with typography that adapts to its length.
Figure 9: Text editing data construction pipeline. First, we use an OCR tool to extract candidate text regions for editing. Second, we generate the editing instructions. Third, we use a generative model to produce the edited ground truth. Through these three steps, we synthesize high-quality text-editing triplets.
Figure 10: Science data generation pipeline. For general science T2I, we collect web images and open-source datasets and design automatic filtering and annotating with open-source models. For physics, we obtain images from documents by PaddleOCR, and propose an SVG-based pipeline for high-quality and affordable image-pair generation. For computer science, we define tasks and render images with Python libraries.
Figure 11: Examples of science-centric data. General science T2I features dense and detailed instructions and requires depicting multidisciplinary concepts. Physics and computer science focus on image editing with disciplinary reasoning.
Figure 12: Examples of spatial-centric data. We consider three spatial-centric scenarios: solid geometry ( e.g . , revolution, symmetry), multi-view CAD (three-view drawing), and spatial rotation of 3D objects.
Figure 13: Overview of our spatial rotation editing data synthesis pipeline. Stage 1 prepares a pool of filtered reference images. Subsequently, Stage 2 generates the final editing pairs via either the Object-First strategy for object-context integration, or the Background-First strategy for strict background preservation.
Figure 14: Meme data example for image generation and editing. The meme data here captures elements of everyday human humor and expressive nuances commonly found in daily life.
Figure 15: Meme data synthesis pipeline. It comprises five stages, leveraging chain-of-thought reasoning to process internet memes and synthesize high-quality training data for meme generation and editing.
Figure 16: Examples of CoT reasoning and enhancement for general image generation and editing . The original prompt is enriched through our chain-of-thought reasoning, adding finer-grained details that enable the model to perform generation and editing with greater accuracy and improved fidelity.
Figure 17: Examples of CoT reasoning and enhancement for knowledge-infused T2I and editing. After CoT reasoning, abstract knowledge concepts are concretized, enabling more precise generation and editing.
Figure 18: Examples of CoT reasoning and enhancement for Science T2I and Editing. Through CoT reasoning, scientific knowledge is injected into the generation process in a more explicit and detailed manner ( e.g . , how elements should be depicted and how operations should be executed).
Configuration
Visual Und. Encoder
Context Backbone
Visual Gen. Head
# Layers
24
28
20
# Num Heads (Q / KV)
16 / 16
16 / 8
12 / 12
Head Size
64
128
128
Intermediate Size
4,096
6,144
6,144
Patch / Scale Factor
14
-
2
# Parameters
0.3B
2B
1.7B
Table 2: Detailed configuration of InternVL-U architecture.
Hyperparameters
Stage 1
Stage 2
Stage 3
Trainable modules
Backbone
×
×
✓
Visual gen. head
✓
✓
✓
Learning rate
3e-04
1e-04
1e-05
LR scheduler
Constant
Cosine
Cosine
Weight decay
0.0
0.0
0.01
Table 3: Hyperparameters for different training stages .
Figure 19: System prompts and user prompts adopted during training for different tasks , where <img_uncond> is a learnable special token for image generation.
Model
#Params
Understanding
Reasoning
MME-P [ 38 ]
SEED [ 67 ]
ChartQA [ 94 ]
OCRBench [ 86 ]
MMMU [ 156 ]
MathVerse [ 158 ]
LogicVista [ 141 ]
MLLMs w/o Generator
LLaVA-1.5V [ 80 ]
7B
1510.7
65.8
17.8
31.8
35.7
7.6
–
Qwen2.5-VL [ 5 ]
3B
1574.9
73.7
84.0
79.7
53.1
31.2
40.3
InternVL3.5 [ 129 ]
2B
1552.1
75.3
80.7
83.6
59.0
53.4
47.7
UMMs w/ Generator
Table 4: Comparisons between InternVL-U and baseline models on multimodal understanding and reasoning benchmarks. * indicates the results based on our evaluation scripts. Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters.
Model
#Params
Single Object
Two Object
Counting
Colors
Position
Color Attribution
Overall
Generation Models
FLUX.1 [dev] [ 64 ]
12B
0.98
0.81
0.74
0.79
0.22
0.45
0.66
SD3-Medium [ 35 ]
2B
0.99
0.94
0.72
0.89
0.33
0.60
0.74
Seedream 3.0 [ 40 ]
-
0.99
0.96
0.91
0.93
0.47
0.80
0.84
GPT Image 1 [High] [ 99 ]
-
0.99
0.92
0.85
0.92
0.75
0.61
0.84
Z-Image [ 8 ]
6B
1.00
0.94
0.78
0.93
0.62
0.77
0.84
Table 5: Evaluation of general text-to-image generation ability on GenEval [ 44 ] . Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters.
Model
#Params
Global
Entity
Attribute
Relation
Other
Overall
Generation Models
FLUX.1 [dev] [ 64 ]
12B
82.10
89.50
88.80
91.10
89.40
84.00
SD3-Medium [ 35 ]
2B
87.90
91.01
88.83
80.70
88.68
84.08
GPT Image 1 [High] [ 99 ]
-
88.89
88.94
89.84
92.63
90.96
85.15
Nano Banana Pro [ 29 ]
-
91.00
92.85
91.56
92.39
89.93
87.16
Z-Image [ 8 ]
6B
93.39
91.22
93.16
92.22
91.52
88.14
Table 6: Evaluation of general text-to-image generation ability on DPG-Bench [ 53 ] . Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters.
Model
#Params
Basic Following
Advanced Following
Designer
Overall
Avg
Attr
Rel
Reas
Avg
A+R
A+Re
R+Re
Style
Text
RW
Generation Models
FLUX.1 [dev] [ 64 ]
12B
83.1
87.1
87.3
75.0
65.8
67.1
73.8
69.1
66.7
43.8
70.7
71.1
Z-Image [ 8 ]
6B
78.4
79.5
80.5
75.1
72.9
72.9
67.0
73.9
90.0
94.8
88.1
80.2
Seedream 3.0 [ 40 ]
-
87.1
90.5
89.6
80.9
79.2
79.8
77.2
75.6
100.0
97.2
83.2
86.0
Qwen-Image [ 136 ]
20B
86.2
90.5
88.2
79.8
79.3
79.2
78.9
75.6
100.0
92.8
90.3
86.1
Table 7: Evaluation of general text-to-image generation ability on TIIF [ 135 ] (Short Prompts). Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters. Abbreviations: Attr =Attribute, Rel =Relation, Reas =Reasoning, A+R =Attribute+Relation, A+Re =Attribute+Reasoning, R+Re =Relation+Reasoning, RW =Real World.
Model
#Params
Basic Following
Advanced Following
Designer
Overall
Avg
Attr
Rel
Reas
Avg
A+R
A+Re
R+Re
Style
Text
RW
Generation Models
FLUX.1 [dev] [ 64 ]
12B
78.7
83.2
80.4
72.4
68.5
73.7
73.3
71.6
66.7
52.8
71.5
71.8
Z-Image [ 8 ]
6B
82.8
86.5
79.9
81.9
77.0
77.6
73.8
75.6
93.3
93.2
85.5
83.0
Seedream 3.0 [ 40 ]
-
84.9
90.1
85.9
78.9
80.6
81.8
78.9
78.6
93.3
87.8
83.6
84.3
Qwen-Image [ 136 ]
20B
87.2
91.5
90.8
79.4
80.9
79.8
81.7
78.6
100.0
89.1
91.4
86.8
Table 8: Evaluation of general text-to-image generation ability on TIIF [ 135 ] (Long Prompts). Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters. Abbreviations: Attr =Attribute, Rel =Relation, Reas =Reasoning, A+R =Attribute+Relation, A+Re =Attribute+Reasoning, R+Re =Relation+Reasoning, RW =Real World.
Model
#Params
Alignment
Text
Reasoning
Style
Diversity
Overall
Generation Models
SDXL [ 35 ]
2.6B
0.69
0.03
0.24
0.33
0.30
0.32
FLUX.1 [dev] [ 64 ]
12B
0.79
0.52
0.25
0.37
0.24
0.43
Qwen-Image [ 136 ]
20B
0.88
0.89
0.31
0.42
0.18
0.54
Z-Image [ 8 ]
6B
0.88
0.99
0.28
0.39
0.19
0.55
Seedream 4.5 [ 7 ]
-
0.89
1.00
0.35
0.43
0.21
0.58
Table 9: Evaluation of general text-to-image generation ability on OneIG-EN [ 11 ] . Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters.
Model
#Params
Alignment
Text
Reasoning
Style
Diversity
Overall
Generation Models
Qwen-Image [ 136 ]
20B
0.83
0.96
0.27
0.41
0.21
0.53
Z-Image [ 8 ]
6B
0.79
0.99
0.27
0.39
0.24
0.54
Seedream 4.5 [ 7 ]
-
0.83
0.99
0.30
0.43
0.21
0.55
Nano Banana Pro [ 29 ]
-
0.84
0.98
0.31
0.46
0.24
0.57
Unified Models
Table 10: Evaluation of general text-to-image generation ability on OneIG-ZH [ 11 ] . Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters.
Figure 20: Visualizations of general image generation. Compared to other open-source models, InternVL-U demonstrates exceptional fidelity in rendering intricate textures and nuanced lighting effects, capturing the precise intent of each instruction.
Model
#Params
NED
CLIPScore
Word Accuracy
2 regions
3 regions
4 regions
5 regions
average
Generation Models
FLUX.1 [dev] [ 64 ]
12B
0.688
0.740
0.609
0.553
0.466
0.432
0.497
Nano Banana Pro [ 29 ]
-
0.875
0.737
0.737
0.775
0.786
0.793
0.779
Qwen-Image [ 136 ]
20B
0.912
0.802
0.837
0.836
0.831
0.816
0.829
Z-Image [ 8 ]
6B
0.937
0.797
0.901
0.872
0.865
0.851
0.867
Table 11: Evaluation of text-centric text-to-image generation ability on CVTG-2k [ 33 ] . Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters.
Model
#Params
LongText-Bench-EN
LongText-Bench-ZH
Generation Models
FLUX.1 [dev] [ 64 ]
12B
0.607
0.005
Z-Image [ 8 ]
6B
0.943
0.946
Qwen-Image [ 136 ]
20B
0.943
0.946
Nano Banana Pro [ 29 ]
-
0.981
0.949
Seedream 4.5 [ 7 ]
-
0.989
0.987
Table 12: Evaluation of text-centric text-to-image generation ability on LongText-Bench [ 42 ] . Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters.
Figure 21: Visualizations of text-centric image generation. The results show that our InternVL-U has excellent ability in rendering symbols of Chinese, English, and numerical formulas. Compared with the open-source unified multimodal models BAGEL and Ovis-U1, it has better rendering ability and comparable performance with Qwen Image, a large parameter and commercial closed source model Nano Banana Pro.
Model
#Params
Cultural
Time
Space
Biology
Physics
Chemistry
Overall
Generation Models
SD3-Medium [ 35 ]
2B
0.43
0.50
0.52
0.41
0.53
0.33
0.45
FLUX.1 [dev] [ 64 ]
12B
0.48
0.58
0.62
0.42
0.51
0.35
0.50
Qwen-Image [ 136 ]
20B
0.63
0.62
0.76
0.60
0.72
0.39
0.63
Unified Models
Janus-Pro [ 16 ]
7B
0.30
0.37
0.49
0.36
0.42
0.26
0.35
Table 13: Evaluation of knowledge-informed text-to-image generation ability on WISE [ 96 ] . Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters.
Model
#Params
Math
Phy
Chem
Bio
Geo
Comp
Eng
Econ
Music
Hist
Overall
Generation Models
FLUX.1 [dev] [ 64 ]
12B
12.2
14.4
12.5
22.8
36.4
11.0
14.0
9.2
21.3
21.7
17.6
HunyuanImage-3.0 [ 9 ]
-
17.0
17.2
18.8
18.7
30.4
15.5
16.9
11.7
23.9
20.4
19.1
Qwen-Image [ 136 ]
20B
18.9
26.3
15.3
32.1
49.6
18.9
32.0
20.3
23.4
38.6
27.5
Seedream 4.5 [ 7 ]
-
44.7
63.4
48.9
75.8
67.6
57.9
69.7
67.3
38.0
55.0
58.8
GPT-Image-1.5 [ 136 ]
-
65.8
85.4
78.1
91.9
92.5
75.8
86.4
85.5
70.8
90.9
82.3
Table 14: Evaluation of knowledge-informed text-to-image generation ability on GenExam [ 132 ] (Relaxed Scores). Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters. Abbreviations: Math =Mathematics, Phy =Physics, Chem =Chemistry, Bio =Biology, Geo =Geography, Comp =Computer Science, Eng =Engineering, Econ =Economics, Hist =History.
Figure 22: Visualizations of knowledge-informed image generation. InternVL-U exhibits superior capability in accurate knowledge rendering. By effectively integrating domain knowledge, our model produces visually faithful results for complex prompts, significantly outperforming baselines that lack specific world knowledge.
Model
#Params
Add
Adjust
Extract
Replace
Remove
Background
Style
Hybrid
Action
Overall
Generation Models
FLUX.1 Kontext [ 63 ]
12B
4.25
4.15
2.35
4.56
3.57
4.26
4.57
3.68
4.63
4.00
GPT-Image-1 [High] [ 99 ]
-
4.61
4.33
2.90
4.35
3.66
4.57
4.93
3.96
4.89
4.20
Qwen-Image-Edit [ 136 ]
20B
4.38
4.16
3.43
4.66
4.14
4.38
4.81
3.82
4.69
4.27
Z-Image-Edit [ 8 ]
6B
4.40
4.14
4.30
4.57
4.13
4.14
4.85
3.63
4.50
4.30
Unified Models
Table 15: Evaluation of general image editing ability on ImgEdit [ 153 ] . Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters.
Models
#Params
BC
CA
MM
MC
PB
ST
SA
SR
SRp
TM
TT
Avg/G_O
Generation Models
GPT Image 1 [ 99 ]
-
6.96
6.85
7.10
5.41
6.74
7.44
7.51
8.73
8.55
8.45
8.69
7.49
Qwen-Image-Edit [ 136 ]
20B
8.23
8.30
7.33
8.05
7.49
6.74
8.57
8.09
8.29
8.48
8.50
8.01
Unified Models
Lumina-DiMOO [ 145 ]
8B
3.43
4.27
3.08
2.77
4.74
5.19
4.44
3.80
4.38
2.68
4.20
3.91
Ovis-U1 [ 124 ]
2.4B+1.2B
7.49
6.88
6.21
4.79
5.98
6.46
7.49
7.25
7.27
4.48
6.31
6.42
Table 16: Evaluation of general image editing ability on GEdit-Bench [ 83 ] . Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters. Abbreviations: BC =Background Change, CA =Color Alteration, MM =Material Modification, MC =Motion Change, PB =Portrait Beautification, ST =Style Transfer, SA =Subject Addition, SR =Subject Removal, SRp =Subject Replacement, TM =Text Modification, TT =Tone Transfer.
Figure 23: Visualizations of general image editing. InternVL-U excels in generating realistic textures and styles while maintaining high fidelity to the original image’s lighting and structural details, demonstrating superior performance across various tasks compared to other open-source models.
Models
# Params
Real
Virtual
OA
OP
OR
F1
NED
CLIP
AES
OA
OP
OR
F1
NED
CLIP
AES
Generation Models
Qwen-Image-Edit [ 136 ]
20B
0.75
0.68
0.66
0.67
0.71
0.75
5.72
0.78
0.75
0.73
0.74
0.75
0.81
5.21
GPT-Image-1.5 [ 98 ]
-
0.74
0.69
0.67
0.68
0.68
0.75
5.78
0.73
0.72
0.71
0.71
0.70
0.80
5.28
Nano Banana Pro [ 29 ]
-
0.77
0.72
0.70
0.71
0.72
0.75
5.79
0.80
0.78
0.77
0.78
0.78
0.81
5.28
Unified Models
Table 17: Evaluation of text-centric image editing on TextEdit (Classic Metrics). Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters. “Real” refers to source images from real-world scene, while “Virtual” refers to images from virtual scene. Abbreviations: OA =OCR Accuracy, OP =OCR Precision, OR =OCR Recall, F1 =OCR F1-Score, NED =ROI-Aware NED, CLIP =CLIPScore, AES =Aesthetic Score. For detailed evaluation metric, please refer to Appendix Section A .
Models
# Params
Real
Virtual
TA
TP
SI
LR
VC
Avg
TA
TP
SI
LR
VC
Avg
Generation Models
Qwen-Image-Edit [ 136 ]
20B
0.92
0.82
0.75
0.57
0.80
0.77
0.57
0.79
0.92
0.80
0.77
0.77
GPT-Image-1.5 [ 98 ]
-
0.96
0.94
0.86
0.80
0.93
0.90
0.82
0.93
0.96
0.91
0.87
0.90
Nano Banana Pro [ 29 ]
-
0.96
0.95
0.85
0.88
0.93
0.91
0.87
0.92
0.96
0.94
0.89
0.92
Unified Models
Table 18: Evaluation of text-centric image editing on TextEdit (MLLM-based Metrics). Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters. “Real” refers to source images from real-world scene, while “Virtual” refers to images from virtual scene. Abbreviations: TA : Text Accuracy, TP : Text Preservation, SI : Scene Integrity, LR : Local Realism, VC : Visual Coherence, Avg : MLLM Overall Average. For detailed evaluation metric, please refer to Appendix Section A .
Figure 24: Visualizations of text-centric image editing . InternVL-U demonstrates more accurate and faithful text editing capabilities, while maintaining strong consistency for both textual and visual content outside the target editing region.
Models
#Params
Temporal
Causal
Spatial
Logical
Overall
IR
AC
VP
Generation Models
FLUX.1 Kontext [ 63 ]
12B
2.3
5.5
13.0
1.2
5.8
26.0
71.6
85.2
Qwen-Image-Edit [ 136 ]
20B
4.7
10.0
17.0
2.4
8.9
37.2
66.4
86.9
Seedream 4.0 [ 110 ]
-
12.9
12.2
11.0
7.1
10.8
58.9
67.4
91.2
Nano Banana Pro [ 29 ]
-
41.2
61.1
48.0
37.6
47.2
77.0
85.5
94.4
GPT-Image-1.5 [ 98 ]
-
54.1
60.0
62.0
21.2
50.0
69.7
92.5
94.9
Table 19: Evaluation of reasoning-informed image editing ability on RISEBench [ 161 ] . Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters. Abbreviations: IR =Instruction Reasoning, AC =Appearance Consistency, VP =Visual Plausibility.
Figure 25: Visualizations of reasoning-informed image editing. InternVL-U outperforms state-of-the-art models in handling complex prompts requiring multi-step reasoning. The results demonstrate our model’s superior ability to accurately interpret and execute diverse logical constraints, spanning temporal calculations for updating calendar dates, spatial and cultural understanding for poem placement, and precise algorithmic rules for binary search tree insertion.
Figure 26: Visualizations of more special image editing examples. InternVL-U demonstrates its advanced spatial reasoning and precise control in complex editing tasks. The results highlight the model’s superior ability to accurately identify graph properties (e.g., node degrees), generate humor-centric content with appropriate expressions, and execute precise 3D geometric transformations based on coordinate vectors, showing its broad applicability in specialized domains.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmarks
Type
Size
Human Filter
GT Ann.
Sub-class
Traditial Eval.
LLM Eval.
AnyText [ 122 ]
Text Generation
2,000
✗
✗
-
✓
✗
LongText [ 42 ]
Text Generation
320
✓
✗
8
✗
✓
CVTG-2K [ 33 ]
Text Generation
2,000
✓
✗
2
✓
✗
MARIO-Eval-edit [ 66 ]
Text Edit
4,000
✗
✗
-
✓
✗
TextEdit (Ours)
Text Edit
2,148
✓
✓
18
✓
✓
Appendix
Table 20: Key attributes comparison of open-source text generation or editing benchmarks.
Figure 27: Data Distribution of TextEdit benchmark.
Major
ID
Category (Mid)
ID
Specific Scene (Sub)
Count
Virtual Scenes
1.1
Poster Scenes
1.1.1
Activities & Promotions Posters
57
1.1.2
Product & Advertising Posters
74
1.1.3
Movie & Art Posters
107
1.2
Comic Scenes
1.2.1
Dialogue & Narration
65
1.2.2
Onomatopoeia & Special-effects Text
26
1.3
Slide / Presentation
1.3.1
Titles & Subtitles
71
Appendix
Table 21: Unified benchmark taxonomy with data statistics.
Category
Metric
What it Measures
Classic (Text-Centric)
OCR Accuracy
Maximum similarity between the generated text in the target region and the ground-truth string.
OCR Precision
Accuracy of background text preservation (penalizes hallucinated or incorrect background text).
OCR Recall
Completeness of background text preservation (penalizes missing background text).
OCR F1-Score
Harmonic mean of OCR Precision and OCR Recall.
ROI-Aware NED
Normalized Edit Distance specifically within the source text bounding box.
Classic (General)
CLIPScore
Semantic alignment between the edited image and the target caption.
Appendix
Table 22: Overview of evaluation metrics. We categorize metrics into objective (Text-Centric, General) and perceptual (MLLM-based) dimensions. Detailed formulations are provided in Section A.2.2 .
Figure 28: The system prompt template used for MLLM-based automated evaluation , covering the analyst persona, task definition, and the first four scoring dimensions (Q1–Q4).
Figure 29: The continuation of the evaluation prompt , detailing the final scoring dimension (Q5) and the strict JSON output schema required for parsing results.
Models
# Params
Real
Virtual
OA
OP
OR
F1
NED
CLIP
AES
OA
OP
OR
F1
NED
CLIP
AES
Generation Models
Qwen-Image-Edit [ 136 ]
20B
0.76
0.69
0.67
0.67
0.70
0.75
5.81
0.74
0.71
0.70
0.70
0.70
0.80
5.27
GPT-Image-1.5 [ 98 ]
-
0.72
0.68
0.66
0.67
0.67
0.75
5.85
0.68
0.69
0.68
0.68
0.65
0.80
5.32
Nano Banana Pro [ 29 ]
-
0.76
0.71
0.69
0.70
0.70
0.75
5.86
0.77
0.76
0.75
0.75
0.76
0.81
5.32
Unified Models
Appendix
Table 23: Evaluation of text-centric image editing on TextEdit MiniSet-500 (Classic Metrics). Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters. “Real” refers to source images from real-world scenes, while “Virtual” refers to images from synthetic virtual. Abbreviations: OA =OCR Accuracy, OP =OCR Precision, OR =OCR Recall, F1 =OCR F1-Score, NED =ROI-Aware NED, CLIP =CLIPScore, AES =Aesthetic Score.
Models
# Params
Real
Virtual
TA
TP
SI
LR
VC
Avg
TA
TP
SI
LR
VC
Avg
Generation Models
Qwen-Image-Edit [ 136 ]
20B
0.93
0.85
0.77
0.55
0.78
0.80
0.60
0.82
0.91
0.81
0.74
0.76
GPT-Image-1.5 [ 98 ]
-
0.97
0.94
0.86
0.79
0.92
0.91
0.85
0.93
0.95
0.92
0.83
0.88
Nano Banana Pro [ 29 ]
-
0.96
0.95
0.85
0.86
0.92
0.91
0.87
0.92
0.96
0.93
0.87
0.92
Unified Models
Appendix
Table 24: Evaluation of text-centric image editing on TextEdit MiniSet-500(MLLM-based Metrics). Sizes of unified models in “A + B” indicate separate understanding (A) and generation (B) parameters. “Real” refers to source images from real-world scenes, while “Virtual” refers to images from virtual scene. Abbreviations: TA : Text Accuracy, TP : Text Preservation, SI : Scene Integrity, LR : Local Realism, VC : Visual Coherence, Avg : MLLM Overall Average.
Unified models (UMs) hold promise for their ability to understand and generate content across heterogeneous modalities. Compared to merely generating visual content, the use of UMs for interleaved cross-modal reasoning is more promising and valuable, e.g., for solving understanding problems that require dense visual thinking, improving visual generation through self-reflection, or modeling visual dynamics of the physical world guided by stepwise action interventions. However, existing UMs necessitate pixel decoding as a bridge due to their disjoint visual representations for understanding and generation, which is both ineffective and inefficient. In this paper, we introduce LatentUM, a novel unified model that represents all modalities within a shared semantic latent space, eliminating the need for pixel-space mediation between visual understanding and generation. This design naturally enables flexible interleaved cross-modal reasoning and generation. Beyond improved computational efficiency, the shared representation substantially alleviates codec bias and strengthens cross-modal alignment, allowing LatentUM to achieve state-of-the-art performance on the Visual Spatial Planning benchmark, push the limits of visual generation through self-reflection, and support world modeling by predicting future visual states within the shared semantic latent space.
Jiachun Jin, Zetong Zhou, Xiao Yang +4
Shanghai Jiao Tong University · Tsinghua University · UCSD
While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision--language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner--executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.
Penghao Wu, Haiwen Diao, Weichen Fan +3
1S-Lab, Nanyang Technological University · 2SenseTime Research
Unified multimodal models are envisioned to bridge the gap between understanding and generation. Yet, to achieve competitive performance, state-of-the-art models adopt largely decoupled understanding and generation components. This design, while effective for individual tasks, weakens the connection required for mutual enhancement, leaving the potential synergy empirically uncertain. We propose to explicitly restore this synergy by introducing Understanding-Oriented Post-Training (UNO), a lightweight framework that treats understanding not only as a distinct task, but also a direct supervisory signal to steer generative representations. By incorporating objectives that encode semantic abstraction (captioning) and structural details (visual regression), we enable effective gradient flow from understanding to generation. Extensive experiments on image generation and editing demonstrate that understanding can serve as an effective catalyst for generation.
Zeyu Liu, Zanlin Ni, Yang Yue +5
1Tsinghua University · 2Kolors Team, Kuaishou Technology