Fine-grained visual understanding requires distinguishing subtle differences in local attributes and spatial relations, yet remains challenging for existing vision-language models. Unified multimodal models (UMMs) offer a new opportunity through generation pathways that preserve rich visual information complementary to understanding. However, existing generation-assisted understanding methods mainly rely on generative supervision or explicit generation and have not systematically explored direct use of native generation-side representations. To address this gap, we systematically analyze what generation-side representations offer for fine-grained understanding, finding that they are more sensitive to fine-grained semantic changes, capture task-relevant changes missed by understanding-side representations, and depend on correct spatial organization for this advantage. Guided by these findings, we design GAFU, a post-training framework that unlocks native generation-side information for fine-grained understanding. It compresses generation-side visual representations into compact G tokens that preserve fine-grained visual details, enabling task-conditioned retrieval by the understanding pathway. Experiments show that GAFU consistently outperforms SFT baselines trained on the same data across diverse visual understanding tasks and UMM architectures, with maximum relative improvements of 20.00%, 17.94%, and 8.62% on VStar-Bench, OmniSpatial, and VSR-Bench, respectively. Overall, GAFU converts complementary information from the generation pathway into gains in fine-grained visual understanding, providing an effective path toward synergy between generation and understanding in UMMs.
Figures & tables
Figure 1: Control-corrected generation–understanding sensitivity gaps and strong-response complementarity across six models and four domains. (a) Solid bars show the unpermuted G−U gap, while hatched bars show the gap after spatial shuffle. (b) Stacked bars show the fractions of samples with generation-side-only and understanding-side-only strong responses.
Figure 2: Overview of GAFU, comprising generation representation compression and alignment (Stage 1) and task-conditioned injection (Stage 2).
Type
Model
Size
VStar-Bench ↑
OmniSpatial ↑
MMVP ↑
PathVQA ↑
VSR-Bench ↑
Avg. ↑
VLM
InternVL3
1B
33.51
38.81
61.67
18.74
30.65
36.67
InternVL3
2B
41.36
36.79
67.00
28.56
41.87
43.12
Qwen3-VL
2B
45.03
30.99
68.33
29.94
47.65
44.39
Qwen3-VL
4B
45.55
22.44
77.33
33.73
49.99
45.81
BAGEL
14B
36.65
30.79
71.67
39.27
69.98
49.67
BAGEL (SFT)
14B
39.27
31.90
74.00
51.42
71.39
53.60
Table 1: Visual understanding results.
Dimension
Setting
VStar-Bench ↑
OmniSpatial ↑
MMVP ↑
Alignment objective
w/o spatial
46.07
33.14
75.00
w/o relation
45.55
34.44
74.33
Reconstruction only
45.03
32.81
74.33
Ours (full)
47.12
34.51
79.67
Injection interface
Token-level append
40.84
32.55
77.33
Entry fusion
41.88
22.37
75.00
Table 2: Ablation results.
Figure 3: GAFU enhances fine-grained visual understanding. Examples show better recognition of fine-grained details, spatial relations, and domain-specific cues over the base model and SFT.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Domain
PG-only(d)
PU-only(d)
Δcomp(d)
BAGEL
attribute
32.99%
2.04%
+30.95 pp
BAGEL
object
28.37%
2.94%
+25.43 pp
BAGEL
count
9.04%
7.12%
+1.92 pp
BAGEL
relation
19.26%
0.84%
+18.43 pp
Janus-Pro
attribute
51.02%
1.36%
+49.66 pp
Janus-Pro
object
48.27%
0.52%
+47.75 pp
Appendix
Table 3: Strong-response complementarity between generation-side and understanding-side representations.
Model
Domain
128
256
512
BAGEL
attribute
0.391
0.334
0.468
BAGEL
object
0.158
0.149
0.303
BAGEL
count
0.063
0.102
0.148
BAGEL
relation
0.221
0.183
0.238
Janus-Pro
attribute
0.787
0.783
0.803
Janus-Pro
object
0.682
0.735
0.727
Appendix
Table 4: Resolution stability of the control-corrected generation–understanding response gap.
Model
Domain
G
Gshuffle
U
Ushuffle
G−U
(G−U)shuffle
BAGEL
attribute
0.764
0.454
0.297
0.339
0.468
0.116
BAGEL
object
0.667
0.402
0.364
0.296
0.303
0.106
BAGEL
count
0.473
0.466
0.325
0.380
0.148
0.086
BAGEL
relation
0.418
0.045
0.180
0.173
0.238
-0.128
Janus-Pro
attribute
1.095
0.264
0.293
0.157
0.803
0.107
Janus-Pro
object
1.088
0.236
0.361
0.172
0.727
0.063
Appendix
Table 5: Spatial dependence of control-corrected representation sensitivity.
Figure 4: Stage 1 reconstruction visualization. Each case compares the original image, the VAE upper bound obtained by decoding the native generation representation, and the reconstruction from the compressed G tokens.
Figure 5: Fine-grained visual reasoning example from FineVision.
Figure 6: Fine-grained local spatial-relation example from VSR.
Figure 7: Fine-grained pathological example from PathVQA.
Figure 8: Stage 2 validation dynamics under different fine-tuning objectives. The three columns correspond to CE only, CE + DPO, and CE + DPO + KL, respectively. The first row reports the answer-token CE of the real and shuffle- G branches, while the second row reports their CE gap, defined as CEshuffle−G−CEreal . The real branch uses matched G tokens, whereas the shuffle- G branch uses mismatched G tokens from another sample. A positive CE gap indicates that the model assigns lower answer-token cross-entropy to matched G tokens.
While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision--language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner--executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.
Penghao Wu, Haiwen Diao, Weichen Fan +3
1S-Lab, Nanyang Technological University · 2SenseTime Research
Unified multimodal models (UMMs) can perform both understanding and generation, raising a central question: can visual generation improve understanding? Existing evaluations provide mixed evidence, but confound task difficulty, reasoning paradigms, and the closed-loop interaction between generation and understanding. We introduce VGAU-Diag, a fine-grained evaluation framework for vision generation-assisted understanding. It stratifies samples by difficulty, enables unified evaluation of multiple reasoning paradigms, and uses Oracle-Assisted Reference Protocols. Our analysis shows that generated visual aids help on easier instances but become unreliable as reasoning complexity increases. Oracle-assisted diagnosis further reveals that the main bottleneck often lies on the visual-understanding side rather than the visual-generation side, as current UMMs struggle to leverage even faithful visual aids. We also show that effective visual generation should target visual-understanding bottlenecks rather than add more reasoning steps, and identify a three-stage transition from task-irrelevant noise, to misleading plausible guidance, and finally to useful assistance. These findings would be useful to guide the development of better UMMs.
Yubo Zhu, Zhehan Kan, Jingyi Yang +6
Nanjing University · TongYi Lab · Tsinghua University +1
Unified multimodal models have recently demonstrated strong generative capabilities, yet whether and when generation improves understanding remains unclear. Existing benchmarks lack a systematic exploration of the specific tasks where generation facilitates understanding. To this end, we introduce UniG2U-Bench, a comprehensive benchmark categorizing generation-to-understanding (G2U) evaluation into 7 regimes and 30 subtasks, requiring varying degrees of implicit or explicit visual transformations. Extensive evaluation of over 30 models reveals three core findings: 1) Unified models generally underperform their base Vision-Language Models (VLMs), and Generate-then-Answer (GtA) inference typically degrades performance relative to direct inference. 2) Consistent enhancements emerge in spatial intelligence, visual illusions, or multi-round reasoning subtasks, where enhanced spatial and shape perception, as well as multi-step intermediate image states, prove beneficial. 3) Tasks with similar reasoning structures and models sharing architectures exhibit correlated behaviors, suggesting that generation-understanding coupling induces class-consistent inductive biases over tasks, pretraining data, and model architectures. These findings highlight the necessity for more diverse training data and novel paradigms to fully unlock the potential of unified multimodal modeling.
Zimo Wen, Boxiu Li, Wanbo Zhang +11
Microsoft Research Asia · Shanghai Jiao Tong University · Fudan University +2