What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models

Tengju Ru; Weitong Lian; Zecong Tang; Zhiheng Liu; Lingyi Meng; Haoran Li; Kaixuan Wang; Bohao Li

cs.CVOct 14, 2026

What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models

Authors: Tengju Ru, Weitong Lian, Zecong Tang, Zhiheng Liu, Lingyi Meng, Haoran Li, Kaixuan Wang, Bohao Li,

Organizations: Zhejiang University · Shenzhen Loop Area Institute · The University of Hong Kong · The Chinese University of Hong Kong, Shenzhen

Abstract

Fine-grained visual understanding requires distinguishing subtle differences in local attributes and spatial relations, yet remains challenging for existing vision-language models. Unified multimodal models (UMMs) offer a new opportunity through generation pathways that preserve rich visual information complementary to understanding. However, existing generation-assisted understanding methods mainly rely on generative supervision or explicit generation and have not systematically explored direct use of native generation-side representations. To address this gap, we systematically analyze what generation-side representations offer for fine-grained understanding, finding that they are more sensitive to fine-grained semantic changes, capture task-relevant changes missed by understanding-side representations, and depend on correct spatial organization for this advantage. Guided by these findings, we design GAFU, a post-training framework that unlocks native generation-side information for fine-grained understanding. It compresses generation-side visual representations into compact $G$ tokens that preserve fine-grained visual details, enabling task-conditioned retrieval by the understanding pathway. Experiments show that GAFU consistently outperforms SFT baselines trained on the same data across diverse visual understanding tasks and UMM architectures, with maximum relative improvements of 20.00%, 17.94%, and 8.62% on VStar-Bench, OmniSpatial, and VSR-Bench, respectively. Overall, GAFU converts complementary information from the generation pathway into gains in fine-grained visual understanding, providing an effective path toward synergy between generation and understanding in UMMs.

Figures & tables

Figure 1: Control-corrected generation–understanding sensitivity gaps and strong-response complementarity across six models and four domains. (a) Solid bars show the unpermuted $G-U$ gap, while hatched bars show the gap after spatial shuffle. (b) Stacked bars show the fractions of samples with generation-side-only and understanding-side-only strong responses.

Figure 2: Overview of GAFU, comprising generation representation compression and alignment (Stage 1) and task-conditioned injection (Stage 2).

Type	Model	Size	VStar-Bench $\uparrow$	OmniSpatial $\uparrow$	MMVP $\uparrow$	PathVQA $\uparrow$	VSR-Bench $\uparrow$	Avg. $\uparrow$
VLM	InternVL3	1B	33.51	38.81	61.67	18.74	30.65	36.67
	InternVL3	2B	41.36	36.79	67.00	28.56	41.87	43.12
	Qwen3-VL	2B	45.03	30.99	68.33	29.94	47.65	44.39
	Qwen3-VL	4B	45.55	22.44	77.33	33.73	49.99	45.81
	BAGEL	14B	36.65	30.79	71.67	39.27	69.98	49.67
	BAGEL (SFT)	14B	39.27	31.90	74.00	51.42	71.39	53.60

Table 1: Visual understanding results.

Dimension	Setting	VStar-Bench $\uparrow$	OmniSpatial $\uparrow$	MMVP $\uparrow$
Alignment objective	w/o spatial	46.07	33.14	75.00
	w/o relation	45.55	34.44	74.33
	Reconstruction only	45.03	32.81	74.33
	Ours (full)	47.12	34.51	79.67
Injection interface	Token-level append	40.84	32.55	77.33
	Entry fusion	41.88	22.37	75.00

Table 2: Ablation results.

Figure 3: GAFU enhances fine-grained visual understanding. Examples show better recognition of fine-grained details, spatial relations, and domain-specific cues over the base model and SFT.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Domain	$P_{G\text{-only}}^{(d)}$	$P_{U\text{-only}}^{(d)}$	$\Delta_{\mathrm{comp}}^{(d)}$
BAGEL	attribute	32.99%	2.04%	$+30.95$ pp
BAGEL	object	28.37%	2.94%	$+25.43$ pp
BAGEL	count	9.04%	7.12%	$+1.92$ pp
BAGEL	relation	19.26%	0.84%	$+18.43$ pp
Janus-Pro	attribute	51.02%	1.36%	$+49.66$ pp
Janus-Pro	object	48.27%	0.52%	$+47.75$ pp

Appendix

Table 3: Strong-response complementarity between generation-side and understanding-side representations.

Model	Domain	128	256	512
BAGEL	attribute	0.391	0.334	0.468
BAGEL	object	0.158	0.149	0.303
BAGEL	count	0.063	0.102	0.148
BAGEL	relation	0.221	0.183	0.238
Janus-Pro	attribute	0.787	0.783	0.803
Janus-Pro	object	0.682	0.735	0.727

Appendix

Table 4: Resolution stability of the control-corrected generation–understanding response gap.

Model	Domain	$G$	$G_{\mathrm{shuffle}}$	$U$	$U_{\mathrm{shuffle}}$	$G-U$	$(G-U)_{\mathrm{shuffle}}$
BAGEL	attribute	0.764	0.454	0.297	0.339	0.468	0.116
BAGEL	object	0.667	0.402	0.364	0.296	0.303	0.106
BAGEL	count	0.473	0.466	0.325	0.380	0.148	0.086
BAGEL	relation	0.418	0.045	0.180	0.173	0.238	-0.128
Janus-Pro	attribute	1.095	0.264	0.293	0.157	0.803	0.107
Janus-Pro	object	1.088	0.236	0.361	0.172	0.727	0.063

Appendix

Table 5: Spatial dependence of control-corrected representation sensitivity.

Figure 4: Stage 1 reconstruction visualization. Each case compares the original image, the VAE upper bound obtained by decoding the native generation representation, and the reconstruction from the compressed $G$ tokens.

Figure 5: Fine-grained visual reasoning example from FineVision.

Figure 6: Fine-grained local spatial-relation example from VSR.

Figure 7: Fine-grained pathological example from PathVQA.

Figure 8: Stage 2 validation dynamics under different fine-tuning objectives. The three columns correspond to CE only, CE + DPO, and CE + DPO + KL, respectively. The first row reports the answer-token CE of the real and shuffle- $G$ branches, while the second row reports their CE gap, defined as $\mathrm{CE}_{\mathrm{shuffle}-G}-\mathrm{CE}_{\mathrm{real}}$ . The real branch uses matched $G$ tokens, whereas the shuffle- $G$ branch uses mismatched $G$ tokens from another sample. A positive CE gap indicates that the model assigns lower answer-token cross-entropy to matched $G$ tokens.

Multimodal Understanding

Multimodal understanding studies how AI systems interpret and reason across text, images, video, audio, and other signals.

See all 479 papers

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System
Sep 1, 2026Penghao Wu, Haiwen Diao, Weichen Fan +3Unified Multimodal Models Multimodal Generation
1S-Lab, Nanyang Technological University · 2SenseTime Research
When Does Visual Generation Help Visual Understanding in Unified Multimodal Models?
Aug 23, 2026Yubo Zhu, Zhehan Kan, Jingyi Yang +6Unified Multimodal Models Multimodal Understanding
Nanjing University · TongYi Lab · Tsinghua University +1
UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?
Mar 3, 2026Zimo Wen, Boxiu Li, Wanbo Zhang +11Unified Multimodal Models Multimodal Model Evaluation
Microsoft Research Asia · Shanghai Jiao Tong University · Fudan University +2

Sep 1, 2026J/K move · Enter open · S save

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

Penghao Wu, Haiwen Diao, Weichen Fan, Lewei Lu+2

1S-Lab, Nanyang Technological University · 2SenseTime Research

Unified Multimodal Models Multimodal Generation Multimodal Learning Multimodal Understanding

Open paper AskPDF arXiv

While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision--language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner--executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.

Abstract

Multimodal Understanding

Explore similar work

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

When Does Visual Generation Help Visual Understanding in Unified Multimodal Models?

UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?