Can Visual Generation Expand What Unified Multimodal Models Know?

Wenkai Li; Wenyi Hong; Zhuoyi Yang; Jie Tang

cs.CVOct 15, 2026

Can Visual Generation Expand What Unified Multimodal Models Know?

Authors: Wenkai Li, Wenyi Hong, Zhuoyi Yang, Jie Tang

Organizations: Tsinghua University

Abstract

Unified multimodal models (UMMs) receive image-generation supervision that exposes them to visual information not fully documented in text. Can learning to generate images expand what pretrained models know? We introduce Knowledge Transfer Probing (KTP), a controlled protocol that supplies new knowledge through image-generation training and tests its accessibility through text-only question answering. Across seven knowledge categories and three UMMs, we observe direct but uneven transfer from visual supervision to language. Motivated by these observations, we successively introduce Textual Alignment and Interleaved Image--Text Alignment, which connect generation and answering through shared knowledge and interleaved responses, respectively. Guidance on selected entities can improve knowledge access on other entities without QA supervision, with additional gains from interleaved alignment. Further analysis shows that knowledge correctly reproduced in images is often more accessible through text, although substantial gaps remain between the two forms of access. These findings support visual generation as a source of knowledge beyond textual supervision, and motivate joint investigation of training data design and model architecture to make this knowledge more broadly accessible.

Figures & tables

Figure 1: Can visual generation expand what unified multimodal models know? Left: Images contain rich visual information that accompanying text may leave unspecified. Middle: Knowledge reproduced in generated images is not necessarily accessible through text-only QA. Right: We first supply new facts only through training images and test text-only QA across seven knowledge categories. Direct transfer is partial; alignment on a subset of entities can improve access on the remaining entities, with interleaved image–text alignment yielding higher average accuracy than textual alignment.

Figure 2: Overview of the text-to-image dataset. Here we show examples from the seven visual knowledge categories. Each panel shows an entity, a training image, and the target attribute value.

Model	Metric	color	count	number	shape	size	spatial	text	Avg
Baseline	random guessing	16.7	25.0	0.0	20.0	50.0	0.0	0.0	15.96
BAGEL	textual accessibility	14.0	14.0	0.0	10.0	24.0	0.0	0.0	8.86
BAGEL	visual accessibility	20.0	42.0	4.0	58.0	60.0	0.0	36.0	31.43
GLM-Omni	textual accessibility	16.0	24.0	2.0	20.0	40.0	0.0	0.0	14.57
GLM-Omni	visual accessibility	70.0	46.0	72.0	88.0	24.0	62.0	16.0	54.00
SenseNova-U1	textual accessibility	20.0	32.0	0.0	20.0	68.0	0.0	0.0	20.00

Table 1: Knowledge accessibility under direct transfer among 7 categories. Target knowledge is supplied through T2I supervision without target-specific QA supervision. Textual and visual accessibility measure correctness in text-only QA and image generation, respectively.

Figure 3: Visual and textual accessibility after T2I training. Target knowledge is supplied only through training images. Filled blue circles show visual accessibility; hollow red circles show textual accessibility. The accessibility gap

\Delta=A_{\mathrm{vis}}-A_{\mathrm{text}}

is reported in percentage points.

Figure 4: Overview of the KTP training and evaluation protocol. (a) Training: All entities receive T2I supervision, with target attribute values shown in images but omitted from captions. A subset of entities additionally receives textual or interleaved image–text alignment supervision. (b) Evaluation: For entities without alignment supervision, we measure textual accessibility through text-only QA and visual accessibility through image generation.

Model	Training Method	Color	Count	Number	Shape	Size	Spatial	Avg
BAGEL	direct	14.0	14.0	0.0	10.0	24.0	0.0	8.86
	textual alignment	4.0	28.0	0.0	24.0	52.0	0.0	15.43
	interleaved alignment	12.0	32.0	0.0	20.0	56.0	0.0	17.14
GLM-Omni	direct	16.0	24.0	2.0	20.0	40.0	0.0	14.57
	textual alignment	16.0	20.0	0.0	12.0	40.0	0.0	12.57
	interleaved alignment	24.0	20.0	0.0	64.0	40.0	40.0	26.86

Table 2: Textual accessibility (%) under three training conditions. “interleaved alignment” indicates interleaved image-text alignment; “direct” indicates direct knowledge transfer. All conditions are evaluated using text-only QA.

Figure 5: Image-loss ablation in interleaved alignment on GLM-Omni. Masking the image loss retains the interleaved sequences and text supervision but substantially lowers performance.

Model	Inference	color	count	number	shape	size	spatial	Avg
BAGEL	Text-only	12.0	32.0	0.0	20.0	56.0	0.0	17.14
BAGEL	Visual CoT	16.0	20.0	4.0	56.0	48.0	0.0	20.57
GLM-Omni	Text-only	24.0	20.0	0.0	64.0	40.0	40.0	26.86
GLM-Omni	Visual CoT	24.0	48.0	36.0	68.0	48.0	64.0	41.14
SenseNova-U1	Text-only	4.0	16.0	0.0	84.0	96.0	0.0	28.57
SenseNova-U1	Visual CoT	8.0	20.0	20.0	92.0	96.0	8.0	34.86

Table 3: Accuracy of text-only QA and Visual CoT (%).

Model	Training	$\Pr(T\mid I)$ (%)	$\Pr(T\mid\neg I)$ (%)	$\Delta_{\mathrm{assoc}}$ (pp)
BAGEL	Textual Alignment	23.91	12.40	$+11.51$
BAGEL	Interleaved Alignment	32.61	11.63	$+20.98$
GLM-Omni	Textual Alignment	6.90	18.18	$-11.28$
GLM-Omni	Interleaved Alignment	31.25	23.16	$+8.09$
SenseNova-U1	Textual Alignment	46.39	5.13	$+41.26$
SenseNova-U1	Interleaved Alignment	44.00	8.00	$+36.00$

Table 4: Conditional T2T accuracy given T2I correctness.

T

and

I

denote correct T2T answers and correct T2I generations, respectively. Gap is measured in percentage points (pp).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Attribute	Knowledge	Value range	Images
Color	The color of a fictional font	blue, green, black, red, white, yellow	1,226
Count	The count of a fictional collection	2, 3, 4, 5	1,195
Shape	The shape of a fictional badge	circle, five-pointed star, pentagon, square, triangle	1,248
Spatial	The stripe arrangement of a fictional flag	—	1,220
Size	The larger country on a fictional map	—	1,218
Number	The number of a fictional person’s jersey	0–99	1,244

Appendix

Table 5: The knowledge of entity, the range of attribute value, and image counts for the seven knowledge categories.

Model	Image generation	Textual alignment	Interleaved alignment
BAGEL	0.956	0.022	0.022
GLM-Omni	0.990	0.005	0.005
SenseNova-U1	0.851	0.063	0.086

Appendix

Table 6: Data sampling ratios when training with all three data sources.

Hyperparameter	BAGEL	SenseNova-U1	GLM-Omni
Learning rate	$1.0\times 10^{-5}$	$1.0\times 10^{-5}$	$1.0\times 10^{-5}$
LR scheduler	constant with linear warm-up	constant	constant with linear warm-up
Weight decay	0	0	0.1
Gradient clipping norm	1.0	1.0	1.0
Warm-up steps	100	0	100
Training steps	4000	1500	1500

Appendix

Table 7: Training hyperparameters for BAGEL, SenseNova-U1, and GLM-Omni.

Figure 6: An example of an entity with visually accessible but textually inaccessible knowledge.

Figure 7: An example of an entity with textually inaccessible knowledge that becomes accessible with visual CoT.

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 282 papers

Transferability Between Understanding and Generation in Unified Multimodal Models
Jul 5, 2026Jiwon Kang, Heeji Yoon, Jaewoo Jung +5Unified Multimodal Models Multimodal Generation
KAIST AI, South Korea · Blynx, South Korea · Trillionlabs, South Korea
UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models
Feb 9, 2026Cheng Yang, Chufan Shi, Bo Shui +7Unified Multimodal Models Image Generation
University of California San Diego · University of Southern California · University of Illinois Urbana-Champaign +1
Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMs
May 30, 2026Xin Gao, Cheng Yang, Chufan Shi +1Knowledge Editing Unified Multimodal Models
University of California San Diego · University of Southern California

Jul 5, 2026J/K move · Enter open · S save

Transferability Between Understanding and Generation in Unified Multimodal Models

Jiwon Kang, Heeji Yoon, Jaewoo Jung, Jaewon Min+4

KAIST AI, South Korea · Blynx, South Korea · Trillionlabs, South Korea

Unified Multimodal Models Multimodal Generation Transfer Learning Multimodal Understanding

Open paper AskPDF arXiv

Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate

\boldsymbol{\mathsf{transferability}}

in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.

Abstract

Unified Multimodal Models

Explore similar work

Transferability Between Understanding and Generation in Unified Multimodal Models

UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models

Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMs