Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models

Zongyang Qiu; Yihan Wu; Kaixuan Fan; Bo Li; Hui Xiong

cs.CVAug 18, 2026

Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models

Authors: Zongyang Qiu, Yihan Wu, Kaixuan Fan, Bo Li, Hui Xiong

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Columbia University · MMLab, CUHK · University of Illinois Urbana-Champaign

Abstract

Unified multimodal models (UMMs) are motivated by the hope that understanding and generation reinforce each other but controlled ablations repeatedly find that adding a generation objective leaves understanding flat. Joint-training studies cannot settle the disagreement: with overlapping supervision, a gain cannot be attributed to the architecture rather than the data. To further investigate the relationship between the two directions in UMMs, we separate them by construction. A novel visual entity, a rendered 3D asset paired with a pseudo-word screened for absence from the frozen model's behavior, is bound through exactly one task direction, and the untrained direction is then measured. We find that the channel is real in both directions, but the directions differ in kind: generation training installs a name the model can only match among candidates; understanding training installs one it can also produce. What governs cross-task usability is where the binding enters the shared computation. An alignment probe predicts export across 36 configurations (Spearman $ρ= +0.68$ ). That objective's alignment term, maximized in closed form over activations with every weight frozen, makes a concept drawable when injected at layer 7 of 28 and is indistinguishable from the base model from layer 14 on, while the weight-based version of the same edit peaks at layers 10-14. In an observational series of four models, this window appears only where the understanding pathway is a semantic vision encoder, suggesting that unified weights are not enough: the two directions must share a semantic format at the entry point. Exploiting the rule, a mid-stack alignment objective acquires the concept for a $0.1\%$ relative loss of the model's general text-to-image ability, against $41\%$ for the standard generative route. Our code is at https://github.com/Zane-ZYQiu/entry-point-umm.

Figures & tables

Figure 1: What the model draws for “a photo of a {name}”. Rows are four novel concepts and the reference column shows a view of the asset each pseudo-word was bound to. On the left side, the base model has never seen the pseudo-word; a placebo that supplies another concept’s address at identical magnitude draws the wrong object; and a closed-form activation edit applied at layer 14 leaves the base model’s output untouched. On the right the binding lands inside the window: the same edit at layer 7 with every weight frozen; training the understanding direction alone; the standard generative route; and the proposed objective, an alignment loss at layer 14 with no generative gradient, for a $0.1\%$ relative loss of general text-to-image ability (§ 7 ). Where the binding enters, not how it is optimized, is what decides.

Figure 2: The contamination-free protocol. A novel entity (left) is bound through exactly one task direction, (a) generation or (b) understanding, and the opposite direction, which received no gradient and no datum, is then probed; the model’s shared blocks are the only path between the two.

condition	und. name matching $\uparrow$	gen. identity $\uparrow$
base model	0.243 [0.19, 0.31] $\pm$ 0.058	0.039 [0.02, 0.06] $\pm$ 0.032
G-inject (flow matching), shared+private	0.520 [0.44, 0.60] $\pm$ 0.121	0.653 [0.56, 0.75] $\pm$ 0.125
G-inject, shared MLP only	0.620 [0.53, 0.70] $\pm$ 0.119	0.717 [0.62, 0.81] $\pm$ 0.122
U-inject (LM CE), shared+private	0.989 [0.98, 1.00] $\pm$ 0.014	0.360 [0.27, 0.46] $\pm$ 0.105
U-inject, shared attention only	0.984 [0.97, 0.99] $\pm$ 0.013	0.078 [0.04, 0.13] $\pm$ 0.069
U-inject, shared MLP only	0.991 [0.98, 1.00] $\pm$ 0.013	0.339 [0.25, 0.44] $\pm$ 0.165

Table 1: Cross-task transfer, 56 concepts in seven groups, 480 steps in both directions. Both columns are accuracy (

\uparrow

); brackets are bootstraps over all 56 concepts and

\pm

is the standard deviation of the seven group means, the run-level spread a concept bootstrap cannot see. Blocks from top to bottom: the base model; generative injection and its substrate variant; understanding injection and its substrate variants; the mid-stack anchoring objective alone (§ 5.1 ) at two depths; and the combination of the two recipes.

condition	LM-CE	4-way	reverse 4-way	context-free	first sub-word
U-inject (LM cross-entropy)	✓	1.000	1.000	0.938	0.922
U-inject, shared attention only	✓	1.000	0.900	0.297	0.688
base model	✗	0.254	0.242	0.141	0.125
G-inject (flow matching)	✗	0.600	0.650	0.141	0.125
anchoring @ layer 14, alone	✗	0.887	1.000	0.125	0.094
activation patch @ layer 3	✗	0.887	1.000	0.234	0.125

Table 2: Matching the name against producing it, for seven conditions spanning both training objectives. All columns are accuracy (

\uparrow

); chance is

0.250

for the 4-way probes and

0.125

for the context-free ones. The last column, first sub-word, is the pre-specified arbiter, immune by construction to the shortcut the probe detects. Rows are grouped by whether the recipe ever computed a language-model cross-entropy.

entry point	shared computation after it	exports	§
input embeddings	all 28 blocks, but nothing to route yet	✗	5.2
activation edit, layer 3	25 blocks	partial (0.53)	5.2
activation edit, layer 7	21 blocks	✓ (0.58)	5.2
activation edit, layer 14	14 blocks	✗ (0.04, at base)	5.2
weight edit, layer 3	25 blocks	partial (0.60)	5.2
weight edit, layers 10–14	14–18 blocks	✓ (0.80)	5.2

Table 3: The entry points this paper compares, placed on one depth axis; the depth sweep behind the middle two blocks is finer than the rows shown and is given in full in Table 4 . Each row puts the same binding at a different point of the shared computation; blocks are ordered by intervention medium, and by depth within each. Export is accuracy on the task direction that received no gradient (§ 3.2 ); the last column points to where each row is measured.

Figure 3: SAR against export across 36 configurations, and the intervention on SAR (arrow): anchoring applied at the readout moves the probe to a perfect

1.000

and leaves export where it was. A quantity can order configurations well and still not be the thing to change.

Figure 4: Export against entry-point depth, for the same anchoring objective at the same site, differing only in whether activations (blue) or weights (purple) may change. Left is export on the axis that cannot be gamed; bands are group-level bootstraps over all seven groups. The orange curve adds a flow-matching gradient and is one group, drawn dashed. Both anchor-only media give a window, and the weight medium’s reaches deeper.

entry point	identity (DINOv2)	(CLIP)	sd	vs. norm	identity	vs. norm
	activations, anchor only (0 gradient steps)				weights, anchor only (480 steps)
layer 0	0.323 [0.22, 0.42]	0.269	0.145	$+0.283$ 7/7	—	—
layer 3	0.528 [0.33, 0.72]	0.433	0.297	$+0.489$ 6/7	0.602 [0.47, 0.75]	$+0.499$ 7/7
layer 7	0.584 [0.46, 0.72]	0.512	0.196	$+0.545$ 7/7	0.769 [0.69, 0.84]	$+0.666$ 7/7
layer 10	0.366 [0.28, 0.45]	0.281	0.127	$+0.327$ 7/7	0.800 [0.70, 0.89]	$+0.698$ 7/7
layer 14	0.040 [0.02, 0.07]	0.054	0.036	$+0.001$ 3/7	0.808 [0.73, 0.88]	$+0.705$ 7/7

Table 4: The depth sweep at 56 concepts, in both media and under two identity encoders. Identity is accuracy (

\uparrow

); the bootstrap is over the seven groups and the “vs. norm” column is paired within group, with

k/7

counting groups agreeing in sign. The CLIP column scores the same generated images under a second vision tower. All twelve sites of the activation sweep are listed; the weight arm was run at the nine sites where it has entries, and a dash means the site was not measured in that medium. The base model scores

0.039

on identity and

0.243

on matching.

Figure 5: The same activation edit on four models, against relative depth so stacks of 24, 28 and 32 blocks are comparable. Curves are grouped by what the understanding pathway encodes: solid for a semantic vision encoder, dashed for a reconstruction codebook. Left is raw identity with each model’s own measured base level as a faint dotted line; right subtracts that level, the only quantity comparable across models whose base levels differ sevenfold. The shaded band is the shallow third, where both semantic-encoder models sit above their base models and the other two never leave theirs.

model	understanding representation	base	visual peak	word	verdict
BAGEL-7B-MoT	ViT, semantic	0.036	0.584	—	window
Janus-Pro-1B	SigLIP, semantic	0.052	0.583	0.625	window
Lumina-DiMOO-8B	VQ codebook	0.104	0.167	0.656	null, instr. works
Omni-Diffusion-7B	MagViT-v2 VQ	0.250	0.250	0.500	null, instr. works

Table 5: Peak and base identity for each of the four models, accuracy (

\uparrow

). Visual peak is the best site on the whole depth sweep and base is the unmodified model measured on the same generations. The word column rotates the pseudo-name onto a real word’s state through the identical code path and is the instrument check (§ 6.1 ). Grouping is by what the understanding pathway encodes.

Figure 6: Mid-stack Semantic Anchoring. (a) Once, before training, each concept’s images are embedded through the frozen understanding pathway and the shared expert’s states are mean-pooled at layer 14 into a frozen visual address $a_{k}$ . (b) At each step, a short prompt containing the name runs to the same layer, the name’s sub-word states are pooled into $h_{k}$ , and a $K$ -way InfoNCE pulls $h_{k}$ onto $a_{k}$ and away from the other concepts’ addresses. No flow-matching gradient is ever computed, which is where the generative route’s $41\%$ GenEval cost originates (Table 6 ); anchoring costs $0.1\%$ and acquires the concept for both directions (Table 1 ).

condition	steps	flow steps	overall	$\Delta$	rel.	identity
base (unmodified)	—	—	0.681	—	—	0.036
activation patch @ layer 3	0	0	0.681	$0.000$	$0.0\%$	0.930
anchoring @ layer 14, alone	480	0	0.680	$-0.001$	$0.1\%$	0.930
U-side, shared MLP only	1600	0	0.680	$-0.001$	$0.1\%$	0.445
U-side then a generation phase	1760	160	0.680	$-0.001$	$0.1\%$	0.891
anchoring @ layer 14 $+$ flow matching	480	480	0.638	$-0.042$	$6.2\%$	0.953

Table 6: General text-to-image ability retained after each recipe. GenEval ( Ghosh et al., 2023 ) prompts, all 553, two images each, identical seeds, all columns accuracy (

\uparrow

). We substitute an OWLv2 detector ( Minderer et al., 2023 ) for the official Mask2Former, so absolute scores are not comparable to published GenEval numbers and every claim here is a

\Delta

against the same base model under the same detector. Steps counts the optimizer steps the recipe takes, and flow steps how many of them compute a flow-matching gradient.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

paired contrast	discovery (g0)	confirmation (g1–g6)	pooled	ratio
alignment @14 vs @ readout, identity	$+0.906$	$+0.672$ [+0.59, +0.74] 6/6	$+0.705$	$1.35\times$
alignment @14 vs @ readout, matching	$+0.637$	$+0.554$ [+0.50, +0.60] 6/6	$+0.566$	$1.15\times$
alignment alone vs G-inject, matching	$+0.362$	$+0.381$ [+0.27, +0.49] 6/6	$+0.379$	$0.95\times$
shared MLP vs shared attention, export	$+0.516$	$+0.219$ [+0.14, +0.31] 6/6	$+0.261$	$2.36\times$
alignment alone vs G-inject, identity	$+0.273$	$+0.135$ [-0.04, +0.29] 5/6	$+0.155$	$2.02\times$
G-inject into shared MLP vs default	$+0.148$	$+0.049$ [-0.06, +0.17] 2/6	$+0.064$	$3.00\times$

Appendix

Table 7: Discovery versus confirmation. g0 is where each contrast was found; g1–g6 were run afterwards against a fixed list. The final column reports the discovery-to-confirmation ratio.

configuration family	$n$	$\rho$ within	$\rho$ without	MAE predicting it
capacity allocation	16	$+0.46$	$+0.78$	0.200
placement / main	10	$+0.60$	$+0.59$	0.137
recipe	6	$+0.71$	$+0.67$	0.194
substrate	4	$+0.80$	$+0.66$	0.122
pooled	36	$\bm{+0.68}$ , family-clustered bootstrap $[+0.37,+0.88]$

Appendix

Table 8: SAR versus export with the dependence structure respected.

\rho

within is the correlation inside one sweep;

\rho

without is the pooled correlation with that sweep deleted; MAE is the error of predicting that sweep’s configurations from a fit that never saw them, in accuracy units.

Figure 7: The same conditions scored with the candidate names in the prompt and with them removed, grouped by evaluation condition. The shaded region is production chance.

Figure 8: The edit in the centered space where the objective is evaluated. The reported experiments use the direction-only edit in panels (a)–(b); panel (c) shows the separately evaluated naive edit.

	generation identity			name matching
entry point	activations	weights	weights	activations	weights	weights
	anchor only	anchor only	+ flow	anchor only	anchor only	+ flow
layer 3	0.930	0.945	0.949	0.887	0.900	0.900
layer 7	0.828	0.914	0.984	0.900	0.925	0.950
layer 10	0.469	0.938	—	0.862	0.962	—
layer 14	0.016	0.930	0.984	0.500	0.887	0.921

Appendix

Table 9: The depth sweep on one group of eight, three arms, accuracy (

\uparrow

). All optimize the same anchoring objective at the same site. The first two differ only in medium; the third adds flow matching. The base model scores

0.036

on identity and

0.254

on matching.

direction	$r_{\text{und}}$	$r_{\text{gen}}$	private share	direct	cross	export eff.
G-inject	32	0	0.00	0.773	0.575	0.58
G-inject	24	8	0.25	0.883	0.700	0.65
G-inject	16	16	0.50	0.781	0.525	0.51
G-inject	8	24	0.75	0.531	0.412	0.53
G-inject	0	32	1.00	0.102	0.138	—
U-inject	32	0	0.00	1.000	0.531	0.60

Appendix

Table 10: Constant-budget capacity allocation, group of eight, accuracy (

\uparrow

); the italicised all-private row is the measured base level of this sweep,

0.138

for matching and

0.016

for retrieval, and is what the last column subtracts; analytic chance is

0.250

and

0.042

. Trainable parameters are identical in every row. Export efficiency is (cross

-

base)/(direct

-

base), blank when the direct task is within

0.15

of that base.

Figure 9: A fixed adapter budget slid between the shared and private experts. In the U-inject panel the trained task is saturated at every allocation while transfer halves, so the shared capacity transfer needs is strictly greater than the capacity the task itself needs.

blocks deleted	where	which blocks	concept identity	$\Delta$ CLIP-T (general ability)
0	—	—	0.828 [0.66, 0.97]	$\pm 0.0000$
2	after the site	8, 9	0.742 [0.56, 0.91]	$-0.0065$
4	after the site	8–11	0.102 [0.02, 0.20]	$-0.0832$
6	after the site	8–13	0.016 [0.00, 0.05]	$-0.1084$
2	before the site	5, 6	0.000 [0.00, 0.00]	$-0.1018$
4	before the site	3–6	0.000 [0.00, 0.00]	$-0.0909$

Appendix

Table 11: Deleting computation with the entry point held fixed at layer 7, identity (

\uparrow

). Every arm’s patch is re-solved under its own ablation.

\Delta

CLIP-T is measured on ordinary prompts containing no pseudo-name.

architecture	params	hook only	grad only	intersection	whole model
		backbone sharing by estimator
BAGEL-7B-MoT ( Deng et al., 2025 )	14.1B	0.500	0.961	0.500	0.484
Janus-Pro-1B ( Chen et al., 2025 )	1.5B	0.873	0.873	0.873	0.720

Appendix

Table 12: Measured cross-task parameter sharing for the two trained models under hook-only, gradient-only, and intersection estimators. Bold marks the reported intersection estimate.

model	backbone	understanding pathway	generation pathway	blocks
BAGEL-7B-MoT	Qwen2.5-7B (MoT)	ViT, semantic	rectified flow, VAE latent	28
Janus-Pro-1B	Llama, 1.5B	SigLIP, semantic	LlamaGen VQ	24
Omni-Diffusion-7B	Dream-7B (Qwen2.5-7B)	MagViT-v2 VQ	masked discrete diffusion	28
Lumina-DiMOO-8B	LLaDA-8B	VQ codebook	discrete diffusion	32

Appendix

Table 13: Every model in the four-architecture study. Injection module is the exact module list the forward hook was registered on; the patch always fires in the language backbone’s residual stream, never in a vision tower or a decoder, so “depth” means the same thing on all four. Sites are absolute block indices;

-2

is the input embedding and

-1

the last block’s output after the final norm.

site	relative depth	Janus-Pro-1B	BAGEL at the nearest relative depth
input embeddings	0.00	0.240	0.023 (embeddings, 1 group)
layer 0	0.04	0.344	0.323 (layer 0)
layer 2	0.12	0.583	0.528 (layer 3, rel. 0.14)
layer 3	0.17	0.427	0.528 (layer 3)
layer 5	0.25	0.281	0.584 (layer 7, rel. 0.29)
layer 8	0.38	0.146	0.366 (layer 10)

Appendix

Table 14: The entry-point window on Janus-Pro against BAGEL at the nearest relative depth, identity (

\uparrow

). The intervention is identical in closed form, with all weights frozen and no gradient step; only the model changes. BAGEL figures are the seven-group values of Table 4 except where marked.

model	site	visual address	real-word target	floor	trained ceiling
BAGEL-7B-MoT	layer 7	0.584 (80%)	—	0.036	0.724
Janus-Pro-1B	layer 2	0.583 (93%)	0.625 (100%)	0.052	0.625
Lumina-DiMOO-8B	layer 4	0.010 ( $<$ floor)	0.656 (6.3 $\times$ floor)	0.104	—
Omni-Diffusion-7B	layer 3	0.000 ( $<$ floor)	0.500 (2.0 $\times$ floor)	0.250	—

Appendix

Table 15: The word-target instrument check of § 6.1 , identity (

\uparrow

). Each row rotates a pseudo-name onto a target at the model’s best site through one code path; only the target differs. Percentages use each model’s measured floor and available trained ceiling. “

<

floor” marks a score below the measured floor.

Figure 10: The same generations scored three ways. Gray is the bank the rest of the paper uses, where no distractor shares the target’s category; blue puts a same-category sibling in the bank; orange compares the target against that sibling alone, where the category carries no information at all.

	generation identity			understanding
condition	coarse	fine	paired drop	vs sibling	matching
base model	0.047	0.031	$+0.016$ [-0.01, +0.04]	0.484	0.529
G-inject (flow matching)	0.906	0.724	+0.182 [+0.09, +0.29]	0.758	0.688
U-inject (LM CE)	0.424	0.344	$+0.081$ [+0.02, +0.16]	0.706	0.925
anchor @14 + flow matching	0.966	0.888	$+0.078$ [+0.00, +0.18]	0.914	0.854
anchor @14 alone	0.802	0.747	$+0.055$ [-0.01, +0.13]	0.878	0.721

Appendix

Table 16: Instance identity, three fine-grained groups (24 concepts in 12 same-category pairs), accuracy (

\uparrow

). Coarse and fine are the same 24-wide width scored on the same generated images and differ only in whether one distractor shares the target’s category. Paired drop is coarse minus fine, differenced per concept. Chance is

0.042

for the 24-way columns and

0.500

for the two-way ones.

control	und. matching	gen. identity	evaluation role
layer 3, real addresses	0.887	0.930	reference condition
layer 3, deranged addresses	0.138	0.000	address-order control
layer 7, quarter magnitude	0.613	0.086	shallow-site magnitude control
layer 24, quarter magnitude	0.138	0.031	deep-site magnitude control
base model	0.254	0.036	unmodified reference

Appendix

Table 17: Protocol-level controls for the entry-point sweep, group of eight, accuracy (

\uparrow

). The base model scores

0.254

on matching and

0.036

on identity.

condition	identity	matching	open naming	real-category
base model	0.036	0.254	0.000	0.931
G-inject	0.758	0.600	0.000	0.931
U-inject	0.562	1.000	1.000	0.924
name shuffling, G-inject	0.016	0.000	0.000	—
text-only injection	0.055	0.238	0.000	—
embedding-only, generation side	0.172	0.238	0.000	—

Appendix

Table 18: Protocol-level controls, group of eight, accuracy (

\uparrow

). Identity is 24-way retrieval on generations and matching is the 4-way naming probe, both as in § 3.2 ; open naming is the fraction of concepts for which the model volunteers the pseudo-word with no candidates in the prompt. Real-category is balanced accuracy on the underlying real categories, the forgetting check. Cross-name retrieval is reported in the text.

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 287 papers

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System
Sep 1, 2026Penghao Wu, Haiwen Diao, Weichen Fan +3Unified Multimodal Models Multimodal Generation
1S-Lab, Nanyang Technological University · 2SenseTime Research
Transferability Between Understanding and Generation in Unified Multimodal Models
Jul 5, 2026Jiwon Kang, Heeji Yoon, Jaewoo Jung +5Unified Multimodal Models Multimodal Generation
KAIST AI, South Korea · Blynx, South Korea · Trillionlabs, South Korea
Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models
Apr 27, 2026Weixing Wang, Liudvikas Zekas, Anton Hackl +5Unified Multimodal Models Cross-Modal Alignment
Hasso Plattner Institute / University of Potsdam, Potsdam, Germany

Sep 1, 2026J/K move · Enter open · S save

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

Penghao Wu, Haiwen Diao, Weichen Fan, Lewei Lu+2

1S-Lab, Nanyang Technological University · 2SenseTime Research

Unified Multimodal Models Multimodal Generation Multimodal Learning Multimodal Understanding

Open paper AskPDF arXiv

While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision--language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner--executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.

Abstract

Unified Multimodal Models

Explore similar work

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

Transferability Between Understanding and Generation in Unified Multimodal Models

Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models