Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Columbia University · MMLab, CUHK · University of Illinois Urbana-Champaign
Unified multimodal models (UMMs) are motivated by the hope that understanding and generation reinforce each other but controlled ablations repeatedly find that adding a generation objective leaves understanding flat. Joint-training studies cannot settle the disagreement: with overlapping supervision, a gain cannot be attributed to the architecture rather than the data. To further investigate the relationship between the two directions in UMMs, we separate them by construction. A novel visual entity, a rendered 3D asset paired with a pseudo-word screened for absence from the frozen model's behavior, is bound through exactly one task direction, and the untrained direction is then measured. We find that the channel is real in both directions, but the directions differ in kind: generation training installs a name the model can only match among candidates; understanding training installs one it can also produce. What governs cross-task usability is where the binding enters the shared computation. An alignment probe predicts export across 36 configurations (Spearman ρ=+0.68). That objective's alignment term, maximized in closed form over activations with every weight frozen, makes a concept drawable when injected at layer 7 of 28 and is indistinguishable from the base model from layer 14 on, while the weight-based version of the same edit peaks at layers 10-14. In an observational series of four models, this window appears only where the understanding pathway is a semantic vision encoder, suggesting that unified weights are not enough: the two directions must share a semantic format at the entry point. Exploiting the rule, a mid-stack alignment objective acquires the concept for a 0.1% relative loss of the model's general text-to-image ability, against 41% for the standard generative route. Our code is at https://github.com/Zane-ZYQiu/entry-point-umm.
Figures & tables
Figure 1: What the model draws for “a photo of a {name}”. Rows are four novel concepts and the reference column shows a view of the asset each pseudo-word was bound to. On the left side, the base model has never seen the pseudo-word; a placebo that supplies another concept’s address at identical magnitude draws the wrong object; and a closed-form activation edit applied at layer 14 leaves the base model’s output untouched. On the right the binding lands inside the window: the same edit at layer 7 with every weight frozen; training the understanding direction alone; the standard generative route; and the proposed objective, an alignment loss at layer 14 with no generative gradient, for a 0.1% relative loss of general text-to-image ability (§ 7 ). Where the binding enters, not how it is optimized, is what decides.
Figure 2: The contamination-free protocol. A novel entity (left) is bound through exactly one task direction, (a) generation or (b) understanding, and the opposite direction, which received no gradient and no datum, is then probed; the model’s shared blocks are the only path between the two.
condition
und. name matching ↑
gen. identity ↑
base model
0.243 [0.19, 0.31] ± 0.058
0.039 [0.02, 0.06] ± 0.032
G-inject (flow matching), shared+private
0.520 [0.44, 0.60] ± 0.121
0.653 [0.56, 0.75] ± 0.125
G-inject, shared MLP only
0.620 [0.53, 0.70] ± 0.119
0.717 [0.62, 0.81] ± 0.122
U-inject (LM CE), shared+private
0.989 [0.98, 1.00] ± 0.014
0.360 [0.27, 0.46] ± 0.105
U-inject, shared attention only
0.984 [0.97, 0.99] ± 0.013
0.078 [0.04, 0.13] ± 0.069
U-inject, shared MLP only
0.991 [0.98, 1.00] ± 0.013
0.339 [0.25, 0.44] ± 0.165
Table 1: Cross-task transfer, 56 concepts in seven groups, 480 steps in both directions. Both columns are accuracy ( ↑ ); brackets are bootstraps over all 56 concepts and ± is the standard deviation of the seven group means, the run-level spread a concept bootstrap cannot see. Blocks from top to bottom: the base model; generative injection and its substrate variant; understanding injection and its substrate variants; the mid-stack anchoring objective alone (§ 5.1 ) at two depths; and the combination of the two recipes.
condition
LM-CE
4-way
reverse 4-way
context-free
first sub-word
U-inject (LM cross-entropy)
✓
1.000
1.000
0.938
0.922
U-inject, shared attention only
✓
1.000
0.900
0.297
0.688
base model
✗
0.254
0.242
0.141
0.125
G-inject (flow matching)
✗
0.600
0.650
0.141
0.125
anchoring @ layer 14, alone
✗
0.887
1.000
0.125
0.094
activation patch @ layer 3
✗
0.887
1.000
0.234
0.125
Table 2: Matching the name against producing it, for seven conditions spanning both training objectives. All columns are accuracy ( ↑ ); chance is 0.250 for the 4-way probes and 0.125 for the context-free ones. The last column, first sub-word, is the pre-specified arbiter, immune by construction to the shortcut the probe detects. Rows are grouped by whether the recipe ever computed a language-model cross-entropy.
entry point
shared computation after it
exports
§
input embeddings
all 28 blocks, but nothing to route yet
✗
5.2
activation edit, layer 3
25 blocks
partial (0.53)
5.2
activation edit, layer 7
21 blocks
✓ (0.58)
5.2
activation edit, layer 14
14 blocks
✗ (0.04, at base)
5.2
weight edit, layer 3
25 blocks
partial (0.60)
5.2
weight edit, layers 10–14
14–18 blocks
✓ (0.80)
5.2
Table 3: The entry points this paper compares, placed on one depth axis; the depth sweep behind the middle two blocks is finer than the rows shown and is given in full in Table 4 . Each row puts the same binding at a different point of the shared computation; blocks are ordered by intervention medium, and by depth within each. Export is accuracy on the task direction that received no gradient (§ 3.2 ); the last column points to where each row is measured.
Figure 3: SAR against export across 36 configurations, and the intervention on SAR (arrow): anchoring applied at the readout moves the probe to a perfect 1.000 and leaves export where it was. A quantity can order configurations well and still not be the thing to change.
Figure 4: Export against entry-point depth, for the same anchoring objective at the same site, differing only in whether activations (blue) or weights (purple) may change. Left is export on the axis that cannot be gamed; bands are group-level bootstraps over all seven groups. The orange curve adds a flow-matching gradient and is one group, drawn dashed. Both anchor-only media give a window, and the weight medium’s reaches deeper.
activations, anchor only (0 gradient steps)
weights, anchor only (480 steps)
entry point
identity (DINOv2)
(CLIP)
sd
vs. norm
identity
vs. norm
layer 0
0.323 [0.22, 0.42]
0.269
0.145
+0.283 7/7
—
—
layer 3
0.528 [0.33, 0.72]
0.433
0.297
+0.489 6/7
0.602 [0.47, 0.75]
+0.499 7/7
layer 7
0.584 [0.46, 0.72]
0.512
0.196
+0.545 7/7
0.769 [0.69, 0.84]
+0.666 7/7
layer 10
0.366 [0.28, 0.45]
0.281
0.127
+0.327 7/7
0.800 [0.70, 0.89]
+0.698 7/7
layer 14
0.040 [0.02, 0.07]
0.054
0.036
+0.001 3/7
0.808 [0.73, 0.88]
+0.705 7/7
Table 4: The depth sweep at 56 concepts, in both media and under two identity encoders. Identity is accuracy ( ↑ ); the bootstrap is over the seven groups and the “vs. norm” column is paired within group, with k/7 counting groups agreeing in sign. The CLIP column scores the same generated images under a second vision tower. All twelve sites of the activation sweep are listed; the weight arm was run at the nine sites where it has entries, and a dash means the site was not measured in that medium. The base model scores 0.039 on identity and 0.243 on matching.
Figure 5: The same activation edit on four models, against relative depth so stacks of 24, 28 and 32 blocks are comparable. Curves are grouped by what the understanding pathway encodes: solid for a semantic vision encoder, dashed for a reconstruction codebook. Left is raw identity with each model’s own measured base level as a faint dotted line; right subtracts that level, the only quantity comparable across models whose base levels differ sevenfold. The shaded band is the shallow third, where both semantic-encoder models sit above their base models and the other two never leave theirs.
model
understanding representation
base
visual peak
word
verdict
BAGEL-7B-MoT
ViT, semantic
0.036
0.584
—
window
Janus-Pro-1B
SigLIP, semantic
0.052
0.583
0.625
window
Lumina-DiMOO-8B
VQ codebook
0.104
0.167
0.656
null, instr. works
Omni-Diffusion-7B
MagViT-v2 VQ
0.250
0.250
0.500
null, instr. works
Table 5: Peak and base identity for each of the four models, accuracy ( ↑ ). Visual peak is the best site on the whole depth sweep and base is the unmodified model measured on the same generations. The word column rotates the pseudo-name onto a real word’s state through the identical code path and is the instrument check (§ 6.1 ). Grouping is by what the understanding pathway encodes.
Figure 6: Mid-stack Semantic Anchoring. (a) Once, before training, each concept’s images are embedded through the frozen understanding pathway and the shared expert’s states are mean-pooled at layer 14 into a frozen visual address ak . (b) At each step, a short prompt containing the name runs to the same layer, the name’s sub-word states are pooled into hk , and a K -way InfoNCE pulls hk onto ak and away from the other concepts’ addresses. No flow-matching gradient is ever computed, which is where the generative route’s 41% GenEval cost originates (Table 6 ); anchoring costs 0.1% and acquires the concept for both directions (Table 1 ).
condition
steps
flow steps
overall
Δ
rel.
identity
base (unmodified)
—
—
0.681
—
—
0.036
activation patch @ layer 3
0
0
0.681
0.000
0.0%
0.930
anchoring @ layer 14, alone
480
0
0.680
−0.001
0.1%
0.930
U-side, shared MLP only
1600
0
0.680
−0.001
0.1%
0.445
U-side then a generation phase
1760
160
0.680
−0.001
0.1%
0.891
anchoring @ layer 14 + flow matching
480
480
0.638
−0.042
6.2%
0.953
Table 6: General text-to-image ability retained after each recipe. GenEval ( Ghosh et al., 2023 ) prompts, all 553, two images each, identical seeds, all columns accuracy ( ↑ ). We substitute an OWLv2 detector ( Minderer et al., 2023 ) for the official Mask2Former, so absolute scores are not comparable to published GenEval numbers and every claim here is a Δ against the same base model under the same detector. Steps counts the optimizer steps the recipe takes, and flow steps how many of them compute a flow-matching gradient.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
paired contrast
discovery (g0)
confirmation (g1–g6)
pooled
ratio
alignment @14 vs @ readout, identity
+0.906
+0.672 [+0.59, +0.74] 6/6
+0.705
1.35×
alignment @14 vs @ readout, matching
+0.637
+0.554 [+0.50, +0.60] 6/6
+0.566
1.15×
alignment alone vs G-inject, matching
+0.362
+0.381 [+0.27, +0.49] 6/6
+0.379
0.95×
shared MLP vs shared attention, export
+0.516
+0.219 [+0.14, +0.31] 6/6
+0.261
2.36×
alignment alone vs G-inject, identity
+0.273
+0.135 [-0.04, +0.29] 5/6
+0.155
2.02×
G-inject into shared MLP vs default
+0.148
+0.049 [-0.06, +0.17] 2/6
+0.064
3.00×
Appendix
Table 7: Discovery versus confirmation. g0 is where each contrast was found; g1–g6 were run afterwards against a fixed list. The final column reports the discovery-to-confirmation ratio.
configuration family
n
ρ within
ρ without
MAE predicting it
capacity allocation
16
+0.46
+0.78
0.200
placement / main
10
+0.60
+0.59
0.137
recipe
6
+0.71
+0.67
0.194
substrate
4
+0.80
+0.66
0.122
pooled
36
+0.68 , family-clustered bootstrap [+0.37,+0.88]
Appendix
Table 8: SAR versus export with the dependence structure respected. ρ within is the correlation inside one sweep; ρ without is the pooled correlation with that sweep deleted; MAE is the error of predicting that sweep’s configurations from a fit that never saw them, in accuracy units.
Figure 7: The same conditions scored with the candidate names in the prompt and with them removed, grouped by evaluation condition. The shaded region is production chance.
Figure 8: The edit in the centered space where the objective is evaluated. The reported experiments use the direction-only edit in panels (a)–(b); panel (c) shows the separately evaluated naive edit.
generation identity
name matching
entry point
activations
weights
weights
activations
weights
weights
anchor only
anchor only
+ flow
anchor only
anchor only
+ flow
layer 3
0.930
0.945
0.949
0.887
0.900
0.900
layer 7
0.828
0.914
0.984
0.900
0.925
0.950
layer 10
0.469
0.938
—
0.862
0.962
—
layer 14
0.016
0.930
0.984
0.500
0.887
0.921
Appendix
Table 9: The depth sweep on one group of eight, three arms, accuracy ( ↑ ). All optimize the same anchoring objective at the same site. The first two differ only in medium; the third adds flow matching. The base model scores 0.036 on identity and 0.254 on matching.
direction
rund
rgen
private share
direct
cross
export eff.
G-inject
32
0
0.00
0.773
0.575
0.58
G-inject
24
8
0.25
0.883
0.700
0.65
G-inject
16
16
0.50
0.781
0.525
0.51
G-inject
8
24
0.75
0.531
0.412
0.53
G-inject
0
32
1.00
0.102
0.138
—
U-inject
32
0
0.00
1.000
0.531
0.60
Appendix
Table 10: Constant-budget capacity allocation, group of eight, accuracy ( ↑ ); the italicised all-private row is the measured base level of this sweep, 0.138 for matching and 0.016 for retrieval, and is what the last column subtracts; analytic chance is 0.250 and 0.042 . Trainable parameters are identical in every row. Export efficiency is (cross − base)/(direct − base), blank when the direct task is within 0.15 of that base.
Figure 9: A fixed adapter budget slid between the shared and private experts. In the U-inject panel the trained task is saturated at every allocation while transfer halves, so the shared capacity transfer needs is strictly greater than the capacity the task itself needs.
blocks deleted
where
which blocks
concept identity
Δ CLIP-T (general ability)
0
—
—
0.828 [0.66, 0.97]
±0.0000
2
after the site
8, 9
0.742 [0.56, 0.91]
−0.0065
4
after the site
8–11
0.102 [0.02, 0.20]
−0.0832
6
after the site
8–13
0.016 [0.00, 0.05]
−0.1084
2
before the site
5, 6
0.000 [0.00, 0.00]
−0.1018
4
before the site
3–6
0.000 [0.00, 0.00]
−0.0909
Appendix
Table 11: Deleting computation with the entry point held fixed at layer 7, identity ( ↑ ). Every arm’s patch is re-solved under its own ablation. Δ CLIP-T is measured on ordinary prompts containing no pseudo-name.
backbone sharing by estimator
architecture
params
hook only
grad only
intersection
whole model
BAGEL-7B-MoT ( Deng et al., 2025 )
14.1B
0.500
0.961
0.500
0.484
Janus-Pro-1B ( Chen et al., 2025 )
1.5B
0.873
0.873
0.873
0.720
Appendix
Table 12: Measured cross-task parameter sharing for the two trained models under hook-only, gradient-only, and intersection estimators. Bold marks the reported intersection estimate.
model
backbone
understanding pathway
generation pathway
blocks
BAGEL-7B-MoT
Qwen2.5-7B (MoT)
ViT, semantic
rectified flow, VAE latent
28
Janus-Pro-1B
Llama, 1.5B
SigLIP, semantic
LlamaGen VQ
24
Omni-Diffusion-7B
Dream-7B (Qwen2.5-7B)
MagViT-v2 VQ
masked discrete diffusion
28
Lumina-DiMOO-8B
LLaDA-8B
VQ codebook
discrete diffusion
32
Appendix
Table 13: Every model in the four-architecture study. Injection module is the exact module list the forward hook was registered on; the patch always fires in the language backbone’s residual stream, never in a vision tower or a decoder, so “depth” means the same thing on all four. Sites are absolute block indices; −2 is the input embedding and −1 the last block’s output after the final norm.
site
relative depth
Janus-Pro-1B
BAGEL at the nearest relative depth
input embeddings
0.00
0.240
0.023 (embeddings, 1 group)
layer 0
0.04
0.344
0.323 (layer 0)
layer 2
0.12
0.583
0.528 (layer 3, rel. 0.14)
layer 3
0.17
0.427
0.528 (layer 3)
layer 5
0.25
0.281
0.584 (layer 7, rel. 0.29)
layer 8
0.38
0.146
0.366 (layer 10)
Appendix
Table 14: The entry-point window on Janus-Pro against BAGEL at the nearest relative depth, identity ( ↑ ). The intervention is identical in closed form, with all weights frozen and no gradient step; only the model changes. BAGEL figures are the seven-group values of Table 4 except where marked.
model
site
visual address
real-word target
floor
trained ceiling
BAGEL-7B-MoT
layer 7
0.584 (80%)
—
0.036
0.724
Janus-Pro-1B
layer 2
0.583 (93%)
0.625 (100%)
0.052
0.625
Lumina-DiMOO-8B
layer 4
0.010 ( < floor)
0.656 (6.3 × floor)
0.104
—
Omni-Diffusion-7B
layer 3
0.000 ( < floor)
0.500 (2.0 × floor)
0.250
—
Appendix
Table 15: The word-target instrument check of § 6.1 , identity ( ↑ ). Each row rotates a pseudo-name onto a target at the model’s best site through one code path; only the target differs. Percentages use each model’s measured floor and available trained ceiling. “ < floor” marks a score below the measured floor.
Figure 10: The same generations scored three ways. Gray is the bank the rest of the paper uses, where no distractor shares the target’s category; blue puts a same-category sibling in the bank; orange compares the target against that sibling alone, where the category carries no information at all.
generation identity
understanding
condition
coarse
fine
paired drop
vs sibling
matching
base model
0.047
0.031
+0.016 [-0.01, +0.04]
0.484
0.529
G-inject (flow matching)
0.906
0.724
+0.182 [+0.09, +0.29]
0.758
0.688
U-inject (LM CE)
0.424
0.344
+0.081 [+0.02, +0.16]
0.706
0.925
anchor @14 + flow matching
0.966
0.888
+0.078 [+0.00, +0.18]
0.914
0.854
anchor @14 alone
0.802
0.747
+0.055 [-0.01, +0.13]
0.878
0.721
Appendix
Table 16: Instance identity, three fine-grained groups (24 concepts in 12 same-category pairs), accuracy ( ↑ ). Coarse and fine are the same 24-wide width scored on the same generated images and differ only in whether one distractor shares the target’s category. Paired drop is coarse minus fine, differenced per concept. Chance is 0.042 for the 24-way columns and 0.500 for the two-way ones.
control
und. matching
gen. identity
evaluation role
layer 3, real addresses
0.887
0.930
reference condition
layer 3, deranged addresses
0.138
0.000
address-order control
layer 7, quarter magnitude
0.613
0.086
shallow-site magnitude control
layer 24, quarter magnitude
0.138
0.031
deep-site magnitude control
base model
0.254
0.036
unmodified reference
Appendix
Table 17: Protocol-level controls for the entry-point sweep, group of eight, accuracy ( ↑ ). The base model scores 0.254 on matching and 0.036 on identity.
condition
identity
matching
open naming
real-category
base model
0.036
0.254
0.000
0.931
G-inject
0.758
0.600
0.000
0.931
U-inject
0.562
1.000
1.000
0.924
name shuffling, G-inject
0.016
0.000
0.000
—
text-only injection
0.055
0.238
0.000
—
embedding-only, generation side
0.172
0.238
0.000
—
Appendix
Table 18: Protocol-level controls, group of eight, accuracy ( ↑ ). Identity is 24-way retrieval on generations and matching is the 4-way naming probe, both as in § 3.2 ; open naming is the fraction of concepts for which the model volunteers the pseudo-word with no candidates in the prompt. Real-category is balanced accuracy on the underlying real categories, the forgetting check. Cross-name retrieval is reported in the text.
While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision--language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner--executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.
Penghao Wu, Haiwen Diao, Weichen Fan +3
1S-Lab, Nanyang Technological University · 2SenseTime Research
Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate transferability in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.
Jiwon Kang, Heeji Yoon, Jaewoo Jung +5
KAIST AI, South Korea · Blynx, South Korea · Trillionlabs, South Korea
Unified Multimodal Models (uMMs) aim to support both visual understanding and visual generation within a shared representation. However, existing evaluation protocols assess these two capabilities independently and do not examine whether they are semantically aligned. As a result, it remains unclear whether current uMMs learn coherent unified representations that remain consistent across tasks given a visual concept. We introduce XTC-Bench, a scene-graph-grounded evaluation framework that measures cross-task visual semantic consistency. By deriving both generation prompts and understanding queries from a structured scene graph, our framework enables fact-level alignment analysis across objects, attributes, and relations. We propose Continuous Cross-Task Agreement (CCTA), a fine-grained metric that quantifies semantic agreement between generation and understanding over matched atomic facts, isolating internal consistency from standalone task accuracy. Extensive experiments on eight open-source and one commercial unified models reveal that high generation or understanding performance does not imply strong cross-task alignment, and architectural analysis shows consistency is governed by how tightly learning objectives are coupled across modalities, not by architectural unification alone. XTC-Bench provides a reproducible and model-agnostic framework for diagnosing representation-level misalignment, offering a concrete direction for advancing unified multimodal modeling beyond isolated task performance.
Weixing Wang, Liudvikas Zekas, Anton Hackl +5
Hasso Plattner Institute / University of Potsdam, Potsdam, Germany