Organizations: University of California, Los Angeles · Aimakj · College of Computer Science and Technology National University of Defense Technology · Key Laboratory of Advanced Microprocessor Chips and Systems College of Computer Science and Technology National University of Defense Technology
Unified multimodal models (UMMs) train image understanding and autoregressive image generation on shared parameters, and the two objectives are known to interfere. Existing diagnoses and remedies operate at the resolution of layers or experts, measuring conflict per layer and resolving it by separating parameters. We argue that this resolution hides an orthogonal axis. Generation in a UMM is next-token prediction over a raster sequence of visual tokens whose roles vary systematically with position, so how strongly a generation gradient interferes with understanding should depend on where in the sequence it originates. We introduce a position-resolved interference map that attributes understanding-generation gradient conflict to visual-token positions within every layer, computed from a single backward pass at 1.2× the cost of a standard backward pass. On Show-o and Janus-Pro, position explains a large share of conflict variance after controlling for depth (partial η2=0.31 vs. 0.35 for layer on Show-o; 0.15 vs. 0.30 on Janus-Pro): the first quarter of the sequence has a mean gradient cosine of −0.18 against understanding, the last quarter −0.02. The dependence survives per-position gradient-norm normalization, retaining 80 of its effect size, and conflict strength tracks semantic content (Spearman ρ=0.64). Building on the map, we propose position-aware modulation (PAM), which removes the anti-aligned component of generation gradients only at high-conflict positions without changing the architecture. Under a matched trainable-parameter budget, PAM improves over layer-wise separation by +21 MME and +2.4 GenEval points on Show-o while matching it on POPE and overall FID; a random-position control recovers about 31 of the gain. Position-based and layer-based separation are complementary degrees of freedom and can be combined.
Figures & tables
Figure 1: Position-resolved interference maps. (a) Show-o-1.3B and (b) Janus-Pro-7B: cosine between the generation gradient attributed to position bucket b and the understanding gradient, per layer; dashed lines separate shallow/middle/deep bands; the white cell marks the strongest conflict (cosine −0.42 and −0.37 ). (c,d) Position marginals with quarter means −0.18/−0.09/−0.05/−0.02 (Show-o) and −0.11/−0.05/−0.03/−0.01 (Janus-Pro); error bars are ± 1 s.e. over diagnostic batches.
Model
ηlayer2
ηpos2
ηint2
Flayer
Fpos
Fint
Show-o-1.3B
0.353
0.312
0.085
1040.1
1389.9
9.8
Janus-Pro-7B
0.299
0.150
0.068
808.5
678.8
9.0
Table 1: Two-way ANOVA of the interference map (factors: layer, position). Partial η2 with F statistics; error df from L×K×5 modules ×24 batch-seed replicates minus cells. Both main effects and the interaction are significant at p<0.001 .
Figure 2: Magnitude vs. direction. (a) Per-bucket conflict on Show-o before and after per-position gradient-norm normalization. (b) Relative gradient-norm profiles: 2.63× (Show-o) and 1.92× (Janus-Pro) decline from first to last bucket. (c) Variance decomposition of the position effect into direction and magnitude components, with the layer effect for comparison.
Model
ηpos2 raw
ηpos2 norm
direction
magnitude
retention
norm decline
Show-o-1.3B
0.312
0.249
80%
20%
0.797
2.63×
Janus-Pro-7B
0.150
0.114
76%
24%
0.758
1.92×
Table 2: Norm–direction decomposition of the position effect. Retention is ηpos2 after per-position normalization divided by the raw value.
Figure 3: Attribution. (a) Conflict magnitude vs. semantic decodability of each position (pooled over layer bands and models, n=96 ; Spearman ρ=0.64 ). (b) Decodability by bucket with quarter means (dashed). (c) Conflict vs. high-frequency reconstruction residual ( ρ=−0.58 ): texture positions are cheap, semantic positions are contested.
Figure 4: Position-aware modulation. (a) The raster sequence with the modulated buckets S highlighted ( 6/16 buckets, 37.5% of visual tokens). (b) The projection operator g′=g−min(0,g⋅u^)u^ : the anti-aligned component of the bucket gradient is removed, the rest preserved. (c) Conflict scores per bucket over the conflict-prone band; buckets above the threshold τ=0.13 are selected ( m=6 ).
Arm
MME ↑
POPE ↑
GenEval ↑
FID ↓
joint + norm (baseline)
1339
84.9
0.550
6.84
layer-wise LoRA separation
1397
86.4
0.603
6.05
Pam (ours)
1418
86.6
0.627
5.97
random-position control
1364
85.3
0.574
6.50
layer-wise + Pam
1424
86.7
0.632
5.94
understanding-only reference
1431
87.6
–
–
Table 3: Show-o-1.3B, matched 12.6M trainable parameters, mean of 3 seeds. Δ rows: Pam vs. layer-wise, 95% paired-bootstrap CIs. Single-task references bound each axis.
Figure 5: Main results. (a) Show-o-1.3B and (b) Janus-Pro-7B: GenEval vs. MME for the five arms (mean of 3 seeds; error bars ± 1 seed-std); dashed lines are the single-task ceilings on Show-o. (c) Per metric deltas of Pam over layer-wise separation with 95% paired-bootstrap CIs; POPE and overall FID are statistical parity.
Figure 6: Ablations on Show-o-1.3B. (a) GenEval gain vs. joint baseline as a function of the number of modulated buckets m (shaded: ± 1 seed-std); the star marks the map-selected m=6 . (b) Robustness to the bucket count K . (c) Fraction of the Pam gain recovered by the random-position control: ≈31% on Show-o, ≈62% on Janus-Pro.
Arm
LF-FID
MF-FID
HF-FID
overall FID
joint + norm
2.41
3.11
2.87
6.84
layer-wise
2.02
2.76
2.66
6.05
Pam (ours)
2.00
2.75
2.77
5.97
random-position
2.23
2.94
2.83
6.50
Table 4: Frequency-band FID on Show-o-1.3B (three-level Laplacian decomposition of MJHQ-30k; lower is better). The high-frequency cost and low-frequency gain are pre-declared metrics (Section 1 ).
Arm
MME ↑
POPE ↑
GenEval ↑
FID ↓
joint + norm (baseline)
1372
87.9
0.662
5.63
layer-wise LoRA sep.
1379
88.1
0.670
5.48
Pam (ours)
1387
88.3
0.681
5.44
random-position control
1381
88.2
0.674
5.57
layer-wise + Pam
1390
88.3
0.684
5.42
Δ ( Pam − layer-wise)
+8 [2, 14]
+0.2 [ −0.3 , 0.7]
+1.1 [0.3, 1.9]
−0.04 [ −0.12 , 0.04]
Table 5: Janus-Pro-7B, matched 31.5M trainable parameters, mean of 3 seeds. CIs as in Table 3 .
Figure 7: Position profiles of the interference map per layer band (shallow / middle / deep), both models; horizontal lines mark band means.
Figure 8: (a) Backward cost of the probe vs. alternatives. (b) H200-hour budget by stage; the total is 200 hours.
Figure 9: Frequency-band FID (a) and overall FID (b) on Show-o-1.3B for the four arms of Table 4 .
Figure 10: Cross-architecture comparison. (a) ANOVA effect sizes by factor. (b) Per-metric deltas of Pam over layer-wise separation with 95% CIs; on Janus-Pro the SigLIP understanding branch bypasses the visual token sequence and the effect halves. (c) Random-control recovery of the Pam gain (MME).
Unified multimodal models (UMMs) are increasingly designed around gradient conflict between understanding and generation objectives. The premise that reducing these metrics improves the downstream understanding-generation trade-off has never been tested directly. We audit it in a controlled testbed, GRIDUMM, which mirrors key structural ingredients of UMM training while making the ground-truth trade-off exactly computable. Across 63 configurations and 372 measured checkpoints, no directional conflict metric reaches an absolute Spearman correlation of 0.3 with a confidence interval excluding zero for conflict measured during training against the eventual trade-off. A dose-response intervention that monotonically suppresses conflict leaves the trade-off flat, separating correlation from causation. The norm ratio is a generation-failure detector and becomes null among configurations that master generation. Functional interference measures outperform directional conflict metrics, while training loss tracks the trade-off strongly. Our results do not show that conflict is useless; they show that its validity as a diagnostic target must be established, not assumed, and we release the audit protocol as a reusable standard.
Shuyang Jiang, Fucheng Deng, Yuchuan Luo +1
University of California, Los Angeles · Aimakj · College of Computer Science and Technology National University of Defense Technology +1
While unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model, functional unification does not guarantee learning synergy: the two objectives may reinforce each other, compete for capacity, or merely coexist. We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting without pretrained vision priors. At the representation level, we find that each objective provides useful signal to the other: generation enriches the visual features learned for understanding, while understanding strengthens vision--language alignment for generation. However, when both objectives are forced through the same computation path, one tends to dominate. A task-decoupled architecture that specializes conflicting visual computation while preserving semantic interaction avoids this asymmetric degradation. At the task level, through three case studies, we find positive bidirectional transfer when understanding and generation tasks rely on shared knowledge. At the system level, we show that an end-to-end UMM outperforms a matched planner--executor pipeline on complex tasks that explicitly require both image understanding and generation. Together, these results show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.
Penghao Wu, Haiwen Diao, Weichen Fan +3
1S-Lab, Nanyang Technological University · 2SenseTime Research
Unified multimodal models (UMMs) aim to handle perception and generation in a single model. Yet existing UMMs still rely on a frozen, separately pretrained VAE for image generation, imposing a structural bottleneck. Naively removing it introduces a quality gap, as the model must learn both high-level structure and low-level details from raw pixels. In this paper, we propose Representation Forcing (RF), a technique that closes this gap by making representation prediction a native capability of the model. Concretely, RF forces the decoder to autoregressively predict visual representations as intermediate tokens before pixels; these tokens then stay in context to guide pixel diffusion within the same backbone. By turning representations from perception outputs into generation targets, RF eliminates the need for any external generative latent space. We find that RF benefits both understanding and generation. On image generation, our pixel-space model with RF matches state-of-the-art VAE-based unified models. On image understanding, pixel-space RF generally outperforms its VAE-based variant. Together, these results offer an effective step toward end-to-end, bottleneck-free UMMs.
Yuqing Wang, Zhijie Lin, Ceyuan Yang +10
University of Hong Kong · ByteDance Seed · The Chinese University of Hong Kong +2