Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models

Shuyang Jiang; Fucheng Deng; Yuchuan Luo; Zhenyu Wu

cs.CVSep 29, 2026

Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models

Authors: Shuyang Jiang, Fucheng Deng, Yuchuan Luo, Zhenyu Wu

Organizations: University of California, Los Angeles · Aimakj · College of Computer Science and Technology National University of Defense Technology · Key Laboratory of Advanced Microprocessor Chips and Systems College of Computer Science and Technology National University of Defense Technology

Abstract

Unified multimodal models (UMMs) train image understanding and autoregressive image generation on shared parameters, and the two objectives are known to interfere. Existing diagnoses and remedies operate at the resolution of layers or experts, measuring conflict per layer and resolving it by separating parameters. We argue that this resolution hides an orthogonal axis. Generation in a UMM is next-token prediction over a raster sequence of visual tokens whose roles vary systematically with position, so how strongly a generation gradient interferes with understanding should depend on where in the sequence it originates. We introduce a position-resolved interference map that attributes understanding-generation gradient conflict to visual-token positions within every layer, computed from a single backward pass at $1.2\times$ the cost of a standard backward pass. On Show-o and Janus-Pro, position explains a large share of conflict variance after controlling for depth (partial $η^2=0.31$ vs. $0.35$ for layer on Show-o; $0.15$ vs. $0.30$ on Janus-Pro): the first quarter of the sequence has a mean gradient cosine of $-0.18$ against understanding, the last quarter $-0.02$ . The dependence survives per-position gradient-norm normalization, retaining $80%$ of its effect size, and conflict strength tracks semantic content (Spearman $ρ=0.64$ ). Building on the map, we propose position-aware modulation (PAM), which removes the anti-aligned component of generation gradients only at high-conflict positions without changing the architecture. Under a matched trainable-parameter budget, PAM improves over layer-wise separation by $+21$ MME and $+2.4$ GenEval points on Show-o while matching it on POPE and overall FID; a random-position control recovers about $31%$ of the gain. Position-based and layer-based separation are complementary degrees of freedom and can be combined.

Figures & tables

Figure 1: Position-resolved interference maps. (a) Show-o-1.3B and (b) Janus-Pro-7B: cosine between the generation gradient attributed to position bucket $b$ and the understanding gradient, per layer; dashed lines separate shallow/middle/deep bands; the white cell marks the strongest conflict (cosine $-0.42$ and $-0.37$ ). (c,d) Position marginals with quarter means $-0.18/-0.09/-0.05/-0.02$ (Show-o) and $-0.11/-0.05/-0.03/-0.01$ (Janus-Pro); error bars are $\pm$ 1 s.e. over diagnostic batches.

Model	$\eta^{2}_{\mathrm{layer}}$	$\eta^{2}_{\mathrm{pos}}$	$\eta^{2}_{\mathrm{int}}$	$F_{\mathrm{layer}}$	$F_{\mathrm{pos}}$	$F_{\mathrm{int}}$
Show-o-1.3B	0.353	0.312	0.085	1040.1	1389.9	9.8
Janus-Pro-7B	0.299	0.150	0.068	808.5	678.8	9.0

Table 1: Two-way ANOVA of the interference map (factors: layer, position). Partial

\eta^{2}

with

F

statistics; error df from

L{\times}K{\times}5

modules

\times\,24

batch-seed replicates minus cells. Both main effects and the interaction are significant at

p<0.001

Figure 2: Magnitude vs. direction. (a) Per-bucket conflict on Show-o before and after per-position gradient-norm normalization. (b) Relative gradient-norm profiles:

2.63\times

(Show-o) and

1.92\times

(Janus-Pro) decline from first to last bucket. (c) Variance decomposition of the position effect into direction and magnitude components, with the layer effect for comparison.

Model	$\eta^{2}_{\mathrm{pos}}$ raw	$\eta^{2}_{\mathrm{pos}}$ norm	direction	magnitude	retention	norm decline
Show-o-1.3B	0.312	0.249	80%	20%	0.797	$2.63\times$
Janus-Pro-7B	0.150	0.114	76%	24%	0.758	$1.92\times$

Table 2: Norm–direction decomposition of the position effect. Retention is

\eta^{2}_{\mathrm{pos}}

after per-position normalization divided by the raw value.

Figure 3: Attribution. (a) Conflict magnitude vs. semantic decodability of each position (pooled over layer bands and models,

n=96

; Spearman

\rho=0.64

). (b) Decodability by bucket with quarter means (dashed). (c) Conflict vs. high-frequency reconstruction residual (

\rho=-0.58

): texture positions are cheap, semantic positions are contested.

Figure 4: Position-aware modulation. (a) The raster sequence with the modulated buckets $\mathcal{S}$ highlighted ( $6/16$ buckets, $37.5\%$ of visual tokens). (b) The projection operator $g^{\prime}=g-\min(0,\,g\!\cdot\!\hat{u})\hat{u}$ : the anti-aligned component of the bucket gradient is removed, the rest preserved. (c) Conflict scores per bucket over the conflict-prone band; buckets above the threshold $\tau=0.13$ are selected ( $m=6$ ).

Arm	MME $\uparrow$	POPE $\uparrow$	GenEval $\uparrow$	FID $\downarrow$
joint + norm (baseline)	1339	84.9	0.550	6.84
layer-wise LoRA separation	1397	86.4	0.603	6.05
Pam (ours)	1418	86.6	0.627	5.97
random-position control	1364	85.3	0.574	6.50
layer-wise $+$ Pam	1424	86.7	0.632	5.94
understanding-only reference	1431	87.6	–	–

Table 3: Show-o-1.3B, matched 12.6M trainable parameters, mean of 3 seeds.

\Delta

rows: Pam vs. layer-wise,

95\%

paired-bootstrap CIs. Single-task references bound each axis.

Figure 5: Main results. (a) Show-o-1.3B and (b) Janus-Pro-7B: GenEval vs. MME for the five arms (mean of 3 seeds; error bars

\pm

1 seed-std); dashed lines are the single-task ceilings on Show-o. (c) Per metric deltas of Pam over layer-wise separation with

95\%

paired-bootstrap CIs; POPE and overall FID are statistical parity.

Figure 6: Ablations on Show-o-1.3B. (a) GenEval gain vs. joint baseline as a function of the number of modulated buckets

m

(shaded:

\pm

1 seed-std); the star marks the map-selected

m=6

. (b) Robustness to the bucket count

K

. (c) Fraction of the Pam gain recovered by the random-position control:

\approx 31\%

on Show-o,

\approx 62\%

on Janus-Pro.

Arm	LF-FID	MF-FID	HF-FID	overall FID
joint + norm	2.41	3.11	2.87	6.84
layer-wise	2.02	2.76	2.66	6.05
Pam (ours)	2.00	2.75	2.77	5.97
random-position	2.23	2.94	2.83	6.50

Table 4: Frequency-band FID on Show-o-1.3B (three-level Laplacian decomposition of MJHQ-30k; lower is better). The high-frequency cost and low-frequency gain are pre-declared metrics (Section 1 ).

Arm	MME $\uparrow$	POPE $\uparrow$	GenEval $\uparrow$	FID $\downarrow$
joint + norm (baseline)	1372	87.9	0.662	5.63
layer-wise LoRA sep.	1379	88.1	0.670	5.48
Pam (ours)	1387	88.3	0.681	5.44
random-position control	1381	88.2	0.674	5.57
layer-wise $+$ Pam	1390	88.3	0.684	5.42
$\Delta$ ( Pam $-$ layer-wise)	$+8$ [2, 14]	$+0.2$ [ $-0.3$ , 0.7]	$+1.1$ [0.3, 1.9]	$-0.04$ [ $-0.12$ , 0.04]

Table 5: Janus-Pro-7B, matched 31.5M trainable parameters, mean of 3 seeds. CIs as in Table 3 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
Backbones	Show-o-1.3B ( $L{=}24$ , $d{=}2048$ , $T{=}256$ ); Janus-Pro-7B ( $L{=}30$ , $d{=}4096$ , $T{=}576$ )
Position buckets	$K=16$ contiguous raster buckets
LoRA	rank 16, $\alpha$ 32, dropout 0.05, $q,k,v,o$ , all layers
Trainable parameters	12.6M (Show-o), 31.5M (Janus-Pro)
Optimization	AdamW, lr $10^{-4}$ , cosine, warmup 500, 20k steps, batch 128
Task mixture	1:1 understanding / generation

Appendix

Table 6: Shared configuration.

Figure 7: Position profiles of the interference map per layer band (shallow / middle / deep), both models; horizontal lines mark band means.

Figure 8: (a) Backward cost of the probe vs. alternatives. (b) H200-hour budget by stage; the total is 200 hours.

Figure 9: Frequency-band FID (a) and overall FID (b) on Show-o-1.3B for the four arms of Table 4 .

Figure 10: Cross-architecture comparison. (a) ANOVA effect sizes by factor. (b) Per-metric deltas of Pam over layer-wise separation with

95\%

CIs; on Janus-Pro the SigLIP understanding branch bypasses the visual token sequence and the effect halves. (c) Random-control recovery of the Pam gain (MME).

Explore similar work

Does Gradient Conflict Predict the Understanding--Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Models
Sep 29, 2026Shuyang Jiang, Fucheng Deng, Yuchuan Luo +1Dose
University of California, Los Angeles · Aimakj · College of Computer Science and Technology National University of Defense Technology +1
Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System
Sep 1, 2026Penghao Wu, Haiwen Diao, Weichen Fan +3Data Generation
1S-Lab, Nanyang Technological University · 2SenseTime Research
Representation Forcing for Bottleneck-Free Unified Multimodal Models
May 29, 2026Yuqing Wang, Zhijie Lin, Ceyuan Yang +10Multimodal Model Video Latents
University of Hong Kong · ByteDance Seed · The Chinese University of Hong Kong +2

Sep 29, 2026, cs.LGJ/K move · Enter open · S save

Does Gradient Conflict Predict the Understanding--Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Models

Shuyang Jiang, Fucheng Deng, Yuchuan Luo, Zhenyu Wu

University of California, Los Angeles · Aimakj · College of Computer Science and Technology National University of Defense Technology+1

Dose

Open paper AskPDF arXiv

Unified multimodal models (UMMs) are increasingly designed around gradient conflict between understanding and generation objectives. The premise that reducing these metrics improves the downstream understanding-generation trade-off has never been tested directly. We audit it in a controlled testbed, GRIDUMM, which mirrors key structural ingredients of UMM training while making the ground-truth trade-off exactly computable. Across 63 configurations and 372 measured checkpoints, no directional conflict metric reaches an absolute Spearman correlation of 0.3 with a confidence interval excluding zero for conflict measured during training against the eventual trade-off. A dose-response intervention that monotonically suppresses conflict leaves the trade-off flat, separating correlation from causation. The norm ratio is a generation-failure detector and becomes null among configurations that master generation. Functional interference measures outperform directional conflict metrics, while training loss tracks the trade-off strongly. Our results do not show that conflict is useless; they show that its validity as a diagnostic target must be established, not assumed, and we release the audit protocol as a reusable standard.

Abstract

Explore similar work

Does Gradient Conflict Predict the Understanding--Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Models

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

Representation Forcing for Bottleneck-Free Unified Multimodal Models