Does Gradient Conflict Predict the Understanding--Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Models

Shuyang Jiang; Fucheng Deng; Yuchuan Luo; Zhenyu Wu

cs.LGSep 29, 2026

Does Gradient Conflict Predict the Understanding--Generation Trade-off? A Controlled Audit of Conflict-Metric Validity in Unified Multimodal Models

Authors: Shuyang Jiang, Fucheng Deng, Yuchuan Luo, Zhenyu Wu

Organizations: University of California, Los Angeles · Aimakj · College of Computer Science and Technology National University of Defense Technology · Key Laboratory of Advanced Microprocessor Chips and Systems College of Computer Science and Technology National University of Defense Technology

Abstract

Unified multimodal models (UMMs) are increasingly designed around gradient conflict between understanding and generation objectives. The premise that reducing these metrics improves the downstream understanding-generation trade-off has never been tested directly. We audit it in a controlled testbed, GRIDUMM, which mirrors key structural ingredients of UMM training while making the ground-truth trade-off exactly computable. Across 63 configurations and 372 measured checkpoints, no directional conflict metric reaches an absolute Spearman correlation of 0.3 with a confidence interval excluding zero for conflict measured during training against the eventual trade-off. A dose-response intervention that monotonically suppresses conflict leaves the trade-off flat, separating correlation from causation. The norm ratio is a generation-failure detector and becomes null among configurations that master generation. Functional interference measures outperform directional conflict metrics, while training loss tracks the trade-off strongly. Our results do not show that conflict is useless; they show that its validity as a diagnostic target must be established, not assumed, and we release the audit protocol as a reusable standard.

Figures & tables

Figure 1: GridUMM audit protocol. (a) Two task streams use the same embedding, transformer and language-model head; the colored terminal nodes denote outputs, not task-specific parameter branches. The grid and network glyphs are schematic. (b) A fixed probe batch yields task gradients grouped by shared parameters; geometry is matched to run identity and checkpoint. (c) In E3 only, the $\alpha$ -projection removes a component parallel to $g_{u}$ before combination. Full projection illustrates $\alpha=1$ ; it does not preserve the gradient norm. Held-out understanding and generation scores evaluate the resulting model, without selecting updates. No experimental data are depicted in this conceptual illustration.

Method	understanding	generation (obj.)	joint score	$\cos(g_{u},g_{g})$	rate	norm ratio
Naive	0.630	0.998	0.814	0.001	0.417	0.025
Token-normalized	0.622	0.607	0.614	0.000	0.542	0.339
GradNorm-style	0.633	0.996	0.815	0.002	0.514	0.112
PCGrad-style	0.641	0.999	0.820	0.000	0.639	0.027
CAGrad	0.636	0.999	0.818	-0.003	0.708	0.029
MGDA	0.389	1.000	0.694	0.123	0.208	0.004

Table 1: Main results by method (base scale, final checkpoint; mean

\pm

s.d. over ratios and seeds,

n{=}9

per method). Object-cell accuracy is the generation axis; the joint score is their mean.

Figure 2: The core audit: probe conflict at step 200 (associated with the step-225 evaluation checkpoint) against the final joint score at step 900, for all 63 configurations (colors: methods; shapes: data ratios). The rank correlation is

\rho=-0.198{}

[-0.463, 0.079].

Predictor	$\rho$	95% CI
Probe separability	0.534	[0.406, 0.639]
Subspace overlap	0.132	[-0.037, 0.281]
Effective rank	0.702	[0.603, 0.782]
Self-consistency	-0.117	[-0.259, 0.034]
Norm ratio	-0.651	[-0.721, -0.573]
Conflict magnitude	-0.187	[-0.318, -0.046]

Table 2: Predictor competition at the checkpoint level: Spearman

\rho

with the Pareto score over all 252 checkpoint rows, cluster-bootstrap CIs. Functional measures in the upper block, gradient-geometric conflict metrics in the lower block.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	Base	Large (E5)
Transformer	$d{=}256$ , 6 blocks, 4 heads, FFN 512	$d{=}320$ , 8 blocks, FFN 640
Parameters	3.2M	6.7M
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.999$ , wd $0.01$ )	same
Learning rate	$1{\times}10^{-3}$ , warmup 100, cosine to $0.1\times$	same
Batch	64 (mixed tasks)	same
Steps	900	1080

Appendix

Table 3: Training and evaluation hyperparameters. Base scale is the default; the large scale (E5) differs only in the listed rows.

Population (E1)	7 methods $\times$ 3 ratios $\times$ 3 seeds $=63$ runs
Training budget	900 steps, batch 64, AdamW $1{\times}10^{-3}$
Conflict snapshots	every 50 steps on a fixed probe batch
Checkpoints	steps 225/450/675/900 (fraction of budget 0.25/0.5/0.75/1)
Interventions (E3)	$\alpha\in\{0.25,0.5,0.75,1.0\}$ , 3 seeds
Cross-scale (E5)	$d{=}320$ , 8 layers, 18 runs
Reliability	5 independent held-out resamples per final checkpoint

Appendix

Table 4: Audit design. All runs use the GridUMM testbed of Section 3 ; hyperparameters in Appendix B .

Metric	$\rho_{\text{pred}}$	95% CI	$\rho_{\text{conc}}$	95% CI	partial $\rho$ (ckpt)
Global cosine	-0.146	[-0.411, 0.147]	-0.321	[-0.523, -0.097]	-0.147
Conflict rate	0.022	[-0.210, 0.260]	0.338	[0.131, 0.537]	0.124
Conflict magnitude	-0.198	[-0.463, 0.079]	-0.009	[-0.266, 0.268]	-0.192
Norm ratio	-0.006	[-0.264, 0.246]	-0.293	[-0.507, -0.035]	-0.528
Logged energy statistic	0.052	[-0.185, 0.287]	0.366	[0.159, 0.559]	0.178
Depth-weighted cosine	—	—	-0.295	[-0.535, -0.040]	—

Appendix

Table 5: Conflict metrics vs. downstream Pareto score.

\rho_{\text{pred}}

: early probe conflict against the final score (

N=63{}

configurations);

\rho_{\text{conc}}

: both measured at the final checkpoint (

N=63

). Partial correlations control for training progress (252 rows); CIs are percentile bootstrap,

B{=}5000

( Efron, 1979 ) .

Figure 3: Signed correlations with 95% bootstrap intervals. (a) Early probe geometry against final score, one observation per run. (b) Concurrent association across all E1 checkpoints. Orange indicates an interval excluding zero. Unavailable weighted estimates are marked “not estimated”, not substituted from another analysis.

Figure 4: Final joint loss vs. Pareto score. Red markers: configurations whose training loss is better than another configuration’s but whose trade-off is Pareto-dominated by it (90/1953 pairs).

Figure 5: Per-layer correlation profile: Spearman

\rho

between each block’s task-gradient cosine and the Pareto score (configuration level, concurrent), with 95% CIs. The shallow-and-deep signature reported by Uni-X ( Hao et al., 2026 ) is not established by this concurrent profile: blocks 5 and 6 have intervals excluding zero. The plot is an association analysis, not a prospective layer-wise prediction test.

Figure 6: The conflict regime across training: conflict rate (left) and generation-to-understanding norm ratio (right), median and interquartile range across all E1 configurations. The rate, a sign count over near-zero cosines, stays elevated throughout; the norm ratio decays as the generation objective converges.

Figure 7: Dose–response. Left: conflict magnitude under the

\alpha

-projection, learning-phase mean over snapshots at steps 200–500 (the manipulation check): the effective (post-intervention) level follows the dose, the raw level shows co-adaptation. Right: the downstream Pareto score at the final checkpoint (mean

\pm

s.d. over seeds), the causal readout.

Figure 8: Predictor competition at the checkpoint level: Spearman

\rho

with the Pareto score, cluster-bootstrap CIs. Green: functional interference measures; orange: gradient-geometric conflict metrics.

Figure 9: Cross-scale check: per-metric correlations at the base and large scales.

Figure 10: The two-axis understanding/generation plane at the final checkpoint over all E1 configurations, with the empirical Pareto frontier (3 configurations: pcgrad (2), cagrad (1)). Colors: methods; shapes: data ratios.

Explore similar work

Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models
Sep 29, 2026Shuyang Jiang, Fucheng Deng, Yuchuan Luo +1Multimodal Model Autoregressive Image Generation
University of California, Los Angeles · Aimakj · College of Computer Science and Technology National University of Defense Technology +1
Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System
Sep 1, 2026Penghao Wu, Haiwen Diao, Weichen Fan +3Data Generation
1S-Lab, Nanyang Technological University · 2SenseTime Research
Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation
Jun 25, 2026Jinyu Liu, Xincheng Shuai, Henghui Ding +1Multimodal Model Synergy
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China · Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China

Sep 29, 2026, cs.CVJ/K move · Enter open · S save

Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models

Shuyang Jiang, Fucheng Deng, Yuchuan Luo, Zhenyu Wu

University of California, Los Angeles · Aimakj · College of Computer Science and Technology National University of Defense Technology+1

Multimodal Model Autoregressive Image Generation Long Visual-Token Sequences

Open paper AskPDF arXiv

Unified multimodal models (UMMs) train image understanding and autoregressive image generation on shared parameters, and the two objectives are known to interfere. Existing diagnoses and remedies operate at the resolution of layers or experts, measuring conflict per layer and resolving it by separating parameters. We argue that this resolution hides an orthogonal axis. Generation in a UMM is next-token prediction over a raster sequence of visual tokens whose roles vary systematically with position, so how strongly a generation gradient interferes with understanding should depend on where in the sequence it originates. We introduce a position-resolved interference map that attributes understanding-generation gradient conflict to visual-token positions within every layer, computed from a single backward pass at

1.2\times

the cost of a standard backward pass. On Show-o and Janus-Pro, position explains a large share of conflict variance after controlling for depth (partial

η^2=0.31

vs.

0.35

for layer on Show-o;

0.15

vs.

0.30

on Janus-Pro): the first quarter of the sequence has a mean gradient cosine of

-0.18

against understanding, the last quarter

-0.02

. The dependence survives per-position gradient-norm normalization, retaining

80%

of its effect size, and conflict strength tracks semantic content (Spearman

ρ=0.64

). Building on the map, we propose position-aware modulation (PAM), which removes the anti-aligned component of generation gradients only at high-conflict positions without changing the architecture. Under a matched trainable-parameter budget, PAM improves over layer-wise separation by

+21

MME and

+2.4

GenEval points on Show-o while matching it on POPE and overall FID; a random-position control recovers about

31%

of the gain. Position-based and layer-based separation are complementary degrees of freedom and can be combined.

Abstract

Explore similar work

Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models

Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System

Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation