Refinement Symmetry in Multimodal Transformers

Yuhao Du; Shunian Chen

cs.AISep 26, 2026

Refinement Symmetry in Multimodal Transformers

Authors: Yuhao Du, Shunian Chen

Organizations: The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute

Abstract

Attention weights depend on token counts, which change with the representation of a signal. We study refinement symmetry: splitting a representation while preserving content, position, visible context, and total mass should preserve its contribution. Building on proportional and quadrature attention, we show that split invariance forces the local mass factor to be linear for any fixed positive attention kernel, provided that factor is nondecreasing. For changed representations, a physical coupling bounds attention error by separating feature change from weight reallocation. In Qwen2.5-Omni-7B, duplicating half the visual tokens threefold changes 255 of 3,586 MVBench answers under standard attention; measure weighting preserves every answer under matched visibility. Under natural frame resampling, it reduces distributional drift. At twofold merging of a frozen video encoding, a five-seed evaluation shows an all-partition-correct accuracy gain of 1.04 percentage points over global count weighting (average group mass) and 0.93 points over standard attention. The advantage over global count also holds on WorldSense but depends on the compression budget. The result is a representation principle with a measured benefit in robustness across partitions.

Figures & tables

Figure 1: From token multiplicity to correctness across partitions. (a) With equal scores, duplicating only the value-one token shifts standard attention from

1/2

2/3

; dividing its mass between copies preserves

1/2

. (b) In Qwen2.5-Omni-7B, duplicating half the visual representations changes 150 and 255 of 3,586 MVBench answers under standard attention, and none under measure weighting with matched positions and visibility. (c) At twofold merging, measure weighting improves correctness under all four tested partitions over global count weighting on two benchmarks. Points show mean paired effects across five training seeds; bars are descriptive, unadjusted 95% crossed source-video/seed bootstrap intervals.

Split	Standard changes	Measure changes	C $\to$ W	W $\to$ C	W $\to$ W
2	150	0	63	55	32
3	255	0	111	90	54

Table 1: Same content, changed predictions. Qwen2.5-Omni-7B on 3,586 usable MVBench questions (14 of 3,600 excluded). Half the visual tokens are replaced by the stated number of copies, with matched positions and visibility. C/W denote correct/incorrect; transitions describe standard attention. Measure weighting changes no answers.

Figure 2: Distributional stability under natural resampling. Qwen2.5-Omni-7B on MVBench. (a) Standard minus measure-weighted

\operatorname{KL}(p_{16}^{\rm std}\|p_{b}^{a})

from the native 16-frame option distribution, with paired 95% intervals over 540 questions; the open marker is the reference budget, zero by construction. (b) Empirical cumulative distributions at 48 frames; the horizontal axis is linear near zero and logarithmic beyond

10^{-5}

nats. Per-budget intervals describe question variability; the primary repeated-budget test clusters by source video.

MVBench
Reduction	Rule	Mean accuracy $\uparrow$	Robust accuracy $\uparrow$	JSD $\downarrow$
2 $\times$	Standard	$64.13\pm 0.65$	$56.50\pm 0.64$	$0.0360\pm 0.0046$
	Global count	$64.64\pm 0.54$	$56.39\pm 0.48$	$0.0413\pm 0.0051$
	Measure	$\mathbf{64.82}\pm 0.58$	$\mathbf{57.44}\pm 0.60$	$\mathbf{0.0350}\pm 0.0044$
4 $\times$	Standard	$61.26\pm 0.59$	$53.08\pm 0.61$	$\mathbf{0.0409}\pm 0.0051$
	Global count	$\mathbf{62.41}\pm 0.56$	$52.53\pm 0.82$	$0.0512\pm 0.0052$

Table 2: Robustness under post-encoder merging. Qwen2.5-Omni-7B on 2,489 MVBench and 2,188 WorldSense questions; means

\pm

sample SD over five training seeds. Mean accuracy averages uniform, motion, entropy, and keyframe partitions; robust accuracy requires correctness under all four (both in %). JSD is mean pairwise Jensen–Shannon divergence in nats. Bold marks the best mean within each benchmark and budget; paired inference is in Table 3 .

MVBench: 2,489 questions, 2,290 source videos
Comparator	Endpoint	Difference [95% interval]	Adjusted $p$
Global count	Robust acc.	+1.04 [+0.45, +1.65]	0.0082
Global count	JSD	-0.0063 [-0.0080, -0.0047]	$7.7\times 10^{-4}$
Standard	Robust acc.	+0.93 [+0.16, +1.69]	0.024
Standard	JSD	-0.0010 [-0.0029, +0.0008]	0.90
WorldSense: 2,188 questions, 1,158 source videos

Table 3: Primary paired effects at twofold reduction. Qwen2.5-Omni-7B, five trained seeds per benchmark. Robust accuracy requires correctness under all four partitions. Effects pair the same checkpoints and test questions. Differences are measure minus comparator, computed before rounding: robust accuracy uses pp; Jensen–Shannon divergence (JSD) uses nats. Intervals are descriptive, unadjusted 95% crossed source-video/seed bootstrap intervals; adjusted

p=\min(1,4\max(p_{\rm source},p_{\rm seed}))

within each benchmark.

Figure 3: Exact worst-case bound and natural partition error. (a) A two-value construction attains

D_{v}\tanh(\Delta/4)

with

D_{v}=1

and no value perturbation. (b,c) Mean absolute error against the dense reference over 512 periodic signals, with 95% intervals across signals. Standard and count weighting coincide. Adaptive cells change their contents as they change width; measure weighting removes the count prior while retaining approximation error.

Figure 4: Transfer to unseen rate imbalance. The task predicts the temporal gap between events in two modalities. (a) Measure minus standard accuracy (pp), with continuum RoPE shared by both arms, across four slow rates and six rate ratios; cells average ten paired seeds. The dotted divider separates the two training ratios from unseen ratios. (b) Cross-modal log-odds shift against rate ratio, with 95% intervals over six seeds. The dashed line is the fixed-score prediction $\log q$ ; trained arms may also differ in their scores.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Convergence to different limits. Each estimator is compared with the intended functional

\mathcal{A}_{\mu}

and the sampling-density functional

\mathcal{A}_{\rho\mu}

. The same smooth signal is evaluated on increasingly fine nonuniform partitions. Reference integrals use adaptive quadrature. The dashed reference has slope

-2

; the nonvanishing error reflects the difference between the two limits. The construction above specifies the signal and partitions.

Figure 6: Two-sided integration squares the support factor and narrows its frequency response. A uniform cell gives a sinc factor; a variance-matched Gaussian gives a smooth nonnegative factor. At equal query/key extents, the squared single-support Gaussian is the two-sided convention used throughout. These are analytic response curves.

Operation	Count form	Measure-aware form
Attention	$4.7\times 10^{-1}$	$2.2\times 10^{-16}$
Mean pooling	$5.5\times 10^{-1}$	$1.1\times 10^{-16}$
Per-token loss	$8.4\times 10^{-1}$	$8.9\times 10^{-16}$
Token-axis normalization	$4.4\times 10^{-1}$	$6.7\times 10^{-16}$
MoE load-balancing loss	$1.3\times 10^{-1}$	$6.7\times 10^{-16}$
Expert capacity	$4.4$	$0$

Appendix

Table 4: Measure-aware operations preserve the tested duplicate splits. Representative duplicate-refinement checks. Deviations are absolute and use each operation’s own units; their magnitudes should not be compared across rows. The measure-aware capacity result concerns the constructed admission instance described in the text.

Figure 7: Physical position drives cross-rate transfer. Points and error bars show seed means and 95% intervals for held-out accuracy. Blue denotes measure weighting, gold standard attention, and grey feature baselines; marker shape encodes the positional family. The dotted line marks chance. Sample sizes and in-distribution scores appear in Table 5 ; comparisons between independently listed summaries are descriptive.

Arm	Seeds	Trained (%)	Held out (%)	95% CI
Index RoPE	5	92.5	21.4	$[20.6,22.2]$
Index + modality bias	5	93.1	21.1	$[20.2,21.9]$
Index + measure weighting	5	91.8	21.7	$[21.2,22.2]$
Continuum RoPE	5	97.2	83.2	$[81.1,85.3]$
Continuum + measure	5	97.1	83.4	$[81.4,85.5]$
Shared physical clock	8	97.4	81.1	$[79.6,82.7]$

Appendix

Table 5: Physical clocks carry cross-rate accuracy. Training rates are

8\times 4

8\times 8

, and

4\times 4

tokens/s; the held-out column averages five other rates within each seed before computing its 95% interval. The eight-seed paired clock comparison is reported separately.

Slow rate	Ratio	Gain	95% CI	Slow rate	Ratio	Gain	95% CI
2	1	-1.25	$[-8.19,+5.69]$	8	1	+0.20	$[-0.05,+0.44]$
2	2	+0.43	$[-3.36,+4.22]$	8	2	+0.14	$[-0.34,+0.63]$
2	4	+2.40	$[-0.62,+5.42]$	8	4	+1.07	$[-1.21,+3.34]$
2	8	+5.22	$[+2.83,+7.61]$	8	8	+3.75	$[-2.74,+10.24]$
2	16	+10.47	$[+5.53,+15.40]$	8	16	+7.68	$[-1.65,+17.02]$
2	32	+15.94	$[+9.88,+22.00]$	8	32	+12.45	$[+1.62,+23.28]$

Appendix

Table 6: Cellwise gains across rate imbalance. Cellwise accuracy gain from measure weighting, in pp. Each interval uses ten paired seeds and is pointwise, not simultaneous across cells.

Figure 8: Support-aware positional models retain more accuracy under merging. (a) Accuracy at each merge factor for five positional and attention configurations, matched across eight seeds. (b) Paired gain of a width-tracking Gaussian over no attenuation, a Gaussian retaining the original extent, and a width-tracking box profile. These four profiles are separately trained with standard attention using eight matched initialization and data seeds; merging is applied at inference. Error bars are 95% intervals. In (a), blue denotes measure weighting and gold standard attention; line style encodes the positional family.

Figure 9: Measure weighting reduces rate-dependent attention allocation. Qwen2.5-Omni-7B’s video share of audio/video attention, averaged over 16 clips with fixed audio tokenization. Error bars show 95%

t

intervals across clips. Orange circles denote standard attention; blue squares denote native-calibrated measure weighting. The dotted line marks the 2 fps calibration rate.

fps	Standard (%)	MC	Count (uncal.)	Count (cal.)	$\Delta$ accuracy [95% CI]
2	34.7	9	65	9	$+0.00\;[-0.66,+0.67]$
0.5	36.2	48	18	48	$-0.67\;[-2.51,+1.17]$
0.25	33.3	70	22	70	$+0.50\;[-1.81,+2.84]$

Appendix

Table 7: Calibrated attention under WorldSense rate changes. Each rate evaluates the same 600 questions. The change columns count answers differing from standard attention for measure weighting (MC) and for the uncalibrated and calibrated count arms. Calibrated count and measure weighting give identical answers. Accuracy differences (measure minus standard) are in pp with 95% bootstrap intervals grouped by source video.

Probe	7B	3B
Duplication std flips at factor two (of 3,586)	150	140
Duplication std flips at factor three	255	236
Duplication mc flips at factors two and three	0, 0	0, 0
Causal-copy mc flips at factors two and three (of 720)	4, 4	4, 5
Budget KL reduction, nats [95% interval]	0.0168	0.0154
	[0.0077, 0.0259]	[0.0082, 0.0226]

Appendix

Table 8: Duplication and resampling across model scales. Probes on Qwen2.5-Omni-7B and Qwen2.5-Omni-3B under the identical harness. Flips count changed answers against each model’s own un-duplicated reference; std is standard attention, mc the measure-weighted arm. The KL row reports standard minus measure-weighted

\operatorname{KL}(p_{16}^{\rm std}\|p_{b}^{a})

, averaged over the five nonreference budgets, with 95% video-clustered intervals. Both arms share the standard-attention reference.

Arm	Consistency (%)	Accuracy (%)	$\Delta$ consistency (pp)	$p$
Standard	86.1	67.7	$+0.00$	1.000
Modality count	87.0	68.9	$+0.93\;[-1.73,+3.58]$	0.608
Measure	87.0	68.4	$+0.93\;[-1.88,+3.73]$	0.627
Local count (0.75 s)	86.6	68.1	$+0.46\;[-1.47,+2.39]$	0.815

Appendix

Table 9: Consistency across content-adaptive tokenizations. Evaluation on 432 MVBench questions. Consistency is agreement across all three tokenizers; accuracy is averaged within question over tokenizers. The interval and nominal exact McNemar test compare consistency with standard attention. The local-count window is selected descriptively from eight tested windows.

$n$	$q$	Standard (%)	MC (%)	$\Delta$ (pp) [95% CI]	W/L	$p$
120	1	61.67	61.67	$+0.00$	0/0	1.000
120	8	61.67	64.17	$+2.50\;[-2.45,+7.45]$	6/3	0.508
360	1	64.44	63.61	$-0.83\;[-1.78,+0.11]$	0/3	0.250
360	8	63.89	62.78	$-1.11\;[-3.79,+1.57]$	10/14	0.541
2786	1	62.42	62.67	$+0.25\;[-0.11,+0.62]$	17/10	0.248
2786	8	62.31	63.03	$+0.72\;[-0.16,+1.59]$	87/67	0.125

Appendix

Table 10: Accuracy after low-rank adaptation. LoRA evaluations on MVBench. The 120- and 360-question slices use 400 training examples and 200 steps; the 2,786-question slice uses 600 examples and 300 steps. Each is one trained seed per arm. Intervals describe paired question variability conditional on those checkpoints. W/L counts correctness gains/losses for measure weighting;

p

is nominal exact McNemar.

Figure 10: Loss weighting remains task dependent. Each cell shows mean held-out loss above the best tested weight at the same rate, across five seeds: (a) per-token objective, (b) measure-normalized objective. Circles mark the grid minima. The shared color scale covers the full observed range; each objective contains 225 runs. The measure-objective minimum reaches the search boundary at $q=16$ , so no extrapolated optimum or fitted scaling law is reported.

Rate ratio $q$	Loss difference	95% CI	$p$
1	+0.0000	$[+0.0000,+0.0000]$	1.000
2	+0.0011	$[-0.0016,+0.0038]$	0.312
4	+0.0068	$[+0.0040,+0.0096]$	0.003
8	+0.0154	$[+0.0114,+0.0193]$	$4.2\times 10^{-4}$
16	+0.0206	$[+0.0149,+0.0262]$	$5.3\times 10^{-4}$

Appendix

Table 11: Inverse-rate loss weighting increases held-out loss at ratios 4–16. Held-out loss at

w=1/q

minus loss at

w=1

in the per-token objective, paired across five seeds. Positive values favor

w=1

. Intervals are 95%; tests are nominal and two-sided. The

q=1

comparison is identical by construction.

Figure 11: Only off-lattice declared rates perturb answers: 14 of 2,160, none on-lattice. Left: changed answers at each declared rate, with fixed image inputs and 540 paired questions per rate; off-lattice rates are shaded. Right: median, 10th–90th percentiles, and maximum KL at the four rates that do not align with the lattice. The intervals describe the distribution of perturbations, not confidence in a mean.

Size set	Arm	Accuracy (%)	$\Delta$ (pp) [95% CI]	$p$
Narrow	Standard	54.25	$+0.00$	1.000
Narrow	Measure	55.00	$+0.75\;[-0.72,+2.22]$	0.508
Narrow	Reverse groups	55.75	$+1.50\;[-0.20,+3.20]$	0.146
Wide	Standard	56.25	$+0.00$	1.000
Wide	Measure	54.75	$-1.50\;[-3.91,+0.91]$	0.307
Wide	Reverse groups	56.25	$+0.00\;[-2.09,+2.09]$	1.000

Appendix

Table 12: Accuracy across overlapping resolutions. Four hundred questions per configuration. Accuracy changes compare with standard attention within the same size set; intervals are paired and tests nominal. Each size set combines overlapping views and falls outside partition refinement.

Reduction	Rule	Decoder (ms)	Stage sum (ms)	Peak (GiB)
1 $\times$	Standard	303.6	4710.5	19.34
1 $\times$	Global count	294.9	4698.4	19.34
1 $\times$	Measure	285.4	4686.3	19.34
2 $\times$	Standard	143.1	4523.8	19.34
2 $\times$	Global count	147.5	4530.0	19.34
2 $\times$	Measure	152.0	4536.1	19.34

Appendix

Table 13: Merging cuts decoder latency far more than total cost. Measured inference costs for the point-support panel. Stage sum includes acquisition, saliency, grouping and decoder time; acquisition dominance keeps the stage-sum saving at 5.2–5.6% from the native partition to eightfold merging. Peak is allocated CUDA memory, including the frozen dense encoder.

	$2\times$	$4\times$	$8\times$
Input-embedding deviation $\epsilon_{h}$ by partition
uniform	45.5	65.3	74.7
motion	55.8	66.6	74.8
entropy	51.5	66.1	75.0
keyframe	62.2	71.0	75.6
Embedding-scaled attention TV, $D_{h}T$ , by rule (partition range)

Appendix

Table 14: Input perturbation and attention redistribution under merging. Mean maximal input-embedding deviation

\epsilon_{h}

(400 questions; standard error

\leq 0.72

; identical across rules) and the embedding-scaled attention diagnostic

D_{h}T

, averaged over layers and heads (ranges across four partitions). The mean native video-embedding diameter is

D_{h}=135.9

. The quantities in the first two blocks have input-embedding units.

Partition	Standard	Global count	Measure	$\Delta$ vs. std. (pp)	$\Delta$ vs. count (pp)
uniform	64.94	65.42	65.42	+0.47	0.00
motion	64.19	64.36	64.69	+0.51	+0.33
entropy	64.37	65.01	65.12	+0.75	+0.11
keyframe	63.03	63.78	64.07	+1.04	+0.28
worst partition (keyframe)	63.03	63.78	64.07	+1.04	+0.28

Appendix

Table 15: Measure-versus-global-count differences arise on adaptive partitions. Pooled MVBench test accuracy (%) for Qwen2.5-Omni-7B at

K=8

, over 12,445 question–seed pairs per cell. Differences are measure minus comparator in pp, computed before rounding. On the uniform partition the measure and count rules coincide exactly. The final row tabulates the worst-partition-accuracy endpoint, which the keyframe partition attains for every arm at this budget.

Attention Mechanisms

Attention mechanisms let neural networks weight information from other tokens or regions according to relevance, enabling context-dependent processing.

See all 513 papers

Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models
Aug 19, 2026Pardis Taghavi, Reza Langari, Gaurav PandeySparse Attention Block-Sparse Attention
Texas A&M University
QUEST: A robust attention formulation using query-modulated spherical attention
Mar 31, 2026Hariprasath Govindarajan, Per Sidén, Jacob Roll +1Transformer Attention Attention Mechanisms
Linköping University, Sweden · Qualcomm Auto Ltd Sweden Filial
RefAM: Attention Magnets for Zero-Shot Referral Segmentation
Sep 26, 2025Anna Kukleva, Enis Simsar, Alessio Tonioni +4Attention Mechanisms Image Segmentation
Max Planck Institute for Informatics, SIC · ETH Zürich · Google +1

Aug 19, 2026J/K move · Enter open · S save

Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models

Pardis Taghavi, Reza Langari, Gaurav Pandey

Texas A&M University

Sparse Attention Block-Sparse Attention Video Generation

Open paper AskPDF arXiv

Block-sparse attention accelerates video transformers by grouping queries and keys and computing attention only between selected groups. However, it faces two challenges: (1) queries sharing a block route may attend to different keys; and (2) retaining most attention mass does not necessarily preserve the attention output. We find that partition choice affects both the overlap among grouped queries' preferred supports and how well an affine function of the sparse output can represent the difference between dense and sparse outputs. We introduce SparsePR, a training-free method combining Response-Coupled Partitioning with Probe-Fitted Residual Reconstruction. We first group paired key/value tokens by how their keys respond to sampled queries. We then group queries by their responses to the resulting key-group centroids for shared sparse routing. We compute exact attention on a small set of query rows and fit an affine correction to the differences between their dense and sparse outputs using weighted ridge regression. The fitted map predicts a residual for each unprobed query, which we add to its sparse output. We measure attention-output error as the relative L2 error with respect to each query's dense output. Across four video generation and world models, SparsePR reduces this error, averaged over unprobed queries, by 61.6--90.5% compared with semantic block-sparse attention without residual correction, at 22% executed-pair density including exact probes. SparsePR achieves 1.48x--2.61x end-to-end speedups over dense attention, with dense-reference PSNR of 24.42--31.84 dB at 21.9--26.0% realized executed-pair density. Project page: https://pardistaghavi.github.io/SparsePR-website/

Abstract

Attention Mechanisms

Explore similar work

Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models

QUEST: A robust attention formulation using query-modulated spherical attention

RefAM: Attention Magnets for Zero-Shot Referral Segmentation