TileShare: Training-Free Acceleration of Overlapping Tiled ViT Decoding

Xiangzhong Ye; Zhongshen Li; Shen You; Ziyang Meng; Liang Xu; Junkai Ji; Qiuzhen Lin; Xiangtao Li

cs.CVOct 23, 2026

TileShare: Training-Free Acceleration of Overlapping Tiled ViT Decoding

Authors: Xiangzhong Ye, Zhongshen Li, Shen You, Ziyang Meng, Liang Xu, Junkai Ji, Qiuzhen Lin, Xiangtao Li,

Abstract

Fixed-window Vision Transformer (ViT) decoders run at deployment resolution as overlapping tiles: every token in an overlap band is processed by several full decoder instances. In the 2.4B-parameter MiniMax-H3 video decoder, about half of all per-token computation is duplicated. Removing the tiles fails because attention, global tokens and coordinates are bound to them. TileShare keeps each released tile's attention and computes every other operation once per token. The released computation changes at one kind of step only: an overlap token's per-tile results are blended into one shared state with the released weights. Interventions show that this blend costs quality where attention reads the shared states, mainly in the first few layers, which TileShare therefore keeps per tile. Without training, keeping six layers per tile decodes 1.38-1.51x faster than batched and compiled tiling on two GPU types (up to 1.96x with FP8) and is equivalent to the released decoder in PSNR, LPIPS and DISTS on DAVIS videos. On unseen UVG videos with the setting frozen beforehand, it matches PSNR and DISTS and improves LPIPS. TileShare carries over to two further decoders and is again confirmed on unseen data. It is equivalent in PSNR and DISTS on MAGI-1. On the ViTok-v2 image autoencoder at 2K-4K, it matches overlapping tiling in PSNR, improves LPIPS and DISTS and decodes 1.5-1.8x faster. How many layers stay per tile is measured once per model (for decoders, without labels); we report it for 14 decoders and encoders. Optionally, two GPU-hours of self-distillation let MiniMax-H3 share every layer without a loss in PSNR, and a coordinate field trained while the network stays frozen brings fully shared CLIP-based open-vocabulary segmentation to parity with sliding windows on seven benchmarks.

Figures & tables

Figure 2: Inside one shared layer. Each token has one state: normalization, QKV projections and QK-normalization run once per token (teal), as do the out-projection and MLP later. Attention still runs per released tile, with the tile’s own global token

g

and coordinates; rulers mark each window’s extent. The overlap tokens 3–5 are stored once but attended in both windows, and their two attention outputs are blended with the released weights

\bar{w}

, the only approximate step.

	$k=4$		$k=0$
Shared operations	$\Delta$ PSNR	FLOPs	$\Delta$ PSNR	FLOPs
none (per-tile states throughout)	$-0.008$	1.000	$-0.008$	1.000
QKV only	$-0.070$	0.924	$-3.262$	0.914
out-projection only	$-0.785$	0.975	$-0.290^{\ast}$	0.971
MLP only	$-0.016$	0.695	$-0.022$	0.657
all (TileShare)	$-0.003$	0.593	$-0.290$	0.543

Table 1: Where the approximation acts (MiniMax-H3, All-38;

\Delta

PSNR in dB relative to the released decoder). Top: from layer

k

on, each tile keeps its own residual stream and only the listed operations are shared; attention always runs per tile. FLOPs: block FLOPs relative to the baseline at 704p. ∗ Equal to TileShare-0 , since identical per-tile states stay identical. Bottom: rule for blending an overlap token’s tiles.

Figure 3: Depth. (a)

\Delta

PSNR relative to tiled decoding for TileShare-

k

against

k/L

(MiniMax-H3: All-38, 95% CIs; MAGI-1: DAVIS-10; ViTok-v2 350M: DIV8K-40 at

4096^{2}

, CIs within the markers); shaded:

\pm 0.05

dB. (b) Merging the per-tile states once at layer

\ell

, then continuing per tile: PSNR of the output to the released output (solid), and for an isotropic perturbation of the same size (dashed); means over 30 (MiniMax-H3) and 42 (MAGI-1) temporal windows.

	DAVIS-30		UVG (prospective)
Method	$\Delta$ PSNR	P / L / D	$\Delta$ PSNR, 704p	$\Delta$ PSNR, 1080p	P / L / D	Speed-up
Released tiling (batched)	0		0	0		1.00 / 1.00 / 1.00
Hard stitching	$-0.199$ [ $-0.212,-0.186$ ]	$\times$ / – / –	–	–		–
TileShare-0	$-0.302$ [ $-0.343,-0.260$ ]	$\times$ / $\times$ / $\times$	–	–		1.69 / 1.54 / 1.61
TileShare-4	$-0.011$ [ $-0.030,+0.009$ ]	✓ / $\times$ / $\times$	$-0.006$ [ $-0.031,+0.018$ ]	$-0.002$ [ $-0.027,+0.024$ ]	✓ / ✓ / ✓	1.56 / 1.44 / 1.50
TileShare-6	$\mathbf{+0.009}$ [ $-0.006,+0.025$ ]	✓ / ✓ / ✓	$+0.018$ [ $+0.000,+0.036$ ]	$+0.031$ [ $+0.013,+0.049$ ]	✓ / $\uparrow$ / ✓	1.51 / 1.41 / 1.45

Table 2: MiniMax-H3 without training.

\Delta

PSNR (dB) vs. ground truth relative to the released tiled decoder [90% CI]. DAVIS-30: held out from the choice of

k=4

, not of

k=6

. UVG: evaluated once with both settings frozen. P / L / D (UVG: both resolutions): ✓ equivalent, the 90% CI within the margin (

\pm 0.05

dB;

\pm 1.16\%

of the baseline mean for LPIPS, DISTS);

\uparrow

better, the CI leaving the margin only on the side of improvement;

\times

neither; – not measured. Speed-up of the compiled decoder alone on an RTX PRO 6000 vs. batched tiling at 704p / 768p / 1080p.

Figure 4: Quality vs. decoder speed-up on MiniMax-H3 (All-38, 704p; larger tiles and whole-frame decoding on 18 clips). Top: the

\pm 0.05

dB band, magnified on the same speed axis; open markers add FP8 MLP layers; bars are 95% CIs.

	Fastest third (All-38)	Corners (1080p)	Steps (‰)	HF (%)
TileShare-4	$-0.051$ [ $-0.077,-0.023$ ]	$-0.162$	4.8 / 5.7 / 2.6	$-4.7$ / $-3.1$
TileShare-6	$-0.030$ [ $-0.047,-0.011$ ]	$-0.107$	2.0 / 1.3 / 0.7	$-2.6$ / $-1.6$

Table 3: Beneath the mean. Fastest third:

\Delta

PSNR (dB) [95% CI] on the third of All-38 with the largest optical flow. Corners: four-tile corners of 1080p-26. Steps: net excess luminance steps of more than 6 gray levels across tile-start lines, per mille of 32-px segments (Gen-44 / 1080p-26 / UVG 1080p; supp. B). HF: high-frequency energy relative to the released decoder (All-38 / UVG 1080p).

Figure 5: The largest local step among held-out real clips, selected on TileShare-4 alone (1080p soapbox, frame 65, start row of a four-tile corner). Top: 80 $\times$ 80 px crops of ground truth, the released decoder, TileShare-4 and TileShare-6 , with the excess step over the released decoder in gray levels. Bottom: tiles per pixel and the absolute difference to the released output ( $\times 8$ ); ticks mark tile-start lines.

Decoder, data	$k$	$\Delta$ PSNR	P / L / D	Whole	Speed-up
MAGI-1, DAVIS-10	3	$-0.029$ [ $-0.035,-0.023$ ]	✓ / ✓ / ✓	$-4.1^{\dagger}$	$1.13$ / $1.14$
MAGI-1, UVG-16	3	$-0.027$ [ $-0.032,-0.023$ ]	✓ / $\times$ / ✓	–	–
ViTok-v2, DIV8K-40	0	$+0.023$ [ $+0.014,+0.031$ ]	✓ / $\uparrow$ / $\uparrow$	$-0.95$	$1.77$ / $2.54$
		$+0.024$ [ $+0.016,+0.033$ ]	✓ / $\uparrow$ / $\uparrow$	$-2.00$	$1.52$ / $7.35$
ViTok-v2, Commons-40	0	$+0.026$ [ $+0.017,+0.037$ ]	✓ / $\uparrow$ / $\uparrow$	$-1.25$	–
		$+0.024$ [ $+0.014,+0.033$ ]	✓ / $\uparrow$ / $\uparrow$	$-2.03$	–

Table 4: MAGI-1 and ViTok-v2 without training.

\Delta

PSNR (dB) relative to overlapping tiled decoding [90% CI]; P / L / D as in Tab. 2 . For each decoder, the first set chose

k

and the second was evaluated once with

k

frozen. MAGI-1: released setting, UVG at 704p; ViTok-v2 350M:

2048^{2}

4096^{2}

. Whole: whole-input coding relative to tiled coding. Speed-up: decoder only, vs. batched tiling / vs. whole-input decoding (MAGI-1: 704p / 720p). † On eight other clips.

Model	Positions	Whole $-$ tiled	$k^{\star}/L$	Speed-up
Whole input as good in quality
DINOv3-B/16 (enc.) [ 45 ]	RoPE	$-0.09^{\dagger}$	8/12	–
DINOv3-L/16 (enc.)	RoPE	$-0.11^{\dagger}$	16/24	–
PE-Spatial-B/16 (enc.) [ 6 ]	RoPE + abs.	$+0.13$	0/12	1.32
ViTok-v2 5B (image dec.) ¶ [ 16 ]	RoPE	$+0.36^{\ddagger}$	0/40	1.61
Tiling needed, small $k^{\star}$

Table 5: Fourteen configurations. Whole

-

tiled: whole-input inference relative to tiled inference (dB PSNR for decoders, mIoU for encoders).

k^{\star}

: smallest per-tile depth that meets the criterion of its model family without training (supp. F). Speed-up of TileShare-

k^{\star}

vs. batched tiling; – none or not measured. † Inconclusive; better whole at short side 896. ‡

4.1\times

slower than tiled decoding with TileShare at 4K. § Batch 8 (

0.89\times

at batch 1). ¶ See Sec. 5.5 ; ViTok-v2 at

2048^{2}

–

4096^{2}

MiniMax-H3, All-38
Variant	Trained	$\Delta$ [95% CI]	Worst	Speed-up
TileShare-0	–	$-0.290$ [ $-0.337,-0.242$ ]	$-0.54$	1.69
TileShare-0 + LoRA self-distillation	51.9M	$\mathbf{+0.008}$ [ $-0.017,+0.032$ ]	$\mathbf{-0.01}$	1.69
TileShare-0 + coordinate field	1.3M	$-0.086$ [ $-0.122,-0.050$ ]	$-0.17$	–
CLIP open-vocabulary segmentation, mean over seven benchmarks
SCLIP, TileShare-0	–	$-2.03$ [ $-2.29,-1.78$ ]	–	1.71–2.15

Table 6: Sharing every layer with training (Trained: new parameters; the coordinate field leaves the network frozen). MiniMax-H3:

\Delta

PSNR (dB) relative to the released decoder; worst coverage zone; decoder speed-up at 704p. CLIP:

\Delta

of the mean mIoU over seven benchmarks relative to sliding windows; worst benchmark; backbone speed-up.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Operation	Layer 0	Layers $\geq k$
post_quant_conv , proj_in	once per token; same input in all tiles, exact	–
Registers / zero token	one set per tile, as released	per tile; attend only to their own tile
Norm1, QKV, QK-RMSNorm	once per token; exact	once, on the shared state
RoPE	per tile, on tile coordinates, as released	as released
Attention (dense, per tile)	released key set, coordinates and registers; exact	keys and values from shared states
Blend of attention outputs	the only approximation (overlap tokens)	in every layer

Appendix

Table S1: Each operation of a MiniMax-H3 block under TileShare.

Method	DAVIS-30	Dev-8	All-38 [95% CI]	1080p-26	UVG 704p	UVG 1080p	$\Delta$ LPIPS	$\Delta$ DISTS
Hard stitching	$-0.199$	$-0.175$	$-0.194$ [ $-0.209,-0.179$ ]	–	–	–	$+0.0010$	–
TileShare-0	$-0.302$	$-0.243$	$-0.290$ [ $-0.337,-0.242$ ]	$-0.353$	–	–	$+0.0037$	$+0.0030$
TileShare-4	$-0.011$	$+0.029$	$-0.003$ [ $-0.026,+0.021$ ]	$-0.017$	$-0.006$	$-0.002$	$+0.0009$	$+0.0004$
TileShare-6	$+0.009$	$+0.041$	$+0.015$ [ $-0.003,+0.034$ ]	$+0.009$	$+0.018$	$+0.031$	$\mathbf{+0.0002}$	$\mathbf{-0.0002}$
FP8 MLP layers only	$-0.008$	$-0.010$	$-0.008$ [ $-0.014,-0.004$ ]	–	–	–	$\mathbf{+0.0000}$	–
TileShare-4 + FP8	$-0.020$	$+0.014$	$-0.012$ [ $-0.040,+0.014$ ]	$-0.026$	–	–	$+0.0009$	$+0.0004$

Appendix

Table S2: MiniMax-H3 without training, all evaluation sets.

\Delta

PSNR (dB). Last two columns:

\Delta

LPIPS and

\Delta

DISTS on All-38 (bold: within the strict margin of 1.16% of the baseline’s mean).

Figure S1 : Per-video

\Delta

PSNR of TileShare-4 and TileShare-6 relative to the released decoder on DAVIS-30, 1080p-26 and UVG at 704p and 1080p; bars: mean with 90% CI; shaded:

\pm 0.05

dB.

Set	Method	Slow	Medium	Fast	$\rho$
All-38	TileShare-4	$+0.041$	$+0.003$	$-0.051$ [ $-0.077,-0.023$ ]	$-0.50$
All-38	TileShare-6	$+0.055$	$+0.022$	$-0.030$ [ $-0.047,-0.011$ ]	$-0.60$
All-38	spatio-temporal	$+0.017$	$-0.028$	$-0.076$ [ $-0.102,-0.046$ ]	–
1080p-26	TileShare-4	$+0.011$	$-0.030$	$-0.032$ [ $-0.067,+0.007$ ]	$-0.45$
1080p-26	TileShare-6	$+0.034$	$+0.002$	$-0.009$ [ $-0.037,+0.022$ ]	$-0.51$

Appendix

Table S3: Motion strata.

\Delta

PSNR (dB) by thirds of mean optical-flow magnitude (All-38: boundaries 4.7 / 13.3 px per frame; 1080p-26: 10.1 / 23.4).

\rho

: Spearman correlation of

\Delta

PSNR with flow magnitude.

Set	Method	WE (%)	Flicker	Jump
All-38	TileShare-4	$-1.80$ / $-0.68$	$+0.016$	$-0.100$
All-38	TileShare-6	$-0.83$ / $+0.54$	$+0.012$	$-0.035$
All-38	spatio-temporal	$-0.57$ / $+0.60$	$+0.043$	$-0.040$
1080p-26	TileShare-4	$-1.26$ / $+1.60$	$+0.087$	$-0.102$
1080p-26	TileShare-6	$-0.56$ / $+2.23$	$+0.060$	$-0.036$
1080p-26	spatio-temporal	$-0.38$ / $+2.53$	$+0.112$	$-0.062$

Appendix

Table S4: Temporal consistency and zones (relative to V0). WE: warping error with ground-truth flow (%), full frame / four-tile corners. Flicker: residual frame difference in four-tile corners (

\times 255

). Jump: excess frame difference at temporal-window boundaries (

\times 255

Method	704p	768p	1080p	Mem.
V0, per tile, eager (released)	28.31 (0.32)	27.42 (0.23)	29.76 (0.26)	12.6
V0, batched, compiled	8.92 (1.00)	6.25 (1.00)	7.60 (1.00)	14.9
V0 + FP8	6.59 (1.35)	4.62 (1.35)	5.63 (1.35)	16.2
TileShare-0	5.28 (1.69)	4.06 (1.54)	4.72 (1.61)	15.7
TileShare-4	5.71 (1.56)	4.34 (1.44)	5.07 (1.50)	15.1–16.8
TileShare-6	5.92 (1.51)	4.43 (1.41)	5.23 (1.45)	15.1

Appendix

Table S5: Decoder time (s) and speed-up vs. batched tiling, RTX PRO 6000. Peak allocated memory (GB) at 704p.

Component	V0	TileShare-4	Saved	Ratio
MLP GEMMs	4674	2596	2079	0.56
QKV + out-projection GEMMs	1596	913	683	0.57
Attention (SDPA)	851	842	9	0.99
RMSNorm + residual	626	410	216	0.66
QK-norm + RoPE (+ gather)	334	311	23	0.93
SwiGLU gate	597	327	270	0.55

Appendix

Table S6: Where the time goes (704p, GPU kernel time per decode, ms, torch.profiler ). Ideal: linear-layer FLOP ratio 0.549 applied to all per-token work, attention unchanged.

Figure S2 : Where the decoding time goes (MiniMax-H3, 704

\times

1280

\times

175, compiled, GPU kernel time per decode). Per-token GEMMs and element-wise work shrink in proportion to the token count; attention is unchanged; the blend and scatter of TileShare are new.

Shared	$k=4$	FLOPs	$k=0$	FLOPs
none	$-0.008$	1.000	$-0.008$	1.000
q	$-0.070$	0.924	$-2.666$ ( $-3.262$ )	0.914
o	$-0.785$	0.975	$-0.243$	0.971
m	$-0.016$	0.695	$+0.006$ ( $-0.022$ )	0.657
q + o	$-0.036$	0.898	$-0.243$	0.886
q + m ( $\equiv$ all)	$-0.003$	0.619	$-0.243$	0.571

Appendix

Table S7: Shared operations, all combinations (MiniMax-H3,

\Delta

PSNR in dB;

k=4

: All-38;

k=0

: Dev-8, except where All-38 is given in parentheses). q: Norm1 + QKV + QK-norm; o: out-projection; m: MLP. Per-tile residual streams are kept from layer

k

on.

$k$	$\Delta$ PSNR [95% CI]	$\Delta$ LPIPS	$\Delta$ DISTS	HF (%)	Corners
0	$-0.290$ [ $-0.337,-0.242$ ]	$+0.0037$	$+0.0030$	$-4.5$	$-0.544$
2	$-0.223$ [ $-0.266,-0.180$ ]	$+0.0030$	$+0.0021$	$-5.3$	$-0.431$
3	$-0.113$ [ $-0.147,-0.079$ ]	$+0.0024$	$+0.0014$	$-6.4$	$-0.270$
4	$-0.003$ [ $-0.026,+0.021$ ]	$+0.0009$	$+0.0004$	$-4.8$	$-0.059$
5	$+0.013$ [ $-0.008,+0.033$ ]	$+0.0005$	$+0.0001$	$-3.9$	$-0.035$
6	$+0.015$ [ $-0.003,+0.034$ ]	$+0.0002$	$-0.0002$	$-2.6$	$-0.032$

Appendix

Table S8: Per-tile-depth sweep on MiniMax-H3 (All-38). LPIPS and DISTS as differences; HF: high-frequency energy relative to V0, on float outputs (the main paper’s Tab. 4 uses uint8 frames, hence

-4.7

there for

k=4

); zones: four-tile corners.

Figure S3 : Distortion and perception along the per-tile-depth sweep (MiniMax-H3, All-38, 95% CIs). Between

k=8

and 18, TileShare-

k

is about 0.05 dB below the released decoder in PSNR but sharper than it, with lower LPIPS.

Figure S4 : A first-order account of the merge cost (MiniMax-H3: 24 temporal windows; MAGI-1: 14 segments). (a) Merging the per-tile states once at layer

\ell

: PSNR of the output to the unmerged output (solid), of the first-order term

\sum_{t}w_{t}J_{t}\delta_{t}

computed by forward-mode differentiation (dashed), and of a random perturbation of the same size on the same windows (dotted). (b) Cancellation ratio

C

(1: no cancellation; band:

\pm 1

s.d. over windows).

	SCLIP				NACLIP				ClearCLIP
Benchmark	V0	W	$k_{0}$	F	V0	W	$k_{0}$	F	V0	W	$k_{0}$	F
VOC21	59.6	$-1.0$	$-2.2$	$+0.2$	57.6	$+1.2$	$-2.2$	$+0.2$	51.8	$-0.9$	$-1.4$	$+1.0$
Context60	32.6	$-2.2$	$-1.4$	$+0.0$	33.4	$-0.4$	$-1.9$	$-0.0$	33.8	$-1.1$	$-1.1$	$+0.1$
COCO-Obj.	33.5	$-1.7$	$-2.2$	$+0.1$	33.2	$-0.3$	$-2.4$	$-0.2$	33.0	$-2.1$	$-1.7$	$-0.6$
VOC20	81.5	$+0.6$	$-2.2$	$+0.8$	77.1	$+2.6$	$-2.5$	$+0.9$	80.9	$-1.5$	$-1.7$	$-0.1$
Context59	34.1	$-2.4$	$-1.8$	$+0.0$	35.2	$-0.3$	$-2.1$	$-0.0$	35.8	$-1.1$	$-1.2$	$+0.2$

Appendix

Table S9: CLIP open-vocabulary segmentation (

\Delta

mIoU vs. sliding windows). W: whole image.

k_{0}

: TileShare-0 without the field. F: TileShare-0 with the coordinate field trained on SCLIP.

Change	$\Delta$ PSNR	Speed-up
Whole-frame decoding	$-11.93$	$0.99$ / $0.81$
Larger tiles (320 / 384 / 512 px)	$-2.41$ / $-6.86$ / $-9.03$	$0.93$ – $1.41$
Less overlap (16 / 0 px)	$-0.342$ / $-0.747$	$1.55$ / $1.86$ (704p)
Temporal re-chunking	$-3$ to $-9$	–
Best single-pass sliding window	$-0.67$	$1.01$ / $0.92$

Appendix

Table S10: Naive deduplication of MiniMax-H3.

\Delta

PSNR (dB); tile sizes and whole frame on 18 clips, overlap widths on All-38, sliding window on Dev-8. Speed-up vs. batched and compiled tiling at 704p / 768p.

Set, size	Variant	$\Delta$ PSNR	$\Delta$ LPIPS / $\Delta$ DISTS ( $10^{-3}$ )	P / L / D
DIV8K, 2048	Whole	$-0.945$ [ $-1.292,-0.653$ ]	$-0.0$ / $-1.3$	$\times$ / $\times$ / $\uparrow$
	Hard	$+0.009$ [ $-0.001,+0.020$ ]	$-0.2$ / $-0.3$	✓ / ✓ / $\uparrow$
	TileShare-0	$+0.023$ [ $+0.014,+0.031$ ]	$-1.0$ / $-1.6$	✓ / $\uparrow$ / $\uparrow$
	TileShare-2	$+0.023$ [ $+0.015,+0.031$ ]	$-1.0$ / $-1.6$	✓ / $\uparrow$ / $\uparrow$
	TileShare-4	$+0.020$ [ $+0.012,+0.028$ ]	$-1.1$ / $-1.7$	✓ / $\uparrow$ / $\uparrow$
	TileShare-8	$+0.051$ [ $+0.042,+0.061$ ]	$-0.5$ / $-1.1$	$\uparrow$ / $\uparrow$ / $\uparrow$

Appendix

Table S11: ViTok-v2 in detail. Difference to tiled coding [90% CI], 40 images each; equivalence marks for PSNR / LPIPS / DISTS as in the main paper (LPIPS and DISTS margins: 0.00035–0.00046 and 0.00023–0.00040). Whole: the released whole-image coding. Hard: each pixel from its dominant tile. +enc: TileShare also in the encoder.

Model, size	Scope	Tiled	Whole	TileShare	vs. tiled	vs. whole
350M, 2048	decoder	76.9	110.2	43.5	$1.77\times$	$2.54\times$
	pipeline	89.9	128.3	55.7 (50.5)	$1.61\times$ ( $1.78\times$ )	$2.30\times$ ( $2.54\times$ )
350M, 4096	decoder	271.9	1310.6	178.4	$1.52\times$	$7.35\times$
	pipeline	322.8	1553.9	229.2 (216.5)	$1.41\times$ ( $1.49\times$ )	$6.78\times$ ( $7.18\times$ )
5B, 4096	decoder	3036.5	7828.7	1885.2	$1.61\times$	$4.15\times$
	pipeline	3401.9	8636.5	2212.1 (2089.5)	$1.54\times$ ( $1.63\times$ )	$3.90\times$ ( $4.13\times$ )

Appendix

Table S12: ViTok-v2 timing (ms, median of ten compiled runs, bf16, one image, RTX PRO 6000 with no other process). Decoder: latents to blended pixels; pipeline: encoder and decoder. In parentheses: TileShare also in the encoder.

Figure S5 : ViTok-v2 at $4096^{2}$ : DIV8K image 0880 at the $256\times 256$ window where tiled coding gains most over whole-image coding, chosen by the preregistered rule; it is the most extreme case, not a typical one (on average, the gain is 2.0 dB). Top: decoded crops with local PSNR, displayed with values $\times 4$ because the window is dark; bottom: absolute error to the ground truth, computed on the original values ( $\times 8$ ). Whole-image coding (released usage) shows a 16 px token grid; overlapping tiles and TileShare-0 match the ground truth.

Efficient ViTs

Efficient Vision Transformers reduce the compute, memory, energy, or latency needed to process images and video while retaining useful accuracy.

See all 243 papers

Aug 9, 2026

Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding

Distributed deployment of large vision foundation models often partitions a ViT backbone and exchanges intermediate token features between computing nodes, making efficient feature compression critical under bandwidth and computation constraints. Existing ViT feature codecs typically flatten heterogeneous global and patch tokens into an L x C pseudo image, causing entropy models to mainly capture sequence-axis dependencies while overlooking the native two-dimensional patch-grid structure. In this paper, we show that ViT patch tokens retain strong local spatial correlations on the original grid. To exploit this structural prior, we propose the Visual Token Codec (VTC), a dual-path learned codec that separates global and patch tokens into dedicated coding paths. Global tokens are compressed with a lightweight factorized prior, whereas patch tokens are encoded on the patch-token grid using a spatial-channel context entropy model. To support intermediate-layer compression and practical rate adaptation, VTC further incorporates feature-matching supervision after subsequent ViT blocks and variable-rate modules within a single codec. Experiments on DINOv2 and SAM3 show that VTC consistently outperforms representative ViT feature coding baselines on classification, segmentation, and detection tasks. At 90% of uncompressed-feature performance, VTC reduces bitrate by 15.7x-37.4x across these tasks. We further provide intermediate-layer rate-utility analyses for practical transmission- and storage-oriented deployment scenarios.

Donghui Feng, Fengxi Zhang, Changsheng Gao +6

School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China · College of Computing and Data Science, Nanyang Technological University, Singapore · Pengcheng Laboratory, Shenzhen, Guangdong, China +1

Entropy Modeling Visual Token Compression

Ask PDF

Oct 26, 2026

Triage: Skip, Skim, or Refine Attention Tiles in Video Diffusion Transformers

Attention over tens of thousands of visual tokens dominates the inference of video diffusion transformers (DiTs). Existing methods reduce this cost through sparsity, low precision, or both, but use a single precision for all kept tiles. We observe that the output error introduced by approximation varies widely across tiles. Many tiles can be skipped with little error, others must be computed but gain little beyond low precision, and a few retain substantial error even at low precision. We present Triage, a training-free attention method that assigns each tile one of three approximation tiers: skipping it by replacing it with its block means, skimming it in NVFP4, or refining it with NVFP4 residual correction. A lightweight router predicts each tile's error under each tier from block statistics and moves a tile to a more accurate tier only when the predicted error reduction justifies the extra compute, with a single price setting this trade-off. A fused kernel executes all three tiers in a single pass on the FP4 tensor cores of RTX Blackwell GPUs. Across Wan2.2, HunyuanVideo, and LTX-2.5, Triage accelerates end-to-end generation by up to 2.32x over dense FlashAttention, and in each of five settings it runs faster than every baseline while reaching 0.3-3.0 dB higher PSNR than the highest-PSNR baseline.

Yeonjae Kim, Omin Kwon, Seong Hoon Seo +1

Department of Computer Science and Engineering, Seoul National University, Seoul, Republic of Korea.

Diffusion Model Inference Acceleration Video Diffusion Models Adaptive Computation for Diffusion Models Diffusion Transformer Diffusion Model Quantization

Ask PDF

Jun 16, 2026

TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization

Video tokenization is fundamental to scalable video generation, as the number of tokens directly determines the computational cost and the length of videos that can be modeled. Existing tokenizers mainly improve scalability by compressing videos into fewer tokens, but they often continue to represent persistent content, such as static backgrounds and consistent object appearances, repeatedly across frames and chunks. In this paper, we propose TivTok (Time-Invariant Tokenizer), a reuse-aware video tokenizer that makes persistent information reusable across time. TivTok represents a clip with Time-Invariant (TIV) tokens that encode information shared across frames and Time-Variant (TV) tokens that encode frame-specific residuals. To obtain this factorization, we introduce Scope-Induced Factorization (SIF), which assigns different attention scopes to the two token groups: TIV tokens attend to the full clip, whereas each TV token only accesses its corresponding frame together with the TIV tokens. In the decoder, Invariant Broadcasting (IB) reuses the same TIV tokens across frames and chunks for parallel reconstruction and long-video tokenization. Experiments show that TivTok achieves an rFVD of 12.65 on the standard

16{\times}256{\times}256

benchmark and improves compression efficiency by 2.91

\times

for 128-frame videos compared with the evaluated baselines, while using only 1.1% of the tokens required by downsample-based tokenizers in our evaluation.

Weiliang Chen, Yuanhui Huang, Xuebo Wang +1

Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China · Department of Automation, Tsinghua University, Beijing, 100084, China · Kuaishou Technology, Beijing, China

Visual Tokenization Video Token Compression Visual Token Compression

Ask PDF

Abstract

Efficient ViTs

Explore similar work

Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding

Triage: Skip, Skim, or Refine Attention Tiles in Video Diffusion Transformers

TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization