Later Is Better: Token Reduction for ViTs Under Distribution Shift

Hyeongheon Cha; Hyungjun Yoon; Sung-Ju Lee

cs.CVOct 6, 2026

Later Is Better: Token Reduction for ViTs Under Distribution Shift

Authors: Hyeongheon Cha, Hyungjun Yoon, Sung-Ju Lee

Organizations: KAIST

Abstract

Training-free token reduction accelerates vision transformers by removing redundant tokens across layers, recovering most of the original accuracy at a fraction of the compute. These methods, however, are designed and evaluated primarily on clean data, and under real-world distribution shift their accuracy gap to the uncompressed model widens with the removal rate. We show that this gap is governed by the reduction schedule, the depth profile of removal, usually left fixed as an implementation detail. Concretely, we introduce a one-parameter late-concentrated power-law schedule that consistently improves out-of-distribution accuracy over flat at no extra inference cost. On ImageNet-C with DeiT-S, the late schedule closes 83% of that gap at a 26% compute reduction (+1.17pp), and 99% of it at a lighter 7% reduction (+0.26pp). The gain cannot be attributed to retaining more tokens or using extra compute: held to flat's compute, the late schedule removes more tokens in total and leaves fewer tokens at the end, yet still wins. Single-layer probes point to a mechanism: earlier reductions perturb features that pass through more remaining layers, front-loading reduction error in depth. The effect is broad, holding across five token-reduction methods (ToMe, EViT, ATS, ATC, PiToMe), nine backbones, all ImageNet-C corruption types, eight further shift suites, and two further modalities, video and vision-language QA. It is also specific to shift, still positive on clean and rising monotonically to ~4x that at the highest severity 5. The schedule keeps its gain under six test-time adaptation methods, and needs no per-input or per-domain tuning.

Figures & tables

Figure 1: Token reduction increasingly sacrifices out-of-distribution accuracy; a late-concentrated schedule recovers it at no extra compute . (a, b) DeiT-S top-1 vs. the per-layer token removal rate $r_{\mathrm{base}}$ on clean and gaussian-noise ImageNet-C. (c) each ribbon’s height is the tokens remaining, so the pale band above is what was removed; flat removes the same number at every layer, ours follows the late-concentrated power law. Ours ends with far fewer tokens, yet closer to uncompressed.

Figure 2: The late-concentrated schedule and its effect (DeiT-S, ImageNet-C severity 5). (a) schedule shape: tokens removed per layer

r_{\ell}\!\propto\!(\ell/L)^{\gamma}

vs. layer index (at

r_{\mathrm{base}}{=}8

), for

\gamma{=}0

(flat),

1

, and our default

2

; larger

\gamma

defers removal to later layers. (b) 15-corruption-mean top-1 vs. compute (GFLOPs), each curve a schedule swept over compression (points labeled by

r_{\mathrm{base}}

);

\star

marks the uncompressed model. (c) gain over flat vs.

\gamma

, one curve per compression level

r_{\mathrm{base}}\!\in\!\{4,8,12\}

Figure 3: Per-layer reduction distortion and amplification (DeiT-S, ImageNet-C severity 5; means over

N{=}1024

images, standard error

<\!2\%

). (a) The directly-measured distortion

D(\ell)

of reducing at layer

\ell

only (relative output-logit change, gaussian noise), normalized to layer 1, for all five reducers. (b) The amplification

A(\ell)

(clean and one corruption per ImageNet-C group). (c)

D(\ell)

against the compute a removal at layer

\ell

saves,

S(\ell)

, both normalized;

D

falls far faster than

S

Figure 4: Oracle clean merge decisions recover less than the late schedule at every

r_{\mathrm{base}}

, the gap widening with compression.

			late-concentrated $\gamma$ ( $\Delta$ vs. flat)
$r_{\mathrm{base}}$	GFLOPs (cut)	flat	$0.5$	$1.0$	$1.5$	$2.0$	$2.5$	$3.0$
$2$	3.97G ( $-7\%$ )	39.14	$+0.09$	$+0.20$	$+0.22$	$+0.26$	$+0.27$	$\underline{+0.28}$
$4$	3.69G ( $-13\%$ )	38.83	$+0.23$	$+0.38$	$+0.50$	$+0.54$	$\underline{+0.61}$	$+0.60$
$6$	3.34G ( $-21\%$ )	38.44	$+0.34$	$+0.69$	$+0.81$	$\underline{+0.88}$	$+0.83$	$+0.69$
$8$	3.15G ( $-26\%$ )	37.99	$+0.57$	$+0.90$	$+1.11$	$\underline{\mathbf{+1.17}}$	$+1.14$	$+1.16$
$10$	2.90G ( $-32\%$ )	37.41	$+0.67$	$+1.17$	$\underline{+1.28}$	$+1.25$	$+1.15$	$+0.92$

Table 1:

\gamma

sensitivity across operating points at iso-GFLOPs (DeiT-S, ImageNet-C severity 5, 15-corruption mean top-1, %). The GFLOPs column is the default

\gamma{=}2

schedule’s compute and its reduction against the uncompressed model (

39.40

4.25

G; per-

\gamma

schedules in Table 31 ). flat is the const-

r_{\mathrm{base}}

baseline in absolute top-1; the

\gamma

columns are

\Delta

= late

-

flat. Shaded = the default

\gamma{=}2

; bold = main result (

r_{\mathrm{base}}{=}8

); underline = per-row optimum

\gamma^{\star}

Sev.	flat	late	$\Delta$	$\Delta/\Delta_{0}$
0 (clean)	79.40	79.68	$+0.28$	1.0 $\times$
1	70.52	71.04	$+0.52$	1.9 $\times$
2	64.33	64.99	$+0.66$	2.4 $\times$
3	59.06	59.89	$+0.83$	3.0 $\times$
4	50.06	51.12	$+1.06$	3.8 $\times$
5	37.99	39.16	$+1.17$	4.2 $\times$

Table 2: Per-severity 15-corruption mean (%), default

\gamma{=}2

r_{\mathrm{base}}{=}8

, DeiT-S; severity 0 is clean ImageNet val.

\Delta

= late

-

flat,

\Delta/\Delta_{0}

its multiple of the clean gain.

DeiT-S
	$r_{\mathrm{base}}{=}4$ ( $-13\%$ )			$r_{\mathrm{base}}{=}6$ ( $-21\%$ )			$r_{\mathrm{base}}{=}8$ ( $-26\%$ )
Method	flat	late	$\Delta$	flat	late	$\Delta$	flat	late	$\Delta$
ToMe	38.83	39.37	$+0.54$	38.44	39.32	$+0.88$	37.99	39.16	$+1.17$
EViT	37.50	38.97	$+1.47$	36.69	38.91	$+2.22$	35.98	38.50	$+2.52$
ATS	34.95	35.43	$+0.48$	34.50	35.21	$+0.71$	33.92	34.88	$+0.96$
ATC	38.28	38.73	$+0.45$	37.96	38.74	$+0.78$	37.55	38.65	$+1.10$

Table 3: Late-concentrated schedule applied to five training-free token-reduction methods at the default

\gamma{=}2

. DeiT-S/B, ImageNet-C severity 5, 15-corruption mean top-1 (%), iso-GFLOPs; each header’s % is the

\gamma{=}2

compute reduction vs. the uncompressed model. For each operating point we report flat, late (bold), and

\Delta

= late

-

flat (pp); The

\gamma{=}1

and larger

r_{\mathrm{base}}

results are in Appendix J .

Table 9

	ViT-S			ViT-B
Method	flat	late	$\Delta$	flat	late	$\Delta$
Source (inference-only)	38.05	39.46	$+1.41$	53.39	54.96	$+1.57$
Tent ( Wang et al., 2021 )	48.47 $\pm 0.00$	52.04 $\pm 0.00$	$+3.57$ $\pm 0.00$	54.49 $\pm 0.00$	55.72 $\pm 0.00$	$+1.23$ $\pm 0.00$
EATA ( Niu et al., 2022 )	52.65 $\pm 0.16$	53.66 $\pm 0.26$	$+1.00$ $\pm 0.38$	59.09 $\pm 0.62$	60.11 $\pm 0.37$	$+1.01$ $\pm 0.43$
SAR ( Niu et al., 2023 )	51.52 $\pm 0.00$	51.96 $\pm 0.00$	$+0.43$ $\pm 0.00$	59.71 $\pm 0.00$	61.07 $\pm 0.00$	$+1.35$ $\pm 0.00$
FOA ( Niu et al., 2024 )	42.42 $\pm 0.48$	42.81 $\pm 0.30$	$+0.39$ $\pm 0.60$	59.80 $\pm 0.10$	60.41 $\pm 0.14$	$+0.61$ $\pm 0.09$

Table 6: Late-concentrated schedule layered under TTA methods. ImageNet-C severity 5, 15-corruption mean top-1,

r_{\mathrm{base}}{=}8

, iso-GFLOPs; each method is run on the flat and the late (

\gamma{=}2

) schedule, which cuts GFLOPs by

26\%

against the same method on full tokens (Appendix K ).

\Delta

= late

-

flat (pp);

\pm

= std over 3 seeds.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Sev	flat	$\gamma{=}1$	$\Delta_{\gamma 1}$	$\gamma{=}2$	$\Delta_{\gamma 2}$
0 (clean)	79.40	79.67	$+0.27$	79.68	$+0.28$
1	70.52	70.95	$+0.43$	71.04	$+0.52$
2	64.33	64.82	$+0.49$	64.99	$+0.66$
3	59.06	59.71	$+0.65$	59.89	$+0.83$
4	50.06	50.88	$+0.82$	51.12	$+1.06$
5	37.99	38.89	$+0.90$	39.16	$+1.17$

Appendix

Table 7: Per-severity 15-corruption mean (%), DeiT-S,

r_{\mathrm{base}}{=}8

, iso-GFLOPs. flat = const (

\gamma{=}0

);

\Delta

= late

-

flat for

\gamma{=}1

and the default

\gamma{=}2

; sev=0 is clean ImageNet val.

Backbone	flat	late $\gamma{=}1$ ( $\Delta$ )	late $\gamma{=}2$ ( $\Delta$ )
DeiT-S	36.67	37.88 ( $+1.21$ )	37.77 ( $+1.10$ )
DeiT-B	41.44	42.96 ( $+1.52$ )	42.74 ( $+1.30$ )
ViT-S	36.07	37.48 ( $+1.41$ )	36.92 ( $+0.85$ )
ViT-B	50.93	52.71 ( $+1.78$ )	52.15 ( $+1.22$ )

Appendix

Table 8: Cross-backbone generalization at the aggressive rate

r_{\mathrm{base}}{=}12

(ImageNet-C sev 5, 15-corruption mean top-1, %, iso-GFLOPs).

\Delta

= late

-

flat.

Corruption	flat	late	$\Delta$	flat	late	$\Delta$
	ViT-B			ViT-S
gaussian_noise	53.64	56.30	$+$ 2.66	31.51	33.38	$+$ 1.87
shot_noise	53.85	56.16	$+$ 2.31	30.93	32.54	$+$ 1.61
impulse_noise	54.11	56.92	$+$ 2.82	31.08	32.93	$+$ 1.85
defocus_blur	44.74	46.98	$+$ 2.23	30.26	31.78	$+$ 1.52
glass_blur	35.04	35.57	$+$ 0.53	22.61	23.00	$+$ 0.39

Appendix

Table 9: Per-corruption top-1 (%), ViT-B and ViT-S, sev=5, 50k samples per corruption, at the main operating point

r_{\mathrm{base}}{=}8

. flat = const (

\gamma{=}0

); late =

\gamma{=}2

\Delta

= late

-

flat. The late schedule uses no more GFLOPs than flat on each backbone, and all 15 corruptions improve on both.

	ImageNet		DN target domains			DN in-domain
Backbone	A	V2	sketch	clipart	painting	real
ViT-S	$+1.24^{*}$	$+0.59^{*}$	$+0.63^{*}$	$+0.56^{*}$	$+0.90^{*}$	$+0.04$
ViT-B	$+2.60^{*}$	$+0.09$	$+0.35^{*}$	$+0.28^{*}$	$+0.46^{*}$	$-0.04$
DeiT-S	$+1.67^{*}$	$+0.38^{*}$	$-0.10$	$-0.01$	$+0.62^{*}$	$+0.11$
DeiT-B	$+1.96^{*}$	$+0.55^{*}$	$+0.47^{*}$	$+0.33^{*}$	$+0.82^{*}$	$+0.24$
CLIP ViT-B	$+2.56^{*}$	$+0.16$	$+0.29^{*}$	$+0.19^{*}$	$+0.56^{*}$	$-0.03$

Appendix

Table 10: Natural shift across backbones,

\Delta

= late

\gamma{=}2

-

flat (pp) at iso-GFLOPs,

r_{\mathrm{base}}{=}8

. ∗ marks an exact McNemar

p<0.05

; those cells also have a

95\%

paired bootstrap interval excluding zero. DN = DomainNet-126, whose real column is the in-domain control.

Corruption	flat	late	$\Delta$	flat	late	$\Delta$
	DeiT-S			ViT-S
gaussian_noise	23.41	24.88	$+$ 1.47	14.85	16.04	$+$ 1.19
shot_noise	23.78	24.95	$+$ 1.17	16.23	17.83	$+$ 1.60
impulse_noise	26.02	27.47	$+$ 1.45	15.60	16.76	$+$ 1.16
defocus_blur	16.49	16.70	$+$ 0.21	6.98	7.03	$+$ 0.05
glass_blur	16.92	17.53	$+$ 0.61	17.29	17.40	$+$ 0.11

Appendix

Table 11: NaBirds-C per-corruption top-1 (%), DeiT-S and ViT-S (NaBirds-finetuned, 555 classes), sev=5, 24,633 images. flat = const (

r_{\mathrm{base}}{=}8

); late =

\gamma{=}2

(the default,

\bar{r}{=}16

, 3.147G

\leq

flat).

\Delta

= late

-

flat.

Corruption	flat	late	$\Delta$	flat	late	$\Delta$
	Severity 3			Severity 5
Shot noise	67.40	68.28	$+0.88$	50.04	51.68	$+1.64$
Fog	70.92	71.36	$+0.44$	66.80	67.96	$+1.16$
Contrast	76.08	76.08	$+0.00$	73.20	73.08	$-0.12$
Brightness	75.84	75.52	$-0.32$	71.72	71.80	$+0.08$
Saturate	75.64	75.76	$+0.12$	65.60	65.52	$-0.08$

Appendix

Table 12: Cross-modality, per-corruption. VideoMAE ViT-B on Kinetics-400 with ToMe, inference-only, iso-GFLOPs (flat

r{=}48

vs. late

\gamma{=}2

, both

\approx

142.6 GFLOPs,

\sim

20% reduction), six spatial corruptions of Mini-Kinetics-C ( Yi et al., 2021 ) , 2500 clips.

\Delta

= late

-

flat.

Backbone	flat	late	$\Delta$	flat	late	$\Delta$
	ImageNet-R			ImageNet-Sketch
DeiT-S	42.14	42.45	$+0.31$	29.40	29.68	$+0.28$
DeiT-B	44.56	44.63	$+0.07$	32.21	32.42	$+0.21$
ViT-S	45.90	46.57	$+0.67$	32.89	33.33	$+0.44$
ViT-B	58.77	58.85	$+0.08$	44.92	45.09	$+0.17$

Appendix

Table 13: ImageNet-R / ImageNet-Sketch, all backbones.

r_{\mathrm{base}}{=}8

, late

\gamma{=}2

, iso-GFLOPs.

\Delta

= late

-

flat (top-1 pp). R = 30,000 images, Sketch = 50,889.

Corruption	flat	late	$\Delta$	flat	late	$\Delta$
	DeiT-S			ViT-S
blue noise	50.36	50.62	$+0.26$	53.08	55.10	$+2.02$
brownish noise	66.40	67.36	$+0.96$	60.42	62.34	$+1.92$
caustic refr.	53.60	54.46	$+0.86$	51.32	53.12	$+1.80$
checkerboard	62.66	63.58	$+0.92$	64.06	65.20	$+1.14$
sine waves	31.42	31.70	$+0.28$	24.62	26.42	$+1.80$

Appendix

Table 14: ImageNet-

\overline{\mathrm{C}}

per-corruption top-1 (%), DeiT-S and ViT-S,

r_{\mathrm{base}}{=}8

, late

\gamma{=}2

, iso-GFLOPs, severity 5, 5k-image subset.

\Delta

= late

-

flat.

Corruption	flat	late	$\Delta$	flat	late	$\Delta$
	DeiT-S			ViT-S
bit error	15.50	16.04	$+0.54$	16.70	16.98	$+0.28$
color quant	59.82	59.80	$-0.02$	41.56	41.44	$-0.12$
far focus	50.96	51.62	$+0.66$	53.30	54.14	$+0.84$
flash	39.28	39.84	$+0.56$	38.98	39.70	$+0.72$
fog 3d	38.72	38.52	$-0.20$	31.34	31.96	$+0.62$

Appendix

Table 15: ImageNet-3DCC per-corruption top-1 (%), DeiT-S and ViT-S,

r_{\mathrm{base}}{=}8

, late

\gamma{=}2

, iso-GFLOPs, severity 5, 5k-image subset.

\Delta

= late

-

flat.

Corruption	flat	late	$\Delta$
Speckle noise	41.67	43.98	$+2.30$
Gaussian blur	23.38	24.09	$+0.71$
Spatter	47.15	49.72	$+2.57$
Saturate	62.10	63.17	$+1.08$
Mean	43.58	45.24	$+1.66$

Appendix

Table 16: The four ImageNet-C corruptions outside the standard fifteen (DeiT-S, severity 5, 50k images each,

r_{\mathrm{base}}{=}8

, iso-GFLOPs, inference-only).

\Delta

= late

-

flat.

	$r_{\mathrm{base}}{=}6$			$r_{\mathrm{base}}{=}8$			$r_{\mathrm{base}}{=}10$
Method	flat	late	$\Delta$	flat	late	$\Delta$	flat	late	$\Delta$
DeiT-S
ToMe	38.44	39.13	$+0.69$	37.99	38.89	$+0.90$	37.41	38.58	$+1.17$
EViT	36.69	38.10	$+1.41$	35.98	37.87	$+1.89$	35.16	37.62	$+2.46$
ATS	34.50	34.99	$+0.49$	33.92	34.61	$+0.69$	33.35	34.38	$+1.03$
ATC	37.96	38.53	$+0.57$	37.55	38.35	$+0.80$	37.04	38.26	$+1.22$

Appendix

Table 17: Late-concentrated at

\gamma{=}1

across five training-free reducers (companion to the

\gamma{=}2

body Table 3 ). DeiT-S/B, ImageNet-C sev=5, 15-corruption mean top-1 (%), full 50k, iso-GFLOPs (late

\leq

flat).

\Delta

= late

-

flat (pp); all 30 cells positive. Operating points

r_{\mathrm{base}}\in\{6,8,10\}

Figure 7: 15-corruption-mean top-1 vs. compute for each of the five training-free reducers (DeiT-S, ImageNet-C severity 5), each curve a schedule swept over compression (points labeled by

r_{\mathrm{base}}

). Flat and late

\gamma{=}2

r_{\mathrm{base}}\in\{4,6,8\}

are the cells of Table 3 , late

\gamma{=}1

r_{\mathrm{base}}\in\{6,8,10\}

those of Table 17 ; each late point is drawn at the GFLOPs of the flat schedule it is calibrated to (its own cost is at or below that).

ViT-S
		$r_{\mathrm{base}}=4$			$r_{\mathrm{base}}=8$
Method	full	flat	late	$\Delta$	flat	late	$\Delta$
Tent	50.19 $\pm 0.00$	49.27 $\pm 0.00$	50.84 $\pm 0.00$	$+1.57$ $\pm 0.00$	48.47 $\pm 0.00$	52.04 $\pm 0.00$	$\mathbf{+3.57}$ $\pm 0.00$
EATA	55.20 $\pm 0.02$	54.32 $\pm 0.11$	54.84 $\pm 0.13$	$+0.52$ $\pm 0.14$	52.65 $\pm 0.16$	53.66 $\pm 0.26$	$+1.00$ $\pm 0.38$
SAR	53.39 $\pm 0.00$	52.77 $\pm 0.00$	53.22 $\pm 0.00$	$+0.45$ $\pm 0.00$	51.52 $\pm 0.00$	51.96 $\pm 0.00$	$+0.43$ $\pm 0.00$
FOA	45.27 $\pm 0.23$	43.97 $\pm 0.28$	44.68 $\pm 0.15$	$+0.71$ $\pm 0.36$	42.42 $\pm 0.48$	42.81 $\pm 0.30$	$+0.39$ $\pm 0.60$

Appendix

Table 18: TTA

\times

schedule, per-rate means (

\gamma{=}2

). ImageNet-C sev=5, 15-corruption mean top-1, 3 seeds. full = the same method with no reduction (

197

tokens in every block); flat = const (

\gamma{=}0

); late =

\gamma{=}2

, iso-GFLOPs.

\Delta

= late

-

flat; small

\pm

values are seed stds. (

r_{\mathrm{base}}{=}8

also appears in the body, Table 6 .)

ViT-S
		gauss	shot	imp	defoc	glass	motn	zoom	snow	frost	fog	brt	cont	elas	pix	jpeg	mean
Tent	flat	42.4	44.3	44.5	46.5	45.2	51.3	18.2	10.9	55.8	60.8	73.4	48.9	57.5	64.2	63.3	48.5
Tent	late	45.5	47.1	47.2	47.7	46.1	52.2	45.1	21.7	56.8	61.3	73.5	51.0	57.4	64.5	63.5	52.0
EATA	flat	42.1	44.1	44.2	46.3	45.6	50.5	46.8	55.5	54.9	59.8	73.6	41.6	57.8	63.9	63.1	52.7
EATA	late	45.0	46.5	46.6	46.7	46.1	51.1	47.4	55.8	55.8	60.6	73.8	44.1	57.7	64.1	63.5	53.7
SAR	flat	40.9	42.5	43.1	45.1	43.9	50.0	45.0	53.5	53.8	58.1	71.9	45.1	56.0	62.5	61.5	51.5

Appendix

Table 19: TTA

\times

schedule, per-corruption top-1 (%) at

r_{\mathrm{base}}{=}8

\gamma{=}2

(sev=5, 3-seed mean). For each method, flat = const, late =

\gamma{=}2

at iso-GFLOPs. Knife-edge collapses (

<\!20

%) are where adaptation, not the schedule, fails; the schedule shifts them between methods (e.g.

\gamma{=}2

repairs Tent’s ViT-S zoom,

18.2{\to}45.1

). NEO’s ViT-S contrast cell is low but is not one of them: it optimizes nothing and so cannot diverge, and it improves that cell over the

14.4

of no adaptation.

	GFLOPs / step			ms / step
Method	full	late	cut	full	late	cut
ViT-S
Tent	1216	890	$26.8\%$	156.9	124.5	$20.7\%$
EATA	1216	890	$26.8\%$	171.5	139.2	$18.8\%$
SAR	2431	1781	$26.8\%$	318.5	256.1	$19.6\%$
FOA	15555	11533	$25.9\%$	1927.4	1590.8	$17.5\%$

Appendix

Table 20: Cost of one adaptation step on full tokens against the late schedule (

r_{\mathrm{base}}{=}8

\gamma{=}2

), ImageNet-C severity 5, batch 64. Each percentage is the fraction the quantity shrank. Full tokens =

197

in every block.

Reducer	Family	noise	blur	weather	digital
ToMe	merging	$31.5$	$21.1$	$35.2$	$20.8$
EViT	pruning	$13.4$	$10.5$	$15.8$	$13.5$
ATS	sampling	$10.9$	$8.9$	$9.5$	$10.1$
ATC	clustering	$38.2$	$22.6$	$38.2$	$21.6$
PiToMe	merging	$14.2$	$6.6$	$9.0$	$10.4$

Appendix

Table 21: Front-loading of the per-layer reduction distortion

D(\ell)

under each ImageNet-C group (DeiT-S,

N{=}1024

, severity 5): early/late ratio (early

=\ell\in\{1,\ldots,4\}

, late

=\ell\in\{9,\ldots,12\}

means) for one representative corruption per group (noise = gaussian, blur = defocus, weather = fog, digital = jpeg). Every cell is strongly front-loaded (ratio

\geq 6\times

; Spearman against depth

\leq-0.90

throughout).

Condition	early $A(\ell)$	late $A(\ell)$	Spearman
clean	0.60	0.25	$-0.92$
gaussian noise (noise)	0.92	0.20	$-1.00$
defocus blur (blur)	0.74	0.20	$-1.00$
fog (weather)	0.83	0.17	$-1.00$
jpeg compression (digital)	0.77	0.23	$-1.00$

Appendix

Table 22: Amplification

A(\ell)

summary (DeiT-S,

N{=}1024

, compute-neutral), for clean inputs and one corruption from each ImageNet-C group (severity 5).

A(\ell)

is front-loaded (early

\gg

late) in every condition, with clean the flattest, so the network’s amplification concentrates a reduction’s distortion in the early layers. early

=\ell\in\{1,\ldots,4\}

, late

=\ell\in\{9,\ldots,12\}

(means); Spearman is over all 12 layers.

$r_{\mathrm{base}}$	own-path (flat)	oracle clean-decision	late schedule ( $\gamma{=}2$ )
$4$	$38.83$	$39.22$ $(+0.39)$	$\mathbf{39.37}$ $(+0.54)$
$6$	$38.44$	$39.03$ $(+0.59)$	$\mathbf{39.32}$ $(+0.88)$
$8$	$37.99$	$38.79$ $(+0.80)$	$\mathbf{39.16}$ $(+1.17)$
$10$	$37.41$	$38.36$ $(+0.95)$	$\mathbf{38.66}$ $(+1.25)$

Appendix

Table 23: Clean merge-path control (15-corruption ImageNet-C mean, severity 5, DeiT-S, iso-GFLOPs). Forcing the clean (oracle) merge decisions recovers some OOD accuracy, but less than the late schedule at every

r_{\mathrm{base}}

. Parentheses give the gain over the own-path baseline; the late schedule uses its own (corrupted) decisions and needs no clean reference.

Schedule	Family	GFLOPs	mean	$\Delta$ vs. flat
flat (const $r_{\mathrm{base}}{=}8$ )	–	$3.199$	$37.99$	–
power law ( $\gamma{=}2$ )	power law (convex)	$3.147$	$39.16$	$+1.17$
exponential ( $\beta{=}3$ )	exponential (convex)	$3.194$	$39.04$	$+1.05$
two-level step	step (non-smooth)	$3.178$	$39.17$	$+1.18$

Appendix

Table 24: Late-concentration is robust to the schedule’s functional form (DeiT-S, ToMe, ImageNet-C sev=5, 15-corruption mean top-1, %;

r_{\mathrm{base}}{=}8

, inference-only, iso-GFLOPs

\leq

flat). Each alternative shape is center-of-mass-matched to the

\gamma{=}2

schedule.

\Delta

= shape

-

flat.

	gauss	shot	imp	defoc	glass	motn	zoom	snow	frost	fog	brt	cont	elas	pix	jpeg	mean
uncompressed	33.89	33.79	34.92	27.73	18.92	32.23	28.41	47.53	53.16	54.22	70.25	38.55	34.75	29.87	52.72	39.40
flat	31.62	31.70	32.40	26.53	18.42	31.11	27.31	46.37	51.10	51.43	69.57	36.59	34.58	28.88	52.19	37.99
power law ( $\gamma{=}2$ )	33.79	33.58	34.85	27.57	18.72	32.24	28.75	47.62	52.45	52.97	70.18	37.31	34.79	29.74	52.82	39.16
exponential	33.43	33.28	34.53	27.42	18.72	32.11	28.36	47.49	52.42	52.85	70.05	37.43	35.07	29.63	52.77	39.04
two-level step	33.58	33.45	34.59	27.61	18.73	32.20	28.67	47.71	52.53	52.94	70.09	37.87	35.13	29.72	52.79	39.17

Appendix

Table 25: Per-corruption top-1 (%) for the alternative late shapes (DeiT-S, ToMe, sev=5,

r_{\mathrm{base}}{=}8

, iso-GFLOPs). uncompressed, flat and the power law (

\gamma{=}2

) are the same runs as Figure 6 . All 15 corruptions improve over flat under every late shape.

layer	2	3	4	5	6	7	8	9	10	11	12	total	last 3
DeiT-S, searched	2	6	12	11	12	10	12	18	5	31	72	191	57%
DeiT-B, searched	1	2	2	7	8	11	11	17	49	29	55	192	69%
power law, $\gamma{=}2$ (both)	1	3	5	7	11	14	19	24	30	36	42	192	56%

Appendix

Table 26: Per-layer tokens removed at the main operating point (DeiT,

197

tokens). DiffRate’s searched schedules, from a labeled search run per architecture and budget, and the closed-form

\gamma{=}2

schedule; all three concentrate removal in the last layers, the searched ones most at the final layer, as Section 4 predicts.

Figure 8: Tokens remaining after each layer at

r_{\mathrm{base}}{=}12

(DeiT-S), for the early-concentrated, flat and late-concentrated schedules; the legend gives each schedule’s GFLOPs.

layer $\ell$	clean	gaussian
1	16.4	25.8
2	12.7	18.4
3	8.3	10.9
4	6.3	5.9
5	4.1	5.7
6	3.4	4.7

Appendix

Table 27: Distortion per unit of compute saved,

\rho(\ell)=D(\ell)/S(\ell)

(

\times 10^{3}

), from the directly-measured per-layer reduction distortion

D(\ell)

(relative output-logit change from removing tokens at layer

\ell

only; DeiT-S,

N{=}1024

; Appendix L ) and the exact marginal saving

S(\ell)=(L-\ell)+c

c{=}0.61

(the layer’s own-MLP fraction, since ToMe merges before the MLP). The distortion is steeply front-loaded in both conditions and steepens under noise. Layer

L{=}12

is the degenerate free reduction:

D\approx 0

confirms

A(L){=}0

, giving

\rho(L){\approx}0

despite

S(L){=}c{>}0

. Among the interior layers the minimum ( bold ) is late (

\ell{=}11

) in both conditions; it is per-layer capacity, not a blow-up of

\rho

, that keeps the optimum off the free last layer. The table is a directional diagnostic, showing the cheap-to-reduce region sits in the late (but capacity-bounded) layers, not a predictor of the exact optimal

\gamma

corruption	flat	late	$\Delta$ full	$\Delta$ excl-1	$\Delta$ excl-2
gaussian noise	31.62	33.79	+2.17	+2.16	+1.89
shot noise	31.70	33.58	+1.88	+1.92	+1.70
impulse noise	32.40	34.85	+2.45	+2.33	+2.21
defocus blur	26.53	27.57	+1.04	+1.11	+1.13
glass blur	18.42	18.72	+0.30	+0.27	+0.29
motion blur	31.11	32.24	+1.13	+1.16	+0.98

Appendix

Table 28: Excluding the final layer(s) from both schedules (DeiT-S, full ImageNet-C, severity 5,

50

k images; flat const

r_{\mathrm{base}}{=}8

3.20

G, late

\gamma{=}2

3.15

G, late

\leq

flat). flat/late columns are top-1 (%) for the full schedule;

\Delta

= late

-

flat for the full, exclude-last, and exclude-last-two variants, each recalibrated to the same compute. The full column reproduces Table 2 ; the gain persists when the (free) final layer(s) are removed.

Budget
	MAE ViT-B			MAE ViT-L
	flat	late	$\Delta$	flat	late	$\Delta$
GFLOPs	12.67 ( $-25\%$ )	12.62 ( $-25\%$ )	–	44.98 ( $-25\%$ )	44.90 ( $-25\%$ )	–
tokens removed	96	169	–	96	180	–
final tokens	100	27	–	100	16	–
Top-1 accuracy (%) by corruption

Appendix

Table 29: Global-average-pooling readout (MAE ViT-B and ViT-L, full ImageNet-C, severity 5,

50

k images). Each model’s flat / late columns carry, in the upper rows, its budget (GFLOPs, with the reduction from the uncompressed model in parentheses,

16.9

G for ViT-B and

59.7

G for ViT-L; tokens removed and final, out of

196

patches) and, in the lower rows, its per-corruption top-1 (%); late is

\gamma{=}2

calibrated to the closest schedule at or below the flat GFLOPs. At iso-GFLOPs (late

\leq

flat) late removes more tokens in total yet keeps fewer at the end, and still beats flat on all

15

corruptions for both scales (mean

+1.28

+1.22

): the gain is not from a larger token budget and reproduces the class-token result.

backbone	pair ( $r_{\mathrm{base}}$ , $\gamma$ )	net	recovery	new correctness
DeiT-S	$4$ , $\gamma{=}1$	$+0.375$	$\mathbf{+0.740}$	$-0.366$
DeiT-S	$4$ , $\gamma{=}2$	$+0.542$	$\mathbf{+1.169}$	$-0.627$
DeiT-S	$8$ , $\gamma{=}1$	$+0.902$	$\mathbf{+1.257}$	$-0.355$
DeiT-S	$8$ , $\gamma{=}2$	$+1.173$	$\mathbf{+1.798}$	$-0.625$
DeiT-S	$12$ , $\gamma{=}1$	$+1.213$	$\mathbf{+1.417}$	$-0.204$
DeiT-S	$12$ , $\gamma{=}2$	$+1.103$	$\mathbf{+1.331}$	$-0.228$

Appendix

Table 30: Two-channel decomposition of the late-minus-flat accuracy difference (pp), full ImageNet-C severity 5. recovery = cases the unreduced model already got right; new correctness = cases it got wrong. In all twelve cells the recovery channel is positive and larger than the net, and the new-correctness channel is negative: the gain comes from not destroying predictions the model already had, not from finding new ones.

Figure 9: Per-image agreement with the unreduced model (DeiT-S, $r_{\mathrm{base}}{=}8$ , iso-GFLOPs, $50{,}000$ images per panel). Horizontal axis: the probability the unreduced model assigns to its own predicted class $c^{\star}$ ; vertical axis: the probability the reduced model assigns to that same class; exact agreement lies on the dashed line. Color is the local density of cases, on a shared scale across panels, saturating above its $75$ th percentile. Reduction is nearly harmless on clean input under either schedule and remains so for ours under corruption; the flat schedule departs from the unreduced model only where reduction and corruption meet. KL is the mean per-case $\mathrm{KL}(p_{\text{unreduced}}\,\|\,p_{\text{reduced}})$ over the full class distribution on the same input. Gaussian noise at severity 5 is shown for legibility; the aggregate over all $15$ corruptions is quoted above.

$r_{\mathrm{base}}$	flat (GFLOPs)	$\gamma{=}0.5$	$\gamma{=}1.0$	$\gamma{=}1.5$	$\gamma{=}2.0$	$\gamma{=}2.5$	$\gamma{=}3.0$
2	2 (3.99G)	3 ( $-1.6\%$ )	3 ( $-0.0\%$ )	4 ( $-1.4\%$ )	4 ( $-0.4\%$ )	5 ( $-1.3\%$ )	5 ( $-0.6\%$ )
4	4 (3.72G)	5 ( $-0.7\%$ )	6 ( $-1.1\%$ )	7 ( $-0.7\%$ )	8 ( $-0.9\%$ )	9 ( $-1.1\%$ )	10 ( $-1.4\%$ )
6	6 (3.46G)	8 ( $-2.7\%$ )	9 ( $-1.2\%$ )	10 ( $-0.2\%$ )	13 ( $-3.6\%$ )	16 ( $-5.2\%$ )	20 ( $-7.6\%$ )
8	8 (3.20G)	10 ( $-1.2\%$ )	12 ( $-2.0\%$ )	14 ( $-1.8\%$ )	16 ( $-1.6\%$ )	18 ( $-0.5\%$ )	21 ( $-0.7\%$ )
10	10 (2.94G)	13 ( $-3.6\%$ )	15 ( $-2.2\%$ )	17 ( $-0.6\%$ )	21 ( $-1.2\%$ )	26 ( $-1.8\%$ )	33 ( $-2.9\%$ )
12	12 (2.67G)	15 ( $-2.2\%$ )	18 ( $-1.6\%$ )	22 ( $-0.4\%$ )	27 ( $-0.3\%$ )	34 ( $-0.1\%$ )	44 ( $-0.4\%$ )

Appendix

Table 31: Calibrated iso-GFLOPs schedules used throughout the paper. For each operating point

r_{\mathrm{base}}

and exponent

\gamma

we list the late-concentrated per-layer-mean removal budget

\bar{r}

, chosen so the schedule’s GFLOPs do not exceed the flat (const

r_{\mathrm{base}}

) baseline. Parentheses give the GFLOPs deviation from flat (DeiT-S, THOP): every entry is

\leq 0

, so every late schedule is at most as expensive as its flat counterpart (strictly conservative).

\bar{r}

is backbone-independent (197 tokens, 12 layers at 224px), so the same schedules apply to DeiT-S/B and ViT-S/B; the flat column shows DeiT-S GFLOPs. The larger under-shoots at small

r_{\mathrm{base}}

with high

\gamma

(e.g.

r_{\mathrm{base}}{=}6

\gamma{=}3

) are an integer-granularity effect and remain conservative.

$r_{\mathrm{base}}$	schedule	gaussian top-1	$\Delta$
6	const	32.35	–
6	late $\gamma{=}2.0$	33.84	$+$ 1.49
8	const	31.62	–
8	late $\gamma{=}1.0$	33.12	$+$ 1.50
8	late $\gamma{=}1.5$	33.49	$+$ 1.86
8	late $\gamma{=}2.0$	33.79	$+$ 2.17

Appendix

Table 32: iso-GFLOPs calibration on gaussian noise (sev=5, 50k), DeiT-S. const rows are flat baselines; late rows are calibrated to the matching flat GFLOPs.

\Delta

= late

-

flat.

backbone	schedule	GFLOPs	ms / batch	img / s
$d{=}384$ (DeiT-S, ViT-S)	uncompressed	4.25	67.1	954
	flat (const $r{=}8$ )	3.20	54.6	1172
	late ( $\gamma{=}2$ )	3.15	54.6	1172
$d{=}768$ (DeiT-B, ViT-B)	uncompressed	16.86	212.0	302
	flat (const $r{=}8$ )	12.67	167.0	383
	late ( $\gamma{=}2$ )	12.46	167.5	382

Appendix

Table 33: Wall-clock latency at the main operating point (

r_{\mathrm{base}}{=}8

), batch 64, single GPU (

200

timed iterations,

30

warmup). At iso-GFLOPs the late (

\gamma{=}2

) and flat schedules match in throughput to within

0.5\%

. The

384

-dim row covers DeiT-S and ViT-S; the

768

-dim row covers DeiT-B and ViT-B (identical architecture, hence identical latency).

Explore similar work

Jun 6, 2026cs.CV

RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT

Vision Transformers (ViTs) achieve strong performance but suffer from high computational costs due to quadratic self-attention complexity. Although token reduction techniques such as pruning and merging mitigate this, they typically overlook how representations evolve across network depth. We propose RAPID, a depth-aware token reduction framework that adapts reduction strategies to the layer-wise characteristics of token representations. The primary methodological contribution is a bifurcated strategy: in shallow-to-middle layers, RAPID employs a redundancy-similarity aware pruning metric to eliminate over-represented local patterns. As features transition to global semantic concepts in deeper layers, the framework shifts to an importance-similarity aware merging mechanism. This stage leverages classification (CLS) token attention weights to protect semantically critical tokens while fusing less important but similar neighbors. Empirical validation on ImageNet-1K using ViT and DeiT architectures demonstrates that RAPID establishes a superior accuracy-compression Pareto frontier compared to plug-and-play baselines such as ToMe and ToFu. RAPID is particularly robust in aggressive compression regimes, achieving up to 4.29% higher accuracy than ToMe at extreme reduction rates. Our framework provides a training-free template for optimizing vision models by aligning reduction strategies with hierarchical feature evolution.

Kyumin Choi, Ikbeom Jang

Hankuk University of Foreign Studies Seoul, South Korea

Vision Transformer Efficient ViTs Visual Token Merging Visual Token Pruning Image Classification Visual Token Compression

Ask PDF

Apr 17, 2026cs.AI

Why Training-Free Token Reduction Collapses: The Inherent Instability of Pairwise Scoring Signals

Training-free token reduction methods for Vision Transformers (ToMe, ToFu, PiToMe, and MCTF) employ different scoring mechanisms, yet they share a closely matched cliff-like collapse at high compression. This paper explains \emph{why}. We develop a diagnostic framework with two tools, ranking consistency

ρ_s

and off-diagonal correlation

ρ_\text{off}

, that decomposes the collapse into (1)a signal-agnostic error amplifier inherent to layer-wise reduction, predicting convex Pareto curves and

r_{\text{crit}} \propto 1/L

; and (2)shared reliance on \emph{pairwise} similarity signals whose ranking consistency degrades from

ρ_s{=}0.88

0.27

in deep layers. Pairwise rankings are inherently unstable (

O(N_p^2)

joint perturbations) while unary signals enjoy greater stability (

O(N_p)

perturbations, CLT). From three design principles derived from this diagnosis, we construct CATIS as a constructive validation: unary signals raise the trigger threshold, triage suppresses the gain. On ViT-Large at 63% FLOPs reduction, CATIS retains 96.9% of vanilla accuracy (81.0%) on ImageNet-1K where all baselines collapse to 43--65%.

Yang Shanglin

Visual Token Pruning Visual Token Compression

Ask PDF

May 26, 2026cs.CV

AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers

The quadratic cost of self-attention in Vision Transformers (ViTs) constitutes a fundamental bottleneck for practical deployment, motivating a vibrant line of research on token reduction. Among existing approaches, token merging (ToMe) has emerged as an elegant training-free solution; yet its design rests on an unspoken premise of token equality, which contravenes the well-documented non-uniformity of self-attention and leads to information loss in high-salience tokens under aggressive compression. We address this limitation with AdaMerge, a token-merging framework based on two complementary mechanisms. First, salience-weighted similarity leverages column-wise feature-affinity centrality as a token-importance proxy and incorporates the resulting salience scores into the bipartite matching score, ensuring that pivotal tokens contribute more strongly to the merged representation. Second, adaptive merging intensity uses pre-computed layer-wise similarity statistics to dynamically modulate the per-layer reduction count in accordance with input-specific redundancy. On ImageNet-1k with ViT-B/16, AdaMerge consistently outperforms ToMe, PiToMe, and DSM across all FLOPs-matched regimes. The accuracy gap widens monotonically with compression: at the 13.4G FLOPs operating point, AdaMerge sustains a Top-1 degradation of only -1.06%, compared to -1.45% for PiToMe and -4.62% for DSM. To our knowledge, AdaMerge is the first to combine salience-weighted similarity and adaptive per-layer reduction into a single training-free token merging framework, advancing the accuracy-FLOPs Pareto frontier of ViT acceleration.

Semi Lee, Hyejin Go, Hyesong Choi

Electronic Engineering · Soongsil University · Seoul, South Korea

Token Merging Self-Attention Vision Transformer Efficient ViTs Visual Token Merging Visual Token Compression

Ask PDF