Variable-Granularity Tokenization for High-Resolution Object Detection

Khayrul Islam

cs.CVAug 27, 2026

Variable-Granularity Tokenization for High-Resolution Object Detection

Authors: Khayrul Islam

Organizations: Lawrence Livermore National Laboratory 7000 East Ave, Livermore, CA 94550

Abstract

ViT detectors fix a uniform token grid before any learned stage. A native-resolution aerial detector must then choose between resolving few-pixel objects and staying inside compute and memory limits. We introduce VGTok, a training-free tokenizer that sets patch granularity per region from pixels, ahead of the encoder. VGTok scores each region by multi-scale morphological top-hat separability from its surround, then thresholds those scores at a per-image percentile, which fixes the token budget. A structure-tensor gate ( $λ_{\min}$ ) refines only where two-dimensional object structure supports it, leaving one-dimensional clutter coarse. The resulting token set is a strict partition of the image. In a Co-DETR detector with an EVA-02 ViT-L encoder, VGTok clears every published VisDrone-val AP and AP $_S$ at every budget from 40% to 100% of tokens. At 40% it records 44.22 AP with three fifths of the sequence discarded before the first transformer block; dense, it reaches 48.38 AP, $6.08$ above the strongest published entry. VGTok transfers to AI-TOD-v2 untouched, same scorer and same rank, and sets a new state of the art at 37.27 AP and 19.51 AP $_{vt}$ . As a pure drop-in into a frozen checkpoint it reaches 36.29 AP at 78.5% of tokens, above every published entry, where our 376.3M-parameter detector clears a 3.0B multi-expert model. We show that a token budget fixed before the backbone, from local separability and structure geometry alone, holds accuracy on the tiny-object regimes that dominate aerial detection, at $3.1\times$ less encoder compute and $1.9\times$ less encoder memory. Code and models are available at github.com/khayrulbuet13/vgtok and huggingface.co/khayrulbuet13/vgtok.

Figures & tables

Figure 1: VGTok on a VisDrone frame at $70\%$ tokens. Contrast alone spends $204$ tokens on painted crosswalk bars holding no object; the $\lambda_{\min}$ gate emits the same crop in $174$ . (a) Ground truth: $317$ instances, $280$ of them under $32\times 32$ px. (b) The finest patch size stays on the vehicles while empty roadway, plaza and building faces coarsen, at $47.76$ AP zero-shot with no weight changed, $5.46$ AP above the strongest published val result; Tab. 4 prints this budget trained. (c) The boxed region, enlarged: paint on plain asphalt. (d) Contrast refines that paint to the finest size. (e) The gate stops the descent, one coarse token in place of four fine ones.

Figure 2: VGTok in four stages: score, split, gate, pack. On this frame, at $p=40$ and the shipped $r=0.20$ , $77$ nodes are gated: the ungated descent emits $1842$ tokens, VGTok $1500$ at the same $\tau$ . Stage $1$ scores every pixel by a top-hat at structuring-element widths $5$ , $9$ and $17$ ; stage $2$ sets $\tau$ from one per-image percentile and splits the busy nodes; stage $3$ stops that descent where $1/(1+\lambda_{\min})$ ranks high, emitting the node at its own size and dropping its $4$ children; stage $4$ packs the retained tokens. No pixel is described twice.

Figure 3: Operating range on VisDrone2019-DET val. VGTok clears every published bar from

40\%

of tokens upward. AP (a) and AP S (b) against the measured retained-token fraction,

20\%

100\%

of the dense grid; each point is one budget from that tokenizer’s measured retention table. Solid, finetuned for

8

epochs: VGTok (

\mathcal{T}+g

) over its ungated control Top-hat (

\mathcal{T}

), whose gap is the gate and widens as the budget tightens. Dashed, zero-shot on the dense checkpoint: Top-hat is worth

+7.90

AP and

+9.32

AP S over Single-pass (

\mathcal{S}

) at

40\%

, from the splitter alone; trained, that same contrast is

+0.62

AP ( Tab. 4 ). Dotted: EVORL at

42.2

AP [ 96 ] , Dome-DETR-L at

40.5

[ 41 ] , YOLC with multi-scale testing at

32.9

AP S [ 53 ] . Finetuned points are three-seed means, as in Tab. 4 .

Method	Backbone	Input	AP	AP 50	AP S
DroneFINE-L [ 85 ]	Swin-L (G-DINO)	concurrent	42.3	65.8	—
EVORL [ 96 ]	CSPDarkNet	multi-crop	42.2	66.0	—
Focus-and-Detect [ 48 ]	ResNeXt-101	2-stage crop	42.06	66.12	32.0
Dome-DETR-L [ 41 ]	HGNetv2-B4	$1200{\times}800$	40.5	64.5	—
AdaZoom [ 100 ]	ResNeXt-101	multi-crop	40.3	66.9	—
UFPMP-Det + MS [ 43 ]	ResNeXt-101	crop + MS	40.1	66.8	—

Table 1: Comparison with the published record on VisDrone2019-DET val. Every token budget from

40\%

100\%

exceeds every published val AP and every published val AP S , including entries that use multi-scale testing. We reach these numbers at native

2048{\times}1152

in one forward pass from a single checkpoint, setting the token count at inference; no published entry above sets its own token budget, and most re-run the detector over crops, zooms, or scales. Zero-shot ∗ : VGTok inserted into the released dense checkpoint at inference, nothing retrained, no head added, no loss changed.

Method	Backbone	Input	AP	AP 50	AP vt
ScaleBridge-Det (3.0B) [ 101 ]	MoE 3.0B	$1024^{2}$	35.7	72.1	16.2
Dome-DETR-L [ 41 ]	HGNetv2-B4	$800^{2}$	34.6	69.2	19.0
Dome-DETR-M [ 41 ]	HGNetv2-B2	$800^{2}$	34.0	68.4	18.4
ScaleBridge-Det (2.0B) [ 101 ]	MoE 2.0B	$1024^{2}$	33.6	71.8	14.2
Dome-DETR-S [ 41 ]	HGNetv2-B0	$800^{2}$	33.3	67.5	17.8
D 3 R-DETR [ 84 ]	HGNetv2-B0	$800^{2}$	31.3	65.1	16.6

Table 2: Transfer to AI-TOD-v2 test, no retuning. At

100\%

of tokens VGTok takes every column:

37.27

AP against ScaleBridge-Det’s

35.7

76.48

AP 50 against its

72.1

, and

19.51

AP vt against Dome-DETR-L’s

19.0

. The tokenizer and rank

r=0.20

are carried from VisDrone under the same four budgets; protocol as in Sec. 4.1 , and AP vt is the very-tiny band (

<8^{2}

px). Italics mark our operating point,

70\%

here against VisDrone’s

40\%

. Zero-shot ∗ : nothing retrained.

78.5\%

is the loosest budget the gate reaches at

r=0.20

Tokenizer	Tokens	GFLOPs	ms	GB	GFLOPs	ms
		Encoder			End-to-end
Dense	$100.0\%$	8282	92.3	0.466	12955.5	182.8
VGTok	$80.0\%$	4498	69.8	0.408	9172.4	158.9
VGTok	$70.0\%$	4093	66.9	0.372	8767.2	154.9
VGTok	$40.0\%$	2634	53.0	0.251	7307.7	136.9
Control	$70.0\%$	8293	104.0	0.498	12967.0	193.3

Table 3: Forward-pass cost at four token budgets. Packing is what turns a token cut into a compute cut: the scatter control holds the same gate and the same retained tokens but returns them to the dense grid before the first block, and costs more than dense. Batch

1

, fp16 with the variable-length kernel, one exclusively held H100 80GB, median over three repeats on all

548

val images with the reported checkpoints loaded; every end-to-end budget is re-measured and lands within

0.0004

of its target. Encoder columns use the fused-score variant, whose

40\%

row sits at

46.7\%

retention, so the reported

3.1\times

is a lower bound.

Tokenizer (AP)	$80\%$	$70\%$	$60\%$	$40\%$	$20\%$
Single-pass ( $\mathcal{S}$ )	47.56	47.30	46.64	43.11	unr.
Top-hat ( $\mathcal{T}$ )	47.04	46.92	46.38	43.73	33.89
Top-hat ( $\mathcal{T}$ ) $+\,g$	47.11	47.04	46.65	44.22	35.17
gate effect	$+0.06$	$+0.12$	$+0.27$	$+0.49$	$+1.28$
Tokenizer (AP S )	$80\%$	$70\%$	$60\%$	$40\%$	$20\%$
Single-pass ( $\mathcal{S}$ )	39.00	38.66	38.00	33.93	unr.

Table 4: Gate ablation, VisDrone2019-DET val, five matched budgets. The

\lambda_{\min}

gate’s effect grows monotonically as the budget tightens, reaching

+1.28

AP and

+1.51

AP S at

20\%

; at the

40\%

operating point the three finetunes per cell do not overlap,

[44.11,\,44.30]

against

[43.66,\,43.77]

. The monotone run begins at

+0.06

AP and

-0.03

AP S at

80\%

, where the budget does not bind. AP above, AP S below,

8

finetune epochs per row, every cell a three-seed mean ( Sec. 4.1 ). The two

\mathcal{T}

rows differ only in the gate at rank

r=0.20

, and gate effect is their difference. unr. marks budgets Single-pass cannot reach. Bold compares the two

\mathcal{T}

rows, which differ only in the gate, and marks only leads that clear the seed floor.

Five candidates, at the $70\%$ budget
	VisDrone		AI-TOD-v2
	32 px	64 px	32 px	64 px
$1/(1+\lambda_{\min})$	0.712	0.731	0.748	0.750
node vs. surround $\times$ homogeneity	0.599	0.639	0.612	0.648
within-node homogeneity	0.602	0.624	0.606	0.634
node vs. surround	0.552	0.601	0.580	0.616

Table A1 : Gate statistic discrimination, with a budget sweep. AUROC for ranking clutter nodes, those no ground-truth box overlaps, above object-bearing ones: only

1/(1+\lambda_{\min})

clears chance meaningfully, at

0.712

or better in every column. Two levels and not three: the gate is a stop rule, so it acts at

64

and

32

px, and the

16

px leaf has no split to stop, so that column would be structurally empty, not unrun. Pools are

150

VisDrone val and

150

AI-TOD-v2 val images. Bold is the best of the five candidates in an upper-block column, fixed at

70\%

Statistic	AP 50	AP 75	AP S
$\lambda_{\min}$ (structure tensor)	73.67	50.91	38.97
top-hat	73.50	50.69	38.85
max deviation, Lab	73.57	50.55	38.84
max deviation	73.03	50.12	38.43
upsample MSE ∗	72.84	49.87	38.09
gradient energy	72.66	49.92	38.01

Table A2 : Eight patch scores on the metrics beyond AP. The ranking is the same on every metric we measured: against the AP order AP S transposes nothing, AP 50 transposes one adjacent pair by

0.07

and AP 75 one by

0.05

, both inside the

0.16

AP seed floor. Fixed

70\%

budget, zero-shot on the dense checkpoint under Single-pass; VisDrone val,

548

images, maxDets

500

, with detector, budget and protocol identical across rows. Rows are in AP order and AP itself is not reprinted; the main paper quotes its spread in prose. Bold is the best value in a column. ∗ Upsample MSE is a valid measurement at this budget but cannot be budget-matched under a quadtree, for the reason below.

AP
Tokenizer	80%	70%	60%	40%	20%
Top-hat ( $\mathcal{T}$ )	47.97	47.58	46.37	39.45	20.22
Top-hat ( $\mathcal{T}$ ) $+\,g$	48.04	47.76	46.85	40.44	21.18
gate effect	$+0.07$	$+0.18$	$+0.48$	$+1.00$	$+0.96$
AP S
Top-hat ( $\mathcal{T}$ )	39.29	38.92	37.91	31.30	13.84

Table A3 : Zero-shot gate effect at five matched budgets. The effect grows as the budget tightens, peaks at

+1.00

AP and

+1.15

AP S at

40\%

, then flattens to

+0.96

and

+0.97

20\%

. The quadtree splitter on the released dense checkpoint. VisDrone val,

548

images, maxDets

500

. We reach each budget by inverting that tokenizer’s own measured retention table, so the two rows hold the same token count in every column.

Tokenizer (AP)	$80\%$	$70\%$	$60\%$	$40\%$	$20\%$
Top-hat ( $\mathcal{T}$ ) $\oplus\lambda$	47.12 $\pm$ 0.04	46.95 $\pm$ 0.08	46.36 $\pm$ 0.13	43.85 $\pm$ 0.18	34.84 $\pm$ 0.12
Top-hat ( $\mathcal{T}$ ) $\oplus\lambda+g$	47.12 $\pm$ 0.02	47.00 $\pm$ 0.08	46.66 $\pm$ 0.05	43.94 $\pm$ 0.03	35.05 $\pm$ 0.04
gate effect	$-0.01$	$+0.04$	$+0.30$	$+0.09$	$+0.20$
Tokenizer (AP S )	$80\%$	$70\%$	$60\%$	$40\%$	$20\%$
Top-hat ( $\mathcal{T}$ ) $\oplus\lambda$	38.42 $\pm$ 0.06	38.32 $\pm$ 0.12	37.64 $\pm$ 0.17	35.10 $\pm$ 0.21	25.12 $\pm$ 0.09
Top-hat ( $\mathcal{T}$ ) $\oplus\lambda+g$	38.42 $\pm$ 0.05	38.27 $\pm$ 0.11	38.05 $\pm$ 0.16	35.18 $\pm$ 0.08	25.48 $\pm$ 0.02

Table A4 : Fused pixel score

\mathcal{T}\oplus\lambda

, gated and ungated. The gate is where

\lambda_{\min}

earns its cost: blending it into the pixel score as well ties the shipped tokenizer of Tab. 4 at four of five budgets and trails it by

0.28

AP and

0.38

AP S at the fifth, so one use of the statistic is the one we ship. VisDrone val,

548

images, maxDets

500

, each budget set by inverting its own retention table; every cell is a three-finetune mean with its sample

\sigma

, none outstanding. Bold marks the best value in a column within a metric block; a column whose top two values sit closer than the seed floor (

0.16

AP,

0.22

AP S ; Tab. A10 ) carries none.

Tokenizer (AP)	$80\%$	$70\%$	$60\%$	$40\%$	$20\%$
Single-pass ( $\mathcal{S}$ )	47.56 $\pm$ 0.06	47.30 $\pm$ 0.06	46.64 $\pm$ 0.12	43.11 $\pm$ 0.09	unr.
Top-hat ( $\mathcal{T}$ )	47.04 $\pm$ 0.17	46.92 $\pm$ 0.16	46.38 $\pm$ 0.12	43.73 $\pm$ 0.06	33.89 $\pm$ 0.07
Top-hat ( $\mathcal{T}$ ) $+\,g$	47.11 $\pm$ 0.10	47.04 $\pm$ 0.03	46.65 $\pm$ 0.07	44.22 $\pm$ 0.10	35.17 $\pm$ 0.11
Tokenizer (AP S )	$80\%$	$70\%$	$60\%$	$40\%$	$20\%$
Single-pass ( $\mathcal{S}$ )	39.00 $\pm$ 0.04	38.66 $\pm$ 0.06	38.00 $\pm$ 0.09	33.93 $\pm$ 0.11	unr.
Top-hat ( $\mathcal{T}$ )	38.41 $\pm$ 0.26	38.25 $\pm$ 0.18	37.78 $\pm$ 0.18	34.99 $\pm$ 0.15	24.15 $\pm$ 0.07

Table A5 : Seed spread behind every cell of Tab. 4 . The widest range over three finetunes is

0.341

AP and

0.466

AP S , on the ungated control at

80\%

, which we do not ship. VisDrone val,

548

images, maxDets

500

; each

\sigma

is the sample deviation of three

8

-epoch finetunes differing only in seed, and we match each budget by inverting that tokenizer’s own retention table. unr. marks a budget Single-pass cannot reach at any threshold, since two scales floor its retention at a quarter of the dense grid.

Tokenizer	AP	AP 50	AP 75	AP S	AP M	AP L
Top-hat ( $\mathcal{T}$ )	43.73 $\pm$ 0.06	69.29 $\pm$ 0.06	45.99 $\pm$ 0.17	34.99 $\pm$ 0.15	56.27 $\pm$ 0.08	70.58 $\pm$ 0.78
Top-hat ( $\mathcal{T}$ ) $+\,g$	44.22 $\pm$ 0.10	69.78 $\pm$ 0.07	46.51 $\pm$ 0.23	35.56 $\pm$ 0.18	56.53 $\pm$ 0.08	71.19 $\pm$ 0.78
gate effect	$+0.49$ $\pm$ 0.12	$+0.49$ $\pm$ 0.09	$+0.52$ $\pm$ 0.29	$+0.56$ $\pm$ 0.23	$+0.26$ $\pm$ 0.11	$+0.61$ $\pm$ 1.10

Table A6 : All six COCO bands at the

40\%

operating point. In five of the six columns the gain clears each tokenizer’s own

\sigma

and runs

1.8\times

5.2\times

the propagated

\sigma_{\Delta}

. VisDrone val,

548

images, maxDets

500

. Each mean is three independent

8

-epoch finetunes at a budget matched to

0.4001

retained, with

\sigma

their sample standard deviation. The two rows differ only in the gate

g

. On the difference row

\sigma_{\Delta}=\sqrt{\sigma_{1}^{2}+\sigma_{2}^{2}}

, the propagated spread of a single-run difference. AP L covers

1068

38{,}759

scored instances; we report the remaining five bands, where the gain clears

\sigma

Tokenizer	Budget	AP	AP vt	AP t	AP s	AP m
VGTok, $100\%$ tokens	100%	37.27	19.51	37.47	43.57	56.04
Top-hat ( $\mathcal{T}$ )	70%	35.37	17.22	35.66	40.89	52.35
Top-hat ( $\mathcal{T}$ ) $+\,g$	70%	35.66	18.46	35.96	41.46	52.79
Top-hat ( $\mathcal{T}$ ) $\oplus\,\lambda$	70%	35.47	18.13	35.90	41.15	52.74
Top-hat ( $\mathcal{T}$ ) $\oplus\,\lambda+g$	70%	35.65	18.52	36.00	41.27	52.71
Top-hat ( $\mathcal{T}$ )	40%	30.09	14.61	30.14	34.59	45.22

Table A7 : AI-TOD-v2 area bands, by budget. The gate lifts every band at the

70\%

operating point we report on this benchmark; below it the very-tiny band alone turns over, the one place the ordering runs opposite to VisDrone, and every cell is printed in full. AI-TOD-v2 test, the

14{,}018

-image split, maxDets

1500

2048

; AP vt is very-tiny (under

8^{2}

px), then tiny, small and medium, no large band here. Every row is zero-shot on one shared NWD dense checkpoint, budgets matched by inverting each tokenizer’s own retention table ( Tab. A12 ). There is no

80\%

block: at

r=0.20

the gate alone caps retention at

0.785

here.

	vehicle	person	ship	storage-tank	bridge	swimming-pool	airplane	wind-mill	AP vt
very-tiny boxes	38,504	5,934	2,930	1,522	41	39	17	8	48,995
VGTok, $100\%$ tokens	19.39	11.07	35.07	32.20	4.84	27.74	20.34	5.46	19.51
Top-hat ( $\mathcal{T}$ )	19.50	10.46	31.64	31.93	3.14	26.50	9.01	5.59	17.22
Top-hat ( $\mathcal{T}$ ) $+\,g$	19.52	10.56	32.05	32.12	3.23	26.37	19.31	4.55	18.46
gate effect	$+0.02$	$+0.10$	$+0.41$	$+0.19$	$+0.09$	$-0.13$	$+10.30$	$-1.04$	$+1.24$
share of the mean	$+0.002$	$+0.013$	$+0.051$	$+0.024$	$+0.011$	$-0.017$	$+1.288$	$-0.130$	$+1.242$

Table A8 : Per-class AP on the very-tiny band. The

+1.24

AP vt the gate is worth at

70\%

is one class: airplane moves

+10.30

17

boxes. AI-TOD-v2 test,

14{,}018

images, maxDets

1500

; both tokenizer rows are zero-shot on the same NWD dense checkpoint at a budget matched to

70\%

by inverting each tokenizer’s own retention table, so the gate is the only difference between them. Very-tiny boxes is that class’s instance count in the band, counted from the test annotations. AP vt is an unweighted mean over the eight classes, so each carries one eighth of it however few boxes it holds, and the share row is that eighth.

	AP, at a matched token budget
Gate rank $r$	70%	40%	20%
0.05	47.72	40.06	20.81
0.10	47.74	40.40	21.14
0.20 (shipped)	47.76	40.44	21.18
0.30	47.71	40.14	20.96

Table A9 : Rank sweep at three matched budgets. The shipped

r=0.20

is the argmax of AP in all three columns and AP falls on both sides of it, so this is an interior optimum rather than an untested default. Twelve zero-shot evaluations of one released dense checkpoint, tokenizer swapped in and no weight changed: VisDrone val,

548

images, maxDets

500

, native

2048\times 1152

, splitter and scorer fixed so that

r

is the only variable. We match budgets per rank by inverting each rank’s own retention table, so the percentiles Tab. A12 prints differ by row. Bold is the best value in a column among the four comparable rows. No training seed enters a zero-shot evaluation, so these differences are exact rather than draws against the seed floor. A fifth rung at

r=0.40

falls further, to

40.05

and

20.80

. The rank was chosen on VisDrone val, then carried onto AI-TOD-v2 and never re-tuned.

Figure A1 : Dense epoch ladder, eleven checkpoints of one training run. The ladder saturates: everything after epoch

12

is worth

+0.32

AP in total, and the last twelve epochs of that only

+0.10

. Every point is scored under the reported protocol: VisDrone val,

548

images, maxDets

500

, ignore regions at IoF

\geq 0.5

, native

2048\times 1152

. The ringed point is the checkpoint we report and the maximum of both curves.

Tokenizer (AP)	Run 1	Run 2	Run 3	Mean	$\sigma$
Top-hat ( $\mathcal{T}$ )	43.66	43.76	43.77	43.73	0.06
Top-hat ( $\mathcal{T}$ ) $+\,g$	44.11	44.26	44.30	44.22	0.10
Top-hat ( $\mathcal{T}$ ) $\oplus\,\lambda$	43.65	43.90	43.99	43.85	0.18
Top-hat ( $\mathcal{T}$ ) $\oplus\,\lambda+g$	43.96	43.95	43.91	43.94	0.03
Tokenizer (AP S )	Run 1	Run 2	Run 3	Mean	$\sigma$
Top-hat ( $\mathcal{T}$ )	34.89	34.93	35.16	34.99	0.15

Table A10 : Seed replicates at the

40\%

budget. The gated and ungated tokenizers do not overlap on either metric: worst gated

44.11

against best ungated

43.77

on AP, and

35.35

against

35.16

on the noisier AP S , so the reported gate effect is not seed noise. VisDrone2019-DET val,

548

images, maxDets

500

; all twelve runs sit at one budget,

0.4001

retained ( Tab. A11 ), and the two blocks score them on AP and AP S . Run 1 is the original cell and the rest independent repeats;

\sigma

is their sample standard deviation. We pool both seed floors over the factorial instead of reading them off one cell: within-cell

\sigma

over the

24

trained VisDrone cells is

0.0974

AP and

0.1342

AP S , so a difference of two three-seed means has standard error

0.0795

and

0.1095

; twice that,

0.16

AP and

0.22

AP S , is the smallest difference we argue for.

Tokenizer	Threshold	Tokens	Retained
Top-hat ( $\mathcal{T}$ )	$p\,81.88$	3687.9	0.4001
Top-hat ( $\mathcal{T}$ ) $+\,g$	$p\,74.82$	3687.3	0.4001
Top-hat ( $\mathcal{T}$ ) $\oplus\,\lambda$	$p\,81.96$	3687.6	0.4001
Top-hat ( $\mathcal{T}$ ) $\oplus\,\lambda+g$	$p\,74.70$	3688.1	0.4001

Table A11 : Token-budget audit of the

40\%

column. All four tokenizers land on

0.4001

of the dense grid, so the gate’s margin at this budget is measured at matched tokens. We measure the retained fraction over all

548

VisDrone val images at

2048\times 1152

against a dense grid of

9216

tokens, for each tokenizer at the exact threshold we deployed it at; Tokens is the per-image mean.

VisDrone val , $548$ images
Gate rank $r$	80%	70%	60%	40%	20%
none (= $\mathcal{T}$ )	37.91	51.53	63.44	81.88	94.58
0.05	32.16	47.12	60.28	80.45	94.21
0.10	25.56	42.03	56.54	78.82	93.79
0.20 (shipped)	5.62	29.06	47.01	74.82	92.76
0.30	unreachable	8.66	33.43	69.16	91.38

Table A12 : Inverted retention tables, five budgets by five gate ranks. Rank and threshold are not interchangeable: raising

r

shifts the whole table left until the loosest budgets fall off it, and by

r=0.30

the

80\%

budget is unreachable on both datasets. Each cell is the percentile that delivers the column’s budget at that rank, from a

32

-rung measured sweep over all

548

VisDrone val images and

2804

AI-TOD-v2 images. The italic row is the shipped setting,

r=0.20

, used unchanged on both. Unreachable is structural rather than unrun: at that rank the gate alone caps retention below the budget, which is why Tab. A7 has no

80\%

block. We invert percentiles on the AI-TOD-v2 val split and verify them against the

14{,}018

-image test split both tokenizers are scored on, where each overshoots its target by at most

0.15

percentage points.

Evaluation resolution	AP	AP S	AP	AP S	gap , AP
	VGTok, $100\%$		VGTok, $70\%$
1536	45.52	35.97	44.25	34.94	$+1.28$
2048 (the protocol)	48.38	39.63	47.58	38.85	$+0.80$
2304	48.86	40.39	48.14	39.85	$+0.72$
2560	49.39	41.04	48.70	40.45	$+0.69$

Table A13 : VGTok at two budgets across four input resolutions. The cost of pruning falls as the input grows: the gap between full and

70\%

retention narrows monotonically from

1.28

AP at

1536

0.69

2560

, so the tokenizer gets cheaper in accuracy terms exactly where high-resolution detection is headed. Same checkpoint, same

548

-image VisDrone val split, maxDets

500

; only the input resolution changes. The

70\%

column is Single-pass evaluated zero-shot, the tokenizer we measured at all four resolutions, rather than the shipped quadtree one; the reading is about how the two columns move together at a fixed retention and does not depend on which tokenizer fills the second pair. Gap is

100\%

minus

70\%

on AP, taken on unrounded values: at

1536

it is

1.28

where the printed cells differ by

1.27

VisDrone val , $150$ images
			coarsened tokens, by distance
Budget	fires on	frees	on an object	1 node	2–3 nodes	farther
70%	4.4%	13.1 pp	6.3%	5.1%	12.3%	76.4%
40%	2.3%	6.9 pp	14.6%	6.0%	9.9%	69.5%
20%	0.9%	2.8 pp	29.0%	6.4%	7.6%	57.0%
AI-TOD-v2 val , $150$ images

Table A14 : Coarsened tokens, by distance to the nearest object. The rate follows the budget, not the statistic: at the

40\%

operating point

14.6\%

of the tokens the gate declines to spend lie on an object, and on AI-TOD-v2 the figure is

1.4\%

or below at every budget we report. The last four columns bin every finest-grid token the gate declines to spend by distance from its node to the nearest ground-truth box; the region is still described, at the gate’s own coarse scale. Shipped statistic at rank

r=0.20

, on

150

VisDrone val and

150

AI-TOD-v2 val images. Fires on is the fraction of gate-visible nodes that fire; frees is the percentage points of the dense grid regained: at

20\%

, firing on

0.9\%

of nodes frees

2.8

points. Rows are rounded independently and need not sum to

100

Visual Tokenization

Visual tokenization turns images and video into compact, structured tokens for efficient perception, compression, and generation.

See all 70 papers

Jul 10, 2026

Subtoken Vision Transformer for Fine-grained Recognition

We present Subtoken Vision Transformer (SubViT), a selective image tokenization method for fine-grained visual recognition. Standard Vision Transformers compress each fixed-size patch into a single token, although fine-grained distinctions often depend on localized variations within only a few patches. SubViT addresses this mismatch by representing discriminative patches with multiple subtokens while retaining the original token sequence for global context, thereby allocating additional capacity where it is most needed. Since attention heads encode complementary semantics and extracting attention maps at inference requires an extra backbone forward, we adopt a two-stage training strategy. Stage 1 fine-tunes the ViT using subdivision regions sampled from random attention heads, exposing the model to diverse subdivision patterns. Stage 2 identifies informative attention maps through feature-degradation distances and distills them into a lightweight single-map router, which directly predicts deterministic token-importance scores without a separate attention forward. We evaluate SubViT on Generalized Category Discovery (GCD), a challenging task requiring both fine-grained discrimination and generalization to unlabeled novel categories. Across CUB, FGVC-Aircraft, and Stanford-Cars, SubViT improves the average novel-category accuracy of DINOv2 from

81.3\%

84.7\%

, with only

0.50

ms additional latency and

3.4\%

more FLOPs, while reducing latency by

73.8\%

relative to Retina Patch. Code: SubViT.

Jie Zhu, Ivy Zhang, Minchul Kim +1

Michigan State University · Cranbrook Kingswood School · University of North Carolina at Chapel Hill

Vision Transformer Visual Tokenization Fine-Grained Image Classification Efficient ViTs

Ask PDF

Apr 2, 2026

SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentation

Foundational Vision Transformers (ViTs) have limited effectiveness in tasks requiring fine-grained spatial understanding, due to their fixed pre-training resolution and inherently coarse patch-level representations. These challenges are especially pronounced in dense prediction scenarios, such as open-vocabulary segmentation with ViT-based vision-language models, where high-resolution inputs are essential for accurate pixel-level reasoning. Existing approaches typically process large-resolution images using a sliding-window strategy at the pre-training resolution. While this improves accuracy through finer strides, it comes at a significant computational cost. We introduce SPAR: Single-Pass Any-Resolution ViT, a resolution-agnostic dense feature extractor designed for efficient high-resolution inference. We distill the spatial reasoning capabilities of a finely-strided, sliding-window teacher into a single-pass student using a feature regression loss, without requiring architectural changes or pixel-level supervision. Applied to open-vocabulary segmentation, SPAR improves single-pass baselines by up to 10.5 mIoU and even surpasses the teacher, demonstrating effectiveness in efficient, high-resolution reasoning. Code: https://github.com/naomikombol/SPAR

Naomi Kombol, Ivan Martinović, Siniša Šegvić +1

Faculty of Electrical Engineering and Computing, University of Zagreb · VRG, Faculty of Electrical Engineering, Czech Technical University in Prague

Vision Transformer Efficient ViTs Open-Vocabulary Semantic Segmentation

Ask PDF

Mar 27, 2026

ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction

We present ARTA, a mixed-resolution coarse-to-fine vision transformer for efficient dense feature extraction. Unlike models that begin with dense high-resolution (fine) tokens, ARTA starts with low-resolution (coarse) tokens and uses a lightweight allocator to predict which regions require more fine tokens. The allocator iteratively predicts a semantic (class) boundary score and allocates additional tokens to patches above a low threshold, concentrating token density near boundaries while maintaining high sensitivity to weak boundary evidence. This targeted allocation encourages tokens to represent a single semantic class rather than a mixture of classes. Mixed-resolution attention enables interaction between coarse and fine tokens, focusing computation on semantically complex areas while avoiding redundant processing in homogeneous regions. Experiments demonstrate that ARTA achieves state-of-the-art results on ADE20K and COCO-Stuff with substantially fewer FLOPs, and delivers competitive performance on Cityscapes at markedly lower compute. For example, ARTA-Base attains 54.6 mIoU on ADE20K in the ~100M-parameter class while using fewer FLOPs and less memory than comparable backbones.

David Hagerman, Roman Naeem, Erik Brorsson +2

Chalmers University of TechnologyChalmers University of technology, 412 96 Gothenburg, Sweden

Efficient ViTs Vision Transformer Semantic Segmentation

Ask PDF

Abstract

Visual Tokenization

Explore similar work

Subtoken Vision Transformer for Fine-grained Recognition

SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentation

ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction