Adaptive tokenization aims to reduce the compute spent on uninformative image regions while preserving accuracy. Existing work relies on (i) fixed or frozen scoring rules, which cannot be trained toward the task loss, or (ii) discrete masks trained with straight-through estimators, which yield noisy gradients, while fully differentiable alternatives instead leave the pre-trained regular grid. We revisit adaptive tokenization from a regular-grid perspective and ask: can a two-scale quadtree be learned end-to-end without deleting a single token during training? We show that one coarse token and its four fine children can be made the two endpoints of a single continuous interpolation. We introduce Differentiable Virtual-Slot Partitioning (\dvsp{}), which keeps four virtual slots per 2×2 region and blends their token features and attention mass with a single weight α, exactly reproducing the discrete grid at α∈{0,1}. On COCO detection with EVA-02-B, \dvsp{} retains 97.9% of dense Box AP while removing 64.2% of visual tokens, giving a 1.61× end-to-end speedup, and it outperforms entropy-based APT, DART, and uniform lower-resolution dense adaptation at matched budgets; these gains depend on the conserved attention mass, whose removal costs 1.59 Box AP. On a frozen Segmenter it also improves Cityscapes mIoU, and on frozen DeiT backbones it exceeds entropy selection at identical per-image token counts.
Figures & tables
Fig. 1: DVSP overview. (a) A dense ViT detector processes 9,216 uniform 16×16 tokens. (b) DVSP chooses 16×16 or 32×32 tokens per 2×2 parent with learned continuous weights, keeping fine tokens on the person and bench. (c) On COCO val2017 (EVA-02-B), DVSP retains 97.9% Box AP ( 58.93→57.72 ) with 3,303 tokens per image ( ↓64.2% ) and a measured 1.61× speedup.
Fig. 2: Token allocation paradigms. (a) APT picks coarse/fine regions with a fixed image-entropy rule, so task gradients cannot train the router. (b) Discrete-gate methods (e.g., MSViT) gate in the forward pass and rely on straight-through gradients. (c) DVSP keeps four continuous virtual slots per parent: one router weight α blends token features and attention mass, so exact task gradients reach the router.
Fig. 3: Learned coordinates after training on frozen DeiT-Tiny/Small (ImageNet-1k). Coordinates concentrate at the two binary endpoints; at most 3.5% remain within 0.4 – 0.6 of the deployment threshold, and excursions outside [0,1] are shallow ( 6.4% / 7.6% , almost all within 0.1 of 0 ).
Method
Tokens
Box AP
Δ Box
AP S
Mask AP
Δ Mask
Dense EVA-02-B
9,216 (100%)
58.93
–
42.09
50.69
–
Budget ≈ 4,200 tokens
Dense, adapted at 1,0402
4,225 (45.8%)
57.07
−1.86
37.89
48.52
−2.17
APT, entropy, per budget
∼ 4,200 (45.6%)
57.33
−1.60
39.89
49.40
−1.29
DART-Flow, fixed N
4,235 (46.0%)
55.81
−3.12
36.90
46.99
−3.70
DART-Grid, fixed N
4,225 (45.8%)
55.94
−2.99
35.85
47.12
−3.57
TABLE I: COCO val2017 detection and instance segmentation with EVA-02-B ( 9,216 dense visual tokens). Each budget block lists the same five systems in the same order, so every DVSP row is read against the four rows directly above it. Parentheses give the mean fraction of kept tokens. Δ is relative to the unadapted dense detector. DART-Grid’s N=NhNw grid uses the exact counts 4,225/3,600/3,306, within 0.4% of each tier.
Fig. 4: Token splits on a COCO scene. (a) Dense, (b) APT, (c) DART-Flow, (d) DVSP, with 9,216 , 3,303 , 3,303 , and 3,378 tokens. Boxes with score ≥0.5 . Of the 35 annotated objects, DVSP matches 21 , as many as dense, against 14 for APT and 13 for DART-Flow. More cases in Appendix C .
Spatial score
Tokens (Red.)
mIoU
Δ mIoU
Dense reference
2,304 (–)
76.49
–
APT entropy
1,045 (54.6%)
72.38
−4.11
DVSP
1,045 (54.6%)
74.77
−1.72
TABLE II: Frozen-task Cityscapes spatial selection. Equal fine-parent counts per crop. Binary rows execute all 2,304 slots at the projected binary vertices under the mass rule of Eq. ( 4 ); by Proposition 1 (ii) this equals packed native execution of the listed token counts layer by layer, so counts measure the implied packed length, not executed slots. Δ is relative to dense. Router checkpoint selected on validation.
Fig. 5: Measured EVA-02-B runtime breakdown. COCO val2017, TITAN RTX, FP16, batch one, same compile+TensorRT backend for both systems. Wall time covers transfer, preprocessing, routing, patch construction, the detector, and postprocessing, excluding data loading. Routing and the coarse-token patch branch add 15.6 ms while the transformer drops 98.7→25.2 ms, so wall time falls 157.5→97.6 ms ( 6.35→10.24 images/s, 1.61× ).
Fig. 6: Token budget during router-only training on frozen DeiT backbones (ImageNet-1k). Training-set mean visual tokens; the ℓ1 penalty ramps over epochs 0.5 – 1.5 . Final full-validation Top-1 is 75.36% and 80.86% ; executed validation lengths are 154.0 and 136.4 .
Mass rule
Tokens
Box AP
Mask AP
Unit mass m≡1
3,268
56.13
48.23
Affine m(α) (ours)
3,303
57.72
49.44
TABLE III: Attention-mass conservation on COCO val2017. Same EVA-02-B recipe and postprocessing; only the mass rule differs. Both rows execute in native signed-threshold mode.
Task
Backbone
Continuous
Binary
Δ
ImageNet-1k, top-1
Tiny, run 1
DeiT-T
75.06
75.33
+0.27
Tiny, run 2
DeiT-T
75.02
75.29
+0.27
Small, run 1
DeiT-S
80.74
80.86
+0.12
Cityscapes, mIoU
resize, λ=0.1
seg.-B
74.23
74.89
+0.66
TABLE IV: Continuous field vs. its binary projection. Each pair uses the same router checkpoint and the same executed token count; accuracy is measured with the field executed continuously ( α ) versus projected ( bi=1[αi≥0.5] ).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 7: COCO object detection examples. Each row is one val2017 scene; columns show the Dense detector, APT, and DVSP with their recorded partitions (white lines, 32×32 coarse regions rendered as centered mini patches) and all saved score ≥0.5 predictions. Token plates give each model’s executed count on the full 15362 canvas. Best viewed zoomed in.
Fig. 8: ImageNet-1k classification examples. Five val images from the frozen-backbone router study of Sec. IV-C ; in every row DVSP is correct while the entropy control is wrong. (a) DVSP partition; (b) APT entropy partition. Class names under the panels give each method’s Top-1 prediction, and the ground truth tags each row. DVSP spends its tokens on the object while the entropy control spends them on background texture. Best viewed zoomed in.
Fig. 9: Cityscapes semantic segmentation examples. Six fixed 768×768 val windows (rows) under the frozen Segmenter of Sec. IV-C : two non-overlapping windows from each of three val images. Each window compares three methods at the same fine-parent budget: Dense (uniform 16×16 grid, 2,304 tokens), APT entropy (the Table II baseline), and DVSP ( 1,035 / 1,098 / 1,161 / 1,167 / 1,146 / 1,074 tokens). In each of the three columns the left panel is the input with that method’s executed patch assignment and the right panel the resulting segmentation. Entropy places fine patches on textured facades and foliage, while DVSP keeps them along object edges and small pedestrians; flat road and sky become 32×32 tokens in both adaptive rows. The DVSP partitions use the recorded ∣α∣≥0.1 projection of this frozen-scorer study.