cs.CVNov 8, 2026

Differentiable Virtual-Slot Partitioning for Adaptive Tokenization in Vision Transformers

Authors: Gaoxiang He, Bin Zhu, Bo Xie, Xiaotong Yu, Yi Chen, Hongfei Yan

Organizations: National University of Defense Technology

Abstract

Adaptive tokenization aims to reduce the compute spent on uninformative image regions while preserving accuracy. Existing work relies on (i) fixed or frozen scoring rules, which cannot be trained toward the task loss, or (ii) discrete masks trained with straight-through estimators, which yield noisy gradients, while fully differentiable alternatives instead leave the pre-trained regular grid. We revisit adaptive tokenization from a regular-grid perspective and ask: can a two-scale quadtree be learned end-to-end without deleting a single token during training? We show that one coarse token and its four fine children can be made the two endpoints of a single continuous interpolation. We introduce Differentiable Virtual-Slot Partitioning (\dvsp{}), which keeps four virtual slots per 2×22{\times}2 region and blends their token features and attention mass with a single weight αα, exactly reproducing the discrete grid at α∈{0,1}α\in \{0,1\}. On COCO detection with EVA-02-B, \dvsp{} retains 97.9%97.9\% of dense Box AP while removing 64.2%64.2\% of visual tokens, giving a 1.61×1.61\times end-to-end speedup, and it outperforms entropy-based APT, DART, and uniform lower-resolution dense adaptation at matched budgets; these gains depend on the conserved attention mass, whose removal costs 1.591.59 Box AP. On a frozen Segmenter it also improves Cityscapes mIoU, and on frozen DeiT backbones it exceeds entropy selection at identical per-image token counts.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Foundation Models

    Jun 12, 2025Shicheng Yin, Kaixuan Yin, Yang Liu +2Vision Foundation ModelsEfficient ViTs

  2. Learning Hierarchical Patch Splitting Policies for Faster Vision Transformers

    Nov 3, 2026Aditya Gupta, Jean Sebastien Dandurand, Kai Qiu +2Vision TransformerEfficient ViTs

  3. ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction

    Mar 27, 2026David Hagerman, Roman Naeem, Erik Brorsson +2Efficient ViTsVision Transformer