DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Foundation Models

Shicheng Yin; Kaixuan Yin; Yang Liu; Weixing Chen; Liang Lin

cs.CVJun 12, 2025

DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Foundation Models

Authors: Shicheng Yin, Kaixuan Yin, Yang Liu, Weixing Chen, Liang Lin

Organizations: Sun Yat-sen University, China

Abstract

The content-agnostic, fixed-grid tokenizers used by standard large-scale vision models like Vision Transformer (ViT) and Vision Mamba (Vim) represent a fundamental performance bottleneck, creating a trade-off between capturing fine-grained detail and suffering from redundant computation. To resolve this dilemma, we introduce DART, a fully differentiable Dynamic Adaptive Region Tokenizer. DART employs learnable region scores and quantile-based partitioning to create content-aware patches of varying sizes, intelligently allocating a higher token density to information-rich regions. The impact of this approach is profound: it unlocks a more intelligent scaling paradigm, where a DART-equipped DeiT-Small (22M parameters) matches the performance of a DeiT-Base (86M) with nearly double the inference speed by efficiently capturing high-resolution details in key regions. Furthermore, the principle of adaptive tokenization proves its generality with clear benefits in dense prediction and spatiotemporal video tasks. We argue that by resolving the tokenizer bottleneck at its source, adaptive tokenization is a key component for building the next generation of more efficient and capable foundation models for multimodal AI, robotics, and content generation. Code is available at https://github.com/HCPLab-SYSU/DART.

Figures & tables

Panel of Figure 3 — Figure 3: Illustration of our core differentiable partitioning mechanism. (a) Boundaries for a 1D distribution are found by inverting its piecewise-linear CDF at uniform quantiles (e.g., 1/3, 2/3). (b) The DART-Grid is formed by independently applying this 1D algorithm to the horizontal and vertical marginal distributions of the 2D score map, where the score map produced by the lightweight network is overlaid on the image.

Panel of Figure 4 — Figure 4: The DART-Flow process. The partitioning algorithm is applied sequentially: first to create adaptive horizontal rows, and then to globally allocate tokens across all rows.

Figure 5: Patch partitioning on an example from the SSv2 dataset Goyal et al. (2017) .

Backbone	Tokenizer	Params	Patches	FLOPs	Top-1 (%)
DeiT-S †	Baseline	22M	576	15.5G	81.6
DeiT-S †	DART	24M	288	7.2G	81.5
VideoMamba-Ti †	Baseline	7M	1296	7.11G	79.6
VideoMamba-Ti †	DART	8M	392	2.24G	79.7

Table 1: Unprecedented efficiency on high-resolution inputs. † denotes long-sequence fine-tuning.

Table 7

Table 8

Table 9

Figure 7: Visualization of the learned score map and patch boundaries at different training epochs, showing the model progressively learning to focus on the salient object.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Scoring Network	Top-1 (%)
w/o (Baseline)	72.2
MobileNetV3-S Howard et al. (2019)	73.8 (+1.6)
MnasNet Tan et al. (2019)	74.0 (+1.8)
SqueezeNet Iandola et al. (2017)	74.3 (+2.1)
EfficientNet-B0 Tan & Le (2019)	75.1 (+2.9)

Appendix

Table 9: Impact of the scoring network architecture on DeiT-Ti.

Model	Throughput (img/s)	Latency (ms)
DeiT-S (Baseline)	1110.5	115.3
DeiT-S + DART	1053.2 (-5.2%)	121.5 (+6.2ms)

Appendix

Table 10: DART module overhead on an NVIDIA RTX 3090 (196 tokens, batch size 128).

Model	Feature Extractor
DeiT - Ti	MobileNetV3 Small[:11]
DeiT - S	MobileNetV3 Large[:15]
Vim - Ti	MobileNetV3 Small[:13]
Vim - S	MobileNetV3 Large[:17]
VideoMamba - Ti	MobileNetV3 Small[:13]

Appendix

Table 11: Feature extractor configurations for various backbone models.

Panel of Figure 8 — Figure 8: Qualitative comparison of DART’s behavior on two distinct types of challenging inputs. These cases highlight the model’s adaptive nature, showing how it intelligently shifts between highly focused and uniform partitioning strategies based on image content.

Figure 9: Partition examples produced by our DART model (Part 1).

Figure 10: Partition examples produced by our DART model (Part 2).

Figure 11: A visual comparison of partitioning strategies. Left: Our main DART-Flow method. Middle: The grid-preserving DART-Grid method. Right: A visualization simulating the model’s input for the DART-Grid method, created by reassembling the non-uniform patches into a uniform grid. In this view, high-density regions (e.g., the animal’s body) appear magnified while low-density background areas are compressed. It is important to note that this is purely a visualization aid; the model’s understanding of the original geometry is preserved through transformed positional embeddings and is not distorted.

Vision Foundation Models

Vision foundation models are large pretrained visual systems adapted to recognition, segmentation, geometry, and video tasks.

See all 141 papers

DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunking
Mar 6, 2026Akash Haridas, Utkarsh Saxena, Parsa Ashrafi Fashi +3Adaptive Tokenization Diffusion Transformer
Advanced Micro Devices Inc. (AMD)
ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction
Mar 27, 2026David Hagerman, Roman Naeem, Erik Brorsson +2Efficient ViTs Vision Transformer
Chalmers University of TechnologyChalmers University of technology, 412 96 Gothenburg, Sweden
ChannelTok: Efficient Flexible-Length Vision Tokenization
Jun 3, 2026Sukriti Paul, Arpit Bansal, Tom GoldsteinImage Tokenization Visual Tokenization
University of Maryland, College Park

Mar 6, 2026J/K move · Enter open · S save

DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunking

Akash Haridas, Utkarsh Saxena, Parsa Ashrafi Fashi, Mehdi Rezagholizadeh+2

Advanced Micro Devices Inc. (AMD)

Adaptive Tokenization Diffusion Transformer Diffusion-Based Image Generation Diffusion Model Inference Acceleration Visual Token Compression

Open paper AskPDF arXiv

Diffusion Transformers rely on static patchify tokenization, assigning the same token budget to smooth backgrounds, detailed object regions, noisy early timesteps, and late-stage refinements. We introduce the Dynamic Chunking Diffusion Transformer (DC-DiT), which replaces fixed patchification with a learned encoder-router-decoder scaffold that adaptively compresses the 2D input into a shorter token sequence through a chunking mechanism learned end-to-end with diffusion training. DC-DiT allocates fewer tokens to predictable regions and noisy timesteps, and more tokens to detailed regions and later refinement stages, yielding meaningful spatial segmentations and timestep-adaptive compression schedules without supervision. Furthermore, the router provides an importance ordering over retained tokens, enabling elastic inference: a single checkpoint can be evaluated at flexible compute budgets with a smooth quality-compute tradeoff. Additionally, DC-DiT can be upcycled from pretrained DiT checkpoints and is also compatible with orthogonal dynamic computation approaches. On class-conditional ImageNet generation, DC-DiT reduces inference FLOPs by up to 36.8% and improves FID by up to 37.8% over DiT baselines, yielding a stronger quality--compute Pareto frontier across model scales, resolutions, and guidance settings. More broadly, these results suggest that adaptive tokenization is a general mechanism for making visual generation both more efficient and more flexible at inference time.

Abstract

Vision Foundation Models

Explore similar work

DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunking

ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction

ChannelTok: Efficient Flexible-Length Vision Tokenization