ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction

David Hagerman; Roman Naeem; Erik Brorsson; Fredrik Kahl; Lennart Svensson

cs.CVMar 27, 2026

ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction

Authors: David Hagerman, Roman Naeem, Erik Brorsson, Fredrik Kahl, Lennart Svensson

Organizations: Chalmers University of TechnologyChalmers University of technology, 412 96 Gothenburg, Sweden

Abstract

We present ARTA, a mixed-resolution coarse-to-fine vision transformer for efficient dense feature extraction. Unlike models that begin with dense high-resolution (fine) tokens, ARTA starts with low-resolution (coarse) tokens and uses a lightweight allocator to predict which regions require more fine tokens. The allocator iteratively predicts a semantic (class) boundary score and allocates additional tokens to patches above a low threshold, concentrating token density near boundaries while maintaining high sensitivity to weak boundary evidence. This targeted allocation encourages tokens to represent a single semantic class rather than a mixture of classes. Mixed-resolution attention enables interaction between coarse and fine tokens, focusing computation on semantically complex areas while avoiding redundant processing in homogeneous regions. Experiments demonstrate that ARTA achieves state-of-the-art results on ADE20K and COCO-Stuff with substantially fewer FLOPs, and delivers competitive performance on Cityscapes at markedly lower compute. For example, ARTA-Base attains 54.6 mIoU on ADE20K in the ~100M-parameter class while using fewer FLOPs and less memory than comparable backbones.

Figures & tables

Figure 1: ARTA overview. ARTA has two stages: Adaptive Token Allocation (Stage 1) and Token Refinement (Stage 2). Stage 1 proceeds bottom-up from coarse $32{\times}32$ tokens. A pre-allocation ViT produces boundary-aware features, after which each allocation round applies a token allocation block (rounds 1–2 also use cluster attention). The allocation block scores the finest tokens and allocates finer-grained tokens to the corresponding patches containing class-boundaries (Figure 2 ), progressively building mixed-resolution sets up to $[32{\times}32,16{\times}16,8{\times}8,4{\times}4]$ . Stage 2 proceeds top-down from the final set and refines tokens using cluster attention, with a ViT in the final round. At each refinement round, the finest tokens are output, and the remaining tokens continue. Lateral Stage 1 outputs are fused into Stage 2 by concatenation and projection before attention. The decoder uses these multi-scale features for dense prediction.

Figure 2: Token Allocation Block. The allocator scores the current finest-resolution tokens and selects the corresponding patches containing class-boundaries. Selected patches are split into $2{\times}2$ sub-patches to allocate finer tokens. Each new token is initialized from sub-patch image features (MLP) and combined with a broadcast residual from the parent token. Finally, scale and position embeddings are added to get output tokens.

	Stage 1		Stage 2
Encoder	Patch Size: [ $32^{2}$ , $16^{2}$ , $8^{2}$ , $4^{2}$ ]		Patch Size: [ $4^{2}$ , $8^{2}$ , $16^{2}$ , $32^{2}$ ]
	Dims	Blocks	Dims	Blocks
ARTA-Tiny	[512, 256, 128, 64]	[1, 1, 1, 0]	[64, 128, 256, 512]	[4, 4, 16, 4]
ARTA-Small	[512, 256, 128, 64]	[2, 2, 2, 0]	[64, 128, 256, 512]	[4, 6, 24, 3]
ARTA-Base	[768, 384, 192, 96]	[2, 2, 2, 0]	[96, 192, 384, 768]	[8, 6, 18, 4]

Table 1: Encoder architecture hyperparameters for ARTA-Tiny, ARTA-Small, and ARTA-Base. For each round, we report the token embedding dimensions and the number of attention blocks. Entries are listed in execution order, following ARTA’s coarse-to-fine-to-coarse layout.

Model	Params	FLOPs $\downarrow$	mIoU	mIoU
Model	Params	FLOPs $\downarrow$	(SS) $\uparrow$	(MS) $\uparrow$
ViT-CoMer-T [ 29 ]	38.7M	-	43.0	44.3
SegFormer-B3 [ 30 ]	47.3M	79G	49.4	50.0
SegNeXt-L [ 11 ]	48.9M	70G	51.0	52.1
SegMAN-B [ 10 ]	51.8M	58G	52.6	-
Mask2Former-Swin-T [ 6 ]	46.5M	74G	47.7	49.6

Table 2: Comparisons with state-of-the-art methods on ADE20K val.

Model	Params	COCO-Stuff		Cityscapes
		FLOPs $\downarrow$	mIoU $\uparrow$	FLOPs $\downarrow$	mIoU $\uparrow$
			(SS/MS)		(SS/MS)
SegFormer-B3 [ 30 ]	47.3M	79G	45.5 / -	963G	81.7 / 83.3
SegNext-L [ 11 ]	48.9M	70G	46.5 / 47.2	578G	83.2 / 83.9
SegMAN-B [ 10 ]	51.8M	58G	48.4 / -	479G	83.8 / -
Mask2Former-Swin-T [ 6 ]	46.5M	-	- / -	537G	82.1 / 83.0

Table 3: Comparisons with state-of-the-art methods on the validation sets of COCO-Stuff and Cityscapes. “

\dagger

” indicates that ImageNet22k was used for pre-training.

Figure 3: From left to right: original image, ground truth, $32\times 32$ patches selected for allocation, $16\times 16$ patches selected for allocation, $8\times 8$ patches selected for allocation, and prediction. Black in the ground truth means that the pixel was not labeled.

Model	Params	FLOPs	FPS	Peak mem (GB)
SegFormer-B3 [ 30 ]	47.3M	963G	6.6	21.9
SegNeXt-L [ 11 ]	48.8M	554G	7.3	16.9
SegMAN-B [ 10 ]	51.8M	479G	5.6	23.8
AFF-Tiny [ 35 ]	46.5M	463G	3.8	34.4
ARTA-Tiny	49.5M	286 $\pm$ 20G	6.6	14.6
SegFormer-B4 [ 30 ]	64.1M	1241G	4.7	29.5

Table 4: FPS and peak allocated memory during inference on Cityscapes with a single NVIDIA A100.

Initial/Final block type	Intermediate block type	mIoU
Initial/Final Block Ablation
Cluster Attention [ 35 ]	Cluster Attention [ 35 ]	42.1
ConvNeXt V2 [ 27 ]		41.8
Swin [ 16 ]		42.2
ViT [ 8 ]		42.4
Intermediate Block Ablation

Table 5: Block-type ablation for ARTA-Tiny on ADE20K. “OOM” indicates out of memory.

Table 9

Explore similar work

Vision Transformers Need Better Token Interaction
May 22, 2026Linxiang SuVisual Representation Learning Vision Transformer
University of Szeged
SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentation
Apr 2, 2026Naomi Kombol, Ivan Martinović, Siniša Šegvić +1Vision Transformer Efficient ViTs
Faculty of Electrical Engineering and Computing, University of Zagreb · VRG, Faculty of Electrical Engineering, Czech Technical University in Prague
MixTok: Adaptive Visual Tokenization with Global 1D Tokens and Sparse 2D Refinement
Oct 9, 2026Chenge Li, Yefei He, Xinyu Liu +3Visual Tokenization Visual Token Compression
Zhejiang University · The Hong Kong University of Science and Technology · University of Adelaide

May 22, 2026J/K move · Enter open · S save

Vision Transformers Need Better Token Interaction

Linxiang Su

University of Szeged

Visual Representation Learning Vision Transformer Sparse Attention Semantic Segmentation

Open paper AskPDF arXiv

Vision Transformers (ViTs) can learn strong image-level representations while their patch representations become less effective for dense prediction during prolonged training. We revisit this dense degradation phenomenon and argue that it is not fully explained by high-norm artifacts alone. Instead, we characterize semantic diffusion: an optimization shortcut in which global semantic information spreads through patch tokens beyond what is locally justified. Our analysis shows that dense representation quality is not captured by locality alone: shallow features can remain better aligned with foreground regions yet underperform deeper features, and [CLS] features remain complementary for dense prediction. These observations suggest that the goal should not be to remove global context, but to make token interactions more selective. We therefore study sparse attention as a minimal intervention, replacing softmax attention with entmax-1.5 while preserving global token connectivity. On DINOv1 ViT-S/16 trained for 200 epochs on ImageNet-1K, this change preserves ImageNet linear probing accuracy and substantially improves semantic segmentation performance: VOC mIoU increases from 42.80 to 48.78, ADE20K from 19.85 to 21.97, and Cityscapes from 36.79 to 37.87. These results suggest that selective token mixing is a simple and effective bias for improving dense ViT representations.

Abstract

Explore similar work

Vision Transformers Need Better Token Interaction

SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentation

MixTok: Adaptive Visual Tokenization with Global 1D Tokens and Sparse 2D Refinement