cs.CVOct 14, 2026

Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation

Authors: Utku Cicek, Mahip Singh, Ziyao Shang, Kimathi Kaai, C Thomas, Pedro Pablo Guerrero Vela, Alexander Wong, Sirisha Rambhatla

Organizations: University of Waterloo · Apple

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

While foundation models and vision transformers (ViTs) have redefined visual representations, current active learning (AL) strategies for these backbones largely operate at the patch or region level, where annotation requirements remain high due to extensive pixel labeling. Extending ViT-based AL to the pixel-level, low-budget regime is uniquely challenging: with as little as one pixel query per image, a method must simultaneously identify the most informative locations and train a reliable decoder head from an extremely sparse signal. We introduce TEAL (Token-space Efficient Active Learning), the first pixel-level AL framework built around ViT representations. TEAL uses a frozen ViT backbone and performs diversity-based selection directly on its native token lattice, avoiding the artifacts introduced by interpolating token embeddings to dense pixel resolutions. Our framework applies a two-level MaxHerding strategy over multi-layer token descriptors to select representative candidates, followed by margin-based uncertainty refinement on a finer decoder grid. Across CamVid, Cityscapes, ADE20K, and Pascal Context, TEAL consistently outperforms previous baselines under extreme label scarcity, improving over the strongest baseline by up to +24.36 mIoU with a single labeled pixel (px) per image, and reaching up to 97% of fully supervised performance with just 10 px. To rigorously establish the source of the gains, we further show that the improvements persist when all baselines are equipped with the same DINOv3 backbone, demonstrating that coupling token-space selection with uncertainty refinement on the decoder grid makes ViT token geometry an effective foundation for extremely low-budget active segmentation.

Figures & tables

Figure 1: Token-space acquisition avoids pixel-space interpolation artifacts. (a) Compares query locations from TEAL and the two-stage eDALD baseline Kim et al. (2026) , with both methods using the same frozen DINOv3 backbone for a controlled comparison. Pixel-space selection often concentrates near boundaries, where upsampled token features mix neighboring descriptors, while TEAL selects more representative regions by operating on the native token lattice. As (c) shows that this leads to cleaner segmentation maps under the same sparse pixel-label budget.
Figure 2: One round of TEAL active learning. A Mini-DPT head is trained on the current labeled pixel set Lt\mathcal{L}_{t} using frozen multi-layer DINOv3 token features. Stage 1 performs two-level MaxHerding in the native token space: local herding selects kk representative unlabeled tokens per image, and global herding compresses their union into the dataset-level candidate set Ct\mathcal{C}_{t} . Stage 2 refines these token candidates on the full 64×6464\times 64 Mini-DPT decoder grid; in the figure, this is visualized locally by subdividing each selected candidate token into a 4×44\times 4 block of decoder cells. Margin uncertainty is evaluated only on valid decoder cells that overlap Ct\mathcal{C}_{t} and have unlabeled center pixels. The center pixels of the most uncertain valid cells are queried and added to Lt+1\mathcal{L}_{t+1} for the next round.
DatasetMethodBudget – average pixels per imageCeiling
1102562256^{2}
CamVidPixelPick30.56 ±\pm 1.5837.51 ±\pm 0.6642.85
2-Stage eDALD34.15 ±\pm 0.7240.20 ±\pm 0.6558.27
Teal49.54 ±\pm 0.37 (+15.39)55.93 ±\pm 0.11 (+15.73)62.02
CityscapesPixelPick25.96 ±\pm 0.1631.03 ±\pm 0.3332.38
2-Stage eDALD30.40 ±\pm 0.4134.60 ±\pm 0.4743.89
Table 1: Performance across datasets and labeling budgets . Green values indicate absolute improvement over the strongest baseline. The Ceiling column reports each method’s fully supervised mIoU when trained with dense labels using its own backbone. PixelPick uses a CNN-based DeepLabV3 backbone ( Chen et al., 2017 ) , 2-Stage eDALD uses DDPM ( Ho et al., 2020 ) features, and TEAL uses DINOv3-base ( Siméoni et al., 2025 ) features with our token/grid-based segmentation head. Results are reported as mean ±\pm standard deviation.
DatasetCeilingMethodBudget – average pixels per image% of Sup.
2562256^{2}110@10 px
CamVid62.02PixelPick47.07 ±\pm 0.2853.79 ±\pm 0.2786.7
2-Stage eDALD45.88 ±\pm 1.6853.98 ±\pm 0.5787.0
Teal49.54 ±\pm 0.37 (+3.66)55.93 ±\pm 0.11 (+1.95)90.2
Cityscapes49.91PixelPick40.21 ±\pm 0.2743.28 ±\pm 0.1786.7
2-Stage eDALD40.55 ±\pm 0.4141.68 ±\pm 0.2083.5
Table 2: Matched-backbone comparison across datasets and labeling budgets . Green values indicate absolute improvement over the previous two-stage method ( Kim et al., 2026 ) . Ceiling reports the fully supervised mIoU obtained using dense labels and the DINOv3 backbone, shared by all methods. The final column reports the fraction of the fully supervised performance recovered at the 10-pixel budget. All methods use DINOv3 features for a matched-backbone comparison.
Figure 3: Decomposing the gap from matched-backbone eDALD to Teal on CamVid. Solid segments show the change from the previous configuration ( →\rightarrow replaces, ++ adds; red marks a loss). Pixel herding helps at one pixel per image but hurts at ten, while token-to-cell refinement and token-space herding help at both.
Figure 4: Results at 512×512512\times 512 resolution. Matched-backbone comparison of eDALD DINO, pixel herding (eDALD DINO with margin sampling), and Teal at 1 and 10 pixels per image. Shaded bands show the standard deviation over three seeds.
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Local budget kkmIoU
546.15 ±1.53\pm 1.53
1049.02 ±2.10\pm 2.10
20 (default)49.54 ±0.37\pm 0.37
4049.43 ±0.43\pm 0.43
8048.40 ±0.75\pm 0.75
Appendix
Table 3: Sensitivity to the local budget kk .
Global retention ratiomIoU
0.2549.29 ±0.78\pm 0.78
0.50 (default)49.54 ±0.37\pm 0.37
0.7550.43 ±0.11\pm 0.11
1.0048.80 ±1.21\pm 1.21
Appendix
Table 4: Sensitivity to the global retention ratio.
Scale factor ssmIoU
1.048.48 ±0.78\pm 0.78
2.0 (default)49.54 ±0.37\pm 0.37
3.049.45 ±0.95\pm 0.95
Appendix
Table 5: Sensitivity to the RBF bandwidth scale factor ss , where σ=s⋅\sigma=s\cdot median pairwise token distance.
Herding layersmIoU
Shallow [0,1,2,3][0,1,2,3]30.30 ±3.85\pm 3.85
Balanced - Mini-DPT default [2,5,8,11][2,5,8,11] [ Yang et al., 2025 ]48.80 ±0.58\pm 0.58
Late [5,7,9,11][5,7,9,11]50.21 ±1.29\pm 1.29
Deep [8,9,10,11][8,9,10,11] (ours)49.54 ±0.37\pm 0.37
Appendix
Table 6: Sensitivity to the DINOv3 layers used for herding.
DatasetMethodBudget – average pixels per image
110
CamVidPixel herding47.34 ±\pm 0.2953.38 ±\pm 0.50
Teal49.54 ±\pm 0.37 (+2.20)55.93 ±\pm 0.11 (+2.55)
CityscapesPixel herding40.61 ±\pm 0.3441.80 ±\pm 0.31
Teal42.25 ±\pm 0.37 (+1.64)43.79 ±\pm 0.26 (+1.99)
ADE20KPixel herding32.67 ±\pm 0.3938.31 ±\pm 0.54
Appendix
Table 7: Pixel herding vs. Teal under matched backbones. mIoU at 1 and 10 pixels per image, mean ±\pm std over three seeds. Green values indicate absolute improvement over pixel herding.
DatasetMethodBudget – average pixels per image
110
CamVideDALD DINO49.86 ±\pm 1.1557.82 ±\pm 1.28
Pixel herding51.37 ±\pm 1.7058.07 ±\pm 1.02
Teal55.49 ±\pm 1.4061.38 ±\pm 0.96
CityscapeseDALD DINO53.53 ±\pm 0.8956.76 ±\pm 0.10
Pixel herding53.07 ±\pm 0.4454.93 ±\pm 1.07
Appendix
Table 8: Results at 512×512512\times 512 resolution. Matched-backbone comparison at 1 and 10 pixels per image, mean ±\pm std over three seeds. Pixel herding is eDALD DINO with margin sampling in place of eBALD.
MethodOn token edge (0 px)Within 1 pxWithin 2 px
Pixel herding60.1%87.8%97.0%
Teal0.0%40.7%69.6%
Uniform random23.4%43.8%60.9%
Appendix
Table 9: Fraction of selected pixels lying on or near token boundaries.
Label set (matched backbone)Rare-class shareMulti-label tokensFinal mIoU
Teal , 1 px14.5%2.0%49.54 ±\pm 0.37
eDALD DINO, 1 px10.1%0.0%45.88 ±\pm 1.68
Teal , 10 px12.7%29.8%55.93 ±\pm 0.11
eDALD DINO, 10 px7.5%1.8%53.98 ±\pm 0.57
Appendix
Table 10: Query allocation under matched backbones on CamVid. Rare-class share is the fraction of queries placed on rare classes; multi-label tokens is the fraction of labeled tokens that received more than one label.
Figure 5: Per-class IoU and selection mass at 10 pixels per image. Comparison of token-level and pixel-level herding using DINOv3 features on CamVid. The top panel reports the per-class IoU difference between token and pixel herding (positive values favor token herding), and the bottom panel shows the fraction of selected queries assigned to each class. Token herding improves IoU for most classes, with the largest gains on lower-frequency, spatially localized categories such as sign symbol, fence, pedestrian, and bicyclist. Pixel herding is slightly better only on large surface classes such as road and pavement.
 Per-image latency (ms)Wall-Clock Runtime
MethodStage 1Train stepStage 210-round AL run
eDALD-DDPM184.3184.3112.7112.7555.7555.71h 33min1\text{h}\,33\text{min}
TEAL (ours)7.2\mathbf{7.2}6.5\mathbf{6.5}5.6\mathbf{5.6}0h 23min\mathbf{0\text{h}\,23\text{min}}
Appendix
Table 11: Compute cost of CamVid run across different methods. Per-image latency for Stage 1 herding, head training step, Stage 2 uncertainty, and the total wall-clock for a full 10-round run.
ComponentOriginal Mini-DPTMini-DPT (ours)
Branch resamplingBilinear resize, then learned 3×33\times 3 convTransposed conv (kernel == stride) for ×4\times 4 , ×2\times 2 ; identity for ×1\times 1 ; strided 3×33\times 3 conv for ×12\times\tfrac{1}{2}
Resampling parameters589,824475,136
Pyramid scales[8,4,2,1][8,4,2,1][4,2,1,12][4,2,1,\tfrac{1}{2}]
Decoder grid128×128128\times 128 (2-px cells)64×6464\times 64 (4-px cells)
Appendix
Table 12: Architectural differences between the original Mini-DPT decoder and our Mini-DPT head. Grid sizes are for a 256×256256\times 256 input.
Figure 6: Qualitative segmentation maps at the final active-learning round. We compare Teal against eDALD with DDPM features and a matched-backbone eDALD using DINOv3 features. Teal produces cleaner object regions and fewer boundary artifacts, consistent with the quantitative gains in the low-budget setting.
More in this topic

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 280 papers

Explore similar work

CardsList

  1. SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentation

    Apr 2, 2026Naomi Kombol, Ivan Martinović, Siniša Šegvić +1Vision TransformerEfficient ViTs

    Faculty of Electrical Engineering and Computing, University of Zagreb · VRG, Faculty of Electrical Engineering, Czech Technical University in Prague

  2. ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction

    Mar 27, 2026David Hagerman, Roman Naeem, Erik Brorsson +2Efficient ViTsVision Transformer

    Chalmers University of TechnologyChalmers University of technology, 412 96 Gothenburg, Sweden

  3. When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compression

    Jul 2, 2026Tien-Phat Nguyen, Ngai-Man CheungDistribution Shift RobustnessVision Transformer

    Temasek Laboratories, Singapore University of Technology and Design, Singapore