Vision transformers typically treat every image token as equally important, yet for most tasks in computer vision only a fraction are needed. Adaptive computation methods accelerate inference by choosing which tokens to process, but existing methods struggle at extreme sparsity and require heuristics that may not generalize like token diversity and attention scores. We address these limitations with LookThere, achieving a new pareto frontier in performance-compute trade-offs through an end-to-end reinforcement learning framework that jointly trains a shallow input selector and a deep representation extractor. The selector learns where to look and the extractor learns what to see, together saving computation by selecting only what is worth processing for a given task without relying on auxiliary signals. We show that LookThere only selects the task-specific input, excelling at sparse recognition in high-resolution settings (traffic signs, billiards), and maintaining accuracy with as little as 0.2% of the input. It generalizes across tasks and models, including global recognition (ImageNet classification), local recognition (ADE20K segmentation), zero-shot classification (by distillation), and regression (counting). Across all settings, LookThere surpasses state-of-the-art selection to provide a general and scalable framework for specialized and efficient adaptive computation.
Figures & tables
Figure 1: Sparse vision for efficient recognition . LookThere learns to select a sparse set of relevant tokens for efficient and task-specific adaptive computation. We show traffic sign recognition (left), ImageNet classification (center), and billiard ball reasoning (right) with selected patches brighter.
Figure 3: Inference computation. We plot (left) accuracy against inference FLOPs for adaptive computation methods on Billiards/Birdsnap at 90% / 95% / 99% sparsity, and report (right) the cost per image. LookThere rivals the accuracy of SoTA token reduction/selection at a fraction of the cost, reaching 98.8%/99.8% of its DINOv2 teacher’s accuracy despite seeing a fraction of the patches.
Figure 4: LookThere learns adaptive computation end-to-end by jointly optimizing the interaction between the selector and extractor. The selector acts, predicting where to look from low-resolution input, and the extractor critiques, predicting what to see (e.g., a yellow bird) from selected high-resolution patches. The two learn a shared approximation of the input by joint training via task supervision Ltask and selection feedback Lselection .
Figure 5: Generalization to initialization. ( 5 , 5 ) LookThere outperforms across tasks and pretrained initializations, ( 5 ) closely matches LookWhere’s best performance on ImageNet classification w/ DINOv2 across sparsities, ( 5 ) and learns selection for the task while teacher selection by attention can fail to choose the relevant input.
Figure 6: Generalization to task and architecture. ( 6 ) LookThere prioritizes high-entropy regions containing fine-grained details for instance segmentation. ( 6 ) Evaluated zero-shot with SigLIP 2, LookThere is robust to initialization and task, outperforming LookWhere for sparse distillation.
Figure 7: Trading accuracy for sparsity. We plot accuracy and sparsity for LookThere and LookWhere (bot.). LookThere achieves the Pareto frontier on sparse object recognition (Traffic, Billiards), outperforms on regression (Counting), and matches LookWhere on fine-grained classification (Birds). At extreme sparsity, LookThere picks only the task-relevant billiard balls (top).
Figure 8: Ablating source of reinforce ; Cells show per-table normalized sparsity (darker is sparser). (a,b)/(c,d) sweep Billiards/Birdsnap; (a,c) performance is robust to learning rates and sparsity schedules; (b) GRPO performs best, with hierarchical policies usually sparser than flat ones; (d) larger sparsity coefficients encourage aggressive selection; gated sparsity prevents premature exploitation.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Learning curves following warmup for Billiards, Birdsnap, and Traffic Signs averaged over λ∈{0.5,2,5} and resp. target sparsities of 90%/95%/95% . The shaded area denotes standard error. LookThere is robust to λ and learns quickly, achieving >90% accuracy at >90% sparsity within a few steps of RL finetuning. Accuracy tends to benefit from progressively finer selection grids.
Figure 10: Sparsity-accuracy tradeoff for Counting. LookThere outperforms LookWhere across sparsity levels, achieving lower MAE at equal sparsity, and nearly matching CACViT’s performance with all patches while only using 20 % of them.
Figure 11: Sparsity-accuracy tradeoff on Billiards. LookThere matches LookWhere on all tasks, up to a sparsity of 95% , and outperforms at higher sparsities. LookThere learns to select only task-relevant balls, achieving 90 % accuracy at 99.7% sparsity, while LookWhere degrades to 30 %.
Method
Top- 1 Acc. ( % ) ↑
Mem (GB) ↓
FLOPs (G) ↓
Speed (im/s) ↑
Sparsity
90%
95%
97%
90%
95%
97%
90%
95%
97%
DINOv2 [ 26 ] (dense)
90.0
0.5
120
572.2
PiToMe [ 16 ]
–
–
–
0.5
54
45
39
433.2
449.0
588.2
ATC [ 17 ]
–
–
–
1.2
70
62
58
39.7
40.2
45.4
DTEM [ 18 ]
87.2
88.2
83.7
0.7
78
72
67
212.4
424.1
430.6
Token Cropr [ 34 ]
88.1
86.8
83.6
0.4
68
66
65
646.0
676.9
734.2
Appendix
Table 1: Birdsnap results at varying sparsity. We measure compute on 518×518 px images ( 588×588 px for IPS/DPS to accommodate their larger patches). LookThere offers finer control over the accuracy-efficiency tradeoff, exceeding accuracy at higher efficiency. Token Cropr and DTEM achieve higher accuracy at the expense of efficiency, while DPS achieves higher efficiency at the expense of accuracy.
Method
Top- 1 Acc. ( % ) ↑
Mem (GB) ↓
FLOPs (G) ↓
Speed (im/s) ↑
Sparsity
90%
95%
99%
90%
95%
99%
90%
95%
99%
DINOv2 [ 26 ] (dense)
98.2
2.9
939
22.9
PiToMe [ 16 ]
–
–
–
2.8
297
248
172
26.4
31.0
40.6
ATC [ 17 ]
–
–
–
2.8
409
373
330
1.5
1.6
1.7
DTEM [ 18 ]
94.7
95.2
64.0
4.1
449
417
335
16.1
16.6
21.2
Token Cropr [ 34 ]
91.0
85.9
87.6
0.6
261
251
243
150.1
158.8
163.5
Appendix
Table 2: Billiards at varying sparsity. We measure compute on 1008×1008 px images ( 1028×1028 px for IPS/DPS). LookThere sits on the Pareto frontier, retaining the highest accuracy at every sparsity level, degrading by only 2.5 points from 90% to 99% sparsity, while requiring the fewest FLOPs and near-minimal memory. DPS is faster and marginally leaner in memory, but loses 30.7 points of accuracy at 99% sparsity.
Table 3: Prefix and Hyperparameter Ablations on Birdsnap. Left: Prefix fusion strategies for passing global context from selector to extractor. Learned residual mixing performs best on both accuracy/sparsity. Right: Policy temperature and entropy schedules. A fixed temperature with annealed low entropy performs best. Cells in the right table are shaded by sparsity (darker is sparser).
Figure 12: Hierarchical and Flat Selection. We visualize the learned policies for LookThere targeting 95% sparsity on TT100K and Billiards, using both hierarchical and non-hierarchical (flat) policies. The former introduces spatial correlation, encouraging patches to be dropped in large swathes to yield more semantic and sparse selection.
Figure 13: DINOv2 initialization. We visualize the high-res input, selector maps, and selected patches following Birdsnap training of LookThere and LookWhere with DINOv2 initialization. DINOv2’s attention generalizes, yet LookThere is more focused to the classification task.
Figure 14: EVA initialization. We visualize the high-res input, selector maps, and selected patches following Birdsnap training of LookThere and LookWhere with EVA initialization. EVA is task-specialized; learning strong representations, and achieving competitive performance, by supervision for ImageNet classification. It does so without salient attention. LookWhere struggles without poor selection targets; yet LookThere still learns selection by interaction with the EVA critic/extractor.
Figure 15: SigLIP initialization. We visualize the high-res input, selector maps, and selected patches following Birdsnap training of LookThere and LookWhere with SigLIP initialization. SigLIP is self-supervised but does not use registers, resulting in noisy attention artifacts. LookWhere scatters selection, sometimes missing the bird; LookThere learns more focused selection by interaction.
Figure 16: MAE initialization. We visualize the high-res input, selector maps, and selected patches following Birdsnap training of LookThere and LookWhere with MAE initialization. MAE is self-supervised, and learns salient attention, though most layers pay more attention to background. LookWhere fails to learn selection but LookThere still does by leveraging reinforced interaction.
Figure 17: Adaptive Computation for Traffic Sign Recognition. We visualize both LookThere and LookWhere selector policies of where to compute and the extractor’s sparse input for what to see; each row shows different scenarios, specifically for recognizing traffic signs on TT100K [ 32 ] . While LookWhere’s selection generalizes, LookThere learns selection for the task for even sparser masks.
Figure 18: Selecting the Necessary Billiards for Reasoning. We visualize LookThere’s policies for three tasks on Billiards [ 24 ] . Each column shows a different input. Rows show the original image, alongside LookThere and LookWhere selection. LookWhere generally pays attention to all the balls whereas LookThere goes further, learning to select only the task-relevant subset of balls.
Figure 19: Adaptive Computation for Scene Recognition. We visualize both LookThere and LookWhere selector policies of where to compute and the extractor’s sparse input for what to see; each row shows different scenarios, specifically for recognizing scenes on Places365 [ 55 ] . LookThere outperforms LookWhere, the former learning to scan scenes while the latter distills from self-supervised models that generally attend to foreground objects , missing task-relevant information.
Figure 20: Adaptive Computation for Instance Segmentation. We visualize both LookThere and LookWhere selections, serving as input to the extractor for what to see; each row shows different images from MS-COCO [ 57 ] instance segmentation. LookThere learns to select the task-relevant instances for segmentation, whereas LookWhere occasionally fails to generalize to the task.
Figure 21: Adaptive Computation for ImageNet Classification. We visualize the LookThere and LookWhere selector policies for where to compute, alongside the extractor’s sparse input for what to see; each row shows a different ImageNet object-recognition scenario [ 28 ] . LookWhere attends to objects, but often focuses on specific features or edges. By contrast, LookThere learns to more fully scan objects, acquiring the full picture needed for classification.
Figure 22: Varying Sparsification for Classification. We visualize LookThere and LookWhere selection under different sparsity conditions. Each group of two rows depicts selection on a different input, with sparsity decreasing to the right; null conditioning shows LookThere’s adaptive sparsity selection, while LookWhere plots a matching k . For LookWhere, varying sparsity amounts to changing k in top - k , though the relative importance of patches is heuristic. LookThere instead learns relative importance through interaction and relaxed optimization.
Transformers dominate video recognition. They split videos into tokens, and processing them has expensive superlinear computational cost. Yet videos are filled with redundancy, so we can question the need for this expense. We introduce LookWhen, a selector-extractor framework that factorizes video recognition into learning when, where, and what to compute. Our shallow selector gets a scaled-down video and quickly scores all tokens across space-time, while our deep extractor gets the top-K selected tokens to approximate full-video representations without actually processing all the tokens. A key challenge is defining effective supervision for selection and extraction. For selection pre-training, we introduce a score on representations that ranks tokens by uniqueness using a simple nearest-neighbor distance. For extraction pre-training, we distill both a video teacher and an image teacher, for which we normalize its frame-wise representations to learn what changes within videos. Through these strategies, our selector-extractor learns general and efficient representations for feature extraction or fine-tuning to a task. Through experiments on Kinetics-400, SSv2, Epic-Kitchens, Diving48, Jester, and Charades, we show that LookWhen achieves a better accuracy-computation trade-off than efficient models and upgraded baselines of similar size. LookWhen Pareto-dominates in accuracy-FLOPs on 9 of 12 cases (6 tasks x 2 settings) and roughly matches on 3. In accuracy-throughput, measuring time in practice, LookWhen is more efficient still at 6.7x faster than InternVideo2-B at equal accuracy.
Ali Salamatian, Anthony Fuller, Pritam Sarkar +3
University of British Columbia · Carleton University · Vector Institute
Vision Transformers achieve strong image classification accuracy but process all image regions with nearly the same computation, even when many regions are redundant or uninformative. Recent adaptive inference methods reduce this cost by selectively compressing tokens or terminating inference early, but combining these mechanisms often causes unstable intermediate representations and accuracy degradation. We introduce Fusion, a unified adaptive inference framework that coordinates token merging, early exiting, and token pruning through a simple staged design: tokens are merged first, confidence is evaluated next, and pruning is applied only to samples that continue inference. This ordering allows the three mechanisms to operate cooperatively rather than competitively. Fusion further includes lightweight routing modules that adapt compression strength to each input and support inference-time adjustment of the accuracy--latency trade-off without retraining. On ImageNet-1k with DeiT-S, Fusion matches or surpasses state-of-the-art adaptive ViT methods at comparable compute budgets while reducing calibration error by up to 4× and inference energy by 48%. Experiments across ImageNet-100, CIFAR-100, and ImageNette with multiple ViT backbones demonstrate consistent transferability without dataset-specific tuning.
Aravind Pradeep, Samira Nazari, Mahdi Taheri +1
Brandenburg University of Technology Cottbus-Senftenberg, Germany · University of Zanjan, Iran · Humboldt University, Berlin, Germany +1
Token-pruning policies are usually designed for a single recognition pipeline, but pretrained Vision Transformers are reused across tasks with different spatial demands. We ask which parts of a pruning policy transfer across image classification, semantic segmentation, and object detection. For each pipeline, controlled probes freeze the no-pruning checkpoint and apply a series of parameter-free reduction criteria at one eligible layer at a time without retraining. The probes reveal three differences: segmentation and detection rank the criteria differently, classification is especially sensitive to attention-based pruning in the earliest layers, and the dense tasks prefer opposite recovery endpoints. These findings motivate Task-Adaptive Pruning (TAP). Existing register tokens serve as task-agnostic storage for feature artifacts. TAP instead introduces one task register per task and activates only the current one. Its evolving state ranks tokens, distributes an exact removal budget over depth, and sets the recovery scale for dense features. At a final keep rate of ρ=0.5, our jointly adapted model, TAP-J, reaches 47.0 mIoU at 1.30× encoder throughput on ADE20K and 53.7 box AP at 1.32× encoder throughput on COCO while remaining competitive on ImageNet-1K.
Hongsen Cao, Mona Jaber, Shanxin Yuan +1
School of Electronic Engineering and Computer Science Queen Mary University of London London, United Kingdom