cs.CVSep 4, 2026
SaveLookThere! Sparse Vision by Reinforced Selection
Organizations: Columbia University · Carleton University · Vector Institute · University of British Columbia
Abstract
Vision transformers typically treat every image token as equally important, yet for most tasks in computer vision only a fraction are needed. Adaptive computation methods accelerate inference by choosing which tokens to process, but existing methods struggle at extreme sparsity and require heuristics that may not generalize like token diversity and attention scores. We address these limitations with LookThere, achieving a new pareto frontier in performance-compute trade-offs through an end-to-end reinforcement learning framework that jointly trains a shallow input selector and a deep representation extractor. The selector learns where to look and the extractor learns what to see, together saving computation by selecting only what is worth processing for a given task without relying on auxiliary signals. We show that LookThere only selects the task-specific input, excelling at sparse recognition in high-resolution settings (traffic signs, billiards), and maintaining accuracy with as little as 0.2% of the input. It generalizes across tasks and models, including global recognition (ImageNet classification), local recognition (ADE20K segmentation), zero-shot classification (by distillation), and regression (counting). Across all settings, LookThere surpasses state-of-the-art selection to provide a general and scalable framework for specialized and efficient adaptive computation.
Figures & tables
Figure 1: Sparse vision for efficient recognition . LookThere learns to select a sparse set of relevant tokens for efficient and task-specific adaptive computation. We show traffic sign recognition (left), ImageNet classification (center), and billiard ball reasoning (right) with selected patches brighter.
Figure 3: Inference computation. We plot (left) accuracy against inference FLOPs for adaptive computation methods on Billiards/Birdsnap at / / sparsity, and report (right) the cost per image. LookThere rivals the accuracy of SoTA token reduction/selection at a fraction of the cost, reaching of its DINOv2 teacher’s accuracy despite seeing a fraction of the patches.
Figure 4: LookThere learns adaptive computation end-to-end by jointly optimizing the interaction between the selector and extractor. The selector acts, predicting where to look from low-resolution input, and the extractor critiques, predicting what to see (e.g., a yellow bird) from selected high-resolution patches. The two learn a shared approximation of the input by joint training via task supervision and selection feedback .
Figure 5: Generalization to initialization. ( 5 , 5 ) LookThere outperforms across tasks and pretrained initializations, ( 5 ) closely matches LookWhere’s best performance on ImageNet classification w/ DINOv2 across sparsities, ( 5 ) and learns selection for the task while teacher selection by attention can fail to choose the relevant input.
Figure 6: Generalization to task and architecture. ( 6 ) LookThere prioritizes high-entropy regions containing fine-grained details for instance segmentation. ( 6 ) Evaluated zero-shot with SigLIP 2, LookThere is robust to initialization and task, outperforming LookWhere for sparse distillation.
Figure 7: Trading accuracy for sparsity. We plot accuracy and sparsity for LookThere and LookWhere (bot.). LookThere achieves the Pareto frontier on sparse object recognition (Traffic, Billiards), outperforms on regression (Counting), and matches LookWhere on fine-grained classification (Birds). At extreme sparsity, LookThere picks only the task-relevant billiard balls (top).
Figure 8: Ablating source of reinforce ; Cells show per-table normalized sparsity (darker is sparser). (a,b)/(c,d) sweep Billiards/Birdsnap; (a,c) performance is robust to learning rates and sparsity schedules; (b) GRPO performs best, with hierarchical policies usually sparser than flat ones; (d) larger sparsity coefficients encourage aggressive selection; gated sparsity prevents premature exploitation.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Learning curves following warmup for Billiards, Birdsnap, and Traffic Signs averaged over and resp. target sparsities of . The shaded area denotes standard error. LookThere is robust to and learns quickly, achieving accuracy at sparsity within a few steps of RL finetuning. Accuracy tends to benefit from progressively finer selection grids.
Figure 10: Sparsity-accuracy tradeoff for Counting. LookThere outperforms LookWhere across sparsity levels, achieving lower MAE at equal sparsity, and nearly matching CACViT’s performance with all patches while only using % of them.
Figure 11: Sparsity-accuracy tradeoff on Billiards. LookThere matches LookWhere on all tasks, up to a sparsity of , and outperforms at higher sparsities. LookThere learns to select only task-relevant balls, achieving % accuracy at sparsity, while LookWhere degrades to %.
| Method | Top- Acc. ( ) | Mem (GB) | FLOPs (G) | Speed (im/s) | ||||||
| Sparsity | ||||||||||
| DINOv2 [ 26 ] (dense) | ||||||||||
| PiToMe [ 16 ] | – | – | – | |||||||
| ATC [ 17 ] | – | – | – | |||||||
| DTEM [ 18 ] | ||||||||||
| Token Cropr [ 34 ] | ||||||||||
Table 1: Birdsnap results at varying sparsity. We measure compute on px images ( px for IPS/DPS to accommodate their larger patches). LookThere offers finer control over the accuracy-efficiency tradeoff, exceeding accuracy at higher efficiency. Token Cropr and DTEM achieve higher accuracy at the expense of efficiency, while DPS achieves higher efficiency at the expense of accuracy.
| Method | Top- Acc. ( ) | Mem (GB) | FLOPs (G) | Speed (im/s) | ||||||
| Sparsity | ||||||||||
| DINOv2 [ 26 ] (dense) | ||||||||||
| PiToMe [ 16 ] | – | – | – | |||||||
| ATC [ 17 ] | – | – | – | |||||||
| DTEM [ 18 ] | ||||||||||
| Token Cropr [ 34 ] | ||||||||||
Table 2: Billiards at varying sparsity. We measure compute on px images ( px for IPS/DPS). LookThere sits on the Pareto frontier, retaining the highest accuracy at every sparsity level, degrading by only points from to sparsity, while requiring the fewest FLOPs and near-minimal memory. DPS is faster and marginally leaner in memory, but loses points of accuracy at sparsity.
Table 3: Prefix and Hyperparameter Ablations on Birdsnap. Left: Prefix fusion strategies for passing global context from selector to extractor. Learned residual mixing performs best on both accuracy/sparsity. Right: Policy temperature and entropy schedules. A fixed temperature with annealed low entropy performs best. Cells in the right table are shaded by sparsity (darker is sparser).
Figure 12: Hierarchical and Flat Selection. We visualize the learned policies for LookThere targeting sparsity on TT100K and Billiards, using both hierarchical and non-hierarchical (flat) policies. The former introduces spatial correlation, encouraging patches to be dropped in large swathes to yield more semantic and sparse selection.
Figure 13: DINOv2 initialization. We visualize the high-res input, selector maps, and selected patches following Birdsnap training of LookThere and LookWhere with DINOv2 initialization. DINOv2’s attention generalizes, yet LookThere is more focused to the classification task.
Figure 14: EVA initialization. We visualize the high-res input, selector maps, and selected patches following Birdsnap training of LookThere and LookWhere with EVA initialization. EVA is task-specialized; learning strong representations, and achieving competitive performance, by supervision for ImageNet classification. It does so without salient attention. LookWhere struggles without poor selection targets; yet LookThere still learns selection by interaction with the EVA critic/extractor.
Figure 15: SigLIP initialization. We visualize the high-res input, selector maps, and selected patches following Birdsnap training of LookThere and LookWhere with SigLIP initialization. SigLIP is self-supervised but does not use registers, resulting in noisy attention artifacts. LookWhere scatters selection, sometimes missing the bird; LookThere learns more focused selection by interaction.
Figure 16: MAE initialization. We visualize the high-res input, selector maps, and selected patches following Birdsnap training of LookThere and LookWhere with MAE initialization. MAE is self-supervised, and learns salient attention, though most layers pay more attention to background. LookWhere fails to learn selection but LookThere still does by leveraging reinforced interaction.
Figure 17: Adaptive Computation for Traffic Sign Recognition. We visualize both LookThere and LookWhere selector policies of where to compute and the extractor’s sparse input for what to see; each row shows different scenarios, specifically for recognizing traffic signs on TT100K [ 32 ] . While LookWhere’s selection generalizes, LookThere learns selection for the task for even sparser masks.
Figure 18: Selecting the Necessary Billiards for Reasoning. We visualize LookThere’s policies for three tasks on Billiards [ 24 ] . Each column shows a different input. Rows show the original image, alongside LookThere and LookWhere selection. LookWhere generally pays attention to all the balls whereas LookThere goes further, learning to select only the task-relevant subset of balls.
Figure 19: Adaptive Computation for Scene Recognition. We visualize both LookThere and LookWhere selector policies of where to compute and the extractor’s sparse input for what to see; each row shows different scenarios, specifically for recognizing scenes on Places365 [ 55 ] . LookThere outperforms LookWhere, the former learning to scan scenes while the latter distills from self-supervised models that generally attend to foreground objects , missing task-relevant information.
Figure 20: Adaptive Computation for Instance Segmentation. We visualize both LookThere and LookWhere selections, serving as input to the extractor for what to see; each row shows different images from MS-COCO [ 57 ] instance segmentation. LookThere learns to select the task-relevant instances for segmentation, whereas LookWhere occasionally fails to generalize to the task.
Figure 21: Adaptive Computation for ImageNet Classification. We visualize the LookThere and LookWhere selector policies for where to compute, alongside the extractor’s sparse input for what to see; each row shows a different ImageNet object-recognition scenario [ 28 ] . LookWhere attends to objects, but often focuses on specific features or edges. By contrast, LookThere learns to more fully scan objects, acquiring the full picture needed for classification.
Figure 22: Varying Sparsification for Classification. We visualize LookThere and LookWhere selection under different sparsity conditions. Each group of two rows depicts selection on a different input, with sparsity decreasing to the right; null conditioning shows LookThere’s adaptive sparsity selection, while LookWhere plots a matching . For LookWhere, varying sparsity amounts to changing in top - , though the relative importance of patches is heuristic. LookThere instead learns relative importance through interaction and relaxed optimization.