CanViT: Toward Active-Vision Foundation Models

Yohaï-Eliel Berreby; Sabrina Du; Audrey Durand; B. Suresh Krishna

cs.CVMar 23, 2026

CanViT: Toward Active-Vision Foundation Models

Authors: Yohaï-Eliel Berreby, Sabrina Du, Audrey Durand, B. Suresh Krishna

Organizations: McGill University · Mila - Quebec AI Institute · Université Laval

Abstract

Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses, but lacks scalable general-purpose architectures and pretraining pipelines, leaving Active-Vision Foundation Models (AVFMs) underexplored. We introduce CanViT, the first task- and policy-agnostic AVFM. CanViT uses scene-relative RoPE to bind a retinotopic Vision Transformer backbone and a spatiotopic scene-wide latent workspace, the canvas. Efficient interaction with this high-capacity working memory is supported by Canvas Attention, a novel asymmetric cross-attention mechanism. We decouple thinking (backbone-level) and memory (canvas-level), eliminating canvas-side self-attention and fully-connected layers to achieve fast sequential inference and scalability to high output resolutions. We propose a label-free active vision pretraining scheme, policy-agnostic passive-to-active dense latent distillation: reconstructing scene-wide DINOv3 embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths. We pretrain CanViT-B from a random initialization on 13.2 million ImageNet-21k scenes--an order of magnitude more than previous active models--and 1 billion random glimpses, in 166 hours on a single H100. On ADE20K segmentation, a frozen CanViT-B achieves 38.5% mIoU in a single low-resolution glimpse, outperforming the best active model's 27.6% with 20x fewer inference FLOPs as well as its FLOP- or input-matched DINOv3 teacher. Given additional glimpses, CanViT-B reaches 45.9% ADE20K mIoU. On ImageNet-1k classification, CanViT-B also sets a new active-vision state of the art, with 84.5% top-1 accuracy after fine-tuning. CanViT generalizes to longer rollouts, larger scenes, and new policies. Our work narrows the wide gap between passive and active computer vision, demonstrating the potential of task- and policy-agnostic AVFM pretraining.

Figures & tables

Figure 1: A CanViT rollout. We consider a high-resolution scene ( A ). At each timestep $t$ , CanViT ingests a $128^{2}\,\text{px}$ glimpse ( B , 1st row), a crop extracted at a viewpoint with center $(x_{t},y_{t})\in[-1,+1]^{2}$ and scale (zoom level) $s_{t}\in(0,1]$ . This updates a scene-wide latent representation, the canvas , with which CanViT integrates broad context and fine detail from variable-scale glimpses, extrapolates to unobserved regions, and conditions visual processing on a working understanding of the scene. We visualize the canvas via Principal Component Analysis across tokens ( B , 2nd row), and canvas updates $(\Delta)$ as cosine dissimilarity heatmaps across consecutive timesteps ( B , 3rd row).

Figure 2: CanViT architecture diagram. We adopt a dual-stream structure, equipping a ViT backbone ( purple , left-hand columns), which processes localized glimpses ( blue ), with a canvas ( red , right-hand columns), a fine-grained scene-wide spatio-semantic memory. At each timestep $t$ , a glimpse is extracted from a viewpoint $\mathbf{v}_{t}=(x_{t},y_{t},s_{t})$ , patchified, and processed through the backbone, alongside a recurrent CLS token ( green ) and a Viewpoint Encoding (VPE) token ( orange ). The canvas regularly interacts with the glimpse stream via Canvas Attention (Figure 3 ), alternating between read (R) and write (W) operations to, respectively, condition the backbone’s processing on the canvas and populate the canvas. Both streams are equipped with register tokens ( gray ) 12 .

Figure 3: A. A Canvas Attention Read-Write pair. B. Total inference FLOPs vs output patch grid. C. Relative cost of canvas-side QKVO projections, per Canvas Attention Read/Write pair.

Figure 4: Benchmark results on ADE20K and ImageNet-1k . A. Accuracy–efficiency frontier on ADE20K segmentation. B. ADE20K segmentation mIoU by viewing policy, at

32^{2}

64^{2}

canvas resolution. C. ImageNet-1k classification accuracy by viewing policy, with or without fine-tuning.

Model	ADE20K mIoU	IN1k top-1
Saccader 15	—	75.0
AME 42 (MAE)	24.4	—
AME 42 (SETR)	27.6	—
AdaGlimpse 43	25.7	77.5
AdaptiveNN 62	—	82.2
CanViT-B	45.9 †	81.1 † / 84.5

Table 1: Comparison with prior work.

\dagger

: Frozen weights with linear probing.

Figure 5: Effects of canvas resolution and ground-truth mask area. We evaluate on ADE20K segmentation with canvas resolution in

\{8^{2},16^{2},32^{2},64^{2}\}

, using the EG-C2F policy, frozen weights, and per-resolution linear probes. A. Per-mask IoU given a single full-scene

128^{2}

px input (

8^{2}

input patch grid). B. Per-mask IoU gain given additional glimpses. C. Accuracy–efficiency Pareto frontier, varying FLOPs via timestep count for CanViT and via input resolution (px) for DINOv3.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Backbone	ViT-B/16
Backbone embedding dim	768
Backbone registers (ephemeral)	5
Glimpse patch size	$16^{2}$ px
Canvas embedding dim	1024
Canvas registers (persistent)	16

Appendix

Table 2: CanViT-B architecture.

Hyperparameter	Value
Optimizer	AdamW
Initial learning rate	$1.00\times 10^{-7}$
Peak learning rate	$4.00\times 10^{-4}$
LR schedule	Warmup $\to$ Const.
LR warmup steps	100,000
Weight decay	$1\times 10^{-4}$

Appendix

Table 3: CanViT-B pretraining hyperparameters.

Policy	Canvas	$t{=}0$	$t{=}1$	$t{=}2$	$t{=}3$	$t{=}4$	$t{=}9$	$t{=}16$	$t{=}20$
EG-C2F †	$32^{2}$	38.5	41.1	42.0	42.7	43.2	43.9	44.2	44.1
EG-C2F †	$64^{2}$	39.6	42.2	43.3	44.1	44.7	45.6	45.8	45.7
C2F	$32^{2}$	38.5 $\pm 0.00$	40.3 $\pm 0.09$	41.4 $\pm 0.09$	42.3 $\pm 0.05$	43.2 $\pm 0.03$	43.6 $\pm 0.03$	44.0 $\pm 0.05$	44.2 $\pm 0.02$
C2F	$64^{2}$	39.6 $\pm 0.00$	41.3 $\pm 0.10$	42.5 $\pm 0.08$	43.6 $\pm 0.06$	44.7 $\pm 0.03$	45.1 $\pm 0.05$	45.6 $\pm 0.03$	45.9 $\pm 0.02$
F-IID	$32^{2}$	38.5 $\pm 0.00$	40.0 $\pm 0.10$	40.8 $\pm 0.08$	41.3 $\pm 0.09$	41.7 $\pm 0.08$	42.7 $\pm 0.08$	43.2 $\pm 0.07$	43.3 $\pm 0.06$
F-IID	$64^{2}$	39.6 $\pm 0.00$	41.2 $\pm 0.10$	42.0 $\pm 0.09$	42.5 $\pm 0.10$	43.0 $\pm 0.09$	44.1 $\pm 0.06$	44.7 $\pm 0.08$	44.9 $\pm 0.07$

Appendix

Table 4: ADE20K mIoU (%) by policy and timestep (

T=21

), frozen CanViT-B. Cells:

\pm

95% bootstrap CI half-width (

n=10

runs) for stochastic policies. † : deterministic policy. Bold = best across all policies at that timestep.

Policy	$c=8$	$c=16$	$c=32$	$c=64$
EG-C2F †	31.6	39.3	44.2	45.8
C2F	31.6 $\pm 0.03$	39.1 $\pm 0.04$	44.2 $\pm 0.02$	45.9 $\pm 0.02$
F-IID	30.7 $\pm 0.04$	38.1 $\pm 0.09$	43.3 $\pm 0.06$	44.9 $\pm 0.07$
R-IID	29.5 $\pm 0.10$	37.2 $\pm 0.07$	41.6 $\pm 0.09$	43.1 $\pm 0.13$
RFS †	30.3	36.8	40.0	41.1
F2C	29.2 $\pm 0.14$	36.8 $\pm 0.11$	41.2 $\pm 0.10$	42.9 $\pm 0.15$

Appendix

Table 5: ADE20K mIoU (%) across canvas grid sizes, frozen CanViT-B. One independently-trained linear probe per canvas grid

c

. Cells: best-

t

mIoU mean, with

\pm

95% bootstrap CI half-width (

n=10

runs) for stochastic policies. † : deterministic policy.

Policy	$t{=}0$	$t{=}1$	$t{=}2$	$t{=}3$	$t{=}4$	$t{=}5$	$t{=}9$	$t{=}15$	$t{=}20$
C2F	76.79 $\pm 0.00$	78.62 $\pm 0.05$	79.58 $\pm 0.02$	80.25 $\pm 0.03$	80.77 $\pm 0.02$	80.85 $\pm 0.02$	81.02 $\pm 0.03$	81.12 $\pm 0.03$	81.13 $\pm 0.02$
F-IID	76.79 $\pm 0.00$	78.50 $\pm 0.04$	79.32 $\pm 0.04$	79.81 $\pm 0.05$	80.13 $\pm 0.04$	80.37 $\pm 0.04$	80.86 $\pm 0.03$	81.09 $\pm 0.03$	81.14 $\pm 0.02$
R-IID	47.73 $\pm 0.10$	65.23 $\pm 0.08$	72.23 $\pm 0.09$	75.50 $\pm 0.06$	77.24 $\pm 0.06$	78.22 $\pm 0.06$	79.71 $\pm 0.06$	80.37 $\pm 0.04$	80.56 $\pm 0.05$
RFS †	76.79	77.85	77.99	78.01	78.01	77.99	77.73	77.40	77.16
F2C	32.22 $\pm 0.07$	49.04 $\pm 0.08$	58.57 $\pm 0.07$	64.40 $\pm 0.06$	68.23 $\pm 0.10$	70.81 $\pm 0.10$	75.93 $\pm 0.08$	78.52 $\pm 0.06$	79.78 $\pm 0.05$

Appendix

Table 6: ImageNet-1k top-1 accuracy (%) across timesteps, frozen CanViT-B (

32\times 32

canvas). Cells:

\pm

95% bootstrap CI half-width (

n=10

runs) for stochastic policies. † : deterministic policy. Bold = best policy at each timestep.

Policy	$c=8$	$c=16$	$c=32$	$c=64$
C2F	80.61 $\pm 0.02$	81.08 $\pm 0.02$	81.14 $\pm 0.03$	81.16 $\pm 0.01$
F-IID	80.74 $\pm 0.04$	81.06 $\pm 0.03$	81.14 $\pm 0.02$	81.14 $\pm 0.03$
R-IID	80.25 $\pm 0.03$	80.52 $\pm 0.05$	80.56 $\pm 0.05$	80.58 $\pm 0.04$
RFS †	77.78	77.99	78.01	78.02
F2C	79.37 $\pm 0.04$	79.69 $\pm 0.02$	79.78 $\pm 0.05$	79.80 $\pm 0.03$

Appendix

Table 7: ImageNet-1k top-1 accuracy (%) across canvas grid sizes, frozen CanViT-B. Same IN1k probe across columns, evaluated at canvas grid

c\times c

. Cells: best-

t

top-1 mean, with

\pm

95% bootstrap CI half-width (

n=10

runs) for stochastic policies. † : deterministic policy.

Policy	$t{=}0$	$t{=}1$	$t{=}2$	$t{=}3$	$t{=}4$	$t{=}5$	$t{=}9$	$t{=}15$	$t{=}20$
C2F	80.59 $\pm 0.00$	82.25 $\pm 0.04$	83.21 $\pm 0.03$	83.90 $\pm 0.03$	84.35 $\pm 0.03$	84.43 $\pm 0.03$	84.51 $\pm 0.01$	84.53 $\pm 0.01$	84.51 $\pm 0.03$
F-IID	80.59 $\pm 0.00$	82.24 $\pm 0.03$	82.95 $\pm 0.04$	83.40 $\pm 0.03$	83.70 $\pm 0.03$	83.89 $\pm 0.04$	84.25 $\pm 0.04$	84.42 $\pm 0.03$	84.43 $\pm 0.04$
R-IID	50.99 $\pm 0.11$	68.90 $\pm 0.07$	75.90 $\pm 0.09$	79.07 $\pm 0.08$	80.66 $\pm 0.06$	81.56 $\pm 0.04$	82.97 $\pm 0.05$	83.50 $\pm 0.05$	83.67 $\pm 0.04$
RFS †	80.59	81.59	81.81	81.76	81.62	81.53	81.21	80.69	80.43
F2C	35.77 $\pm 0.10$	53.47 $\pm 0.07$	63.01 $\pm 0.07$	68.78 $\pm 0.08$	72.47 $\pm 0.09$	74.91 $\pm 0.09$	79.38 $\pm 0.07$	81.40 $\pm 0.05$	82.38 $\pm 0.04$

Appendix

Table 8: ImageNet-1k top-1 accuracy (%) across timesteps, fine-tuned CanViT-B (

32\times 32

canvas). Cells:

\pm

95% bootstrap CI half-width (

n=10

runs) for stochastic policies. † : deterministic policy. Bold = best policy at each timestep.

Model	Input	Scene grid	GFLOPs	mIoU (%)
DINOv3 ViT-S/16	128 px	$8\times 8$	3.1	25.2
DINOv3 ViT-S/16	144 px	$9\times 9$	3.8	27.1
DINOv3 ViT-S/16	160 px	$10\times 10$	4.7	29.4
DINOv3 ViT-S/16	192 px	$12\times 12$	6.8	32.3
DINOv3 ViT-S/16	256 px	$16\times 16$	12.5	36.9
DINOv3 ViT-S/16	384 px	$24\times 24$	31.2	40.9

Appendix

Table 9: Passive-vision comparison: ADE20K mIoU at

t=0

(single full-scene glimpse). CanViT-B vs. DINOv3 ViT-B/16 and ViT-S/16 at various input and output resolutions. All models use frozen features with a linear segmentation probe.

Figure 6: Impact of canvas grid size on CanViT-B ADE20K mIoU (C2F policy,

T=21

). Mean mIoU across

n=10

evaluation runs, at each timestep, at canvas grids

c\times c

(

c\in\{8,16,32,64\}

Model	IN-ReAL top-1 (official / ours)	IN1k top-1 (ours)
ViT-S/16	87.0% / 87.08%	81.40%
ViT-S+/16	88.0% / 88.08%	82.89%
ViT-B/16	89.3% / 89.54%	85.00%
ViT-L/16	90.2% / 90.42%	87.44%
ViT-H+/16	90.3% / 90.31%	87.65%

Appendix

Table 10: DINOv3 IN1k linear probe accuracy at

512\times 512

input resolution.

Model	Loss	Batch size	Peak LR	Weight decay	$(\beta_{1},\beta_{2})$	DINOv3 init	Epochs (aug., total)
ViT-S/16	softmax	2048	$6.6\times 10^{-3}$	$6.0\times 10^{-2}$	$(0.22,0.71)$	Yes	10, 20
ViT-S+/16	softmax	1024	$1.1\times 10^{-3}$	$3.8\times 10^{-2}$	$(0.10,0.94)$	No	10, 20
ViT-B/16	softmax	1024	$2.8\times 10^{-4}$	$9.2\times 10^{-2}$	$(0.56,0.73)$	Yes	10, 20
ViT-L/16	sigmoid	2048	$3.0\times 10^{-3}$	$3.1\times 10^{-2}$	$(0.40,0.63)$	Yes	10, 20
ViT-H+/16	sigmoid	1024	$8.2\times 10^{-3}$	$4.9\times 10^{-2}$	$(0.78,0.93)$	No	8, 16

Appendix

Table 11: Selected hyperparameters for each DINOv3 IN1k linear probe.

Figure 7: Ablation study: loss curves during ImageNet-21k pretraining. Faint lines: logged per-batch losses. Bold lines: exponential moving average of logged losses.

Variant	Params	GFLOPs	$\Delta$ Sp. Cos	$\Delta$ CLS Cos
CanViT-B (baseline)	95.2M	15.8	ref	ref
a $D_{\mathrm{can}}=256$ , asymmetric ( $n_{h}=2$ )	88.1M	13.2	$-$ 12.0%	$-$ 2.2%
b $D_{\mathrm{can}}=256$ , + QKVO ( $n_{h}=2$ )	88.8M	14.8	$-$ 11.7%	$-$ 1.5%
c $D_{\mathrm{can}}=384$ , + QKVO ( $n_{h}=3$ )	91.0M	17.3	$-$ 5.8%	$-$ 1.1%
d No canvas reads	90.4M	14.2	$-$ 6.6%	$-$ 8.0%
e RW stride $=6$ (1R / 1W)	88.8M	13.7	$-$ 4.0%	$-$ 6.0%

Appendix

Table 12: Ablation study: reconstruction quality on ADE20K validation set. We compare CanViT’s predicted DINOv3-ViT-B features against reference teacher features after 10 R-IID glimpses. GFLOPs: per-timestep forward pass cost.

\Delta

: % change in cosine similarity relative to baseline (mean over

n=5

evaluation runs per variant). Sp. = spatial (dense, patch-level); CLS = CLS-token.

Variant	$t=0$		$t=9$
Variant	Sp.	CLS	Sp.	CLS
CanViT-B (baseline)	46.9–47.3	46.3–46.9	71.3–71.4	69.6–69.6
a $D_{\mathrm{can}}=256$ , asymmetric	42.6–43.1	45.7–46.5	62.7–62.8	68.0–68.1
b $D_{\mathrm{can}}=256$ , + QKVO	42.6–43.2	45.9–46.7	63.0–63.0	68.5–68.6
c $D_{\mathrm{can}}=384$ , + QKVO	44.7–45.0	45.9–46.4	67.1–67.2	68.8–68.9
d No canvas reads	45.3–45.7	44.2–44.8	66.6–66.7	64.0–64.1

Appendix

Table 13: Ablation study: reconstruction quality on ADE20K validation set (supplementary). Cosine similarity (%) with DINOv3-B teacher features; cells are 95% bootstrap CIs across

n=5

independent R-IID runs per variant. Sp. = spatial (patch-level), CLS = CLS-token.

Figure 8: Latency scaling with output resolution. Faint scatter: individual iterations.

Device	Scene grid	CanViT-B		DINOv3 ViT-B/16		DINOv3 ViT-S/16
Device	Scene grid	ms	MB	ms	MB	ms	MB
CPU, 1T	$8\times 8$	143	—	121	—	34.8	—
CPU, 1T	$16\times 16$	154	—	407	—	115	—
CPU, 1T	$32\times 32$	224	—	1975	—	663	—
CPU, 1T	$64\times 64$	486	—	13049	—	4882	—
CPU, 16T	$8\times 8$	29.3	—	27.4	—	10.1	—

Appendix

Table 14: Latency and peak memory by hardware, precision, and output resolution.

Figure 9: Canvas updates within and across glimpses. CanViT performs multiple Canvas Attention Write operations per glimpse, each producing a residual that is summed with the canvas (Figure 2 ). To isolate the contribution of individual Write operations, we capture intermediate residuals and canvases after each Write operation within a $128^{2}$ px glimpse, and visualize these snapshots with Principal Component Analysis (PCA) across tokens. We compute PCA bases independently for all snapshots rather than using a shared basis, in order to visualize the spatial structure of each individual snapshot. We observe an emergent progression in the spatial structure of residuals as within-glimpse processing unfolds from Write 0 to Write 2, from localized and feature-level content to scene-wide and semantically grounded content. Write-0 residuals reflect the structure of the glimpse’s $8\times 8$ patch grid. Write-1 residuals begin to reflect the structure of the objects within the glimpse. Write-2 residuals extrapolate beyond the confines of the glimpse, and delineate sharper object boundaries.

Active Vision

See all 21 papers

Active Spatial Guidance: Eliminating Injected Positional Mechanisms in Vision Transformers
Jul 1, 2026Cong Liu, Xiaofang Li, Simon X. YangVision Transformer
School of Engineering, Yancheng Institute of Technology, No. 1 Xiwang Middle Avenue, Yancheng, 224051, Jiangsu, PR China · School of Computer Information Engineering, Changzhou Institute of Technology, No. 666 Liaohe Road, Xinbei District, Changzhou, 213032, Jiangsu, PR China · Advanced Robotics and Intelligent Systems Laboratory, School of Engineering, University of Guelph, 50 Stone Road East, Guelph, N1G 2W1, Ontario, Canada
Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation
Oct 14, 2026Utku Cicek, Mahip Singh, Ziyao Shang +5Active Learning
University of Waterloo · Apple
SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentation
Apr 2, 2026Naomi Kombol, Ivan Martinović, Siniša Šegvić +1Vision Transformer Efficient ViTs
Faculty of Electrical Engineering and Computing, University of Zagreb · VRG, Faculty of Electrical Engineering, Czech Technical University in Prague

Jul 1, 2026J/K move · Enter open · S save

Active Spatial Guidance: Eliminating Injected Positional Mechanisms in Vision Transformers

Cong Liu, Xiaofang Li, Simon X. Yang

School of Engineering, Yancheng Institute of Technology, No. 1 Xiwang Middle Avenue, Yancheng, 224051, Jiangsu, PR China · School of Computer Information Engineering, Changzhou Institute of Technology, No. 666 Liaohe Road, Xinbei District, Changzhou, 213032, Jiangsu, PR China · Advanced Robotics and Intelligent Systems Laboratory, School of Engineering, University of Guelph, 50 Stone Road East, Guelph, N1G 2W1, Ontario, Canada

Vision Transformer

Open paper AskPDF arXiv

Vision Transformers (ViTs) commonly rely on injected positional mechanisms to address self-attention's permutation invariance. Motivated by the spatial regularities of natural images, we ask whether spatial organization can be induced from data rather than explicitly injected. Under controlled, matched from-scratch training, we propose Active Spatial Guidance (Guidance), a training-only objective that disables positional injection and applies an auxiliary 2D coordinate-regression loss to the final-layer patch tokens. The guidance head is used only during training and removed for inference; the deployed model consists of a positional-injection-free ViT encoder and the task-specific prediction module. Using DINOv3 ViT backbones, Guidance consistently improves performance on ImageNet-100 classification, ADE20K semantic segmentation, and Hypersim monocular depth estimation, outperforming strong injected baselines such as learned absolute positional embeddings and rotary positional embeddings under identical training protocols. On ImageNet-100, broader comparisons against representative injected positional designs further support Guidance's effectiveness. Guidance also improves robustness under resolution transfer, and multi-resolution training further strengthens accuracy across input sizes. Overall, our results suggest that spatial inductive bias in ViTs need not be architecturally injected, but can be shaped through training-time supervision. The code used for training and evaluation is publicly available in https://github.com/cloudlc/asg.

Abstract

Active Vision

Explore similar work

Active Spatial Guidance: Eliminating Injected Positional Mechanisms in Vision Transformers

Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation

SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentation