Adaptive Observation Control for Active Visual Recognition

Dikai Li; Weijia Wang; Yuan-Gen Wang

cs.CVOct 14, 2026

Adaptive Observation Control for Active Visual Recognition

Authors: Dikai Li, Weijia Wang, Yuan-Gen Wang

Organizations: School of Artificial Intelligence, Guangzhou University, Guangzhou, China

Abstract

Visual recognition models usually process a fixed view under a fixed computation budget, although images often benefit from different scales and amounts of spatial evidence. Active vision can adapt by observing a coarse scene and then selecting informative fixations, but the public AdaptiveNN-DeiT-S pipeline still fixes both field of view and observation count. We extend this fixed-step recognizer with a controllable observation interface that adapts both the field of view and the number of fixations to the input. Specifically, the model predicts a per-fixation scale that sets its field of view, refines each selected crop with optional scale-gated foveation and a residual high-resolution detail-token branch, and uses label-grounded stop heads for input-dependent early termination. Experiments across fine-grained benchmarks, full ImageNet-1K, and ImageNet-A/R-derived classification splits show that the interface remains competitive with fixed-step references while exposing a controllable observation budget. The same checkpoint enables threshold-controlled accuracy-compute trade-offs without per-threshold retraining. On sequential ImageNet-1K, the learned stop rule outperforms fixed-step truncation of the same checkpoint, sustaining 81.9% top-1 using 2.45 of the four focus steps and exceeding three-step truncation at lower cost. On each benchmark, a variant combining scale control and detail-token fusion achieves the highest accuracy among the tested component-added non-exit variants. Our code is available at https://github.com/GZHU-DVL/LI-Dikai.

Figures & tables

Figure 1: Overview of the controllable observation loop. A low-resolution glance and the previous-step state drive a policy that predicts crop location and field-of-view scale (+S). The selected crop is optionally foveated (+F), encoded by the shared focus branch, augmented by an inner high-resolution detail-token branch (+D), and routed to classifier and label-grounded stop heads (+EE). Larger $s_{t}$ gives a wider field of view. Here, $(u_{t},v_{t})$ denotes the normalized top-left crop position, $q_{t}$ is the stop logit, $\sigma$ is the sigmoid function, and $\tau$ is the exit threshold. The dashed path continues only when the stopping score $\sigma(q_{t})<\tau$ and $t<T$ , where $T$ is the final focus step. The inner detail crop is sampled directly from $I$ at the outer crop’s center, with scale $0.6s_{t}$ . Green and orange boxes illustrate the outer and inner crop regions, respectively.

Dataset	Official	Best in-codebase non-exit	Full+EE (ours)	$\Delta$ best
Aircraft	77.318/97.000 @75	77.078/96.550 @69 (+S+F+D)	77.558 /96.910 @10	+0.240
Cars	83.076/97.525 @79	82.741/97.675 @77 (+S+F+D)	83.711 /97.650 @15	+0.635
CUB	84.524/96.912 @65	84.369/96.929 @79 (Orig)	84.800 /97.084 @17	+0.276
Dogs	91.981/99.452 @30	92.401/99.464 @22 (+S+D)	92.413 /99.534 @3	+0.012
ImageNet-1K	82.126/95.946 @1	82.282 /96.014 @1 (+S+D)	82.124/95.896 @5	$-0.158$
ImageNet-A	64.815/80.688 @46	66.138/82.011 @42 (+S+D) †	66.534 /81.746 @17	+0.396

Table 1: Main validation accuracy (Acc@1/Acc@5 at the selected 1-based epoch).

\Delta

best is Full+EE minus the stronger of the Official rerun and the best in-codebase non-exit variant (selected row named in parentheses). † +S+F also achieves 66.138% Acc@1 on ImageNet-A. The highest Acc@1 in each row is shown in bold; Table 2 uses the same convention.

Figure 2: Qualitative module contrasts on held-out examples. The Cars rows compare detail fusion (+S+F to +S+F+D) and scale-gated foveation (+S+D to +S+F+D). The ImageNet-1K rows compare Orig with Full+EE as joint-model contrasts rather than controlled, single-component comparisons. Colored boxes and numbers mark focus order; the heat overlay is per-example fixation density; dashed inner boxes mark detail crops (top row) and the sharp foveal core (second row) from step 2 onward.

Dataset	Orig	+S	+S+F	+S+D	+S+F+D	Full+EE
Aircraft	76.358	76.748	76.898	76.628	77.078	77.558
Cars	81.696	82.256	81.771	82.467	82.741	83.711
CUB	84.369	84.179	84.300	84.300	84.334	84.800
Dogs	92.051	92.378	92.238	92.401	92.238	92.413
ImageNet-1K	82.116	82.234	82.146	82.282	82.246	82.124
ImageNet-A	65.873	65.741	66.138	66.138	65.873	66.534

Table 2: Acc@1 ablation across additive model variants. +S, +F, and +D denote learned scale, scale-gated foveation, and detail-token fusion, respectively; Full+EE adds early-exit fine-tuning on top of +S+F+D.

			$\tau=0.70$			$\tau=0.90$
Dataset	Protocol	Full	Acc@1	Avg. step	Ratio	Acc@1	Avg. step	Ratio
Aircraft	Sim.	77.558	77.378	3.669	0.917	77.558	4.000	1.000
Cars	Sim.	83.711	83.686	3.988	0.997	83.711	4.000	1.000
CUB	Sim.	84.800	84.800	3.710	0.927	84.800	4.000	1.000
Dogs	Sim.	92.413	92.424	3.992	0.998	92.413	4.000	1.000
ImageNet-1K	Seq. steps	82.040	81.404	1.731	0.433	81.888	2.446	0.611

Table 3: Confidence-conditioned early exit at

\tau\in\{0.70,0.90\}

. Seq. steps : realized exits (Acc@1 from canonical logits at the exit step; steps measured and encoder FLOPs estimated from those steps). Sim. : diagnostic stop rule on full four-step outputs only; these rows do not measure realized inference cost or FLOP savings. Ratio is average focus steps over four. Full is the full-unroll Acc@1 of the same checkpoint measured in the corresponding protocol environment; for ImageNet-1K this is the sequential evaluation, hence 82.040 versus the main-table 82.124.

Figure 3: Realized sequential accuracy versus average focus steps for the Full+EE checkpoint. Red markers are realized confidence-conditioned exits from one checkpoint with measured average focus steps; dashed gray curves are fixed-step truncation at integer budgets. The supplementary material reports the full mixed diagnostic, including ImageNet-R simulation-only points.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Non-exit BS	Non-exit epochs	Non-exit LR	EE BS	EE epochs	EE LR
Aircraft	108	80	5e-5	108	20	2e-5
Cars	108	80	5e-5	108	20	2e-5
CUB	108	80	5e-5	108	20	2e-5
Dogs	108	80	5e-5	108	20	2e-5
ImageNet-1K	144	50	5e-5	144	15	2e-5
ImageNet-A	108	60	5e-5	108	20	2e-5

Appendix

Table 1: Training schedules for the reported runs. BS is effective batch size. EE denotes early-exit fine-tuning from the +S+F+D checkpoint.

Dataset	Official	Orig	Orig $-$ Official
Aircraft	77.318	76.358	$-0.960$
Cars	83.076	81.696	$-1.380$
CUB	84.524	84.369	$-0.155$
Dogs	91.981	92.051	+0.070
ImageNet-1K	82.126	82.116	$-0.010$
ImageNet-A	64.815	65.873	+1.058

Appendix

Table 2: External official-codebase rerun versus the same-codebase Orig anchor used for ablations (Acc@1 at the selected epoch). The main paper reports Official as the external fixed-step reference and uses Orig only inside in-codebase ablations; this table records their gap explicitly.

Dataset	Orig	Best added non-exit variant	Full+EE
Aircraft	76.358 / 75.848	77.078 / 76.838 (+S+F+D)	77.558 / 77.258
Cars	81.696 / 81.248	82.741 / 82.442 (+S+F+D)	83.711 / 83.648
CUB	84.369 / 84.265	84.334 / 84.110 (+S+F+D)	84.800 / 84.455
Dogs	92.051 / 91.702	92.401 / 91.748 (+S+D)	92.413 / 92.016
ImageNet-1K	82.116 / 81.280	82.282 / 80.896 (+S+D)	82.124 / 81.946
ImageNet-A	65.873 / 64.550	66.138 / 65.212 (+S+D)	66.534 / 65.873

Appendix

Table 3: Best-versus-last Acc@1 sensitivity for the main table rows. Each cell is selected-best / final-epoch Acc@1. The middle column selects among component-added non-exit variants, excluding Orig. +S+F ties +S+D on ImageNet-A.

Dataset	Orig	+S+F+D	Full+EE	Full+EE $-$ Orig	Full+EE $-$ (+S+F+D)	+S+F+D $-$ Orig
Aircraft	76.398 $\pm$ 0.035	76.748 $\pm$ 0.330	77.308 $\pm$ 0.312	+0.910 $\pm$ 0.337	+0.560 $\pm$ 0.092	+0.350 $\pm$ 0.360
CUB	84.210 $\pm$ 0.348	84.302 $\pm$ 0.183	84.745 $\pm$ 0.187	+0.535 $\pm$ 0.165	+0.443 $\pm$ 0.020	+0.092 $\pm$ 0.176
Dogs	92.273 $\pm$ 0.248	92.331 $\pm$ 0.107	92.514 $\pm$ 0.155	+0.241 $\pm$ 0.108	+0.183 $\pm$ 0.059	+0.058 $\pm$ 0.141
Oxford-Pet	96.564 $\pm$ 0.098	96.447 $\pm$ 0.031	96.537 $\pm$ 0.054	$-0.027\pm 0.098$	+0.090 $\pm$ 0.062	$-0.117\pm 0.068$
Cars	81.632 $\pm$ 0.457	82.731 $\pm$ 0.063	84.074 $\pm$ 0.315	+2.442 $\pm$ 0.595	+1.343 $\pm$ 0.327	+1.098 $\pm$ 0.518

Appendix

Table 4: Three-seed paired results (seeds 0–2) for the three main arms. Cells are mean Acc@1

\pm

standard deviation; the last three columns are paired deltas. These support the seed-0 main table and the near-parity reading of Oxford-Pet.

Dataset	Orig+30	+S+F+D+30	Full+EE+30	Scale-cost+30	$\Delta$ Orig+30	$\Delta$ +S+F+D+30
Aircraft	79.718/96.910 @24	80.168/97.120 @30	80.378 /97.000 @28	79.748/97.030 @27	+0.660	+0.210
Cars	86.121/98.222 @27	86.668/98.408 @25	86.880 /98.346 @24	–	+0.759	+0.211
CUB	85.554/97.428 @30	85.623/97.204 @30	85.692 /97.273 @24	85.433/97.290 @29	+0.138	+0.069
Dogs	92.296/99.510 @4	92.226/99.510 @5	92.378 /99.499 @3	–	+0.082	+0.152
ImageNet-A	67.328 /82.407 @23	66.402/82.804 @18	66.005/82.011 @15	66.270/81.746 @29	$-1.323$	$-0.397$
Oxford-Pet	96.645 /99.919 @22	96.429/99.946 @20	96.456/99.756 @14	–	$-0.189$	+0.027

Appendix

Table 5: Budget-matched continuation diagnostic: 30-epoch single-GPU continuation with matched effective batch and learning-rate schedule (Acc@1/Acc@5 at the selected epoch).

\Delta

Orig+30 and

\Delta

+S+F+D+30 are Full+EE+30 minus the corresponding continued baseline. Scale-cost+30 is a diagnostic +S+F+D continuation with an auxiliary scale-cost penalty; it is not used by any reported main-paper run and was run only for Aircraft, CUB, and ImageNet-A. This diagnostic was run only for Aircraft, CUB, and ImageNet-A.

Dataset	Step 1	Step 2	Step 3	Step 4
Aircraft	71.347	74.377	76.508	77.558
Cars	76.200	80.291	82.753	83.711
CUB	82.678	83.834	84.438	84.800
Dogs	90.688	91.713	92.040	92.413
ImageNet-1K	79.704	81.106	81.810	82.040
ImageNet-A	56.746	61.772	63.757	66.534

Appendix

Table 6: Per-step Acc@1 for the full early-exit checkpoint without early stopping.

Dataset	Step 1	Step 2	Step 3	Step 4	Avg. std
Aircraft	1.006	1.015	1.011	1.008	0.007
Cars	1.110	1.167	1.167	1.215	0.077
CUB	1.006	1.010	1.008	1.012	0.008
Dogs	1.002	1.003	1.002	1.002	0.002
ImageNet-1K	1.078	1.162	1.175	1.106	0.055
ImageNet-A	1.004	1.003	1.004	1.003	0.003

Appendix

Table 7: Mean learned scale per focus step for the full early-exit checkpoint without early stopping. The last column is the standard deviation of the learned scale across validation images, averaged over the four steps.

Figure 1: Qualitative fixation trajectories on ImageNet validation examples at $\tau=0.70$ . Each row shows the input and four predicted focus boxes from the Full+EE checkpoint. The overlay reports learned scale $s$ and stop probability $p=\sigma(q_{t})$ ; asterisks identify the displayed stop-probability annotations.

Figure 2: Additional ImageNet-1K fixation-density contrasts, including the adaptive failure case. Fixed density accumulates four fixed-scale crops; adaptive density accumulates only the crops executed at $\tau=0.70$ . Heat intensity remains a per-example fixation-density map rather than a dataset-level fixation distribution.

Figure 3: Stanford Cars validation cases where detail fusion flips the prediction. Each row shows the input and the executed four-step fixations of the no-detail (+S+F) and detail (+S+F+D) ablation checkpoints; numbered boxes mark fixation order, heat shading marks accumulated crop coverage, and each label reports the prediction with its confidence. All three cases are same-make confusions corrected by the detail-enabled checkpoint.

Figure 4: Accuracy versus average focus steps for the Full+EE checkpoint. Dashed: truncating every sequence at a fixed step (1–4). Red markers: confidence-conditioned exit for $\tau\in\{0.30,0.50,0.70,0.90\}$ , using realized sequential-step measurements where available and otherwise the full-forward stop-rule simulation. Sequential-step points measure realized exits; simulated points analyze the stop rule without claiming realized savings.

Figure 5: Simulation-only ImageNet-R stop-rule diagnostic from full four-step rollouts. This panel helps separate the shape of the learned stopping criterion from realized sequential FLOP savings, whose estimates use physically executed step counts only for Seq. rows in the main table.

Dataset	Protocol	Avg. step	GFLOPs	Avg. step	GFLOPs	GFLOPs
		$\tau=0.70$		$\tau=0.90$		Full
Aircraft	Sim.	3.67	9.17	4.00	9.90	9.90
Cars	Sim.	3.99	9.87	4.00	9.90	9.90
CUB	Sim.	3.71	9.26	4.00	9.90	9.90
Dogs	Sim.	3.99	9.88	4.00	9.90	9.90
ImageNet-1K	Seq. steps	1.73	4.91	2.45	6.48	9.90

Appendix

Table 8: Conservative encoder-pass cost estimate (GFLOPs, multiply-adds; non-encoder operations excluded) of the Full+EE checkpoint at each early-exit threshold, computed as glance + average steps

\times

(focus + detail). Simulated rows are analytic stop-rule costs rather than realized sequential savings. The original model at four fixed steps costs 5.50 GFLOPs under the same proxy.

Active Visual Perception

Active visual perception studies how agents choose where, when, and how to look to gather task-relevant evidence efficiently.

See all 49 papers

CanViT: Toward Active-Vision Foundation Models
Mar 23, 2026Yohaï-Eliel Berreby, Sabrina Du, Audrey Durand +1Active Vision Active Perception
McGill University · Mila - Quebec AI Institute · Université Laval
An Exam for Active Observers
Jul 17, 2026Jiarui Zhang, Muzi Tao, Shangshang Wang +3VLM Evaluation Multimodal Large Language Models
University of Southern California
Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth
May 18, 2026Yuhuan Wu, Cong Wei, Fangzhen Lin +2Vision-Language Models VLM Adaptation
Hong Kong University of Science and Technology · University of Waterloo

Mar 23, 2026J/K move · Enter open · S save

CanViT: Toward Active-Vision Foundation Models

Yohaï-Eliel Berreby, Sabrina Du, Audrey Durand, B. Suresh Krishna

McGill University · Mila - Quebec AI Institute · Université Laval

Active Vision Active Perception Vision Foundation Models Self-Supervised Visual Representation Learning

Open paper AskPDF arXiv

Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses, but lacks scalable general-purpose architectures and pretraining pipelines, leaving Active-Vision Foundation Models (AVFMs) underexplored. We introduce CanViT, the first task- and policy-agnostic AVFM. CanViT uses scene-relative RoPE to bind a retinotopic Vision Transformer backbone and a spatiotopic scene-wide latent workspace, the canvas. Efficient interaction with this high-capacity working memory is supported by Canvas Attention, a novel asymmetric cross-attention mechanism. We decouple thinking (backbone-level) and memory (canvas-level), eliminating canvas-side self-attention and fully-connected layers to achieve fast sequential inference and scalability to high output resolutions. We propose a label-free active vision pretraining scheme, policy-agnostic passive-to-active dense latent distillation: reconstructing scene-wide DINOv3 embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths. We pretrain CanViT-B from a random initialization on 13.2 million ImageNet-21k scenes--an order of magnitude more than previous active models--and 1 billion random glimpses, in 166 hours on a single H100. On ADE20K segmentation, a frozen CanViT-B achieves 38.5% mIoU in a single low-resolution glimpse, outperforming the best active model's 27.6% with 20x fewer inference FLOPs as well as its FLOP- or input-matched DINOv3 teacher. Given additional glimpses, CanViT-B reaches 45.9% ADE20K mIoU. On ImageNet-1k classification, CanViT-B also sets a new active-vision state of the art, with 84.5% top-1 accuracy after fine-tuning. CanViT generalizes to longer rollouts, larger scenes, and new policies. Our work narrows the wide gap between passive and active computer vision, demonstrating the potential of task- and policy-agnostic AVFM pretraining.

Abstract

Active Visual Perception

Explore similar work

CanViT: Toward Active-Vision Foundation Models

An Exam for Active Observers

Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth