FoAViS: Analyzing Learned Fixation Strategies in a Foveated Active Vision System

Valentin Hassler; Jonathan Henrich; Yohaï-Eliel Berreby; Haoze Zhu; Alexander Ecker

cs.CVOct 30, 2026

FoAViS: Analyzing Learned Fixation Strategies in a Foveated Active Vision System

Authors: Valentin Hassler, Jonathan Henrich, Yohaï-Eliel Berreby, Haoze Zhu, Alexander Ecker

Organizations: University of Göttingen, Germany · McGill University, Canada · Mila - Quebec AI Institute, Canada · Lamarck Labs, USA ·

Abstract

Human vision is foveated and active: resolution is highest at the fovea and falls off towards the periphery, and fixations are actively directed to locations of interest. Building a machine system that processes information in the same way lets us study gaze behavior in a controlled setting where the system's components and objectives are known. The emergence of gaze characteristics can therefore be examined systematically, which may in turn inform our understanding of biological vision. This requires a model that can sample like the fovea, handle various tasks from classification to dense prediction, and be trained concurrently with a fixation policy. In our study, we present such a system and study its capabilities and the fixation policies that emerge from it. Our system, FoAViS, samples visual input by adopting the established log-polar foveation and accumulates information in a 2D spatial memory. Due to its 2D structure, the same memory can be read out globally for classification or densely for semantic segmentation, without any architectural change. Fixations are drawn from a 2D map by a policy trained concurrently with the task. We find that learned fixation policies select more informative, task-specific locations than non-learned heuristics. These policies also shape the task representations in a beneficial way: for both classification and segmentation, models trained concurrently with a policy outperform models trained with random locations, even when the latter are provided with learned scanpaths. Moreover, these scanpaths exhibit similarities to human gaze behavior by capturing aspects of empirical scanpath characteristics. Our work marks an important step away from standard passive, single-pass architectures, towards biologically grounded models of vision. It opens a path for studying goal-directed gaze behavior and information representation in the visual system.

Figures & tables

Figure 1: Emergent fixation strategies and model performance under concurrent policy and task training. (A) Example scanpaths directing fixations towards informative image areas. (B) Cumulative distribution over fixations two to four (excluding the initial center fixation) across 10k (C-MNIST and ImageNet), 5k (COCO-Stuff) and 2k (ADE20K) images, showing task- and dataset-dependent spatial priors. The colorbar indicates likelihood relative to a uniform baseline, with values reflecting the factor by which probability exceeds chance. The distributions are consistent across models trained with different random seeds ( Section H.2 ). (C) Downstream model performance comparing models trained concurrently with their policy and evaluated with it against models trained on random locations, evaluated either with random locations or via transfer of the learned policy.

Model	Biologically Motivated Foveation	Explicit 2D Spatial Memory	Trainable Fixation Policy	Global Tasks (e. g., Cls.)	Dense Tasks (e. g., Seg.)	Concurrent Policy and Task Training
RAM ( Mnih et al., 2014 )	$\sim$ (Patch Pyramid)	$\times$ (1D Recurrent)	✓	✓	$\times$	✓
DRAM ( Ba et al., 2015 )	$\sim$ (Patch Pyramid)	$\times$ (1D Recurrent)	✓	✓	$\times$	✓
Cheung et al. (2017)	✓(Learned Sampling)	$\times$ (1D Recurrent)	✓	✓	$\times$	✓
Ren and Zemel (2017)	$\times$ (Rect. Crops)	$\sim$ (Prediction-Masks Memory)	✓	✓	✓	✓
Saccader ( Elsayed et al., 2019 )	$\times$ (Feature Sampling)	$\times$ (Mean Pooling)	✓	✓	$\times$	✓
Attend-and-Segment ( Seifi and Tuytelaars, 2020 )	$\sim$ (Patch Pyramid)	✓(Multi-Level Memory)	$\times$ (Uncertainty Map)	$\times$	✓	$\times$ (No Policy)

Table 1: Comparison of active vision architectures across core design axes. Entries are ordered chronologically by publication year. Columns indicate whether a requirement is fully supported (✓), partially supported (

\sim

), or not supported (

\times

). Entries reflect each model as described and evaluated in its original publication. Some models could plausibly be extended to meet further requirements, but this was not explored in the respective work. FoAViS is discussed in more detail in Section 3 .

Figure 2: Overview of FoAViS unrolled over three glimpses. At each step, the current fixation serves as the center for a foveation module that extracts a log-polar glimpse \raisebox{0.5pt}{\scriptsize1}⃝ (sampling grid shown in white). The glimpse is encoded by the backbone \raisebox{0.5pt}{\scriptsize2}⃝ (feature map visualized via PCA), backprojected to Cartesian image space, and accumulated into a 2D spatial memory using a shared per-position LSTM \raisebox{0.5pt}{\scriptsize3}⃝ (memory state visualized via PCA). The memory state is then read out by the policy head \raisebox{0.5pt}{\scriptsize4}⃝ to produce a spatial probability map (logits visualized as a heatmap) from which the next fixation is drawn (purple number), while the task head \raisebox{0.5pt}{\scriptsize5}⃝ uses it to generate the task prediction. Appendix F shows these stages over full example scanpaths as visualizations.

	C-MNIST	ImageNet	ADE20K		COCO-Stuff
# Glimpses	Acc (%)	Acc (%)	mIoU (%)	Px Acc (%)	mIoU (%)	Px Acc (%)
Baseline trained and evaluated with full images
–	99.5	70.5	30.7	74.4	29.7	60.8
Trained and evaluated with random locations
1	$76.6\pm 0.2$	$59.0\pm 0.1$	$19.7\pm 0.3$	$66.0\pm 0.1$	$20.2\pm 0.2$	$51.4\pm 0.2$
4	$90.6\pm 0.3$	$63.7\pm 0.1$	$21.8\pm 0.4$	$68.1\pm 0.0$	$23.2\pm 0.2$	$54.7\pm 0.1$

Table 2: Native policy evaluation. A passive full-image baseline (top), models trained with random locations (middle), and models trained concurrently with a policy (bottom). Values report the mean

\pm

std across three random seeds (single run for passive baselines). All active vision models were trained with a fixed horizon of

T=4

, but are evaluated at different # Glimpses . For the learned-policy models, test-time fixations were selected deterministically via argmax from the policy’s probability map. Passive baselines receive the full image at

128\times 128

pixels.

Trained with random locations
		ImageNet	ADE20K		COCO-Stuff
Evaluation policy	# Glimpses	Acc (%)	mIoU (%)	Px Acc (%)	mIoU (%)	Px Acc (%)
random locations	4	$63.7\pm 0.1$	$21.8\pm 0.4$	$68.1\pm 0.0$	$23.2\pm 0.2$	$54.7\pm 0.1$
triangle	4	$65.0\pm 0.1$	$22.6\pm 0.1$	$68.9\pm 0.1$	$24.2\pm 0.2$	$55.8\pm 0.1$
quadrant	5	$65.9\pm 0.1$	$23.1\pm 0.2$	$69.5\pm 0.1$	$24.8\pm 0.2$	$56.6\pm 0.2$
saliency + IoR	4	$65.6\pm 0.2$	–	–	–	–

Table 3: Policy transfer evaluation. Models trained with random locations (top block) and models trained on a specific dataset with their own learned policy (bottom block) are evaluated with heuristic policies (top sub-block) and with learned policies (bottom sub-block). Shaded cells indicate evaluation with a model’s native policy. Values report the mean

\pm

std across three models trained with different random seeds. The heuristic policies are defined in Appendix G .

Figure 3: Scanpath statistics of the learned policies. Results are reported for a single seed, but remain consistent across training runs with different seeds ( Section H.1 ). Unless stated otherwise, metrics for our models are calculated for the standard training horizon of three fixations (

f_{2},f_{3},f_{4}

) after an initial center fixation (

f_{1}

). The human dataset (MIT1003) was also recorded with an initial center fixation and metrics were calculated for all scanpaths that consist of at least three fixations afterwards. (A) Saccade amplitude across first three saccades. (B) Saccade direction across first three saccades. (C) Mean amplitude of the next saccade as a function of the preceding one (amplitudes standardized within each scanpath). Our models were unrolled for nine fixations (

f_{2}\text{--}f_{10}

) starting from a center fixation (

f_{1}

). For MIT1003, we selected scanpaths with at least nine fixations after the center fixation and truncated them to these nine. The plot is based on the saccades between

f_{2}

and

f_{10}

. The initial saccade from the center to

f_{2}

is excluded because its starting point is fixed. (D) Turning angle between two consecutive saccade vectors, measured between the first and second saccade (at

f_{2}

) and between the second and third saccade (at

f_{3}

Figure 4: Argmax fixation distance. Euclidean distance between fixations from the native reference model and fixations from evaluated models conditioned on the native model’s fixation history, computed across nine fixations

f_{2}

–

f_{10}

following the initial center fixation

f_{1}

(see Section 5.3 for details). Distances are expressed as a percentage of image width (all images are square, so width equals height). Lower values indicate stronger policy alignment. Colored lines indicate comparison models (different tasks or the same task with different random seeds). The gray line denotes the expected distance when locations are drawn uniformly at random. Shaded bands show

\pm 1

std over three model pairs. The dotted line marks the

T=4

glimpses used in training.

Figure 5: Emergent inhibition of return. z-standardized policy logits at visited locations as a function of relative lag

t-k

(see Section 5.3 for details). Thin lines show individual models trained with different random seeds, while bold lines indicate their mean. Negative / positive lags correspond to steps before / after the fixation. Lower / higher logits reflect reduced / increased selection probability at that location.

Figure 6: Training dynamics under different optimization settings. (A) Task performance across models trained with stop-grad (see Section 3.6 ), without stop-grad, and without stop-grad alongside entropy regularization. The triangle marks the checkpoint evaluated in panel (B), matched at roughly 150k training iterations. (B) Distribution of the ratio

q

of policy to task gradient magnitudes for backbone and memory parameters, as a kernel density estimate over

\log_{10}q

with one value per minibatch, computed over 78 (ImageNet, batch size 64) or 312 (ADE20K and COCO-Stuff, batch size 16) training minibatches, i. e. about 5k images per model. The stop-grad model yields zero policy gradient to these components and is omitted.

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Layer	Res. change	Out ch.	Output size
Encoder	ResBlock (stride 1)	–	64	$64\times 64$
	ResBlock (stride 1)	–	64	$64\times 64$
	ResBlock (stride 2)	$\downarrow 2$	128	$32\times 32$
	ResBlock (stride 1)	–	128	$32\times 32$
	ResBlock (stride 2)	$\downarrow 2$	256	$16\times 16$
	ResBlock (stride 1)	–	256	$16\times 16$

Appendix

Table A.1: Backbone architecture. The backbone takes a log-polar transformation of the input image. Each ResBlock uses

3\times 3

convolutions, with a

1\times 1

projection on the skip connection where needed. In a ResBlock, each convolution is followed by Instance Normalization and the first also by a ReLU; the second ReLU follows the residual addition. The bottleneck operates at the encoder’s final resolution. In the decoder, each “Conv

1\times 1

fuse” row concatenates the incoming features with the corresponding encoder feature map (channel counts in parentheses) and projects them to the decoder width, again followed by Instance Normalization and ReLU.

Figure A.1 : Sampling density of the glimpse at the classification (

224\times 224

) and segmentation (

512\times 512

) resolutions. The spacing between neighboring samples, roughly the smallest detail a glimpse resolves, is plotted against the distance from the fixation, both as a percentage of the image width. Up to

11

px (

224

) and

10

px (

512

) from the fixation, the image is oversampled: neighboring samples are less than a pixel apart. The plot floors the spacing at one pixel there (plateaus, labeled full resolution ), since samples closer than a pixel add no detail.

Hyperparameter	ImageNet	C-MNIST	ADE20K	COCO-Stuff
Initialization	random	random	ImageNet random-location model
Optimizer	AdamW, weight decay $0.01$ (all parameters)
$\beta_{1},\beta_{2},\epsilon$	$0.99,\ 0.999,\ 10^{-8}$
Learning rate	$10^{-4}$	$10^{-4}$	$2\times 10^{-4}$	$2\times 10^{-4}$
Warmup	$2000$ steps (linear)
Epochs	$150$	$400$	$500$	$72$

Appendix

Table A.3: Training hyperparameters. Random initialization uses PyTorch’s defaults. The segmentation models take over the backbone and memory of the ImageNet random-location model; their policy and segmentation heads are newly initialized.

Figure B.1 : C-MNIST validation images , the first four of the validation set, with the target digit boxed and its label above. The other characters are EMNIST letters.

Model	Backbone	Init.	Input res.	Pixels (%)	Glimpses	ADE20K mIoU (%)	ImageNet Acc (%)
DRAM ( Ba et al., 2015 )	ResNet-50	ImageNet (3-stage)	$224^{2}$	94.5 a	8	–	67.5 b
Saccader ( Elsayed et al., 2019 )	NASNet	ImageNet (3-stage)	$331^{2}$	69.9 a	6	–	75.0
GFNet ( Wang et al., 2020 )	ResNet-50	ImageNet	$224^{2}$	91.8	5	–	75.9
TNet ( Papadopoulos et al., 2021 )	BagNet-77	from scratch	$224^{2}$	70.9	6	–	74.6
FoveaTer ( Jonnalagadda et al., 2021 )	PatchConvNet	ResNet-18 + DeiT-S	$224^{2}$	n/a	3	–	76.7
Lukanov et al. (2021)	Xception-based CNN	from scratch	$256^{2}$	10.0 a	1	–	65.2 c

Appendix

Table D.1 : Published results of prior active vision models on ADE20K and ImageNet. Each model is listed with its best published result. These works cannot be compared in a controlled way: backbone, initialization, glimpse budget, input resolution and evaluation protocol all differ. Pixels : pixels used for the task, summed over all glimpses and divided by the pixels of the model’s own input image ( Input res. ). Pixels are counted as delivered, after any down-scaling, not by the area they cover. A glance or overview pass, if a model uses one, counts at its own size and is included in Glimpses . Overlapping glimpses are not de-duplicated, so a budget can exceed

100\%

. Two values in a cell are segmentation / classification. A dash marks a benchmark the model does not report.

Figure E.1 : Unrolling beyond the training horizon. Every model is trained at

T=4

(dotted rule) and unrolled to each budget at inference; nothing is retrained. One color per dataset, one line style per setting. Solid is each dataset’s own learned policy, dashed a model trained and evaluated with random locations, dash-dotted each seed’s learned policy transferred to that same seed’s random-location model, and dotted the learned-policy model evaluated with random locations. All four are measured by one protocol (last checkpoints; learned policies fixate the argmax of their probability map, random locations are drawn uniformly once per evaluation, so these points can differ slightly from the corresponding table cells in Tables 2 and 3 ). At

T_{\text{eval}}{=}1

the four curves collapse to two, since the first fixation is the fixed image center and no policy has acted yet. Shading is

\pm 1

std over three models trained with different random seeds, measured on this protocol in every panel.

Panel of Figure 16 — Figure F.1 : Foveation, ImageNet. Same examples and scanpaths as Figure F.5 (top: Japanese spaniel , bottom: paddlewheel ). Rows as described in Section F.1 .

Panel of Figure 17 — Figure F.2 : Foveation, C-MNIST. Same examples and scanpaths as Figure F.7 (ground truth 5 and 4 ). Rows as described in Section F.1 .

Panel of Figure 18 — Figure F.3 : Foveation, ADE20K. Same examples and scanpaths as Figure F.9 . Rows as described in Section F.1 .

Panel of Figure 19 — Figure F.4 : Foveation, COCO-Stuff. Same examples and scanpaths as Figure F.11 . Rows as described in Section F.1 .

Panel of Figure 20 — Figure F.5 : Policy, memory and prediction, ImageNet ( $T=4$ ). Top : ground truth Japanese spaniel . The first two predictions name the related Blenheim spaniel ( $23$ , $84\%$ ); after the third glimpse the prediction switches to Japanese spaniel ( $92\%$ ) and keeps it ( $91\%$ ). Bottom : the predicted class alternates between flagpole ( $33\%$ ), paddlewheel ( $20\%$ ) and flagpole again ( $39\%$ ); the last glimpse raises the ground truth paddlewheel to $86\%$ .

Panel of Figure 21 — Figure F.6 : Policy, memory and prediction, ImageNet (continued). Top : four predictions, four classes, all at low confidence: coho ( $28\%$ ), sturgeon ( $13\%$ ), cowboy hat ( $9\%$ ), and the ground truth gondola ( $23\%$ ) only at the last glimpse. Bottom, failure : lion at $95\%$ after the first glimpse loses probability at every step ( $32$ , $23$ , $2\%$ ) and grey fox is the predicted class from the second glimpse on ( $42$ , $56$ , $68\%$ ).

Panel of Figure 22 — Figure F.7 : Policy, memory and prediction, C-MNIST ( $T=4$ ). Top : 2 for the first three predictions ( $52$ , $40$ , $55\%$ ) while the ground truth 5 stays at $9$ – $14\%$ ; the last glimpse switches to 5 at $100\%$ . Bottom : 9 with rising confidence ( $26$ , $46$ , $87\%$ ) while the ground truth 4 stays at $10$ – $15\%$ ; the last glimpse switches to 4 at $100\%$ .

Panel of Figure 23 — Figure F.8 : Policy, memory and prediction, C-MNIST (continued). Top : 9 ( $64\%$ ), 1 ( $63\%$ ), 2 ( $51\%$ ), then the ground truth 9 again at $100\%$ . Bottom, failure : 9 at $73\%$ after the first glimpse; the three later fixations land on the same location and the prediction moves to 8 ( $96$ , $94$ , $85\%$ ), with the ground truth at $4$ – $15\%$ .

Panel of Figure 24 — Figure F.9 : Policy, memory and prediction, ADE20K ( $T=4$ ). Top : pixel accuracy $97$ – $99\%$ at every step and per-image mIoU from $93\%$ to $96\%$ ; later glimpses relabel at most $2\%$ of the pixels. Bottom : pixel accuracy $80\%$ after the first glimpse and $98\%$ after the second, which relabels $30\%$ of the pixels; per-image mIoU $12\to 40\%$ .

Panel of Figure 25 — Figure F.10 : Policy, memory and prediction, ADE20K (continued). Top : pixel accuracy $73$ , $85$ , $87$ , $86\%$ and per-image mIoU $22\to 49\%$ , with less relabeling at each step ( $14$ , $5$ , $3\%$ of the pixels). Bottom, failure : $93\%$ pixel accuracy after the first glimpse; the next two glimpses relabel $8\%$ and $13\%$ of the pixels, mostly from correct to wrong, and accuracy falls to $88\%$ and $76\%$ ( $78\%$ at the end).

Panel of Figure 26 — Figure F.11 : Policy, memory and prediction, COCO-Stuff ( $T=4$ ). Top : $60$ , $49$ and $63\%$ of the pixels change label at the three later glimpses; pixel accuracy $7$ , $23$ , $28$ , $70\%$ . Bottom : pixel accuracy changes little ( $95$ , $98$ , $98$ , $99\%$ ) while per-image mIoU rises from $33\%$ to $82\%$ .

Panel of Figure 27 — Figure F.12 : Policy, memory and prediction, COCO-Stuff (continued). Top : pixel accuracy $82\%$ after the first glimpse and $98\%$ after the second, which relabels $16\%$ of the pixels ( $98\%$ at the end); per-image mIoU $27\to 55\%$ . Bottom, failure : pixel accuracy above $99\%$ after the first glimpse, then $41$ , $4$ and $3\%$ ; the second and third glimpses relabel $59\%$ and $37\%$ of the pixels.

	ImageNet Acc (%)		ADE20K mIoU (%)		COCO-Stuff mIoU (%)
Evaluation policy	learned	random	learned	random	learned	random
saliency	$64.5\pm 0.2$	$63.7\pm 0.2$	–	–	–	–
saliency + IoR ( $r{=}12.5\%$ )	$66.7\pm 0.2$	$65.2\pm 0.2$	–	–	–	–
saliency + IoR ( $r{=}25\%$ )	$66.7\pm 0.2$	$65.6\pm 0.2$	–	–	–	–
entropy	–	–	$21.9\pm 0.4$	$21.8\pm 0.2$	$23.2\pm 0.2$	$22.7\pm 0.1$
entropy + IoR ( $r{=}12.5\%$ )	–	–	$22.8\pm 0.3$	$22.3\pm 0.2$	$23.7\pm 0.1$	$23.1\pm 0.1$

Appendix

Table G.1 : Further heuristic policies. Accuracy (ImageNet) and mIoU (ADE20K, COCO-Stuff) after

T{=}4

glimpses for the heuristic variants that Table 3 does not show. The IoR radius

r

is given as a fraction of the image width ( Appendix G ). The

r{=}25\%

rows are the ones reported in Table 3 , repeated here to compare the two IoR radii. Saliency is run on ImageNet only and entropy on the segmentation datasets only ( Appendix G ). Learned : learned-policy models, random : random-location models, each evaluated with the heuristic of the row. Mean

\pm

std over three models trained with different random seeds, last checkpoints.

Figure G.1 : Heuristic policies on ImageNet. The learned-policy model evaluated with different policies on two validation images (left and right). Each row is one evaluation policy: the model’s own (native) policy, saliency, saliency with IoR at $r=12.5\%$ and at $r=25\%$ , and the two memory-saliency variants $\mathrm{relu}(\mathbf{H})$ and $|\mathbf{H}|$ , both with IoR at $r=12.5\%$ . For each image, the first column shows the four fixations of the policy, numbered in order (1 is the initial center fixation). The next three columns show the map from which the next fixation was chosen after glimpses 1, 2 and 3, with brighter colors for higher values. In these maps, the cyan cross marks the chosen fixation (the argmax of the map, or in the native-policy row the fixation drawn from the probability map), white crosses mark earlier fixations, and dashed circles the regions excluded by IoR. The native-policy row shows the policy logits.

Figure G.2 : Heuristic policies on ADE20K. As in Figure G.1 , with entropy in place of saliency in rows 2 to 4. The entropy map is shown on its $16{\times}16$ block grid.

Figure G.3 : Heuristic policies on COCO-Stuff , as in Figure G.2 .

Figure H.1 : Scanpath statistics across seeds, C-MNIST (

10\,000

validation images per seed; fixations drawn from the policy). Panels as in Figure 3 ; the three models trained with different random seeds share the dataset color and differ by line style (seed 1 solid, 2 dashed, 3 dotted) and marker. Thick gray line, in every panel: a uniform random policy (initial center fixation, then independent uniform fixations; simulated), the geometric reference for an image of this shape, the same simulation in every dataset’s figure and in Figure 3 . The

y

-axes are scaled per dataset, so the same gray line looks flatter where the models’ curves are more peaked (e.g. the direction row of ADE20K against that of ImageNet).

Figure H.2 : Scanpath statistics across seeds, ImageNet (

10\,000

validation images per seed; fixations drawn from the policy). Panels, line styles and the random-policy reference as in Figure H.1 .

Figure H.3 : Scanpath statistics across seeds, ADE20K (

2000

validation images per seed; fixations drawn from the policy). Panels, line styles and the random-policy reference as in Figure H.1 .

Figure H.4 : Scanpath statistics across seeds, COCO-Stuff (

5000

validation images per seed; fixations drawn from the policy). Panels, line styles and the random-policy reference as in Figure H.1 .

Figure H.5 : Cumulative fixation distribution across seeds. Figure 1 B for the three seeds (rows) of our default models; seed 1 is the model shown in Figure 1 . Each panel is the mean probability map over the validation pool, with fixations drawn from the policy, averaged over the maps after glimpses 1 to 3, which choose fixations 2 to 4 (the initial center fixation is fixed and therefore left out, as is the map after the fourth glimpse). Colors are clipped per panel at the $99.5$ th percentile. Images per seed: C-MNIST $10\,000$ , ImageNet $10\,000$ , ADE20K $2000$ , COCO-Stuff $5000$ .

Figure H.6 : Cumulative fixation distributions with the policy moved across image domains. Each cell is the mean probability map over the image pool, with fixations drawn from the policy, averaged over the maps after glimpses 1 to 3, which choose fixations 2 to 4 (the initial center fixation is fixed and therefore left out, as is the map after the fourth glimpse), i.e. the quantity Figure 1 B plots per dataset. Rows fix the policy (the learned-policy models shown in Figure 1 ), columns fix the image domain ; images per column: ImageNet $10\,000$ , ADE20K $2000$ , COCO-Stuff $5000$ . Each model sees its own training resolution and all priors are compared in the shared $64\times 64$ memory space. The diagonal (colored frame) is each policy on its own training domain.

Figure H.7 : Fixation distribution at each step, for the models and human observers. Figure 1 B, split by step: one row per dataset (top four: models; bottom: human observers), one column per step from $t=1$ to $t=6$ , and the cumulative map. The models are trained with $T=4$ and unrolled to $T_{\text{eval}}=7$ glimpses with fixations drawn from the policy: after the initial center fixation $f_{1}$ , the map at step $t$ chooses $f_{t+1}$ , so the six steps place $f_{2}$ to $f_{7}$ . Each map is the mean of the per-step distributions over the validation images (C-MNIST and ImageNet $10\,000$ , ADE20K $2000$ , COCO-Stuff $5000$ ); the color scale is shared within a row and clipped at the $99.5$ th percentile. The cumulative column averages all six steps ( $t=1$ to $6$ ), more than the cumulative map of Figure 1 B ( $t=1$ to $3$ ). Bottom row: MIT1003 free viewing; column $t$ is the density of the fixation $f_{t+1}$ , as a $64\times 64$ histogram blurred by one cell, and $n$ is the number of fixations per panel.

Figure H.8 : Autocorrelation of saccade amplitudes. Mean amplitude of the next saccade as a function of the preceding one, both z-standardized within each scanpath (own mean subtracted, divided by own std), smoothed with a Gaussian kernel (bandwidth

0.15

) and drawn over the range that lies within the

2

nd to

98

th percentile of the preceding amplitude for every source in the panel. Each panel uses the fixations

f_{2}

f_{k+1}

for one

k=4,\dots,9

, i.e.

k-1

saccades. The initial saccade from the center

f_{1}

to the next fixation

f_{2}

is excluded since its starting position is fixed. For MIT1003,

f_{2}

f_{k+1}

are the first

k

fixations after the initial center fixation, and each panel uses all scanpaths with at least

k

of them (counts in the panel titles). One learned-policy model per dataset (the one shown in Figure 1 , trained with four glimpses) is unrolled for nine fixations (

f_{2}

f_{10}

), drawn from the policy after the initial center fixation, and cut to

f_{2}

f_{k+1}

. Thick gray line: uniform random policy.

Figure H.9 : Per-fixation policy alignment: near hits. Fraction of images on which the two policies’ argmax fixations lie within

5\%

of the image width of each other (about three cells of the

64\times 64

probability map; higher = more aligned). It thresholds the per-image distances that Figure 4 averages. On every dataset and at every fixation, the two seeds of the same task give the most near hits, the other tasks’ policies fewer and uniform fixations the fewest. Layout as Figure 4 : every model is unrolled for nine fixations (

f_{2}

–

f_{10}

) after the initial center fixation; the dotted line marks the end of the

T=4

glimpses used in training. Shading:

\pm 1

std over three model pairs, each averaged over the validation images (ImageNet

10\,000

, ADE20K

2000

, COCO-Stuff

5000

Figure H.10 : Per-fixation policy alignment: probability-map correlation. Correlation between the two policies’ next-fixation probability maps, each first averaged over the images (higher = more aligned): the least localized of the three metrics, and a general impression of where each policy puts its mass rather than of where its peak falls for a specific image. Layout and shading as in Figure H.9 .

Figure H.11 : Inhibition of return, split by visit index. The quantity of Figure 5 with one row per visited fixation $f_{k}$ ( $k=2,\dots,10$ ), one panel per dataset and a shared color scale. Fixations are drawn from the policy. Values are means over the three seeds of each of our default models. The black vertical line separates the steps before the visit (negative lags) from the visit and the steps after it.

Figure H.12 : Inhibition of return on C-MNIST , shown on its own because its values are an order of magnitude larger: its probability maps put nearly all their mass on a single cell. The policy logit $z$ at the location of $f_{2}$ , the first fixation after the initial center fixation, tends to drop sharply at the visit ( $z=20.9\!\to\!15.4$ ) and stays low afterwards (seen in panel B). From $f_{3}$ on, the policy tends to keep returning to the same location, and the logit at the visited cell drops by at most $1.7$ (seen in panel B), so the marginal (seen in panel A) shows little suppression. This reflects the fact that the policy has a tendency to keep looking at the digit once it found it. Inhibition of return therefore does not emerge in the same way in this setting where discriminatory information is highly localized. Values are means over the three seeds of each of our default models. In (A), thin lines are the individual seeds. The black vertical line separates the steps before the visit (negative lags) from the visit and the steps after it.

Choice	Default	Alternative	Analyzed in
Policy gradient	stop-gradient on $\mathbf{H}_{t}$ ( Equation 8 )	full gradient	Appendix J
Entropy regularization	none	entropy target ( $\mathcal{H}^{\star}=3.5$ nats)	Section J.5
Task supervision	mean per-step CE ( Equation 5 )	last step only	Section K.1
Reward	$r_{t}=\mathrm{CE}_{t}-\mathrm{CE}_{t+1}$ ( Equation 6 )	$-\mathrm{CE}_{T}$ for every fixation	Section K.2

Appendix

Table I.1 : Alternatives to the training recipe investigated in Appendices J and K . For each choice of the training recipe, the table gives the setting used in our default models, the alternative we compare it with, and the subsection that reports the results.

\mathrm{CE}_{t}=\mathcal{L}_{\text{CE}}(\hat{\mathbf{y}}_{t},\mathbf{y})

is the task loss after glimpse

t

	Deviation	Description
D1	Terminal reward	Every fixation receives the same reward, $-\mathrm{CE}$ at the last step, instead of its own per-step improvement ( Equation 6 )
D2	Terminal supervision	The task loss is computed only on the prediction after the last glimpse, instead of after every glimpse ( Equation 5 )
D3	Level normalizer	The reward is divided by its mean absolute value, pooled over batch and step, instead of being standardized per step ( Equation 7 )
D4	Gradient on backbone and memory	The policy gradient reaches backbone and memory, instead of stopping at the memory features (stop-gradient in Equation 8 )
D5	Frozen representation	Only the policy head trains, on a frozen backbone, memory and task head, instead of all parameters; accuracy then depends on the fixations alone
D6	Early epoch	Evaluated at a matched epoch before convergence, instead of at the end of the full schedule ( Table A.3 )

Appendix

Table I.2 : Numbered deviations from the main protocol. Every figure and table caption in Appendices J and K lists the numbers of the deviations that apply to its runs. Investigated settings without a number (the small policy head, the entropy target and the policy-loss weight

\lambda

) are named in the captions themselves. A result given only in the text either belongs to the runs of a cited figure or table, or lists its deviations in a footnote.

	C-MNIST	ImageNet	ADE20K	COCO-Stuff
	Acc (%)	Acc (%)	mIoU (%)	mIoU (%)
with stop-grad	$99.3\pm 0.0$	$61.6$	$24.3\pm 0.2$	$24.9\pm 0.2$
w/o stop-grad	$98.8\pm 0.1$	$25.5$	$18.1\pm 0.8$	$17.9\pm 0.5$
w/o stop-grad + small head	$95.1$	$61.4$	$23.1$	$24.1$
w/o stop-grad + entropy	$99.3$	$61.9$	$23.8$	$23.5$

Appendix

Table J.1 : Task performance with the policy gradient on backbone and memory. With stop-grad (reference): the policy gradient stops at the memory features (our default models). w/o stop-grad : no stop-gradient. w/o stop-grad + small head , w/o stop-grad + entropy : no stop-gradient, with one of the two variants of Section J.5 (a

129

-parameter policy head or an entropy target of

3.5

nats). On C-MNIST, ADE20K and COCO-Stuff, with stop-grad are our default models. On ImageNet, with stop-grad is our default ImageNet model after 75 of its 150 epochs, and the other rows are trained only for these 75 epochs.

\pm

: std over seeds where several exist. Within each column the runs share epochs, learning-rate schedule, batch size and initialization and differ only in the setting named in each row. Deviations ( Table I.2 ): gradient on backbone and memory (D4; all rows except with stop-grad ), reduced budget (D11; ImageNet), logged validation score (D12).

	C-MNIST		ImageNet		ADE20K		COCO-Stuff
	Acc (%)	$\Delta$	Acc (%)	$\Delta$	mIoU (%)	$\Delta$	mIoU (%)	$\Delta$
random locations	$90.5$	–	$63.6$	–	$22.2$	–	$23.0$	–
with stop-grad	$98.3$	$+7.8$	$66.0$	$+2.5$	$23.0$	$+0.8$	$23.9$	$+0.8$
w/o stop-grad	$98.3$	$+7.8$	$64.9$	$+1.4$	$23.2$	$+1.0$	$24.1$	$+1.1$
w/o stop-grad + small head	$97.7$	$+7.2$	$66.3$	$+2.8$	$23.0$	$+0.8$	$24.3$	$+1.3$
w/o stop-grad + entropy	$98.4$	$+7.9$	$66.2$	$+2.7$	$23.3$	$+1.1$	$24.3$	$+1.3$

Appendix

Table J.2 : The policies of Table J.1 , transferred to one random-location model. In Table J.1 each row changes both the policy and backbone and memory, so a low cell there can come from either. Here the receiving model is frozen: in every row, the dataset’s default random-location model is evaluated on its validation set, and only the fixations

f_{2}

–

f_{4}

it is given change (

f_{1}

is always the image center). Row 1 is that model under the uniform-random policy it was trained with, so each

\Delta

is the value of the policy alone. Deviations ( Table I.2 ): gradient on backbone and memory (D4; the policies without stop-gradient), policy-transfer score (D10), reduced budget (D11; the ImageNet policies).

Figure J.1 : Training variants on ADE20K. Validation mIoU of the run with stop-gradient, a run without stop-gradient at policy-loss weight

\lambda=0.1

and one at

\lambda=0.01

, otherwise identical. First

65

epochs; single runs. Deviations ( Table I.2 ): gradient on backbone and memory (D4; models without stop-gradient), reduced budget (D11; the run at

\lambda=0.01

ends at epoch

64

), logged validation score (D12).

Logit range
	w/o stop-grad	with stop-grad
ImageNet	$82\,946$	$360$
ADE20K	$1802$	$240$
COCO-Stuff	$3184$	$616$
Jacobian norm of the policy head, ImageNet
first epoch	$137$	$50$

Appendix

Table J.3 : Logit range and Jacobian norm of the policy head, without and with stop-gradient. Logit range : largest minus smallest policy logit of one probability map, median over

500

validation images and the three decision steps, final checkpoints. Jacobian norm :

\|\mathbf{J}_{t}\|

, the largest singular value of the Jacobian of the

64\times 64

policy logits with respect to the memory features (

128\times 64\times 64

), computed by power iteration without forming the matrix; median over

32

validation images and the three decision steps, at the first and the last epoch; ImageNet only. Models as in Table J.1 . Deviations ( Table I.2 ): gradient on backbone and memory (D4; models without stop-gradient), reduced budget (D11; ImageNet).

Figure J.2 : Policy entropy and the entropy weight during training. Models as in Table J.1 on ImageNet, ADE20K and COCO-Stuff (one model each); on ADE20K also the run without stop-gradient with a

10\times

smaller policy-loss weight (

\lambda=0.01

) of Figure J.1 . (A) policy entropy (dashed: uniform policy,

\log 4096\approx 8.3

nats; dotted: the entropy target of

3.5

nats of the entropy-target variant). (B) the entropy weight

\lambda_{\mathcal{H}}

set by the update rule of the entropy-target variant (dotted: the bounds

10^{-6}

and

10^{-1}

within which it is kept); in the second half of training it is about

1.6\times 10^{-4}

on ImageNet,

1.0\times 10^{-3}

on ADE20K and

3.6\times 10^{-5}

on COCO-Stuff. One logged minibatch per epoch; bold lines are rolling medians (window a tenth of the run, at least nine epochs), faint lines the raw values. Deviations ( Table I.2 ): gradient on backbone and memory (D4; all models except the one with stop-gradient), reduced budget (D11; ImageNet and the run at

\lambda=0.01

Figure J.3 : Bursts of the policy gradient. Distribution of the per-sample gradient norm

\|\partial\log\pi_{\theta}(f_{t+1}\mid\mathbf{H}_{t})/\partial\mathbf{H}_{t}\|

into the memory features, for the model without stop-gradient and the two variants of Section J.5 , as a kernel density estimate over

\log_{10}

of the norm. One sample is one fixation drawn from one map (

8

per map;

64

ImageNet or

32

segmentation validation images), computed in fp32. Samples with a gradient of exactly zero cannot be placed on the log axis and are left out of the curves, so each curve integrates to the share of samples with a nonzero gradient. Each panel prints the share of zeros for the model without stop-gradient (e.g.

70\%

on ImageNet). Epochs as in the panel titles; single runs. Deviations ( Table I.2 ): gradient on backbone and memory (D4), early epoch (D6; the epochs in the panel titles), reduced budget (D11; ImageNet).

	Random locations			Learned policy
$T_{\text{eval}}$	Per-step	Last only	$\Delta$	Per-step	Last only	$\Delta$
1	$58.71$	$52.77$	$+5.94$	$55.25$	$46.23$	$+9.02$
4 (trained)	$62.92$	$63.81$	$-0.89$	$64.66$	$64.34$	$+0.32$
10	$65.49$	$64.64$	$+0.85$	$65.21$	$64.14$	$+1.07$
20	$65.73$	$63.18$	$+2.55$	$64.13$	$62.19$	$+1.94$

Appendix

Table K.1 : Supervision at every step against supervision at the last step only. ImageNet accuracy (%) of models trained with

T=4

glimpses and unrolled to

T_{\text{eval}}

glimpses, each evaluated with the fixations it was trained with (random locations or its native policy).

\Delta

is the accuracy in column Per-step minus that in column Last only . The models use an earlier version of the backbone, so their accuracies are not comparable with those of the other tables. Deviations ( Table I.2 ): terminal reward (D1), terminal supervision (D2; the last-only variants), level normalizer (D3), gradient on backbone and memory (D4), best checkpoint (D7; random-location pair), baseline present (D8), different pretrained models (D9), different learning-rate schedule (D13), entropy regularization (D14). In addition, the learned last-only model is evaluated with fixations drawn from its policy instead of argmax fixations.

$T_{\text{eval}}$	Random locations	Terminal	Per-step	$\Delta$
1	$58.89$	$58.89$	$58.89$	$0.00$
2	$61.17$	$63.12\pm 0.06$	$63.47\pm 0.02$	$+0.35\pm 0.08$
4 (trained)	$63.54$	$65.27\pm 0.47$	$\mathbf{65.89\pm 0.07}$	$\mathbf{+0.62\pm 0.40}$
6	$64.65$	$64.97\pm 0.17$	$66.50\pm 0.06$	$+1.53\pm 0.23$
8	$65.26$	$64.56\pm 0.16$	$66.57\pm 0.01$	$+2.01\pm 0.15$
10	$65.80$	$64.21\pm 0.20$	$\mathbf{66.58\pm 0.04}$	$\mathbf{+2.36\pm 0.15}$

Appendix

Table K.2 : Reward rule and horizon on ImageNet. Accuracy (%) with policy heads trained on a frozen random-location model and evaluated with it, unrolled to

T_{\text{eval}}

glimpses. The column Random locations shows that model with its own, randomly drawn locations. Terminal and Per-step are means over two policy heads trained with different random seeds;

\pm

is half the difference between the two.

\Delta

is the accuracy in column Per-step minus that in column Terminal , computed per seed and then averaged. Deviations ( Table I.2 ): terminal reward (D1; the terminal column), level normalizer (D3), frozen representation (D5), baseline present (D8), policy-transfer score (D10), reduced budget (D11; one GPU,

24

epochs of

2000

steps). In addition, the policy head has one instead of three

3\times 3

convolutions.

Active Visual Perception

Active visual perception studies how agents choose where, when, and how to look to gather task-relevant evidence efficiently.

See all 51 papers

May 18, 2026

Why We Look Where We Look: Emergent Human-like Fixations of a Foveated Visual Language Model Maximizing Scene Understanding

When humans view scenes without a specific task (free-viewing), they initially direct their eye movements toward the scene center and then fixate on people, text, objects being gazed at or grasped, and semantically meaningful regions. What these signature fixation patterns reflect and whether they optimize an underlying perceptual task remain unknown. We show that a computational agent with simulated foveation, trained to optimize scene comprehension, exhibits emergent human fixation signature patterns. In contrast, versions of the agent trained to search or classify scenes, or equipped with peripheral vision that was better or worse than human vision, predicted human fixation patterns less accurately. Thus, human free-viewing fixation patterns may emerge as a functional byproduct of optimizing scene comprehension under the biological constraints of foveated vision.

Shravan Murlidaran, Ziqi Wen, Sana Shehabi +1

Psychological & Brain Sciences, University of California, Santa Barbara & 93106, USA. · Computer Science, University of California, Santa Barbara & 93106, USA. · Electrical and Computer Engineering, University of California, Santa Barbara & 93106, USA.

Visual Attention Large Vision-Language Models Active Visual Perception Foveated Vision Human Visual Perception

Ask PDF

Jul 21, 2025

Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers

Human vision is a highly active process driven by gaze, which directs attention to task-relevant regions through foveation, dramatically reducing visual processing. In contrast, robot learning systems typically rely on passive, uniform processing of raw camera images. In this work, we explore how incorporating human-like active gaze into robotic policies can enhance efficiency and robustness. We develop GIAVA (Gaze Integrated Active-Vision ALOHA), a robot vision system that emulates human head and neck movement, and gaze adjustment for foveated processing. Extending the AV-ALOHA robot platform, we introduce a framework for simultaneously collecting eye-tracking, perspective control, and robot manipulation demonstration data from a human operator. We also open-source a simulation benchmark and dataset for training robot policies that incorporate human gaze. Inspired by recent work in foveated image segmentation and given the widespread use of Vision Transformers (ViTs) in robot learning, we integrate gaze information into ViTs using a foveated patch tokenization scheme. Compared to uniform patch tokenization, this significantly reduces the number of tokens, and thus computation. Our results show that our method for foveated robot vision drastically reduces computational overhead, and enhances robustness to background distractors. Notably, on certain high-precision tasks, foveated vision also improves performance, as reflected in higher success rates. Together, these findings suggest that human-inspired foveated visual processing offers untapped potential and should be further considered as a useful inductive bias in robotic vision systems. https://soltanilara.github.io/giava/

Ian Chuang, Jinyu Zou, Andrew Lee +2

University of California, Berkeley · 2Tongji University · University of California, Davis

Visuomotor Policy Learning Efficient ViTs Active Visual Perception Robotic Perception Foveated Vision

Ask PDF

Jul 2, 2026

DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction

Understanding human visual attention on a scene over time has applications in domains such as interface design and inferring cognitive states. Modeling visual scanpaths has historically relied on specialized architectures with hand-crafted priors. While these architectures can model fixation sequences, their rigid structural biases restrict easy extendability and flexible conditioning. For instance, integrating task-specific instructions or adapting to distinct viewer identities requires custom, disjoint architectural additions. We frame scanpath prediction purely as a discrete sequence modeling task. By mapping coordinates into a text vocabulary, we leverage the pretrained representations of Vision-Language Models. This framing absorbs diverse factors of variation: simple prompting allows for global conditioning, such as providing viewer identities to capture personalized biases, or task-specific objectives like visual search. The framework can also integrate per-fixation attributes, such as individual fixation durations, alongside spatial locations. The autoregressive alignment enables the scalable, exact computation of per-fixation log-likelihoods, directly equivalent to the commonly used Information Gain (IG) metric. Our model, DeepGaze3.5-VL, establishes a new state-of-the-art across multiple datasets, achieving 2.18 bits of IG on MIT1003, a 46% improvement over DeepGaze III. This advantage persists even when baselines use identical high-capacity vision encoders. Beyond predictive performance, our generative framework serves as a powerful computational tool for direct behavioral interventions, allowing for controlled in-silico simulations that would be experimentally difficult or impossible to conduct in vivo. We demonstrate this ability by performing controlled interventions on the durations of pre-saccadic fixations, recovering known oculomotor phenomena purely from data.

Susmit Agrawal, Matthias Bethge, Matthias Kümmerer

University of Tübingen · Tübingen AI Center · IMPRS-IS

Visual Attention Scanpath Prediction Attention Mechanisms Autoregressive Modeling

Ask PDF

Abstract

Active Visual Perception

Explore similar work

Why We Look Where We Look: Emergent Human-like Fixations of a Foveated Visual Language Model Maximizing Scene Understanding

Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers

DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction