cs.CVMar 28, 2026
SaveEVA: Bridging Performance and Human Alignment in Hard-Attention Vision Models for Image Classification
Organizations: The University of Tokyo
Abstract
Optimizing vision models purely for classification accuracy can impose an alignment tax, degrading human-like scanpaths and limiting interpretability. We introduce EVA, a neuroscience-inspired hard-attention mechanistic testbed that makes the performance-human-likeness trade-off explicit and adjustable. EVA samples a small number of sequential glimpses using a minimal fovea-periphery representation with CNN-based feature extractor and integrates variance control and adaptive gating to stabilize and regulate attention dynamics. EVA is trained with the standard classification objective without gaze supervision. On CIFAR-10 with dense human gaze annotations, EVA improves scanpath alignment under established metrics such as DTW, NSS, while maintaining competitive accuracy. Ablations show that CNN-based feature extraction drives accuracy but suppresses human-likeness, whereas variance control and gating restore human-aligned trajectories with minimal performance loss. We further validate EVA's scalability on ImageNet-100 and evaluate scanpath alignment on COCO-Search18 without COCO-Search18 gaze supervision or finetuning, where EVA yields human-like scanpaths on natural scenes without additional training. Overall, EVA provides a principled framework for trustworthy, human-interpretable active vision.
Figures & tables
Figure 1: EVA reduces the alignment tax through mechanism-aware design. Left : a schematic trade-off between accuracy and scanpath similarity to humans. EVA improves the trade-off relative to full-image baselines and prior hard-attention models. Right : EVA uses minimal fovea–periphery sensing with variance control and adaptive gating to regulate active sampling dynamics.
Figure 2: EVA model architecture. Top : one glimpse step at time . EVA combines a minimal fovea–periphery retina with a CNN feature extractor and a two-level recurrent backbone. The lower recurrent state updates the glimpse representation and controls the next fixation. An adaptive gate regulates information flow to the upper recurrent classifier. Bottom : a prediction-error signal modulates the fixation variance and the gate, regulating evidence acquisition dynamics. The components are motivated by functional motifs rather than biological fidelity.
| Model | Params | Time | FLOPs | Acc. | DTW | SM | NSS | AUC | GCS |
| (M) | (ms/im) | (B)↓ | (%) ↑ | ↓ | ↑ | ↑ | ↑ | ↑ | |
| CNN (ResNet18) | 11.18 | 0.89 ± 0.02 | 0.34 | 78.00 | - | - | - | - | - |
| CNN (MobileNetV3) | 4.21 | 0.94 ± 0.08 | 0.03 | 78.52 | - | - | - | - | - |
| TinyViT | 4.37 | 3.03 ± 0.22 | 2.66 | 68.21 | - | - | - | - | - |
| DeepGaze IIE | 0.2 | 0.82 ± 0.03 | - | - | 705.48 | 0.311 | 0.251 | 0.601 | 0.039 |
| Gazeformer | 0.27 | 0.04 ± 0.01 | - | - | 669.22 | 0.346 | 0.757 | 0.689 | 0.068 |
Table 1: Mechanistic analysis of the accuracy–alignment trade-off on CIFAR-10. The table is organized into blocks separated by horizontal rules. Bold denotes the best value within each block for a given metric. Light blue marks two EVA variants. EVA-Mobile uses a pretrained MobileNetV3 backbone.
Figure 3: Qualitative comparison of scanpaths on CIFAR-10. Columns show different models and rows show example images. Orange denotes the model scanpath and blue denotes the human scanpath.
Figure 4: Accuracy–alignment trade-off on CIFAR-10. The x-axis is center-debiased GCS and the y-axis is classification accuracy.
Figure 5: ImageNet-100 scalability. Top: ImageNet-100 accuracy and FLOPs. Pre. indicates initialization from ImageNet-1K pretrained weights. Bottom: an example scanpath from EVA where the red rectangle indicates the foveal crop.
| Model | COCO | COCO-Search18 | DTW | SM | NSS | AUC |
| Acc.% | Acc.% | ↓ | ↑ | ↑ | ↑ | |
| CNN MobileNet (pre.) | 58.82 | 27.5 | - | - | - | - |
| Gazeformer | - | - | 168.39 | 0.571 | 1.961 | 0.8 |
| Saccader-Mobile (pre.) | 57.1 | 16.7 | 333 | 0.242 | 0.361 | 0.658 |
| RAM | 34.81 | 12.83 | 500.01 | 0.072 | -0.132 | 0.585 |
| DRAM | 43.32 | 14.34 | 530.78 | 0.077 | -0.124 | 0.587 |
Table 2: COCO-Search18: scanpath metrics under gaze-supervision-free, cross-task evaluation. EVA is trained on COCO classification labels only and evaluated against human search-conditioned scanpaths without finetuning.
Figure 6: PCA of EVA recurrent states on CIFAR-10 using class-wise mean trajectories over glimpse steps. The upper-layer state separates classes more clearly than the lower-layer state, consistent with category-level evidence integration in and sampling-control dynamics in .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Sequential evidence accumulation in EVA on a representative CIFAR-10 sample. The top row shows the cumulatively observed foveal evidence over glimpse steps, and the bottom row shows the corresponding peripheral evidence. The predicted label is shown at each step, illustrating how the model’s hypothesis evolves as additional evidence is acquired.
Figure 8: Comparison between static saliency and scanpath-based summaries on a representative sample. (a) shows a conventional saliency map computed by a computer-vision method. (b) shows a fixation-density map reconstructed from human scanpaths. (c) shows a fixation-density map reconstructed from repeated EVA scanpaths. (d) overlays the human and EVA scanpaths on the image. The comparison highlights that static saliency and scanpath-derived summaries are related but not equivalent: saliency summarizes spatial importance, whereas scanpaths expose the sequential process of evidence acquisition.
Figure 9: CNN module used in EVA. In EVA-Large, the lightweight CNN width is increased from 64-128-256-256 to 96-192-384-384, while the remaining architectural components are kept unchanged.
| Model | Gaze Policy | ||||
| Predicted | Center-fixed | Corner-fixed | Random | Shuffled | |
| RAM | 61.6 | 45.1 (-16.5) | 12.8 (-48.8) | 43.5 (-18.1) | 61.3 (-0.3) |
| DRAM | 62.2 | 41.6 (-20.6) | 11.6 (-50.6) | 38.4 (-23.8) | 61.9 (-0.3) |
| MRAM | 58.3 | 44.1 (-14.2) | 13.6 (-44.7) | 48.7 (-9.6) | 56.9 (-1.4) |
| EVA | 79.8 | 57.6 (-22.2) | 19.6 (-60.2) | 70.5 (-9.3) | 71.3 (-8.5) |
Table 3: Accuracy (%) of hard-attention models under different gaze policies on CIFAR-10. The Predicted column reports the learned policy; the remaining columns replace or perturb the gaze trajectory. Numbers in parentheses indicate the accuracy drop relative to Predicted .
Figure 10: Sensitivity of a fixed EVA checkpoint to test-time initialization. Top: different initialization conditions induce different levels of early exploration, as measured by first-step saccade amplitude and spatial coverage. Bottom: across the tested initialization conditions, broader early exploration is associated with higher classification accuracy. These plots are intended as diagnostic summaries rather than as a statistical claim about population-level correlation.
Figure 11: Representative examples of successful evidence revision in EVA. In (a), the model initially favors ship but switches to airplane once later glimpses reveal more diagnostic object structure. In (b), the model briefly drifts toward ship under partial evidence, then returns to the correct automobile hypothesis after more of the vehicle body becomes visible. These examples show that EVA’s prediction is shaped by the temporal order in which evidence is acquired.
Figure 12: Representative failure cases of EVA. In (a), the model eventually commits to truck after observing evidence sufficient for a coarse vehicle category but insufficient for finer discrimination. In (b), the model revises from dog to cat and then remains trapped in that hypothesis, suggesting that later glimpses reinforce a locally plausible but ultimately misleading interpretation. These failures remain interpretable at the level of the scanpath and the accumulated evidence.
Figure 13: Same image under two different stochastic rollouts. In (a), EVA initially fluctuates among several hypotheses but eventually converges to the correct airplane class once later glimpses reveal more diagnostic structure. In (b), a different rollout remains unstable under ambiguous local evidence and ultimately converges to the incorrect dog class. This example illustrates that different scanpaths can induce different evidence-acquisition scanpaths and therefore different prediction outcomes.