Organizations: Institut de Neurosciences de la Timone, Aix-Marseille Université, CNRS, Marseille, 13005, France · Laboratoire d’Électronique, Antennes et Télécommunications, Université Côte d’Azur, CNRS, Sophia Antipolis, 06903, France
Human vision achieves remarkable perceptual performance while operating under strict metabolic constraints. A key ingredient is the selective attention mechanism, driven by rapid saccadic eye movements that constantly reposition the high-resolution fovea onto task-relevant locations, unlike conventional AI systems that process entire images with equal emphasis. Our work aims to draw inspiration from the human visual system to create smarter, more efficient image processing models. Using DINO, a self-supervised Vision Transformer that produces attention maps strikingly similar to human gaze patterns, we explore a saccade inspired method to focus the processing of information on key regions in visual space. To do so, we use the ImageNet dataset in a standard classification task and measure how each successive saccade affects the model's class scores. This selective-processing strategy preserves most of the full-image classification performance and can even outperform it in certain cases. By benchmarking against established saliency models built for human gaze prediction, we demonstrate that DINO provides superior fixation guidance for selecting informative regions. These findings highlight Vision Transformer attention as a promising basis for biologically inspired active vision and open new directions for efficient, neuromorphic visual processing.
Figures & tables
Figure 1: Schematic illustration of attention map extraction within a single transformer layer. ( a. ) The input set of token embeddings, including the additional [CLS] token, is passed trough the multiple attention heads for parallel processing. ( b. ) Within each head, the tokens are projected through learned linear layers to obtain Query, Key, and Value representations (Value projections omitted in the figure for clarity). ( c. ) Attention weights are computed as the scaled dot product between Queries and Keys (as depicted in the bottom pink frame). The last column of the resulting attention matrix corresponds to the [CLS] token, and yields a vector of attention scores indicating the relative importance assigned to each image patch. ( d. ) This vector is reshaped into a 2-D grid to recover the spatial layout of the original image. This operation outputs one spatial attention map per head.
Figure 2: Saccade selection method: ( a. ) The input image of dimension H×W is split into 16H×nW sized patches and embedded into token vectors. ( b. ) The tokens are passed through the DINO transformer, and attention flow from patch tokens to [CLS] token (white arrows) are extracted and reshaped into one attention map per attention-head. ( c. ) The multiple attention maps are fused into one by taking the maximum value across heads. ( d. ) The highest-attention locations define square regions ( “saccades” ) whose tokens are retained. ( e. ) Selected regions are revealed sequentially, and the image variants are classified by a pre-trained linear head. Image adapted from Wikimedia Commons [ 31 ]
Figure 3: Accuracy score across saccades: For a fovea of size 3x3 tokens ( a. ), 5x5 tokens ( b. ), and the corresponding percentage of the image given to the model throughout saccades ( c. ). The accuracy score when the model is given the full image is shown as a dotted red line. The light curves correspond to the random saccades in every plot.
Figure 4: Histogram of classification dynamics across saccades: a. and b. Cumulative accuracy (percentage of images correctly classified at least once), with the proportion of images that turned correct or incorrect at each saccade. c. and d. . Percentages of images that changed classification at this specific saccade (stacked bars) (Correct → Wrong and Wrong → Correct) and the proportions that have remained incorrect (Stayed Wrong) or remained recovered to correct (Stayed Right). e. and f. Percentage of images distributed according to the number of occurrence of correct classification across saccades, over the full validation dataset, and over the subset of images that were correctly classified during saccades but not when the image was given full ( g. and h. ). The percentage correspond to the percentage over the whole validation dataset in every case. a. , c. , e. and g. : 3×3 tokens fovea. b. , d. , f. and h. : 5×5 tokens fovea
Figure 5: Evolution of the model’s classification certainty across saccades: Curves represent the mean certainty for attention driven ( a. ) and random saccades ( b. ). Colors indicate sets of images correctly classified at a given saccade (dark to light blue: first to tenth saccade). The red dotted line shows the mean certainty of all images, across saccade. Colored bars in the rectangles on the right of each plot shows mean certainty when the full image is given to the model, for each set.
Figure 6: Saccade accuracies for attention maps across layers and input resolutions: Accuracy scores for attention maps from different layers ( a ) and the corresponding percentage of the image revealed during saccades ( b ). Accuracy scores for attention maps from different input resolutions ( c ) and their corresponding revealed pixels ( d ). Red dotted lines indicate results for random fixations.
Figure 7: Entropies of attention maps grouped by the saccade leading to a correct classification : The first column (Saccade 0) shows images correctly classified when given full to the model, but not during saccades. The orange line indicates the mean entropy across saccades. The lower panel shows the number of images processed at each saccade. ( a. attention-driven fixations. b. randomly selected fixations.)
Figure 8: Examples of images correctly classified at different numbers of saccades, with their corresponding attention maps: Upper and lower rows display low- and high-entropy attention maps, respectively. Saccade order is coded as a colored square from light green (first) to dark blue (tenth); fixations leading to correct classification are highlighted. Images adapted from Wikimedia Commons [ 1 , 47 , 100 , 64 , 85 , 54 , 3 , 79 , 41 , 57 , 68 , 6 , 33 , 92 , 89 , 90 , 2 , 58 , 78 , 88 ]
Figure 9: Examples of saliency maps generated by the different models and their corresponding saccade patterns. Images adapted from Wikimedia Commons [ 76 , 51 , 73 ]
Figure 10: Effect of saliency maps generated by the different models: Accuracy scores of the ResNet50 classifier using saliency maps from different models, as well as random and centered fixations ( a ). Accuracy scores of the DINO classifier, over multiple saccades driven by these attention maps ( b ) and the corresponding percentage of the image revealed during saccades ( c ).
Figure 11: Distances between fixations coordinates across saccades: Between saccades generated by different models ( a. ), between the first fixation point and the following saccades ( b. ), and between a given saccade and the preceding one ( c. ).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Examples of the effect of input resolution on the attention patterns. This figure illustrate the attention maps ability to identify informative regions even from low-resolution input. Once obtained, the low-resolution attention maps are resized to a 14×14 grid through bilinear interpolation, and are no longer as coarse. All images are from Wikimedia Commons under CC BY or CC BY-SA licenses [ 20 , 34 , 83 , 30 , 67 , 8 , 74 , 4 ] .
Figure 13: Attention maps across layers, from first to twelfth layer (left to right respectively). All images are from Wikimedia Commons under CC BY or CC BY-SA licenses [ 28 , 72 , 56 , 50 , 42 ] .
Figure 14: Accuracy score over multiple saccades driven by attention maps from different models, for fovea of size 5×5 tokens ( a. ) and the corresponding percentage of the image given to the model throughout saccades ( b. )
Figure 15: Accuracy in CNN classification for 80×80 pixels crops ( 5.8% of the original image) ( a. ). Mean certainty in CNN classification across condition for 48×48 pixels crops ( b. ) and 80×80 pixels crops ( c. ).
Figure 16: Evolution of the model’s classification certainty across saccades: Curves represent the mean certainty across the different saliency models, and the random fixation condition. Colors indicate sets of images correctly classified at a given saccade (dark to light blue: first to tenth saccade). The red dotted line shows the mean certainty of all images, across saccade. Colored bars in the rectangles on the right of each plot shows mean certainty when the full image is given to the model, for each set. For fovea of size 3×3 tokens ( a. ) and 5×5 token ( b. )
Figure 17: Accuracy score across saccades for attention maps of low-resolution 128×128 pixel input, taken at different layers ( a. ), and the corresponding percentage of the image given to the model throughout saccades ( b. ).
Optimizing vision models purely for classification accuracy can impose an alignment tax, degrading human-like scanpaths and limiting interpretability. We introduce EVA, a neuroscience-inspired hard-attention mechanistic testbed that makes the performance-human-likeness trade-off explicit and adjustable. EVA samples a small number of sequential glimpses using a minimal fovea-periphery representation with CNN-based feature extractor and integrates variance control and adaptive gating to stabilize and regulate attention dynamics. EVA is trained with the standard classification objective without gaze supervision. On CIFAR-10 with dense human gaze annotations, EVA improves scanpath alignment under established metrics such as DTW, NSS, while maintaining competitive accuracy. Ablations show that CNN-based feature extraction drives accuracy but suppresses human-likeness, whereas variance control and gating restore human-aligned trajectories with minimal performance loss. We further validate EVA's scalability on ImageNet-100 and evaluate scanpath alignment on COCO-Search18 without COCO-Search18 gaze supervision or finetuning, where EVA yields human-like scanpaths on natural scenes without additional training. Overall, EVA provides a principled framework for trustworthy, human-interpretable active vision.
Vision Transformers (ViTs) have demonstrated remarkable performance in computer vision tasks. However, their self-attention mechanism often diffuses focus across background regions, relying on spurious correlations rather than object-relevant cues. Inspired by inhibitory mechanisms observed in biological vision systems, we propose the Inhibited Self-Attention (ISA), a novel self-attention that integrates inhibitory signals to enhance feature selectivity and suppress spurious responses. In contrast to conventional self-attention, which relies solely on positive attention values due to softmax normalization, our approach retains and utilizes negative attention scores to suppress irrelevant features and sharpen focus on objects of interest. Experiments across multiple datasets, including ImageNet-1k and COCO, and several robustness benchmarks demonstrate that ISA enhances object-centric selectivity, reduces shortcut reliance, and improves out-of-distribution generalization. Our analysis of relevance maps confirms that ViTs with ISA exhibit sharper, more localized focus on object-relevant regions while reducing distractions from non-relevant (background) features, enabling more reliable models. We release our code at https://github.com/prdvanderwal/inhibited-self-attention
Peter R. D. van der Wal, Nicola Strisciuglio, George Azzopardi
University of Groningen · University of Twente · 3Stellenbosch University +1
For state-of-the-art image understanding, Vision Transformers (ViTs) have become the standard architecture but their processing diverges substantially from human attentional characteristics. We investigate whether this cognitive gap can be shrunk by fine-tuning the self-attention weights of Google's ViT-B/16 on human saliency fixation maps. To isolate the effects of semantically relevant signals from generic human supervision, the tuned model is compared against a shuffled control. Fine-tuning significantly improved alignment across five saliency metrics and induced three hallmark human-like biases: tuning reversed the baseline's anti-human large-object bias toward small-objects, amplified the animacy preference and diminished extreme attention entropy. Bayesian parity analysis provides decisive to very-strong evidence that this cognitive alignment comes at no cost to the model's original classification performance on in- (ImageNet), corrupted (ImageNet-C) and out-of-distribution (ObjectNet) benchmarks. An equivalent procedure applied to a ResNet-50 Convolutional Neural Network (CNN) instead degraded both alignment and accuracy, suggesting that the ViT's modular self-attention mechanism is uniquely suited for dissociating spatial priority from representational logic. These findings demonstrate that biologically grounded priors can be instilled as a free emergent property of human-aligned attention, to improve transformer interpretability.