PPS: Plug-and-Play Saccadic Vision for Fine-Grained Classification

Johann Schmidt; Sebastian Stober; Joachim Denzler; Paul Bodesheim

cs.CVSep 19, 2025

PPS: Plug-and-Play Saccadic Vision for Fine-Grained Classification

Authors: Johann Schmidt, Sebastian Stober, Joachim Denzler, Paul Bodesheim

Organizations: Artificial Intelligence Lab Otto-von-Guericke University Magdeburg Germany · Computer Vision Group Friedrich Schiller University Jena Germany

Abstract

Fine-grained visual classification benefits from high-resolution inputs to resolve subtle, localized cues, but at the cost of large compute and memory budgets. Inspired by human saccadic vision, we propose a lightweight extension that turns any pretrained backbone with spatial feature maps into a saccadic classifier through glimpse-induced inference. The only architectural additions are a small priority head and a single cross-attention fusion block. The same weight-tied encoder encodes a downsampled peripheral glance, exposes the spatial feature map from which the priority head predicts a glimpse-priority distribution and re-encodes high-resolution glimpse crops sampled from that distribution in parallel. A sequential sampler with non-maximum suppression draws a runtime-configurable number of glimpses. A residual multi-head cross-attention block then fuses the glance against the glimpses. To improve the quality of the priority maps, we train the priority head by a per-glimpse information-gain target from a lightweight per-image memory bank. This requires no reinforcement learning. Complexity is linear in the number of glimpses and peak memory is decoupled from input resolution. Across nine benchmarks our framework improves consistently over its backbones.

Figures & tables

Figure 1: We reuse the backbone encoder to capture a glance, generate a priority map and encode sampled glimpses. Glance and glimpse representations are fused for prediction.

Figure 2: Visual side-by-side comparison of related glimpse-based approaches covering RAM [ 36 ] , RA-CNN [ 14 ] , GFNet [ 20 ] , DPS [ 7 ] , NTS-Net [ 56 ] , Saccader [ 13 ] and our saccadic vision extension. The sharing of encoders, additional localization networks for glimpse localization, and the classifier are highlighted by different colors. Tags are used to highlight training based on reinforcement learning, whether glimpse encoding is parallelized (or sequential) and whether the approach is agnostic to the input resolution, the number of glimpses or the choice of backbone.

Figure 3: Overview of PPS. A glance $\mathbf{x}_{L}$ is obtained by downsampling the input $\mathbf{x}$ . The backbone $\phi$ encodes it into spatial feature maps $\mathcal{F}$ and a glance embedding $\mathbf{z}_{L}$ . A lightweight priority head $\psi_{\text{prio}}$ turns $\mathcal{F}$ into a priority map $\mathbf{P}$ , read as a spatial distribution over locations. From $\mathbf{P}$ we sample $N$ glimpse locations $\mathcal{O}$ and crop the corresponding patches $\mathcal{X}_{G}$ from the high resolution source $\mathbf{x}$ . The same (weight-tied) backbone $\phi$ encodes each patch into a glimpse embedding, giving $\{\mathbf{z}_{G}\}$ . A cross-attention block $\psi_{\text{fusion}}$ fuses the glimpse set with the glance $\mathbf{z}_{L}$ , and the head $\phi_{\text{head}}$ maps the result to class logits $\mathbf{z}$ .

Figure 4: The priority progression over $N$ glimpses using our NMS with Gaussian kernels on $\mathbf{P}$ . Left an example progression for two different resolutions of $\mathbf{P}$ and on the right the pseudo-code of our sampling method.

Figure 5: The training process of PPS with focus on our auxiliary priority loss.

Figure 6: Test samples from NABirds and ECU with results obtained by our PPS-MPSA-SwinB. From left to right the columns show the input image $\mathbf{x}_{L}$ , MPSA’s multi-granularity feature map (generally $\mathcal{F}$ ), our priority map $\mathbf{P}$ , and two glimpses at $\mathbf{o}_{1}$ and $\mathbf{o}_{2}$ . Annotated on the left is the prediction delta induced by the glimpses compared to the glance.

Method	Backbone	Pretr.	Res.	ECU	EU	WE	NA	CA
Swin-B Liu et al. (2021)	-	Im22k	224	75.41	97.06	94.03	97.57	98.98
MPSA Wang et al. (2024)	SwinB	Im22k	224	77.04	97.64	94.17	97.68	99.07
PPS-MPSA (ours)	SwinB	Im22k	224	80.89	98.18	94.44	97.76	99.08
SPPs-Cond. Korsch et al. (2022)	Inc.V3	Im1k	299	-	91.50	-	-	-
SPPs-Cond. Korsch et al. (2022)	Inc.V3	iNat	299	-	93.13	-	-	-
Swin-B Liu et al. (2021)	-	Im22k	384	77.19	97.10	94.84	97.95	99.02

Table 1: Average top-1 test accuracy on FGVC moth benchmarks.

	Method	Backbone	Pretr.	Acc@1
224 px	RA-CNN Fu et al. (2017)	VGG19	Im1k	85.3
	TASN Zheng et al. (2019)	ResNet50	Im1k	87.0
	NTSN Yang et al. (2018)	ResNet50	Im1k	87.3
	MPSA Wang et al. (2024)	SwinB	Im22k	91.4
	PPS-MPSA (ours)	SwinB	Im22k	91.5
384 px	CNB Woo et al. (2023)	-	Im22k	90.2

Table 2: Average top-1 test accuracy on CUB.

	Method	Backbone	Pretr.	Acc@1
224 px	MetaFormer2 Diao et al. (2022)	-	Im1k	89.7
	MPSA Wang et al. (2024)	SwinB	Im22k	90.2
	PPS-MPSA (ours)	SwinB	Im22k	90.4
384 px	MetaFormer2 Diao et al. (2022)	-	Im22k	92.2
	MPSA Wang et al. (2024)	SwinB	Im22k	92.4
	PPS-MPSA (ours)	SwinB	Im22k	92.5

Table 3: Avg. top-1 test accuracy on NABirds.

	Method	Backbone	Pretr.	Acc@1
224 px	RA-CNN Fu et al. (2017)	VGG-16	Im1k	87.3
224 px	MPSA Wang et al. (2024)	SwinB	Im22k	94.1
	PPS-MPSA (ours)	SwinB	Im22k	94.2
384 px	MPSA Wang et al. (2024)	SwinB	Im22k	94.7
	PPS-MPSA (ours)	SwinB	Im22k	94.9
	PPS-MPSA † (ours)	SwinB	Im22k	95.1

Table 4: Avg. top-1 test acc. on Stanford-Dogs. With † being PPS trained with

N_{\text{train}}=4

Method	Backbone	Pretr.	Res.	Acc@1
FAL-ViT Huang et al. (2025)	ViT-B16	Im1k	448	91.8
MPSA Wang et al. (2024)	SwinB	Im22k	224	93.7
PPS-MPSA (ours)	SwinB	Im22k	224	94.0

Table 5: Avg. top-1 test acc. on Food-101.

	Method	SwinB	MPSA
$\mathbf{P}$	upsampled	90.4	91.4
$\mathbf{P}$	native (ours)	90.7	91.5
$\mathcal{L}$	REINFORCE Elsayed et al. (2019)	90.5	91.4
	only $\mathcal{L}_{\text{task}}$ (Gumbel)	90.6	91.4
	Eq. 8 (ours)	90.7	91.5

Table 6: Avg. top-1 test accuracy on CUB with a priority map and loss ablation.

Sampling	$\mathcal{F}$	$\mathbf{o}\sim\mathbf{F}$		$\mathbf{o}\sim\mathbf{P}$
Sampling	$\mathcal{F}$	SwinB	MPSA	SwinB	MPSA
Random	last	90.3	91.4	90.8	91.3
TopK	last	90.2	91.4	90.5	91.3
NMS	full	90.2	91.3	90.4	91.4
NMS	mask	90.6	91.4	90.3	91.3
NMS	last	90.0	91.5	90.7	91.4

Table 7: Avg. top-1 test accuracy on CUB using a SwinB with and without MPSA. The sampler’s input feature map and the sampler itself are varied.

$N_{\text{test}}$	EU	CUB
2	98.2	91.4
4	98.2	91.5
8	98.2	91.5

Table 8: Avg. top-1 test acc. for MPSA with varying glimpses.

Method	Sampler	Fusion	$N_{\text{train}}$	$N_{\text{test}}$	MPSA
Vanilla	-	-	0	0	96.0
DA	Rnd	Avg	4	0	97.1
TTA	Rnd	Avg	0	4	96.0
DA+TTA	Rnd	Avg	4	4	96.4
PPS (ours)	NMS	Attn	4	4	98.0

Table 9: Avg. top-1 test accuracies on EU between our PPS, (training-time) data augmentation (DA), test-time augmentation (TTA), and a combination of both. DA and TTA use random sampling of target object patches and signal averaging.

Figure 7: Performance and peak GPU memory over input resolutions on EU.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	Variant	#P (M) ( $\downarrow$ )	CUB ( $\uparrow$ )	EU ( $\uparrow$ )
CNB	vanilla	87.9	90.2	96.2
	+ ConvNet (#P-matched)	100.6	90.3	95.3
	+ PPS (ours)	101.0	90.5	96.2
	w/ 2 fusion layers	111.5	89.8	96.9
SwinB	+ PPS (ours)	100.8	90.9	97.1
	w/ 2 fusion layers	110.5	89.8	96.9

Appendix

Table 10: Glimpse encoder and fusion ablation of PPS with a ConvNeXtv2-Base (CNB) and a SwinB backbone. Compared are the number of parameters (#P) in millions (M) and the avg. top-1 test accuracies on CUB and EU.

Model	Prio Enc.	CUB ( $\uparrow$ )	EU ( $\uparrow$ )	PiB CUB ( $\uparrow$ )
SwinB	Ours	90.9	97.1	97.7%
SwinB	CAM	90.7	97.1	88.5%
SwinB	Attention	90.8	96.9	87.9%
SwinB	Spectral	90.4	96.9	47.2%

Appendix

Table 11: Ablation of priority encoders. Alongside average top-1 test accuracy we report peak-in-box (PiB). This is the fraction of images whose hottest priority cell falls inside the ground truth box (flatness-robust), using CUB meta data.

PPS	EU (3 seeds)	CUB (3 seeds)	NAB (3 seeds)
SwinB	97.1 $\pm$ 0.05	90.9 $\pm$ 0.03	90.5 $\pm$ 0.08
MPSA+SwinB	98.2 $\pm$ 0.08	91.6 $\pm$ 0.1	90.4 $\pm$ 0.07

Appendix

Table 12: Performance stability under differently seeded runs.

Model	Data	$s=0.15$	$s=0.25$	$s=0.35$
SwinB+PPS	CUB	90.4	90.9	90.8
SwinB+PPS	NAB	89.4	90.5	89.6

Appendix

Table 13: Glimpse crop size sweep.

Fine-Grained Image Classification

Fine-grained image classification distinguishes visually similar categories, such as species, product varieties, and medical findings.

See all 88 papers

A Saccade-inspired Approach to Image Classification using Vision Transformer Attention Maps
Mar 10, 2026Matthis Dallain, Laurent Rodriguez, Laurent Udo Perrinet +1Active Vision Foveated Vision
Institut de Neurosciences de la Timone, Aix-Marseille Université, CNRS, Marseille, 13005, France · Laboratoire d’Électronique, Antennes et Télécommunications, Université Côte d’Azur, CNRS, Sophia Antipolis, 06903, France
LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models
Feb 26, 2026Yuxiang Shen, Hailong Huang, Zhenkun Gao +6VLM Reasoning Large Vision-Language Models
the Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China, and also with the University of Science and Technology of China, Hefei 230026, China · Zhejiang University, Hangzhou 310027, China · East China Normal University, Shanghai 200241, China +3
SAGE: Saliency-Annealed Gated Encoder for Universal Vision Recognition
Oct 27, 2026Le Yang, Yucheng Huang, Junyu Chen +4Vision Transformer Vision Foundation Model Adaptation
Huawei Technologies Co., Ltd.

Mar 10, 2026J/K move · Enter open · S save

A Saccade-inspired Approach to Image Classification using Vision Transformer Attention Maps

Matthis Dallain, Laurent Rodriguez, Laurent Udo Perrinet, Benoît Miramond

Institut de Neurosciences de la Timone, Aix-Marseille Université, CNRS, Marseille, 13005, France · Laboratoire d’Électronique, Antennes et Télécommunications, Université Côte d’Azur, CNRS, Sophia Antipolis, 06903, France

Active Vision Foveated Vision Visual Attention Active Visual Perception Active Perception Image Classification

Open paper AskPDF arXiv

Human vision achieves remarkable perceptual performance while operating under strict metabolic constraints. A key ingredient is the selective attention mechanism, driven by rapid saccadic eye movements that constantly reposition the high-resolution fovea onto task-relevant locations, unlike conventional AI systems that process entire images with equal emphasis. Our work aims to draw inspiration from the human visual system to create smarter, more efficient image processing models. Using DINO, a self-supervised Vision Transformer that produces attention maps strikingly similar to human gaze patterns, we explore a saccade inspired method to focus the processing of information on key regions in visual space. To do so, we use the ImageNet dataset in a standard classification task and measure how each successive saccade affects the model's class scores. This selective-processing strategy preserves most of the full-image classification performance and can even outperform it in certain cases. By benchmarking against established saliency models built for human gaze prediction, we demonstrate that DINO provides superior fixation guidance for selecting informative regions. These findings highlight Vision Transformer attention as a promising basis for biologically inspired active vision and open new directions for efficient, neuromorphic visual processing.

Abstract

Fine-Grained Image Classification

Explore similar work

A Saccade-inspired Approach to Image Classification using Vision Transformer Attention Maps

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

SAGE: Saliency-Annealed Gated Encoder for Universal Vision Recognition