Vision Hopfield Memory Networks for Image Recognition

Jianfeng Wang; Amine M'Charrak; Luk Koska; Xiangtao Wang; Daniel Petriceanu; Ruizhi Wang; Michael Bumbar; Luca Pinchetti

cs.LGMar 26, 2026

Vision Hopfield Memory Networks for Image Recognition

Authors: Jianfeng Wang, Amine M'Charrak, Luk Koska, Xiangtao Wang, Daniel Petriceanu, Ruizhi Wang, Michael Bumbar, Luca Pinchetti, +1 more

Organizations: Department of Computer Science, University of Oxford, United Kingdom · Faculty of Informatics, Vienna University of Technology, Austria

Abstract

Recent vision backbones, such as Transformer families and state-space models like Mamba, have achieved remarkable progress on image recognition. Despite their empirical success, these architectures remain far from the computational principles of the human brain, often demanding enormous amounts of training data while offering limited interpretability. We propose the Vision Hopfield Memory Network (V-HMN), a brain-inspired vision backbone that integrates hierarchical memory mechanisms across layers with iterative refinement updates. Specifically, V-HMN incorporates local Hopfield modules that provide associative memory dynamics at the image patch level, global Hopfield modules that function as episodic memory for contextual modulation, and a predictive-coding-inspired refinement rule for iterative error correction. By organizing these memory-based modules hierarchically, V-HMN captures both local and global dynamics in a unified framework. Memory retrieval exposes the relationship between inputs and stored patterns, providing a prototype-based form of interpretability through explicit memory retrieval, while the reuse of stored patterns improves data efficiency. This brain-inspired design therefore enhances data efficiency and provides a prototype-based form of interpretability compared to existing self-attention- or state-space-based approaches. We conducted extensive experiments on public image classification benchmarks. V-HMN achieves strong performance on small- and medium-scale benchmarks, and remains competitive with widely adopted backbone architectures on ImageNet despite minimal architectural tuning, while offering improved data efficiency and a prototype-based form of interpretability. These findings highlight the potential of V-HMN as a memory-centric alternative to standard vision backbones, thereby bridging brain-inspired computation with modern machine learning.

Figures & tables

Figure 1: Overview of V-HMN. (a) Each HMN block refines features through local and global Hopfield memory retrieval, rather than convolution or self-attention. (b) A deep backbone is constructed by stacking HMN blocks, with attention pooling and a linear head for image classification.

Fraction of training data	CIFAR-10	CIFAR-100	Fashion-MNIST
10%	$80.22_{\pm 0.29}$	$43.21_{\pm 1.07}$	$89.18_{\pm 0.16}$
30%	$88.67_{\pm 0.21}$	$62.42_{\pm 0.29}$	$91.04_{\pm 0.22}$
50%	$91.19_{\pm 0.38}$	$68.93_{\pm 0.35}$	$91.53_{\pm 0.12}$

Table 1: Data efficiency of V - HMN on CIFAR-10, CIFAR-100, and Fashion-MNIST with different fractions of labeled training samples. Reported values are top-1 test accuracy (%).

Model	CIFAR-10		CIFAR-100		Fashion-MNIST
Model	10%	30%	10%	30%	10%	30%
ViT	$72.73_{\pm 0.42}$	$83.94_{\pm 0.33}$	$40.48_{\pm 0.80}$	$57.40_{\pm 0.79}$	$87.17_{\pm 0.40}$	$89.71_{\pm 0.45}$
Swin-ViT	$70.37_{\pm 0.94}$	$80.89_{\pm 0.24}$	$35.25_{\pm 0.59}$	$51.77_{\pm 0.60}$	$88.42_{\pm 0.48}$	$90.34_{\pm 0.15}$
MLP-Mixer	$76.14_{\pm 0.16}$	$85.53_{\pm 0.40}$	$41.94_{\pm 0.98}$	$56.22_{\pm 0.75}$	$87.16_{\pm 0.63}$	$89.41_{\pm 0.13}$
MetaFormer	$49.92_{\pm 4.08}$	$60.86_{\pm 6.05}$	$19.39_{\pm 1.26}$	$37.47_{\pm 3.19}$	$85.02_{\pm 1.22}$	$88.78_{\pm 1.03}$
Vim	$69.02_{\pm 2.13}$	$79.58_{\pm 0.37}$	$36.61_{\pm 1.25}$	$49.67_{\pm 0.79}$	$86.09_{\pm 1.03}$	$88.18_{\pm 2.43}$

Table 2: Comparison of data efficiency across models with 10% and 30% labeled training data on CIFAR-10, CIFAR-100, and Fashion-MNIST. Top-1 test accuracy (%) are reported as mean

\pm

standard deviation over 3 seeds. Baselines include ViT ( Dosovitskiy et al., 2021 ) , Swin-ViT ( Liu et al., 2021 ) , MLP-Mixer ( Tolstikhin et al., 2021 ) , MetaFormer ( Yu et al., 2024 ) , Vim ( Zhu et al., 2024 ) , and AiT ( Sun et al., 2025 ) .

Model	CIFAR-10	CIFAR-100	SVHN	FashionMNIST	Params (M)
ViT	$91.66_{\pm 0.08}$	$72.56_{\pm 0.01}$	$96.11_{\pm 0.29}$	$91.83_{\pm 0.15}$	7.16
Swin-ViT	$87.94_{\pm 0.10}$	$66.58_{\pm 0.37}$	$95.89_{\pm 0.17}$	$91.79_{\pm 0.17}$	6.92
MLP-Mixer	$92.65_{\pm 0.26}$	$73.35_{\pm 0.39}$	$96.91_{\pm 0.06}$	$91.46_{\pm 0.32}$	8.71
MetaFormer	$75.40_{\pm 1.56}$	$48.76_{\pm 0.75}$	$86.54_{\pm 2.71}$	$91.42_{\pm 0.21}$	6.99
Vim	$82.83_{\pm 3.87}$	$63.29_{\pm 2.83}$	$86.63_{\pm 1.51}$	$88.63_{\pm 1.49}$	7.12
AiT	$92.97_{\pm 0.30}$	$72.91_{\pm 0.17}$	$95.98_{\pm 0.06}$	$91.51_{\pm 0.04}$	7.15

Table 4: Comparison of V-HMN with baseline models on CIFAR-10, CIFAR-100, SVHN, and Fashion-MNIST. Top-1 test accuracy (%) are reported as mean

\pm

Method	Image Size	#Params	Top-1 Acc.
ResNet-50 ( He et al., 2016 )	224	26M	76.1
SENet-50 ( Hu et al., 2018 )	224	28M	76.7
ViT-B/16 ( Dosovitskiy et al., 2021 )	384	86M	77.9
ViT-B/16 † ( Dosovitskiy et al., 2021 )	224	86M	76.2
ViT-L/16 ( Dosovitskiy et al., 2021 )	384	307M	76.5
MLP-Mixer-B/16 ( Tolstikhin et al., 2021 )	224	59M	76.4

Table 5: Comparison on ImageNet-1k. All results are Top-1 accuracy (%). (

\dagger

) indicates our re-implementation under the same setting for fair comparison.

Figure 3: Visualization of retrieved prototypes from local and global memory.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Setting	Sample memory at eval.	Standard Top-1	No local Top-1 drop / changed pred.	No global Top-1 drop / changed pred.	No both Top-1 drop / changed pred.	Shuffled mem. Top-1 drop / changed pred.	Random top- $k$ Top-1 drop / changed pred.
CIFAR-10	Clean labels	populated	91.11 $\pm$ 0.06	61.94/70.86	0.70/3.35	66.17/75.27	42.69/50.80	16.34/23.31
CIFAR-10	Strong aug.	not populated	93.67 $\pm$ 0.02	not applicable (sample memory not populated)
CIFAR-10	Strong aug.+clean mem.	populated	93.56 $\pm$ 0.25	1.24/3.95	0.05/0.28	1.23/3.98	14.69/19.32	23.71/28.84
CIFAR-100	Clean labels	populated	65.45 $\pm$ 0.26	15.91/41.70	5.70/27.46	19.95/48.49	22.61/51.20	13.45/39.36
CIFAR-100	Strong aug.	not populated	75.83 $\pm$ 0.11	not applicable (sample memory not populated)
CIFAR-100	Strong aug.+clean mem.	populated	75.86 $\pm$ 0.10	3.63/14.47	0.27/5.98	3.74/15.15	11.11/26.80	10.74/28.15

Appendix

Table 6: 400-epoch paired retrieval-disruption summary. Each row evaluates the same class-balanced checkpoints under standard retrieval and matched inference-time interventions. Each intervention cell reports Top-1 drop / changed predictions: Top-1 drop is the percentage-point decrease relative to that row’s own Standard Top-1, and changed predictions is the percentage of test examples whose predicted class differs from standard retrieval. Sample memory at evaluation states whether class-indexed sample-memory entries are available for local/global reads. In Strong aug., MixUp/CutMix batches are used for the loss but are not written into class-indexed sample memory, so memory-intervention cells do not provide applicable numerical intervention measurements. Strong aug.+clean mem. uses the same strong-augmentation objective and schedule and adds a clean-label no-gradient pass after each epoch to populate sample memory.

Figure 4: Top-1 accuracy for matched memory constructions under 400-epoch training. Points show the mean over three independent runs and horizontal bars show one standard deviation. Each panel compares the same four memory constructions under the same model, slot budget, latent dimension, retrieval rule, and optimizer. The figure focuses on Top-1 because Top-5 is near-saturated for these CIFAR settings; storage mechanics are summarized in Table 7 .

Design	Example- written	Gradient- learned	Local/global entries	Stored mem. C10/C100 (MB)	Trainable params C10/C100 (M)
CB-ring	yes	no	2500/1000	20.69/20.70	7.12/7.15
U-ring	yes	no	2500/1000	20.69/20.69	7.12/7.15
Learned proto.	no	yes	2500/1000	—	12.50/12.52
EMA proto.	no	no	10/10 (C10) 100/100 (C100)	0.28/1.34	7.12/7.15

Appendix

Table 7: Memory construction and storage summary for the 400-epoch memory-bank comparison. Example-written entries are detached projected features written from labeled training examples: local entries are local-neighborhood features, and global entries are image-level mean-pooled features. These entries are latent vectors, not raw images. Gradient-learned entries are free memory vectors optimized by back-propagation. Local/global entries report the number of local and global memory entries exposed to the same Hopfield-style retrieval rule. CB-ring, U-ring, and Learned proto. use the matched 2500 local and 1000 global entries. EMA proto. stores one running class-mean prototype per class for local and global retrieval, giving 10/10 entries on CIFAR-10 and 100/100 on CIFAR-100. Stored memory is reported separately from trainable model parameters. A dash in stored memory means that no separate example-written memory buffer is used; learned prototype capacity is included in trainable parameters.

Dataset	Model	Trainable params (M)	Approx. FLOPs (G)	Throughput (img/s)	GPU mem. (GB)
CIFAR-10	V-HMN	7.12	1.85	8874	1.08
CIFAR-10	ViT	7.16	0.92	29360	0.38
CIFAR-10	MLP-Mixer	8.71	1.35	19808	0.67
CIFAR-100	V-HMN	7.15	0.96	10692	0.72
CIFAR-100	ViT	7.19	0.92	29114	0.38
CIFAR-100	MLP-Mixer	8.76	1.35	19943	0.67

Appendix

Table 8: Runtime and compute comparison for the populated-memory 400-epoch setting. Approx. FLOPs are profiler-derived per-image floating-point operations. Throughput and GPU memory are measured consistently across the listed models. V-HMN stored-memory and retrieval-time components are separated in Figure 5 and the surrounding text.

Figure 5: Runtime breakdown for V-HMN at CIFAR resolution. “Neighborhood gather” forms each token’s local spatial neighborhood before local retrieval. “Local retrieval” and “global retrieval” are Hopfield-style memory reads from the local and global banks. “Remaining layers” groups patch embedding, positional encodings, projections, fusion, normalization, and classifier head. Stored memory is reported separately from trainable model parameters.

	CIFAR-10	CIFAR-100	Fashion-MNIST
Local Top-1 Hit Rate	30.87	1.92	35.73
Local Top-5 Hit Rate	61.38	6.73	73.44
Global Top-1 Hit Rate	36.32	7.82	68.80
Global Top-5 Hit Rate	81.43	24.19	96.24

Appendix

Table 9: Top-1 and top-5 retrieval hit rate, in percent (%). Measured on CIFAR-10, CIFAR-100 and Fashion-MNIST.

	CIFAR-10	CIFAR-100
Nearest Prototype	$93.74_{\pm 0.09}$	$75.93_{\pm 0.16}$
Top-5 Average	$93.82_{\pm 0.13}$	$76.16_{\pm 0.12}$
Hopfield Retrieval	$93.94_{\pm 0.11}$	$76.58_{\pm 0.09}$

Appendix

Table 10: Comparison of different retrieval methods, in terms of accuracy (%). Measured on CIFAR-10, CIFAR-100. Results are reported as mean

\pm

standard deviation over 3 seeds.

Panel of Figure 6 — Figure 6: Visualization of the memory slot weightings of the last layer of the V-HMN

Figure 7: Robustness analysis of V-HMN’s associative memory under Gaussian noise and occlusion.

Window size $k$	CIFAR-10	CIFAR-100	Fashion-MNIST
3	$93.94_{\pm 0.11}$	$76.58_{\pm 0.09}$	$92.27_{\pm 0.06}$
5	$93.22_{\pm 0.07}$	$74.80_{\pm 0.07}$	$91.94_{\pm 0.14}$
7	$92.60_{\pm 0.67}$	$70.67_{\pm 0.08}$	$91.23_{\pm 0.35}$

Appendix

Table 11: Ablation on spatial window size

k

. Accuracy (%) is reported on CIFAR-10, CIFAR-100, and Fashion-MNIST. Results are reported as mean

\pm

standard deviation over 3 seeds.

Local memory size				Global memory size
Size	CIFAR-10	CIFAR-100	Fashion-MNIST	Size	CIFAR-10	CIFAR-100	Fashion-MNIST
1500	$93.93_{\pm 0.09}$	$76.47_{\pm 0.29}$	$92.32_{\pm 0.07}$	500	$93.91_{\pm 0.10}$	$75.93_{\pm 0.12}$	$92.23_{\pm 0.10}$
2500	$93.94_{\pm 0.11}$	$76.58_{\pm 0.09}$	$92.27_{\pm 0.06}$	1000	$93.94_{\pm 0.11}$	$76.58_{\pm 0.09}$	$92.27_{\pm 0.06}$
3500	$94.14_{\pm 0.01}$	$76.42_{\pm 0.14}$	$92.36_{\pm 0.14}$	1500	$94.15_{\pm 0.17}$	$76.41_{\pm 0.03}$	$92.12_{\pm 0.04}$
4500	$94.04_{\pm 0.05}$	$76.59_{\pm 0.13}$	$92.44_{\pm 0.03}$	2000	$93.94_{\pm 0.14}$	$76.19_{\pm 0.04}$	$92.09_{\pm 0.17}$

Appendix

Table 12: Ablation study on the effect of local and global memory sizes in V-HMN. Top-1 test accuracy (%) are reported as mean

\pm

standard deviation over 3 seeds.

Figure 8: Effect of $\beta$ initialization on model accuracy with refinement iteration fixed to $t=1$ .

Method	Imbalanced CIFAR-10		Imbalanced CIFAR-100
Method	50	100	50	100
ViT	$71.47_{\pm 0.34}$	$64.61_{\pm 0.98}$	$42.91_{\pm 0.29}$	$37.45_{\pm 0.16}$
Swin-ViT	$65.66_{\pm 0.13}$	$34.13_{\pm 0.15}$	$38.37_{\pm 0.08}$	$34.13_{\pm 0.15}$
MLP-Mixer	$72.61_{\pm 0.67}$	$66.89_{\pm 0.14}$	$41.90_{\pm 0.21}$	$37.47_{\pm 0.07}$
Vim	$62.19_{\pm 0.70}$	$57.41_{\pm 0.24}$	$38.69_{\pm 0.12}$	$34.55_{\pm 0.11}$
AiT	$63.47_{\pm 0.13}$	$56.89_{\pm 0.19}$	$38.12_{\pm 0.18}$	$33.49_{\pm 0.22}$

Appendix

Table 13: Performance under class-imbalance on CIFAR-10 and CIFAR-100. All results are top-1 accuracy (%). Baselines include ViT [ 4 ] , Swin-ViT [ 20 ] , MLP-Mixer [ 30 ] , MetaFormer [ 40 ] , Vim [ 44 ] , and AiT [ 28 ] . Results are reported as mean

\pm

standard deviation over 3 seeds.

Explore similar work

Sep 27, 2026

VisionHOPE: Visual Backbones as Self-Modifying Learning Systems

Visual backbones have evolved from Convolutional Neural Networks (CNNs) with local aggregation to Vision Transformers (ViTs) with global interactions, State-Space Models (SSMs) with input-dependent state transitions, and Test-Time Training (TTT) layers that adapt an inner learner while processing an image. Across this progression, visual computation has become increasingly adaptive to each input, yet the rules governing that adaptation remain largely prescribed by the trained backbone. We introduce VisionHOPE, the first generic visual backbone formulated as a self-modifying learning system, in which what the model remembers and how it learns co-evolve within an image. Building on the self-referential construction of Nested Learning (NL), VisionHOPE realizes this co-evolution through five coupled memories that store content, generate key and value representations, and govern learning rate and retention. These memories evolve jointly as visual context accumulates along each scan. However, directly applying the unconstrained self-referential update to a visual backbone leads to instability. We therefore derive a stability-matched step-size control scheme that combines a soft cap on self-referential injection with a spectral clamp on the retained memory transition, and prove that the resulting memory dynamics are non-expansive along each scan. For two-dimensional feature maps, we adapt NL's chunk formulation by aligning chunks with image rows and columns across four directional scans. The proposed VisionHOPE achieves competitive results on ImageNet-1K, COCO, and ADE20K, establishing self-modifying learning systems as a practical foundation for general-purpose visual backbones. The code is available at https://github.com/PSRben/VisionHOPE.

Siran Peng, Tianshuo Zhang, Tianyu Fu +8

MAIS, CASIA · SAI, UCAS · Mininglamp Technology +1

Visual Representation Learning

Ask PDF

Feb 25, 2026

RECAP: Local Hebbian Prototype Learning as a Self-Organizing Readout for Reservoir Dynamics

Robust perception in brains is often attributed to high-dimensional population activity together with local plasticity mechanisms that reinforce recurring structure. In contrast, most modern image recognition systems are trained by error backpropagation and end-to-end gradient optimization, which are not naturally aligned with local computation and local plasticity. We introduce RECAP (Reservoir Computing with Hebbian Co-Activation Prototypes), a bio-inspired learning strategy for robust image classification that couples untrained reservoir dynamics with a self-organizing Hebbian prototype readout. RECAP discretizes time-averaged reservoir responses into activation levels, constructs a co-activation mask over reservoir unit pairs, and incrementally updates class-wise prototype matrices via a Hebbian-like potentiation-decay rule. Inference is performed by overlap-based prototype matching. The method avoids error backpropagation and is naturally compatible with online prototype updates. We illustrate the resulting robustness behavior on MNIST-C, where RECAP remains robust under diverse corruptions without exposure to corrupted training samples.

Heng Zhang

International Research Center for Neurointelligence (IRCN), University of Tokyo, Japan

Image Classification Prototype-Based Classification Neural Network Robustness Reservoir Computing Hebbian Learning

Ask PDF

May 29, 2026

PRISM: Progressive Reasoning through Iterative Slot Memory for Vision

Modern vision models process images in a single feed-forward pass, which limits their ability to recover missing evidence or refine uncertain representations under incomplete observations. Inspired by the iterative nature of human perception, we introduce PRISM (Progressive Reasoning through Iterative Slot Memory), a pyramid vision architecture that reasons over images through iterative refinement. At a high level, PRISM groups visual features into object-centric representations, retrieves relevant patterns from a learned memory, and iteratively refines the representation to resolve ambiguity and recover missing information. This organize-recall-refine process operates recurrently across multiple scales, enabling progressive improvement of visual representations. Across standard vision tasks, including image classification, object detection, and semantic segmentation, PRISM achieves competitive performance while demonstrating improved robustness under incomplete observations such as occlusion. These results suggest that iterative reasoning with structured representations and memory is a promising direction for building more resilient and adaptive vision models. Source code and models will be released.

Ziyu Wang, Shuangpeng Han, Mengmi Zhang

Deep NeuroCognition Lab, Nanyang Technological University, Singapore

Visual Representation Learning Visual Reasoning

Ask PDF