Vision Hopfield Memory Networks for Image Recognition

Jianfeng Wang; Amine M'Charrak; Luk Koska; Xiangtao Wang; Daniel Petriceanu; Ruizhi Wang; Michael Bumbar; Luca Pinchetti

cs.LGMar 26, 2026

Vision Hopfield Memory Networks for Image Recognition

Authors: Jianfeng Wang, Amine M'Charrak, Luk Koska, Xiangtao Wang, Daniel Petriceanu, Ruizhi Wang, Michael Bumbar, Luca Pinchetti, +1 more

Organizations: Department of Computer Science, University of Oxford, United Kingdom · Faculty of Informatics, Vienna University of Technology, Austria

Abstract

Recent vision backbones, such as Transformer families and state-space models like Mamba, have achieved remarkable progress on image recognition. Despite their empirical success, these architectures remain far from the computational principles of the human brain, often demanding enormous amounts of training data while offering limited interpretability. We propose the Vision Hopfield Memory Network (V-HMN), a brain-inspired vision backbone that integrates hierarchical memory mechanisms across layers with iterative refinement updates. Specifically, V-HMN incorporates local Hopfield modules that provide associative memory dynamics at the image patch level, global Hopfield modules that function as episodic memory for contextual modulation, and a predictive-coding-inspired refinement rule for iterative error correction. By organizing these memory-based modules hierarchically, V-HMN captures both local and global dynamics in a unified framework. Memory retrieval exposes the relationship between inputs and stored patterns, providing a prototype-based form of interpretability through explicit memory retrieval, while the reuse of stored patterns improves data efficiency. This brain-inspired design therefore enhances data efficiency and provides a prototype-based form of interpretability compared to existing self-attention- or state-space-based approaches. We conducted extensive experiments on public image classification benchmarks. V-HMN achieves strong performance on small- and medium-scale benchmarks, and remains competitive with widely adopted backbone architectures on ImageNet despite minimal architectural tuning, while offering improved data efficiency and a prototype-based form of interpretability. These findings highlight the potential of V-HMN as a memory-centric alternative to standard vision backbones, thereby bridging brain-inspired computation with modern machine learning.

Figures & tables

Figure 1: Overview of V-HMN. (a) Each HMN block refines features through local and global Hopfield memory retrieval, rather than convolution or self-attention. (b) A deep backbone is constructed by stacking HMN blocks, with attention pooling and a linear head for image classification.

Fraction of training data	CIFAR-10	CIFAR-100	Fashion-MNIST
10%	$80.22_{\pm 0.29}$	$43.21_{\pm 1.07}$	$89.18_{\pm 0.16}$
30%	$88.67_{\pm 0.21}$	$62.42_{\pm 0.29}$	$91.04_{\pm 0.22}$
50%	$91.19_{\pm 0.38}$	$68.93_{\pm 0.35}$	$91.53_{\pm 0.12}$

Table 1: Data efficiency of V - HMN on CIFAR-10, CIFAR-100, and Fashion-MNIST with different fractions of labeled training samples. Reported values are top-1 test accuracy (%).

Model	CIFAR-10		CIFAR-100		Fashion-MNIST
Model	10%	30%	10%	30%	10%	30%
ViT	$72.73_{\pm 0.42}$	$83.94_{\pm 0.33}$	$40.48_{\pm 0.80}$	$57.40_{\pm 0.79}$	$87.17_{\pm 0.40}$	$89.71_{\pm 0.45}$
Swin-ViT	$70.37_{\pm 0.94}$	$80.89_{\pm 0.24}$	$35.25_{\pm 0.59}$	$51.77_{\pm 0.60}$	$88.42_{\pm 0.48}$	$90.34_{\pm 0.15}$
MLP-Mixer	$76.14_{\pm 0.16}$	$85.53_{\pm 0.40}$	$41.94_{\pm 0.98}$	$56.22_{\pm 0.75}$	$87.16_{\pm 0.63}$	$89.41_{\pm 0.13}$
MetaFormer	$49.92_{\pm 4.08}$	$60.86_{\pm 6.05}$	$19.39_{\pm 1.26}$	$37.47_{\pm 3.19}$	$85.02_{\pm 1.22}$	$88.78_{\pm 1.03}$
Vim	$69.02_{\pm 2.13}$	$79.58_{\pm 0.37}$	$36.61_{\pm 1.25}$	$49.67_{\pm 0.79}$	$86.09_{\pm 1.03}$	$88.18_{\pm 2.43}$

Table 2: Comparison of data efficiency across models with 10% and 30% labeled training data on CIFAR-10, CIFAR-100, and Fashion-MNIST. Top-1 test accuracy (%) are reported as mean

\pm

standard deviation over 3 seeds. Baselines include ViT ( Dosovitskiy et al., 2021 ) , Swin-ViT ( Liu et al., 2021 ) , MLP-Mixer ( Tolstikhin et al., 2021 ) , MetaFormer ( Yu et al., 2024 ) , Vim ( Zhu et al., 2024 ) , and AiT ( Sun et al., 2025 ) .

Model	CIFAR-10	CIFAR-100	SVHN	FashionMNIST	Params (M)
ViT	$91.66_{\pm 0.08}$	$72.56_{\pm 0.01}$	$96.11_{\pm 0.29}$	$91.83_{\pm 0.15}$	7.16
Swin-ViT	$87.94_{\pm 0.10}$	$66.58_{\pm 0.37}$	$95.89_{\pm 0.17}$	$91.79_{\pm 0.17}$	6.92
MLP-Mixer	$92.65_{\pm 0.26}$	$73.35_{\pm 0.39}$	$96.91_{\pm 0.06}$	$91.46_{\pm 0.32}$	8.71
MetaFormer	$75.40_{\pm 1.56}$	$48.76_{\pm 0.75}$	$86.54_{\pm 2.71}$	$91.42_{\pm 0.21}$	6.99
Vim	$82.83_{\pm 3.87}$	$63.29_{\pm 2.83}$	$86.63_{\pm 1.51}$	$88.63_{\pm 1.49}$	7.12
AiT	$92.97_{\pm 0.30}$	$72.91_{\pm 0.17}$	$95.98_{\pm 0.06}$	$91.51_{\pm 0.04}$	7.15

Table 4: Comparison of V-HMN with baseline models on CIFAR-10, CIFAR-100, SVHN, and Fashion-MNIST. Top-1 test accuracy (%) are reported as mean

\pm

Method	Image Size	#Params	Top-1 Acc.
ResNet-50 ( He et al., 2016 )	224	26M	76.1
SENet-50 ( Hu et al., 2018 )	224	28M	76.7
ViT-B/16 ( Dosovitskiy et al., 2021 )	384	86M	77.9
ViT-B/16 † ( Dosovitskiy et al., 2021 )	224	86M	76.2
ViT-L/16 ( Dosovitskiy et al., 2021 )	384	307M	76.5
MLP-Mixer-B/16 ( Tolstikhin et al., 2021 )	224	59M	76.4

Table 5: Comparison on ImageNet-1k. All results are Top-1 accuracy (%). (

\dagger

) indicates our re-implementation under the same setting for fair comparison.

Figure 3: Visualization of retrieved prototypes from local and global memory.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Setting	Sample memory at eval.	Standard Top-1	No local Top-1 drop / changed pred.	No global Top-1 drop / changed pred.	No both Top-1 drop / changed pred.	Shuffled mem. Top-1 drop / changed pred.	Random top- $k$ Top-1 drop / changed pred.
CIFAR-10	Clean labels	populated	91.11 $\pm$ 0.06	61.94/70.86	0.70/3.35	66.17/75.27	42.69/50.80	16.34/23.31
CIFAR-10	Strong aug.	not populated	93.67 $\pm$ 0.02	not applicable (sample memory not populated)
CIFAR-10	Strong aug.+clean mem.	populated	93.56 $\pm$ 0.25	1.24/3.95	0.05/0.28	1.23/3.98	14.69/19.32	23.71/28.84
CIFAR-100	Clean labels	populated	65.45 $\pm$ 0.26	15.91/41.70	5.70/27.46	19.95/48.49	22.61/51.20	13.45/39.36
CIFAR-100	Strong aug.	not populated	75.83 $\pm$ 0.11	not applicable (sample memory not populated)
CIFAR-100	Strong aug.+clean mem.	populated	75.86 $\pm$ 0.10	3.63/14.47	0.27/5.98	3.74/15.15	11.11/26.80	10.74/28.15

Appendix

Table 6: 400-epoch paired retrieval-disruption summary. Each row evaluates the same class-balanced checkpoints under standard retrieval and matched inference-time interventions. Each intervention cell reports Top-1 drop / changed predictions: Top-1 drop is the percentage-point decrease relative to that row’s own Standard Top-1, and changed predictions is the percentage of test examples whose predicted class differs from standard retrieval. Sample memory at evaluation states whether class-indexed sample-memory entries are available for local/global reads. In Strong aug., MixUp/CutMix batches are used for the loss but are not written into class-indexed sample memory, so memory-intervention cells do not provide applicable numerical intervention measurements. Strong aug.+clean mem. uses the same strong-augmentation objective and schedule and adds a clean-label no-gradient pass after each epoch to populate sample memory.

Figure 4: Top-1 accuracy for matched memory constructions under 400-epoch training. Points show the mean over three independent runs and horizontal bars show one standard deviation. Each panel compares the same four memory constructions under the same model, slot budget, latent dimension, retrieval rule, and optimizer. The figure focuses on Top-1 because Top-5 is near-saturated for these CIFAR settings; storage mechanics are summarized in Table 7 .

Design	Example- written	Gradient- learned	Local/global entries	Stored mem. C10/C100 (MB)	Trainable params C10/C100 (M)
CB-ring	yes	no	2500/1000	20.69/20.70	7.12/7.15
U-ring	yes	no	2500/1000	20.69/20.69	7.12/7.15
Learned proto.	no	yes	2500/1000	—	12.50/12.52
EMA proto.	no	no	10/10 (C10) 100/100 (C100)	0.28/1.34	7.12/7.15

Appendix

Table 7: Memory construction and storage summary for the 400-epoch memory-bank comparison. Example-written entries are detached projected features written from labeled training examples: local entries are local-neighborhood features, and global entries are image-level mean-pooled features. These entries are latent vectors, not raw images. Gradient-learned entries are free memory vectors optimized by back-propagation. Local/global entries report the number of local and global memory entries exposed to the same Hopfield-style retrieval rule. CB-ring, U-ring, and Learned proto. use the matched 2500 local and 1000 global entries. EMA proto. stores one running class-mean prototype per class for local and global retrieval, giving 10/10 entries on CIFAR-10 and 100/100 on CIFAR-100. Stored memory is reported separately from trainable model parameters. A dash in stored memory means that no separate example-written memory buffer is used; learned prototype capacity is included in trainable parameters.

Dataset	Model	Trainable params (M)	Approx. FLOPs (G)	Throughput (img/s)	GPU mem. (GB)
CIFAR-10	V-HMN	7.12	1.85	8874	1.08
CIFAR-10	ViT	7.16	0.92	29360	0.38
CIFAR-10	MLP-Mixer	8.71	1.35	19808	0.67
CIFAR-100	V-HMN	7.15	0.96	10692	0.72
CIFAR-100	ViT	7.19	0.92	29114	0.38
CIFAR-100	MLP-Mixer	8.76	1.35	19943	0.67

Appendix

Table 8: Runtime and compute comparison for the populated-memory 400-epoch setting. Approx. FLOPs are profiler-derived per-image floating-point operations. Throughput and GPU memory are measured consistently across the listed models. V-HMN stored-memory and retrieval-time components are separated in Figure 5 and the surrounding text.

Figure 5: Runtime breakdown for V-HMN at CIFAR resolution. “Neighborhood gather” forms each token’s local spatial neighborhood before local retrieval. “Local retrieval” and “global retrieval” are Hopfield-style memory reads from the local and global banks. “Remaining layers” groups patch embedding, positional encodings, projections, fusion, normalization, and classifier head. Stored memory is reported separately from trainable model parameters.

	CIFAR-10	CIFAR-100	Fashion-MNIST
Local Top-1 Hit Rate	30.87	1.92	35.73
Local Top-5 Hit Rate	61.38	6.73	73.44
Global Top-1 Hit Rate	36.32	7.82	68.80
Global Top-5 Hit Rate	81.43	24.19	96.24

Appendix

Table 9: Top-1 and top-5 retrieval hit rate, in percent (%). Measured on CIFAR-10, CIFAR-100 and Fashion-MNIST.

	CIFAR-10	CIFAR-100
Nearest Prototype	$93.74_{\pm 0.09}$	$75.93_{\pm 0.16}$
Top-5 Average	$93.82_{\pm 0.13}$	$76.16_{\pm 0.12}$
Hopfield Retrieval	$93.94_{\pm 0.11}$	$76.58_{\pm 0.09}$

Appendix

Table 10: Comparison of different retrieval methods, in terms of accuracy (%). Measured on CIFAR-10, CIFAR-100. Results are reported as mean

\pm

standard deviation over 3 seeds.

Panel of Figure 6 — Figure 6: Visualization of the memory slot weightings of the last layer of the V-HMN

Figure 7: Robustness analysis of V-HMN’s associative memory under Gaussian noise and occlusion.

Window size $k$	CIFAR-10	CIFAR-100	Fashion-MNIST
3	$93.94_{\pm 0.11}$	$76.58_{\pm 0.09}$	$92.27_{\pm 0.06}$
5	$93.22_{\pm 0.07}$	$74.80_{\pm 0.07}$	$91.94_{\pm 0.14}$
7	$92.60_{\pm 0.67}$	$70.67_{\pm 0.08}$	$91.23_{\pm 0.35}$

Appendix

Table 11: Ablation on spatial window size

k

. Accuracy (%) is reported on CIFAR-10, CIFAR-100, and Fashion-MNIST. Results are reported as mean

\pm

standard deviation over 3 seeds.

Local memory size				Global memory size
Size	CIFAR-10	CIFAR-100	Fashion-MNIST	Size	CIFAR-10	CIFAR-100	Fashion-MNIST
1500	$93.93_{\pm 0.09}$	$76.47_{\pm 0.29}$	$92.32_{\pm 0.07}$	500	$93.91_{\pm 0.10}$	$75.93_{\pm 0.12}$	$92.23_{\pm 0.10}$
2500	$93.94_{\pm 0.11}$	$76.58_{\pm 0.09}$	$92.27_{\pm 0.06}$	1000	$93.94_{\pm 0.11}$	$76.58_{\pm 0.09}$	$92.27_{\pm 0.06}$
3500	$94.14_{\pm 0.01}$	$76.42_{\pm 0.14}$	$92.36_{\pm 0.14}$	1500	$94.15_{\pm 0.17}$	$76.41_{\pm 0.03}$	$92.12_{\pm 0.04}$
4500	$94.04_{\pm 0.05}$	$76.59_{\pm 0.13}$	$92.44_{\pm 0.03}$	2000	$93.94_{\pm 0.14}$	$76.19_{\pm 0.04}$	$92.09_{\pm 0.17}$

Appendix

Table 12: Ablation study on the effect of local and global memory sizes in V-HMN. Top-1 test accuracy (%) are reported as mean

\pm

standard deviation over 3 seeds.

Figure 8: Effect of $\beta$ initialization on model accuracy with refinement iteration fixed to $t=1$ .

Method	Imbalanced CIFAR-10		Imbalanced CIFAR-100
Method	50	100	50	100
ViT	$71.47_{\pm 0.34}$	$64.61_{\pm 0.98}$	$42.91_{\pm 0.29}$	$37.45_{\pm 0.16}$
Swin-ViT	$65.66_{\pm 0.13}$	$34.13_{\pm 0.15}$	$38.37_{\pm 0.08}$	$34.13_{\pm 0.15}$
MLP-Mixer	$72.61_{\pm 0.67}$	$66.89_{\pm 0.14}$	$41.90_{\pm 0.21}$	$37.47_{\pm 0.07}$
Vim	$62.19_{\pm 0.70}$	$57.41_{\pm 0.24}$	$38.69_{\pm 0.12}$	$34.55_{\pm 0.11}$
AiT	$63.47_{\pm 0.13}$	$56.89_{\pm 0.19}$	$38.12_{\pm 0.18}$	$33.49_{\pm 0.22}$

Appendix

Table 13: Performance under class-imbalance on CIFAR-10 and CIFAR-100. All results are top-1 accuracy (%). Baselines include ViT [ 4 ] , Swin-ViT [ 20 ] , MLP-Mixer [ 30 ] , MetaFormer [ 40 ] , Vim [ 44 ] , and AiT [ 28 ] . Results are reported as mean

\pm

standard deviation over 3 seeds.

Explore similar work

VisionHOPE: Visual Backbones as Self-Modifying Learning Systems
Sep 27, 2026Siran Peng, Tianshuo Zhang, Tianyu Fu +8Visual Representation Learning
MAIS, CASIA · SAI, UCAS · Mininglamp Technology +1
RECAP: Local Hebbian Prototype Learning as a Self-Organizing Readout for Reservoir Dynamics
Feb 25, 2026Heng ZhangImage Classification Prototype-Based Classification
International Research Center for Neurointelligence (IRCN), University of Tokyo, Japan
PRISM: Progressive Reasoning through Iterative Slot Memory for Vision
May 29, 2026Ziyu Wang, Shuangpeng Han, Mengmi ZhangVisual Representation Learning Visual Reasoning
Deep NeuroCognition Lab, Nanyang Technological University, Singapore

Sep 27, 2026J/K move · Enter open · S save

VisionHOPE: Visual Backbones as Self-Modifying Learning Systems

Siran Peng, Tianshuo Zhang, Tianyu Fu, Weisong Zhao+7

MAIS, CASIA · SAI, UCAS · Mininglamp Technology+1

Visual Representation Learning

Open paper AskPDF arXiv

Visual backbones have evolved from Convolutional Neural Networks (CNNs) with local aggregation to Vision Transformers (ViTs) with global interactions, State-Space Models (SSMs) with input-dependent state transitions, and Test-Time Training (TTT) layers that adapt an inner learner while processing an image. Across this progression, visual computation has become increasingly adaptive to each input, yet the rules governing that adaptation remain largely prescribed by the trained backbone. We introduce VisionHOPE, the first generic visual backbone formulated as a self-modifying learning system, in which what the model remembers and how it learns co-evolve within an image. Building on the self-referential construction of Nested Learning (NL), VisionHOPE realizes this co-evolution through five coupled memories that store content, generate key and value representations, and govern learning rate and retention. These memories evolve jointly as visual context accumulates along each scan. However, directly applying the unconstrained self-referential update to a visual backbone leads to instability. We therefore derive a stability-matched step-size control scheme that combines a soft cap on self-referential injection with a spectral clamp on the retained memory transition, and prove that the resulting memory dynamics are non-expansive along each scan. For two-dimensional feature maps, we adapt NL's chunk formulation by aligning chunks with image rows and columns across four directional scans. The proposed VisionHOPE achieves competitive results on ImageNet-1K, COCO, and ADE20K, establishing self-modifying learning systems as a practical foundation for general-purpose visual backbones. The code is available at https://github.com/PSRben/VisionHOPE.

Abstract

Explore similar work

VisionHOPE: Visual Backbones as Self-Modifying Learning Systems

RECAP: Local Hebbian Prototype Learning as a Self-Organizing Readout for Reservoir Dynamics

PRISM: Progressive Reasoning through Iterative Slot Memory for Vision