VisionHOPE: Visual Backbones as Self-Modifying Learning Systems

Siran Peng; Tianshuo Zhang; Tianyu Fu; Weisong Zhao; Haoyuan Zhang; Jiankuo Zhao; Minghui Wu; Ping Jiang

cs.CVSep 27, 2026

VisionHOPE: Visual Backbones as Self-Modifying Learning Systems

Authors: Siran Peng, Tianshuo Zhang, Tianyu Fu, Weisong Zhao, Haoyuan Zhang, Jiankuo Zhao, Minghui Wu, Ping Jiang, +3 more

Organizations: MAIS, CASIA · SAI, UCAS · Mininglamp Technology · SCSE, FIE, M.U.S.T.

Abstract

Visual backbones have evolved from Convolutional Neural Networks (CNNs) with local aggregation to Vision Transformers (ViTs) with global interactions, State-Space Models (SSMs) with input-dependent state transitions, and Test-Time Training (TTT) layers that adapt an inner learner while processing an image. Across this progression, visual computation has become increasingly adaptive to each input, yet the rules governing that adaptation remain largely prescribed by the trained backbone. We introduce VisionHOPE, the first generic visual backbone formulated as a self-modifying learning system, in which what the model remembers and how it learns co-evolve within an image. Building on the self-referential construction of Nested Learning (NL), VisionHOPE realizes this co-evolution through five coupled memories that store content, generate key and value representations, and govern learning rate and retention. These memories evolve jointly as visual context accumulates along each scan. However, directly applying the unconstrained self-referential update to a visual backbone leads to instability. We therefore derive a stability-matched step-size control scheme that combines a soft cap on self-referential injection with a spectral clamp on the retained memory transition, and prove that the resulting memory dynamics are non-expansive along each scan. For two-dimensional feature maps, we adapt NL's chunk formulation by aligning chunks with image rows and columns across four directional scans. The proposed VisionHOPE achieves competitive results on ImageNet-1K, COCO, and ADE20K, establishing self-modifying learning systems as a practical foundation for general-purpose visual backbones. The code is available at https://github.com/PSRben/VisionHOPE.

Figures & tables

Figure 1: Progression of within-image adaptation from local aggregation to self-modifying learning.

Figure 2: (a) The VisionHOPE operator applies SRNL along four directional scans with spatially aligned chunks, followed by spatial restoration and channel-wise fusion. Five coupled memories evolve through SR-DGD with step-size control. (b) The VisionHOPE block integrates the operator with projections, depthwise convolution, and an FFN in two pre-normalized residual branches.

Method	Type	#P	FLOPs	Acc.
ConvNeXt-T	CNN	29	4.5	82.1
InternImage-T	CNN	30	5.0	83.5
MambaOut-T	CNN	27	4.5	82.7
FasterViT-1	ViT	53	5.3	83.2
TransNeXt-T	ViT	28	5.7	84.0
RMT-S	ViT	27	4.5	84.1

Table 1: ImageNet-1K classification of hierarchical backbones at

224\times 224

. The panels correspond to increasing model scales. Parameters, FLOPs, and Top-1 accuracy are reported in M, G, and %.

Method	Type	#P	FLOPs	Acc.
DeiT-T	ViT	6	1.3	72.2
XCiT-T12/16	ViT	7	1.2	77.1
Agent-DeiT-T	ViT	6	1.2	74.9
Vim-T	SSM	7	1.6	76.1
Mamba ® -T	SSM	9	1.4	77.4
ViT 3 -T	TTT	6	1.2	76.5

Table 2: ImageNet-1K classification of plain backbones at

224\times 224

resolution. The panels correspond to increasing scales. Parameters, FLOPs, and Top-1 accuracy are reported in M, G, and %.

Method	FLOPs	$\mathbf{AP}^{b}$	$\mathbf{AP}^{m}$
ConvNeXt-T	262	44.2	40.1
InternImage-T	270	47.2	42.5
MambaOut-T	262	45.1	41.0
CSWin-T	279	46.7	42.2
SOFT++-S	261	43.8	40.1
MILA-T	255	46.8	42.1

Table 3: COCO object detection and instance segmentation with Mask R-CNN under the

1\times

schedule. The panels correspond to increasing backbone scales. FLOPs and AP are reported in G and %.

Backbone	#P	FLOPs	mIoU
ConvNeXt-T	60	939	46.0
MambaOut-T	54	938	47.4
VVT-S	56	960	46.8
SOFT++-S	60	948	46.5
MambaVision-T	55	945	46.0
H-ViT 3 -T	58	946	48.0

Table 4: ADE20K semantic segmentation with UPerNet. The panels correspond to increasing backbone scales. Complete-model parameters, FLOPs, and mIoU are reported in M, G, and %.

Figure 3: Efficiency comparison among DeiT-B, Vim-B, and P-VisionHOPE-B. We plot FLOPs, throughput, and memory footprint for input resolutions ranging from

224\times 224

1280\times 1280

. Hardware measurements use one NVIDIA A100 GPU with BF16 and a fixed batch size of

8

Variant	Acc.
w/o adaptive K/V & dynamics	81.7
w/o adaptive K/V	81.9
w/o adaptive dynamics	82.0
w/o DGD	81.9
Full model	82.3

Table 5: Ablations on P-VisionHOPE-S. The left, middle, and right panels examine the coupled SRNL memories and DGD, stability-matched step-size control, and directional scanning and fusion. Top-1 accuracy is reported in %. A NaN entry indicates numerical divergence during training.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Size	VisionHOPE-T	VisionHOPE-S	VisionHOPE-B
Stage 1	$\left(\frac{H}{4}\times\frac{W}{4}\right)$	Stem $\downarrow\!4$	Stem $\downarrow\!4$	Stem $\downarrow\!4$
Stage 1	$\left(\frac{H}{4}\times\frac{W}{4}\right)$	$\mathrm{B}(64,32,2)\!\times\!3$	$\mathrm{B}(64,32,2)\!\times\!4$	$\mathrm{B}(96,48,3)\!\times\!4$
Stage 2	$\left(\frac{H}{8}\times\frac{W}{8}\right)$	Down $\downarrow\!2$	Down $\downarrow\!2$	Down $\downarrow\!2$
Stage 2	$\left(\frac{H}{8}\times\frac{W}{8}\right)$	$\mathrm{B}(128,64,4)\!\times\!4$	$\mathrm{B}(160,80,5)\!\times\!8$	$\mathrm{B}(192,96,6)\!\times\!8$
Stage 3	$\left(\frac{H}{16}\times\frac{W}{16}\right)$	Down $\downarrow\!2$	Down $\downarrow\!2$	Down $\downarrow\!2$
Stage 3	$\left(\frac{H}{16}\times\frac{W}{16}\right)$	$\mathrm{B}(256,128,8)\!\times\!18$	$\mathrm{B}(320,160,10)\!\times\!25$	$\mathrm{B}(448,224,14)\!\times\!25$

Appendix

Table 6: Configurations of hierarchical VisionHOPE models. For an input of size

H\times W

\downarrow n

denotes downsampling by a factor of

n

\mathrm{B}(D,D_{m},n_{h})\!\times\!L

denotes

L

repeated VisionHOPE blocks with block width

D

, internal width

D_{m}

, and

n_{h}

heads per direction. The head dimension is

d=16

	Size	P-VisionHOPE-T	P-VisionHOPE-S	P-VisionHOPE-B
Backbone	$\left(\frac{H}{16}\times\frac{W}{16}\right)$	Stem $\downarrow\!16$	Stem $\downarrow\!16$	Stem $\downarrow\!16$
Backbone	$\left(\frac{H}{16}\times\frac{W}{16}\right)$	$\mathrm{B}(192,96,6)\!\times\!12$	$\mathrm{B}(384,192,12)\!\times\!13$	$\mathrm{B}(768,384,24)\!\times\!14$
Classifier	–	Global average pooling, linear

Appendix

Table 7: Configurations of plain VisionHOPE models. For an input of size

H\times W

, Stem

\downarrow 16

produces a feature map at

\left(\frac{H}{16}\times\frac{W}{16}\right)

\mathrm{B}(D,D_{m},n_{h})\!\times\!L

denotes

L

repeated VisionHOPE blocks with block width

D

, internal width

D_{m}

, and

n_{h}

heads per direction. The head dimension is

d=16

Figure 4: Efficiency comparison among DeiT, Vim, and P-VisionHOPE at the Tiny and Small scales. We plot FLOPs, throughput, and memory footprint for input resolutions from

224\times 224

1280\times 1280

. Hardware measurements use one NVIDIA A100 GPU with BF16 and a fixed batch size of

8

Method	FLOPs	$\mathbf{AP}^{b}$	$\mathbf{AP}^{b}_{50}$	$\mathbf{AP}^{b}_{75}$	$\mathbf{AP}^{m}$	$\mathbf{AP}^{m}_{50}$	$\mathbf{AP}^{m}_{75}$
ConvNeXt-T	262	46.2	67.9	50.8	41.7	65.0	44.9
InternImage-T	270	49.1	70.4	54.1	43.7	67.3	47.3
CSWin-T	279	49.0	70.7	53.7	43.6	67.9	46.6
MILA-T	255	48.8	71.0	53.6	43.8	68.0	46.8
VMamba-T	271	48.8	70.4	53.5	43.7	67.4	47.0
LocalVMamba-T	291	48.7	70.1	53.0	43.4	67.0	46.4

Appendix

Table 8: COCO object detection and instance segmentation with Mask R-CNN under the

3\times

schedule. FLOPs and AP are reported in G and %. VisionHOPE delivers competitive performance.

$D_{m}/D$	#P	FLOPs	Acc.
$1/4$	20	4.2	81.5
$1/2$	22	4.7	82.3
$1$	27	6.1	82.3
$2$	44	9.8	82.5

Appendix

Table 9: Configuration ablations on P-VisionHOPE-S. The four panels vary

D_{m}/D

, chunk length,

\gamma_{\eta}

, and

d

, respectively. Parameters, FLOPs, and Top-1 accuracy are reported in M, G, and %.

Method	Acc.
MILA-T (w/o MESA)	83.3
MILA-T (w/ MESA)	83.5
H-ViT 3 -T (w/o MESA)	83.5
H-ViT 3 -T (w/ MESA)	84.0
VisionHOPE-T (w/o MESA)	83.7
VisionHOPE-T (w/ MESA)	84.1

Appendix

Table 10: ImageNet-1K classification of hierarchical models with and without MESA. The panels correspond to Tiny, Small, and Base models from left to right. The results for MILA and H-ViT 3 are taken from their respective papers ( Han et al., 2024a ; Han et al., 2026 ) . Top-1 accuracy is reported in %.

Figure 5: Qualitative comparison of global perception through Effective Receptive Fields (ERFs) at $512\times 512$ resolution. The maps are computed from the geometrically centered $2\times 2$ features at the native output of the second processing block and averaged across the complete ImageNet-1K validation set. Darker colors indicate stronger relative influence on the centered representation.

Figure 6: Spatial variation of the final executed step in P-VisionHOPE-S across several ImageNet-1K validation images. Each two-row block pairs input images with the corresponding $14\times 14$ maps of $\widetilde{\eta}_{t}$ from the final VisionHOPE block, restored to image coordinates and averaged across heads and four scan directions. Darker colors indicate relatively larger executed steps within each image.

Explore similar work

Apr 22, 2026cs.CV

Self-supervised pretraining for an iterative image size agnostic vision transformer

Vision Transformers (ViTs) dominate self-supervised learning (SSL). While they have proven highly effective for large-scale pretraining, they are computationally inefficient and scale poorly with image size. Consequently, foundational models like DINO are constrained to low-resolution processing. A recent foveal-inspired transformer achieves resolution agnosticism by iteratively processing a fixed-size context of multi-zoom patches. This model demonstrated promising results via supervised learning, utilizing a sequential, recurrent-like process without backpropagation through time. To unlock its potential as a foundational backbone, we introduce a novel sequential-to-global SSL framework based on DINO's self-distillation objective. Supported by an efficient integral-image patch extraction method, our approach enables large-scale pretraining for image-size agnostic vision encoders. We achieve competitive performance on ImageNet-1K and downstream classification tasks, maintaining a constant computational budget regardless of input resolution.

Nedyalko Prisadnikov, Danda Pani Paudel, Yuqian Fu +1

INSAIT, Sofia University "St. Kliment Ohridski", Bulgaria

Self-Supervised Vision Transformers Vision Encoders Self-Supervised Learning Dino

Ask PDF

Jun 23, 2026cs.CV

Adaptive Hebbian Memory Routing in Vision Transformers for Few-Shot Learning

Few-shot image recognition requires models to adapt to new classes from a small labeled support set. Hebbian fast-weight memory can provide temporary associative information during an episode, but fixed memory behavior may not be appropriate for every few-shot task. In this work, we propose Adaptive Hebbian Routing for few-shot Vision Transformers. The method uses a lightweight MLP router to control the contribution of Hebbian memory, the strength of memory updates, and the retention of previous memory from support-set features. We study Adaptive Placement, Adaptive Plasticity, and Fully Adaptive Hebbian Routing. Experiments use ViT-Small, DeiT-Small, and Swin-Tiny under 5-way 1-shot evaluation on Omniglot, CIFAR-FS, and cross-domain transfer from CIFAR-FS to Omniglot. In the direct Swin comparison, fixed and adaptive Hebbian variants use the same memory location. Adaptive Plasticity improves the fixed Hebbian result from 96.74% to 96.92%, while Fully Adaptive Routing achieves the best result at 96.94%. The fully adaptive Swin model also reduces inference time from 16.51 ms to 14.05 ms relative to fixed Hebbian Swin. On CIFAR-FS, adaptive variants improve performance across all three backbones, and the multi-shot evaluation shows that these gains remain useful as the number of support examples increases. These results show that adaptive plasticity and adaptive memory activation can improve few-shot Transformer representations beyond fixed Hebbian behavior.

Mohammed Yusuf Mujawar, Noorbakhsh Amiri Golilarz

Department of Computer Science The University of Alabama, Tuscaloosa, AL, USA

Few-Shot Learning Vision Transformer Hebbian Learning Plasticity

Ask PDF

Sep 28, 2026cs.CV

W2Rep: Learning Visual Representations by Watching the World Change

Images capture the world at one moment, whereas video reveals how it changes. Image self-supervision learns spatial structure from a single moment, while video methods commonly learn temporal relationships inside a representation computed jointly from several frames. We ask whether watching a scene change can instead improve features available from one image without sacrificing the ability to represent video. We introduce W2Rep, a masked feature-prediction framework in which an independently encoded source image participates in prediction at the same or another moment. The predictor is conditioned on visible video context, the queried location, and the signed time interval between source and target. This gives the cross-frame objective two complementary roles: the image path learns features that remain useful across time, while the video path must gather evidence that is missing from the source image. Across model scales and downstream tasks, W2Rep improves frozen and fine-tuned recognition under our comparison protocol, while joint video encoding provides further gains over frame-wise aggregation. Controlled experiments show that these gains depend on directly updating the source-image features and on using both video context and temporal displacement. Overall, change across a video can supervise a visual encoder whose representations remain useful at either image or video granularity. Code is available at~\href{https://wenooi.github.io/W2Rep}{https://wenooi.github.io/W2Rep}.

Wen Huang, Hang Guo, Jiarui Yang +3

Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Nankai University, Tianjin, China · Shenzhen University, Shenzhen, China

Fine-Grained Video Understanding Vision Encoders Self-Supervised Representations Visual Representations Self-Supervised Learning World+1

Ask PDF