VisionHOPE: Visual Backbones as Self-Modifying Learning Systems

Siran Peng; Tianshuo Zhang; Tianyu Fu; Weisong Zhao; Haoyuan Zhang; Jiankuo Zhao; Minghui Wu; Ping Jiang

cs.CVSep 27, 2026

VisionHOPE: Visual Backbones as Self-Modifying Learning Systems

Authors: Siran Peng, Tianshuo Zhang, Tianyu Fu, Weisong Zhao, Haoyuan Zhang, Jiankuo Zhao, Minghui Wu, Ping Jiang, +3 more

Organizations: MAIS, CASIA · SAI, UCAS · Mininglamp Technology · SCSE, FIE, M.U.S.T.

Abstract

Visual backbones have evolved from Convolutional Neural Networks (CNNs) with local aggregation to Vision Transformers (ViTs) with global interactions, State-Space Models (SSMs) with input-dependent state transitions, and Test-Time Training (TTT) layers that adapt an inner learner while processing an image. Across this progression, visual computation has become increasingly adaptive to each input, yet the rules governing that adaptation remain largely prescribed by the trained backbone. We introduce VisionHOPE, the first generic visual backbone formulated as a self-modifying learning system, in which what the model remembers and how it learns co-evolve within an image. Building on the self-referential construction of Nested Learning (NL), VisionHOPE realizes this co-evolution through five coupled memories that store content, generate key and value representations, and govern learning rate and retention. These memories evolve jointly as visual context accumulates along each scan. However, directly applying the unconstrained self-referential update to a visual backbone leads to instability. We therefore derive a stability-matched step-size control scheme that combines a soft cap on self-referential injection with a spectral clamp on the retained memory transition, and prove that the resulting memory dynamics are non-expansive along each scan. For two-dimensional feature maps, we adapt NL's chunk formulation by aligning chunks with image rows and columns across four directional scans. The proposed VisionHOPE achieves competitive results on ImageNet-1K, COCO, and ADE20K, establishing self-modifying learning systems as a practical foundation for general-purpose visual backbones. The code is available at https://github.com/PSRben/VisionHOPE.

Figures & tables

Figure 1: Progression of within-image adaptation from local aggregation to self-modifying learning.

Figure 2: (a) The VisionHOPE operator applies SRNL along four directional scans with spatially aligned chunks, followed by spatial restoration and channel-wise fusion. Five coupled memories evolve through SR-DGD with step-size control. (b) The VisionHOPE block integrates the operator with projections, depthwise convolution, and an FFN in two pre-normalized residual branches.

Method	Type	#P	FLOPs	Acc.
ConvNeXt-T	CNN	29	4.5	82.1
InternImage-T	CNN	30	5.0	83.5
MambaOut-T	CNN	27	4.5	82.7
FasterViT-1	ViT	53	5.3	83.2
TransNeXt-T	ViT	28	5.7	84.0
RMT-S	ViT	27	4.5	84.1

Table 1: ImageNet-1K classification of hierarchical backbones at

224\times 224

. The panels correspond to increasing model scales. Parameters, FLOPs, and Top-1 accuracy are reported in M, G, and %.

Method	Type	#P	FLOPs	Acc.
DeiT-T	ViT	6	1.3	72.2
XCiT-T12/16	ViT	7	1.2	77.1
Agent-DeiT-T	ViT	6	1.2	74.9
Vim-T	SSM	7	1.6	76.1
Mamba ® -T	SSM	9	1.4	77.4
ViT 3 -T	TTT	6	1.2	76.5

Table 2: ImageNet-1K classification of plain backbones at

224\times 224

resolution. The panels correspond to increasing scales. Parameters, FLOPs, and Top-1 accuracy are reported in M, G, and %.

Method	FLOPs	$\mathbf{AP}^{b}$	$\mathbf{AP}^{m}$
ConvNeXt-T	262	44.2	40.1
InternImage-T	270	47.2	42.5
MambaOut-T	262	45.1	41.0
CSWin-T	279	46.7	42.2
SOFT++-S	261	43.8	40.1
MILA-T	255	46.8	42.1

Table 3: COCO object detection and instance segmentation with Mask R-CNN under the

1\times

schedule. The panels correspond to increasing backbone scales. FLOPs and AP are reported in G and %.

Backbone	#P	FLOPs	mIoU
ConvNeXt-T	60	939	46.0
MambaOut-T	54	938	47.4
VVT-S	56	960	46.8
SOFT++-S	60	948	46.5
MambaVision-T	55	945	46.0
H-ViT 3 -T	58	946	48.0

Table 4: ADE20K semantic segmentation with UPerNet. The panels correspond to increasing backbone scales. Complete-model parameters, FLOPs, and mIoU are reported in M, G, and %.

Figure 3: Efficiency comparison among DeiT-B, Vim-B, and P-VisionHOPE-B. We plot FLOPs, throughput, and memory footprint for input resolutions ranging from

224\times 224

1280\times 1280

. Hardware measurements use one NVIDIA A100 GPU with BF16 and a fixed batch size of

8

Variant	Acc.
w/o adaptive K/V & dynamics	81.7
w/o adaptive K/V	81.9
w/o adaptive dynamics	82.0
w/o DGD	81.9
Full model	82.3

Table 5: Ablations on P-VisionHOPE-S. The left, middle, and right panels examine the coupled SRNL memories and DGD, stability-matched step-size control, and directional scanning and fusion. Top-1 accuracy is reported in %. A NaN entry indicates numerical divergence during training.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Size	VisionHOPE-T	VisionHOPE-S	VisionHOPE-B
Stage 1	$\left(\frac{H}{4}\times\frac{W}{4}\right)$	Stem $\downarrow\!4$	Stem $\downarrow\!4$	Stem $\downarrow\!4$
Stage 1	$\left(\frac{H}{4}\times\frac{W}{4}\right)$	$\mathrm{B}(64,32,2)\!\times\!3$	$\mathrm{B}(64,32,2)\!\times\!4$	$\mathrm{B}(96,48,3)\!\times\!4$
Stage 2	$\left(\frac{H}{8}\times\frac{W}{8}\right)$	Down $\downarrow\!2$	Down $\downarrow\!2$	Down $\downarrow\!2$
Stage 2	$\left(\frac{H}{8}\times\frac{W}{8}\right)$	$\mathrm{B}(128,64,4)\!\times\!4$	$\mathrm{B}(160,80,5)\!\times\!8$	$\mathrm{B}(192,96,6)\!\times\!8$
Stage 3	$\left(\frac{H}{16}\times\frac{W}{16}\right)$	Down $\downarrow\!2$	Down $\downarrow\!2$	Down $\downarrow\!2$
Stage 3	$\left(\frac{H}{16}\times\frac{W}{16}\right)$	$\mathrm{B}(256,128,8)\!\times\!18$	$\mathrm{B}(320,160,10)\!\times\!25$	$\mathrm{B}(448,224,14)\!\times\!25$

Appendix

Table 6: Configurations of hierarchical VisionHOPE models. For an input of size

H\times W

\downarrow n

denotes downsampling by a factor of

n

\mathrm{B}(D,D_{m},n_{h})\!\times\!L

denotes

L

repeated VisionHOPE blocks with block width

D

, internal width

D_{m}

, and

n_{h}

heads per direction. The head dimension is

d=16

	Size	P-VisionHOPE-T	P-VisionHOPE-S	P-VisionHOPE-B
Backbone	$\left(\frac{H}{16}\times\frac{W}{16}\right)$	Stem $\downarrow\!16$	Stem $\downarrow\!16$	Stem $\downarrow\!16$
Backbone	$\left(\frac{H}{16}\times\frac{W}{16}\right)$	$\mathrm{B}(192,96,6)\!\times\!12$	$\mathrm{B}(384,192,12)\!\times\!13$	$\mathrm{B}(768,384,24)\!\times\!14$
Classifier	–	Global average pooling, linear

Appendix

Table 7: Configurations of plain VisionHOPE models. For an input of size

H\times W

, Stem

\downarrow 16

produces a feature map at

\left(\frac{H}{16}\times\frac{W}{16}\right)

\mathrm{B}(D,D_{m},n_{h})\!\times\!L

denotes

L

repeated VisionHOPE blocks with block width

D

, internal width

D_{m}

, and

n_{h}

heads per direction. The head dimension is

d=16

Figure 4: Efficiency comparison among DeiT, Vim, and P-VisionHOPE at the Tiny and Small scales. We plot FLOPs, throughput, and memory footprint for input resolutions from

224\times 224

1280\times 1280

. Hardware measurements use one NVIDIA A100 GPU with BF16 and a fixed batch size of

8

Method	FLOPs	$\mathbf{AP}^{b}$	$\mathbf{AP}^{b}_{50}$	$\mathbf{AP}^{b}_{75}$	$\mathbf{AP}^{m}$	$\mathbf{AP}^{m}_{50}$	$\mathbf{AP}^{m}_{75}$
ConvNeXt-T	262	46.2	67.9	50.8	41.7	65.0	44.9
InternImage-T	270	49.1	70.4	54.1	43.7	67.3	47.3
CSWin-T	279	49.0	70.7	53.7	43.6	67.9	46.6
MILA-T	255	48.8	71.0	53.6	43.8	68.0	46.8
VMamba-T	271	48.8	70.4	53.5	43.7	67.4	47.0
LocalVMamba-T	291	48.7	70.1	53.0	43.4	67.0	46.4

Appendix

Table 8: COCO object detection and instance segmentation with Mask R-CNN under the

3\times

schedule. FLOPs and AP are reported in G and %. VisionHOPE delivers competitive performance.

$D_{m}/D$	#P	FLOPs	Acc.
$1/4$	20	4.2	81.5
$1/2$	22	4.7	82.3
$1$	27	6.1	82.3
$2$	44	9.8	82.5

Appendix

Table 9: Configuration ablations on P-VisionHOPE-S. The four panels vary

D_{m}/D

, chunk length,

\gamma_{\eta}

, and

d

, respectively. Parameters, FLOPs, and Top-1 accuracy are reported in M, G, and %.

Method	Acc.
MILA-T (w/o MESA)	83.3
MILA-T (w/ MESA)	83.5
H-ViT 3 -T (w/o MESA)	83.5
H-ViT 3 -T (w/ MESA)	84.0
VisionHOPE-T (w/o MESA)	83.7
VisionHOPE-T (w/ MESA)	84.1

Appendix

Table 10: ImageNet-1K classification of hierarchical models with and without MESA. The panels correspond to Tiny, Small, and Base models from left to right. The results for MILA and H-ViT 3 are taken from their respective papers ( Han et al., 2024a ; Han et al., 2026 ) . Top-1 accuracy is reported in %.

Figure 5: Qualitative comparison of global perception through Effective Receptive Fields (ERFs) at $512\times 512$ resolution. The maps are computed from the geometrically centered $2\times 2$ features at the native output of the second processing block and averaged across the complete ImageNet-1K validation set. Darker colors indicate stronger relative influence on the centered representation.

Figure 6: Spatial variation of the final executed step in P-VisionHOPE-S across several ImageNet-1K validation images. Each two-row block pairs input images with the corresponding $14\times 14$ maps of $\widetilde{\eta}_{t}$ from the final VisionHOPE block, restored to image coordinates and averaged across heads and four scan directions. Darker colors indicate relatively larger executed steps within each image.

Explore similar work

Self-supervised pretraining for an iterative image size agnostic vision transformer
Apr 22, 2026Nedyalko Prisadnikov, Danda Pani Paudel, Yuqian Fu +1Self-Supervised Vision Transformers Vision Encoders
INSAIT, Sofia University "St. Kliment Ohridski", Bulgaria
Adaptive Hebbian Memory Routing in Vision Transformers for Few-Shot Learning
Jun 23, 2026Mohammed Yusuf Mujawar, Noorbakhsh Amiri GolilarzFew-Shot Learning Vision Transformer
Department of Computer Science The University of Alabama, Tuscaloosa, AL, USA
W2Rep: Learning Visual Representations by Watching the World Change
Sep 28, 2026Wen Huang, Hang Guo, Jiarui Yang +3Fine-Grained Video Understanding Vision Encoders
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Nankai University, Tianjin, China · Shenzhen University, Shenzhen, China

Apr 22, 2026, cs.CVJ/K move · Enter open · S save

Self-supervised pretraining for an iterative image size agnostic vision transformer

Nedyalko Prisadnikov, Danda Pani Paudel, Yuqian Fu, Luc Van Gool

INSAIT, Sofia University "St. Kliment Ohridski", Bulgaria

Self-Supervised Vision Transformers Vision Encoders Self-Supervised Learning Dino

Open paper AskPDF arXiv

Vision Transformers (ViTs) dominate self-supervised learning (SSL). While they have proven highly effective for large-scale pretraining, they are computationally inefficient and scale poorly with image size. Consequently, foundational models like DINO are constrained to low-resolution processing. A recent foveal-inspired transformer achieves resolution agnosticism by iteratively processing a fixed-size context of multi-zoom patches. This model demonstrated promising results via supervised learning, utilizing a sequential, recurrent-like process without backpropagation through time. To unlock its potential as a foundational backbone, we introduce a novel sequential-to-global SSL framework based on DINO's self-distillation objective. Supported by an efficient integral-image patch extraction method, our approach enables large-scale pretraining for image-size agnostic vision encoders. We achieve competitive performance on ImageNet-1K and downstream classification tasks, maintaining a constant computational budget regardless of input resolution.

Abstract

Explore similar work

Self-supervised pretraining for an iterative image size agnostic vision transformer

Adaptive Hebbian Memory Routing in Vision Transformers for Few-Shot Learning

W2Rep: Learning Visual Representations by Watching the World Change