Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training

Quan Kong; Yanru Xiao; Yuhao Shen; Cong Wang

cs.CVFeb 28, 2026

Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training

Authors: Quan Kong, Yanru Xiao, Yuhao Shen, Cong Wang

Organizations: Zhejiang University, Hangzhou, China · Amazon Web Services, USA

Abstract

Learning efficient and expressive visual representation has long been the pursuit of computer vision research. While Vision Transformers (ViTs) gradually replace traditional Convolutional Neural Networks (CNNs) as more scalable vision learners, their applications are plagued by the quadratic complexity of the self-attention mechanism. To address the challenge, we introduce a new linear-time sequence modeling method Test-Time Training (TTT) into vision and propose Vision-TTT, which treats visual sequences as datasets and compresses the visual token sequences in a novel self-supervised learning manner. By incorporating the dual-dataset strategy and Conv2d-based dataset preprocessing, Vision-TTT effectively extends vanilla TTT to model 2D visual correlations with global receptive fields. Extensive experiments show that Vittt-T/S/B achieve $77.7\%,81.8\%,82.7\%$ Top-1 accuracy on ImageNet classification and also greatly outperform their counterparts on downstream tasks. At $1280\times1280$ resolution, Vittt-T reduces FLOPs by $79.4\%$ and runs $4.72\times$ faster with $88.9\%$ less memory than DeiT-T. These results demonstrate the expressiveness and efficiency of Vision-TTT as a strong candidate for the next-generation generic visual backbone.

Figures & tables

Panel of Figure 1 — Figure 1: Performance and efficiency comparison between DeiT [ 53 ] , Vim [ 69 ] and our Vittt model. Results show that Vittt not only achieves superior performance on ImageNet classification and downstream detection and segmentation tasks, but also is more computation and memory efficient in dealing with high-resolution images.

Figure 2: Basics of TTT. The core idea is to update the hidden state $\mathrm{W}$ with steps of self-supervised gradient descent and then forward the output to the next layer.

Figure 3: Gradient Magnitude Map (illustrated by Eq. ( 6 )). Vision-TTT employs gradients as the explicit indicators to quantify visual token semantics.

Figure 4: Overall architecture of Vision-TTT. It consists of three stages: Patchification, Vision-TTT Encoder, and Task Adapters to learn representation for vision tasks.

Figure 5: Vittt Block. It evolves from vanilla TTT with the dual dataset strategy and the Conv2d-based dataset preprocessing mechanism for inner self-supervised learning.

CNNs
Method	size	#P.	FLOPs	Acc.
RegNetY-4G	$224^{2}$	12M	4G	80.0
RegNetY-8G	$224^{2}$	25M	8G	81.7
RegNetY-16G	$224^{2}$	45M	16G	82.9
EffNet-B4	$380^{2}$	19M	4.2G	82.9
EffNet-B5	$456^{2}$	30M	9.9G	83.6

Table 1: Performance comparison on ImageNet-1K classification. We report the parameters (#P.), FLOPs and Top-1 Accuracy (%) metrics. Left : hierarchical architectures; Right : non-hierarchical architectures.

Method	#Param.	FLOPs	$\mathrm{\ AP^{b}\ }$	$\mathrm{\ AP^{m}\ }$
ViT-T	8M	147.1G	41.6	37.9
VRWKV-T	8M	67.9G	41.7	38.0
Vim-T	9M	76.7G	41.8	38.2
ViT 3 -T	8M	69.2G	42.0	38.3
Vittt-T	9M	74.3G	42.9	38.7
ViT-S	28M	344.5G	44.9	40.1

Table 2: Performance comparison on downstream COCO2017 detection ( Left ) and ADE20K segmentation ( Right ) tasks. We report the parameters (#Param.), FLOPs and

\mathrm{AP^{b}}

\mathrm{AP^{m}}

\mathrm{mIoU}

metrics, respectively. FLOPs are calculated with

1333\times 800

resolution for detection and

512\times 512

resolution for segmentation tasks.

Panel of Figure 6 — Figure 6: Efficiency comparison between DeiT [ 53 ] , Vim [ 69 ] and our Vittt model. We sketch the FLOPs, throughput, and memory footprint curves for different resolutions ranging from $224\times 224$ (i.e. 196 tokens) to $1280\times 1280$ (i.e. 6400 tokens).

Design Route	#Param.	FLOPs	$\mathrm{\ FPS\ }$	$\mathrm{\ Acc.\ }$	$\mathrm{\ AP^{b}\ }$	$\mathrm{AP^{m}\ }$	$\mathrm{\ mIoU\ }$
TTT layer	5.87M	1.14G	3607	74.2	40.4	36.8	41.5
+ Share Q / K	5.43M	1.07G	3892	74.0	40.1	36.5	41.2
+ Gating	5.89M	1.15G	3565	75.2	41.2	37.1	42.0
+ Conv1d (vanilla TTT)	5.90M	1.16G	3346	75.6	41.6	37.6	42.6
+ Dual dataset strategy	6.96M	1.42G	2068	77.5	42.7	38.6	44.1
+ Conv2d dataset preprocessing	6.98M	1.44G	2029	77.7	42.9	38.7	44.3

Table 3: Performance comparison for different snapshots on the design route for Vittt-T . Throughput metric FPS (frames per second) is tested with batch size 64.

Method	#Param.	FLOPs	$\mathrm{\ Acc.\ }$	$\mathrm{\ AP^{b}\ }$	$\mathrm{\ AP^{m}\ }$	$\mathrm{\ mIoU\ }$
HeadClassTok	6.98M	1.44G	76.6	41.6	37.8	43.0
MidClassTok	6.98M	1.44G	76.8	41.7	37.9	43.0
DoubleClassTok	6.98M	1.45G	75.4	41.2	37.2	42.5
MaxPool	6.98M	1.44G	77.2	42.2	38.4	43.8
MeanPool	6.98M	1.44G	77.7	42.9	38.7	44.3

Table 4: Performance comparison for different classification strategies on Vittt-T .

Panel of Figure 7 — Figure 7: Ablation study of (a) mini-batch size $b$ to perform gradient descent along the sequence length $T$ , and (b) number of learnable initial states $\mathrm{W}_{0}$ (indicated by the superscript number 0 1 2 ) for the self-supervised learning of dual datasets.

Figure 8: Comparison of Effective Receptive Field (ERF) [ 38 ] . Pixels with higher intensity indicate larger responses related to the central pixel.

Figure 9: Gradient Magnitude Map of Vittt-T before and after training. Visual tokens with higher intensity indicate steeper gradients (Eq. ( 6 ))and greater importance.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: The Gradient Magnitude Map (GMM) of Vision-TTT, which employs gradient indicator $G_{t}$ to explicitly quantify token importance. Before training (the first row behind the images), the model attributes almost the same gradient magnitude to each token. After training by supervised class label signal (the second row behind the images), the marked Top-30% tokens clearly show the global semantics of the images.

Figure 11: The self-supervised reconstruction loss (L2 norm) of all the 196 tokens for epoch 0, 3, 300 inside 12 layers of Vittt-T (take the Forth TTT as the example), averaged over 128 images. For epoch 0 before training, the reconstruction loss is fairly equal among all the tokens due to random initialization of the network. During training, we observe that the losses initially rise to a certain level before finally dropping to a lower region. This is because at the very beginning the classification signal dominates the optimization to gain more drastic drop of classification loss. As the training progresses, this effect weakens. At a certain stage (around epoch 20-30), the TTT reconstruction loss starts to take over the leadership and performs deeper adaptation. For epoch 300, the final losses among the 12 layers show different patterns for diverse adaptability. Note that they all stay above zero for non-decaying adaptation across mini-batches.

Explore similar work

TuringViT: Making SOTA Vision Transformers Accessible to All
Jun 23, 2026Qiman Wu, Hanlin Chen, Lyujie Chen +19Vision Foundation Models Vision Transformer
Foundation Model Team, Xpeng Inc.
Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective
Jul 4, 2026Zhenfeng Su, Kang Zhao, Han Bao +4Vision Transformer Efficient ViTs
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · Huawei Technologies Co., Ltd., Shenzhen, China · Tsinghua University, Beijing, China +1
Accelerating Vision Foundation Models with Drop-in Depthwise Convolution
May 21, 2026Carmelo Scribano, Mohammad Mahdi, Nedyalko Prisadnikov +5Vision Foundation Models Vision Foundation Model Adaptation
University of Modena and Reggio Emilia, Modena, Italy. · INSAIT, Sofia University “St. Kliment Ohridski”, Sofia, Bulgaria

Jun 23, 2026J/K move · Enter open · S save

TuringViT: Making SOTA Vision Transformers Accessible to All

Qiman Wu, Hanlin Chen, Lyujie Chen, Rui Xin+18

Foundation Model Team, Xpeng Inc.

Vision Foundation Models Vision Transformer Efficient ViTs Linear Attention Vision-Language Pretraining

Open paper AskPDF arXiv

Modern VLMs and VLA systems commonly adopt off-the-shelf ViTs such as SigLIP2 as visual encoders, but diverse downstream requirements in latency, temporal modeling, and VLM integration often call for customized SOTA-level ViTs. Training such encoders remains beyond the reach of much of the community, as it requires massive image-text data, while standard softmax attention makes high-resolution or dynamic-resolution pretraining prohibitively costly and often forces low-resolution pretraining followed by post-hoc adaptation. TuringViT addresses these challenges with three key designs: Turing Linear Attention (TLA) for efficient sequence modeling, VISTA-Curation to construct supervision-rich image-video training data, and native dynamic-resolution pretraining that supports flexible inputs from the start and transfers seamlessly to downstream VLMs. As a result, TuringViT outperforms leading open-source ViT baselines with only 10% of the data, achieves stronger downstream VLM performance, and delivers substantially better latency scaling on high-resolution inputs. Our scaling-law analysis further shows that TuringViT continues to improve predictably with curated data scale, far from saturation. Its fast adaptation, hardware-friendly design, and efficient deployment have made it a unified visual foundation across XPeng's AI systems. More broadly, TuringViT provides a reproducible pipeline that dramatically lowers the cost for the community to train, customize, and deploy SOTA-level ViTs, moving toward making such Vision Transformers accessible to all.

Abstract

Explore similar work

TuringViT: Making SOTA Vision Transformers Accessible to All

Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective

Accelerating Vision Foundation Models with Drop-in Depthwise Convolution