Continuous Token-Level Spatio-Temporal Context Modeling for Visual Object Tracking

Ding Xia; Meiqin Liu; Jing Zhou; Jian Lan

cs.CVSep 7, 2026

Continuous Token-Level Spatio-Temporal Context Modeling for Visual Object Tracking

Authors: Ding Xia, Meiqin Liu, Jing Zhou, Jian Lan

Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an 710049, China. · College of Electrical Engineering, Zhejiang University, Hangzhou 310027, China. · School of Electronics and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China.

Abstract

Spatio-temporal context has become increasingly crucial for visual tracking. However, most existing approaches extract spatio-temporal cues via discrete sampling strategies, which inherently deviate from the continuity of spatio-temporal context, thereby deteriorating tracking performance. To address this challenge, we propose TLCTrack, a novel tracking framework that models token-level spatio-temporal context through continuously updated salient tokens, enabling more accurate target representation. Specifically, TLCTrack incorporates three components: Masked Unidirectional Attention (MUA), Spatial Salient Token Collection (SSTC), and Temporal Salient Token Bank (TSTB) modules. By explicitly integrating spatio-temporal context, MUA extracts discriminative targetaware spatial features in the search region. To avoid the negative impact of background on feature learning, SSTC progressively suppresses background interference, thereby enhancing target spatial representation. Finally, TSTB captures high-quality spatio-temporal information through continuous salient token updates. Extensive experiments on five benchmarks demonstrate that our method achieves superior performance over state-of-the-art trackers. Code and models are available at https://github.com/xiading123/TLCTrack.

Figures & tables

Fig. 1: Comparison of different tracking pipelines. (a): Offline learning. (b)-(c): Sampling-based discrete update strategies. (d): Token-level continuous update strategy.

Fig. 2: Comparison of the response between OSTrack and TLCTrack. TLCTrack consistently focuses on target-relevant region. (a)-(b): Search results with predicted boxes and template image (in the top-left corner). (c): Attention maps of OSTrack at time t. (d): Attention maps of TLCTrack at time t.

Fig. 3: (a) Architecture of the proposed TLCTrack. TLCTrack consists of four parts: a backbone with MUA for visual feature extraction, SSTC for collecting spatial salient tokens, TSTB for storing temporal salient tokens, and a prediction head for making predictions. (b) Unidirectional masking among three kinds of tokens in MUA. (c) The process of collecting spatial salient tokens in SSTC.

Method	GOT10k*			TrackingNet			TNL2K		NFS
Method	AO(%)	$\text{SR}_{0.5}\text{(\%)}$	$\text{SR}_{0.75}\text{(\%)}$	AUC(%)	$\text{P}_{Norm}\text{(\%)}$	P(%)	AUC(%)	P(%)	AUC(%)
OSTrack 384 [ 30 ]	73.7	83.2	70.8	83.9	88.5	83.2	55.9	-	66.5
SeqTrack 256 [ 6 ]	74.7	84.7	71.8	83.3	88.3	82.2	54.9	-	67.6
SeqTrack 384 [ 6 ]	74.5	84.3	71.4	83.9	88.8	83.6	56.4	-	66.7
ARTrack 256 [ 24 ]	73.5	82.2	70.9	84.2	88.7	83.5	57.5	-	64.3
ARTrack 384 [ 24 ]	75.5	84.3	74.3	85.1	89.1	84.8	59.8	-	66.8

TABLE I: Comparison with state-of-the-arts on four benchmarks: GOT10k, TrackingNet, TNL2K and NFS. * denotes for trackers only trained on GOT10k. Best in bold, second best underlined.

Method	Params $\downarrow$	FLOPs $\downarrow$	Speed $\uparrow$	Memory $\downarrow$
SeqTrack 256	89M	66G	52 fps	1653M
SeqTrack 384	89M	148G	22 fps	3855M
ARTrack 256	173M	37G	62 fps	1287M
ARTrack 384	173M	83G	43 fps	1343M
TLCTrack ${}_{\textbf{256}}$	92M	28G	97 fps	951M
TLCTrack ${}_{\textbf{384}}$	92M	58G	67 fps	1122M

TABLE II: Comparison of model parameters, FLOPs, inference speed, and memory overhead on an RTX 3090 GPU.

Fig. 4: State-of-the-art comparison on the OTB100 dataset. TLCTrack outperforms previous SOTA trackers. Best viewed with zooming in.

Fig. 5: Qualitative evaluation with other leading trackers (000078 and 000139 from GOT10k test split). TLCTrack consistently maintains stable target localization. Best viewed with zooming in.

(a) Study on key components					(b) Study on keeping ratio $\rho$				(c) Study on the loss weight $\lambda_{\text{aux}}$
#	Method	AO(%)	$\text{SR}_{0.5}\text{(\%)}$	$\text{SR}_{0.75}\text{(\%)}$	$\rho$	AO(%)	$\text{SR}_{0.5}\text{(\%)}$	$\text{SR}_{0.75}\text{(\%)}$	$\lambda_{\text{aux}}$	AO(%)	$\text{SR}_{0.5}\text{(\%)}$	$\text{SR}_{0.75}\text{(\%)}$
1	baseline	72.0	81.6	69.5	0.7	74.0	83.6	72.8	0.1	74.3	84.5	73.2
2	+TSTB	73.3	82.7	71.6	0.8	75.4	86.3	74.3	0.3	74.9	85.7	73.8
3	+MUA	74.6	84.8	73.2	0.9	74.9	85.3	73.7	0.5	75.4	86.3	74.3
4	+SSTC	75.4	86.3	74.3	1.0	74.6	84.8	73.2	0.7	74.6	85.3	73.7

TABLE III: Three ablation studies on the GOT10k benchmark.

Spatial Context Modeling

See all 16 papers

An Efficient Token Compression Framework for Visual Object Tracking
May 8, 2026Weijing Wu, Qihua Liang, Bineng Zhong +3Visual Object Tracking Efficient ViTs
Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University, Guilin, 541004, China · Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin, 541004, China · University Engineering Research Center of Educational Intelligent Technology, Guangxi Normal University, Guilin, 541004, China +1
Drift-Resilient Temporal Priors for Visual Tracking
Apr 3, 2026Yuqing Huang, Liting Lin, Weijun Zhuang +2Visual Object Tracking
Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · Lero, the Research Ireland Centre for Software, University of Limerick +1
Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking
Aug 29, 2026Han Wang, Yuxuan Liu, Yuhan Sun +5Visual Object Tracking Multimodal Visual Object Tracking
Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences · University of Chinese Academy of Sciences

May 8, 2026J/K move · Enter open · S save

An Efficient Token Compression Framework for Visual Object Tracking

Weijing Wu, Qihua Liang, Bineng Zhong, Haiying Xia+2

Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University, Guilin, 541004, China · Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin, 541004, China · University Engineering Research Center of Educational Intelligent Technology, Guangxi Normal University, Guilin, 541004, China+1

Visual Object Tracking Efficient ViTs Visual Token Pruning Visual Token Compression

Open paper AskPDF arXiv

Refining visual representations by eliminating their internal feature-level redundancy is crucial for simultaneously optimizing the performance and computational cost of models in visual tracking. To enhance their performance, many contemporary Transformer-based trackers leverage a larger number of historical template frames to capture richer spatio-temporal cues. However, this strategy leads to a massive number of input visual tokens. This creates two critical issues: it imposes a quadratic computational burden and can also degrade the tracker's overall performance. To bridge this gap, we propose a compress-then-interact tracking framework, ETCTrack, that learns to efficiently compress template tokens from historical template frames into a robust target representation, moving beyond handcrafted rules. Our method first employs the Adaptive Token Compressor to dynamically construct compact yet highly discriminative template tokens by filtering out redundant visual tokens. These refined template tokens are then processed by our Hierarchical Interaction Encoder to achieve a deep, adaptive interaction with the search features. Refined search features ensure subsequent precise target localization. Experiments on seven benchmarks demonstrate that our method outperforms current state-of-the-art trackers. ETCTrack-B224 reduces the number of template tokens by 60%, leading to a 21.4% reduction in MACs with only a 0.4% drop in accuracy. The source code are available at https://github.com/PJD-WJ/ETCTrack.

Abstract

Spatial Context Modeling

Explore similar work

An Efficient Token Compression Framework for Visual Object Tracking

Drift-Resilient Temporal Priors for Visual Tracking

Efficient Language-to-Vision Feature Injection for Referring Single-Object Tracking