UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking

Hao Wu; Xudong Wang; Jialiang Zhang; Junlong Tong; Xinghao Chen; Junyan Lin; Yunpu Ma; Xiaoyu Shen

cs.CVFeb 27, 2026

UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking

Authors: Hao Wu, Xudong Wang, Jialiang Zhang, Junlong Tong, Xinghao Chen, Junyan Lin, Yunpu Ma, Xiaoyu Shen

Organizations: Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · Shanghai Jiao Tong University · The Hong Kong Polytechnic University ·

Abstract

One-stream Transformer-based trackers achieve advanced performance in visual object tracking but suffer from significant computational overhead that hinders real-time deployment. While token pruning offers a path to efficiency, existing methods are fragmented. They typically prune the search region, dynamic template, and static template in isolation, overlooking critical inter-component dependencies, which yields suboptimal pruning and degraded accuracy. To address this, we introduce UTPTrack, a simple and Unified Token Pruning framework that, for the first time, jointly compresses all three components. UTPTrack employs an attention-guided, token type-aware strategy to holistically model redundancy, a design that seamlessly supports unified tracking across multimodal and language-guided tasks within a single model. Extensive evaluations on 10 benchmarks demonstrate that UTPTrack achieves a new state-of-the-art in the accuracy-efficiency trade-off for pruning-based trackers, pruning 65.4% of vision tokens in RGB-based tracking and 67.5% in unified tracking while preserving 99.7% and 100.5% of baseline performance, respectively. This strong performance across both RGB and multimodal scenarios underlines its potential as a robust foundation for future research in efficient visual tracking. Code will be released at https://github.com/EIT-NLP/UTPTrack.

Figures & tables

Figure 1: UTPTrack supports RGB-based and unified tracking, prunes redundant tokens in the search region (SR), dynamic template (DT), and static template (ST) to improve efficiency, with $r$ indicating the token retention ratio for each component.

Figure 2: Architecture of the proposed UTPTrack. UTPTrack supports both RGB-based and unified tracking. It adopts a one-stream transformer that jointly processes tokens from the search region (SR), dynamic template (DT), and static template (ST). A lightweight Candidate or Template Elimination Module (CTEM) is inserted into encoder layers to prune redundant tokens from all three sources. In the figure, D/T/E denote depth, thermal, and event modalities, respectively.

Method	Base Model	Params (M)	MACs (G)	FPS (GPU)	FPS (CPU)
UTPTrack-O 256	OSTrack-256	92	24 (-11)	95 (+1)	17 (+8)
UTPTrack-O 384	OSTrack-384	92	53 (-25)	47 (+7)	6 (+3)
UTPTrack-S 224	SUTrack-B224	70 (+85)	16 (-7)	43 (+2)	9 (+1)
UTPTrack-S 384	SUTrack-B384	70 (+85)	48 (-19)	31 (+4)	5 (+2)

Table 1: Details of UTPTrack model variants.

Figure 3: Performance comparison of UTPTrack and other pruning methods under each method’s default compression settings at two resolutions. Top (High Resolution): 384 (RGB and Unified). Bottom (Low Resolution): 256 (RGB), and 224 (Unified).

Method	LaSOT	LaSOT ext	TrkNet	GOT-10k	Avg Perf. (%)
Upper Bound, 384 Tokens (100%)
OSTrack 256 (Baseline)	67.9	47.6	84.5	74.9	100%
Retain 87.2% Tokens in Average ( $\approx$ 335 tokens, $\downarrow$ 18.8%)
OSTrack-CE 256	67.1	46.9	84.1	74.9	99.3% ( $\downarrow 0.7\%$ )
OSTrack-ToMe 256	68.0	46.4	83.8	74.3	99.0% ( $\downarrow 1.0\%$ )
OSTrack-EViT 256	66.8	46.7	84.1	75.0	99.0% ( $\downarrow 1.0\%$ )

Table 2: Performance comparisions under different vision token configurations across RGB-based tracking . All methods are applied on the same base model OSTrack 256 . The average performance listed is calculated across all four benchmarks.

	RGB				RGB-D	RGB-T		RGB-E	RGB-Lang
Method	LaSOT	LaSOT ext	TrkNet	GOT-10k	VOT- RGBD22	LasHeR	RGBT234	VisEvent	TNL2K	OTB99	Avg Perf. (%)
Upper Bound, 294 Tokens (100%)
SUTrack 224 (Baseline)	73.7	53.2	85.9	77.5	75.5	59.9	70.0	63.3	67.8	68.4	100%
Retain 71.4% Tokens in Average ( $\approx$ 218 tokens, $\downarrow$ 25.6% )
SUTrack-CE 224	72.4	52.8	85.1	77.4	75.8	59.9	70.0	61.8	66.0	70.5	99.5% ( $\downarrow 0.5\%$ )
SUTrack-ToMe 224	72.1	52.6	85.4	76.9	76.0	60.0	69.0	62.3	66.2	70.4	99.4% ( $\downarrow 0.6\%$ )

Table 3: Performance comparisons under different vision token compression configurations across unified tracking . All methods are applied on the same base model SUTrack 224 . The best result are bolded and the second best results are underlined in all following tables. The average performance listed is calculated across all 10 benchmarks. TrackingNet is abbreviated as TrkNet for brevity.

#	Method	LaSOT	LaSOT ext	TrkNet	GOT-10k	Avg Vis Tok	Cmp Vis Tok	MACs (G)	Avg Perf. (%)	$\Delta$
1	Baseline	67.9	47.6	84.5	74.9	384	384	34.5	100.0	-
2	+ CE	67.4	46.6	83.7	75.5	291	217	27.0	99.3	$\downarrow$ 0.7
3	+ DTE	68.4	46.6	83.8	75.3	271	176	25.4	99.6	$\uparrow$ 0.3
4	+ STE	67.2	46.5	84.3	74.4	252	135	23.8	98.9	$\downarrow$ 0.7
5	+ TTA	67.4	47.3	84.0	75.2	252	135	23.8	99.7	$\uparrow$ 0.8

Table 4: Ablation Study on RGB-based Trackers .

\Delta

denotes the averaged performance change from the to row above.

#	Method	LaSOT	VOT- RGBD22	LasHeR	VisEvent	OTB99	Avg Vis Tok	Cmp Vis Tok	MACs (G)	Avg Perf. (%)	$\Delta$
1	Baseline	73.7	75.5	59.9	63.3	68.4	294	294	21.8	100.0	-
2	+ CE	72.8	76.7	59.6	62.8	70.8	223	166	18.3	99.9	$\downarrow$ 0.1
3	+ DTE	72.8	76.3	60.3	62.5	70.8	206	128	17.3	99.7	$\downarrow$ 0.2
4	+ STE	72.3	76.7	60.1	61.4	70.1	188	90	16.2	99.3	$\downarrow$ 0.4
5	+ TTA	72.4	75.9	59.4	61.9	70.7	188	90	16.2	99.7	$\uparrow$ 0.4
6	+ TG	73.2	76.7	59.3	62.2	71.6	188	90	16.2	100.0	$\uparrow$ 0.3

Table 5: Ablation Study on Unified Trackers .

\Delta

denotes the average performance change from the row above.

Figure 4: Ablation Study on Progressive Pruning. Performance and the number of vision tokens are reported as the keep ratio decreases and CE, DTE, and STE are progressively enabled for the RGB-based tracker (top) and unified tracker (bottom).

#	CE Location	DTE Location	LaSOT	VOT- RGBD22	LasHeR	VisEvent	OTB99	Avg Vis Tok	Cmp Vis Tok	MACs (G)	Avg Perf. (%)
1	-	-	73.7	75.5	59.9	63.3	68.4	294	294	22.8	100.0
2	[6,12,18]	[6,12,18]	71.9	76.0	59.4	62.5	70.4	201	128	17.0	99.8
3	[6,12,18]	[9,15,21]	72.8	76.3	60.3	62.5	70.8	206	128	17.3	100.6
4	[6,12,18]	[3,9,15]	71.8	76.1	60.0	62.5	70.5	196	128	16.7	100.0
5	[3,9,15]	[6,12,18]	71.6	75.9	58.3	62.0	70.4	185	128	16.0	99.2
6	[9,15,21]	[6,12,18]	71.9	76.3	59.7	62.3	70.2	217	128	18.0	99.9

Table 6: Ablation Study on CTEM Location for Unified Trackers with a 24-layer HiViT backbone.

#	Injection Target			LaSOT	VOT- RGBD22	LaHeR	VisEvent	OTB99	Avg Perf. (%)
#	SR	DT	ST	LaSOT	VOT- RGBD22	LaHeR	VisEvent	OTB99	Avg Perf. (%)
1				72.4	75.9	59.4	61.9	70.7	99.7
2	✓			72.7	76.0	59.8	61.7	70.7	99.4
3		✓		73.2	76.7	59.3	62.2	71.6	100.0
4			✓	72.7	76.3	59.0	62.2	70.8	99.3
5	✓	✓		72.2	76.5	59.4	62.0	70.9	99.7

Table 7: Ablation study of selectively injecting language cues into different components for text-guided token pruning. Average performance is reported relative to the baseline of unified trackers.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Train Hours	FPS (GPU)	FPS (CPU)	Lat. (GPU, ms)	Lat. (CPU, ms)
OSTrack 256	38.3	94.0	9.0	15.0	66.5
UTPTrack-O 256	32.6	95.0	16.7	12.7	49.5
OSTrack 384	101.4	39.8	3.2	36.1	340.8
UTPTrack-O 384	80.6	47.3	6.0	30.2	212.6
SUTrack 224	56.8	40.5	7.8	17.4	66.4
UTPTrack-S 224	53.3	42.7	9.3	17.8	56.4

Appendix

Table 8: Efficiency comparison. Lat.: per-layer backbone latency. GPU: NVIDIA 1080Ti; CPU: Intel Xeon Gold [email protected].

Method	LaSOT			LaSOT ext			TrackingNet			GOT-10k
Method	AUC	P Norm	P	AUC	P Norm	P	AUC	P Norm	P	AO	SR 0.5	SR 0.75
Trackers with Resolution 256
OSTrack 256 (Baseline)	67.9	76.9	73.3	47.6	57.5	53.6	84.5	89.0	83.0	74.9	84.2	73.0
OSTrack-CE 256	67.4	76.5	72.8	46.6	56.5	52.3	83.7	88.3	82.4	75.5	85.5	73.8
OSTrack-ToMe 256	67.6	76.6	72.6	46.5	56.0	52.5	84.0	88.6	82.6	73.4	83.4	72.4
OSTrack-EViT 256	68.4	77.7	74.1	46.1	55.7	51.1	84.0	88.7	82.9	75.8	85.9	74.6

Appendix

Table 9: SOTA comparison on RGB-based tracker across RGB-Based tracking.

Method	LaSOT			LaSOT ext			TrackingNet			GOT-10k
Method	AUC	P Norm	P	AUC	P Norm	P	AUC	P Norm	P	AO	SR 0.5	SR 0.75
Trackers with Resolution 224
SUTrack 224 (Baseline)	73.7	83.8	80.7	53.2	64.5	61.6	85.9	90.4	85.7	77.5	86.7	78.1
SUTrack-CE 224	72.8	82.9	80.2	52.9	63.7	60.4	85.3	90.0	84.8	77.6	87.6	78.0
SUTrack-ToMe 224	71.4	81.2	78.3	52.0	62.9	59.3	85.2	90.0	84.6	77.5	87.4	77.5
SUTrack-EViT 224	71.3	81.9	78.3	51.3	62.7	58.5	84.5	89.5	83.5	76.4	86.5	75.7

Appendix

Table 10: SOTA comparison on unified tracker across RGB-Based tracking.

Method	VOT-RGBD22			LasHeR		RGBT234		VisEvent		TNL2K		OTB99
Method	EAO	Acc	Rob	AUC	P	MSR	MPR	AUC	P	AUC	P	AUC	P
Trackers with Resolution 224
SUTrack 224 (Baseline)	75.5	82.5	91.3	59.9	74.8	70.0	92.1	63.3	80.7	67.8	73.8	68.4	91.1
SUTrack-CE 224	76.7	82.7	92.4	59.6	74.0	70.1	92.4	62.8	79.9	65.7	71.1	70.8	92.7
SUTrack-ToMe 224	76.1	82.7	91.8	59.2	73.7	69.4	91.8	61.7	78.9	65.9	71.4	70.5	91.6
SUTrack-EViT 224	75.0	82.1	90.9	58.6	73.0	69.2	91.6	62.0	79.4	65.0	70.0	70.8	91.9

Appendix

Table 11: SOTA comparison on unified tracker across RGB-Depth, RGB-Thermal, RGB-Event, and RGB-Language tracking.

Method	LaSOT			LaSOT ext			TrackingNet			GOT-10k			Avg Perf.(%)
Method	AUC	P Norm	P	AUC	P Norm	P	AUC	P Norm	P	AO	SR 0.5	SR 0.75	Avg Perf.(%)
Upper Bound, 384 Tokens (100%)
OSTrack 256 (Baseline)	67.9	76.9	73.3	47.6	57.5	53.6	84.5	89.0	83.0	74.9	84.2	73.0	100%
OSTrack 256 (Baseline)	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
Retain 87.2% Tokens ( $\approx$ 335 tokens, $\downarrow$ 18.8%)
OSTrack-CE 256	67.1	75.8	72.0	46.9	56.7	52.4	84.1	88.8	83.0	74.9	85.0	73.5	99.3% ( $\downarrow 0.7\%$ )

Appendix

Table 12: Performance comparisons under different vision token compression settings for RGB-based tracking . The baseline uses 384 tokens. The first row shows raw scores, and the second row reports percentages relative to the upper bound.

#	CE Location	DTE Location	LaSOT	LaSOT ext	TrackingNet	GOT-10k	Avg Vis Tok	Cm Vis Tok	MACs (G)	Avg Perf. (%)
1	–	–	67.9	47.6	84.5	74.9	384.0	384	34.5	100.0
2	[3,6,9]	[3,6,9]	67.6	47.0	83.8	74.5	267.8	176	25.1	99.3
3	[3,6,9]	[4,7,10]	68.4	46.6	83.8	75.3	271.2	176	25.4	99.6
4	[3,6,9]	[2,5,8]	67.4	45.7	83.8	75.4	264.3	176	24.8	98.8
5	[2,5,8]	[3,6,9]	68.0	46.4	83.9	75.7	253.8	176	23.9	99.5
6	[4,7,10]	[3,6,9]	67.5	46.5	83.9	73.4	281.7	176	26.3	98.6

Appendix

Table 13: Ablation Study on CTEM Location for RGB-based Trackers with a 12-layer ViT backbone.

Method	LaSOT	LaSOT ext	TrackingNet	GOT-10k
Attn-based Pruning w/o Priors	67.2	46.5	84.3	74.4
Attn-based Pruning w/ Priors	67.4	47.3	84.0	75.2
Random Pruning w/ Priors	67.0	46.8	83.9	74.2

Appendix

Table 14: Effect of spatial priors in attention-guided pruning.

#	Bonus	LaSOT	VOT- RGBD22	LasHeR	VisEvent	OTB99	Avg Perf. (%)
1	Full	71.7	75.7	59.5	61.7	69.7	99.3
2	Soft	72.4	75.9	59.4	61.9	70.7	99.8
3	All	72.2	75.6	59.5	61.2	69.4	99.1

Appendix

Table 15: Ablation Study on bonus for Unified Trackers.

Method	LaSOT			LaSOT ext			TrackingNet			GOT-10k			Avg Perf.(%)
Method	AUC	P Norm	P	AUC	P Norm	P	AUC	P Norm	P	AO	SR 0.5	SR 0.75	Avg Perf.(%)
Upper Bound, 294 Tokens (100%)
SUTrack 224 (Baseline)	73.7	83.8	80.7	53.2	64.5	61.6	85.9	90.4	85.7	77.5	86.7	78.1	100%
SUTrack 224 (Baseline)	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
Retain 71.4% Tokens ( $\approx$ 218 tokens, $\downarrow$ 25.6%)
SUTrack-CE 224	72.4	82.4	79.2	52.8	63.6	60.2	85.1	89.7	84.4	77.4	87.0	78.0	99.5 %( $\downarrow 0.5\%$ )

Appendix

Table 16: Performance comparisons under different vision token compression configurations across unified tracking on RGB-based tracking benchmark. The vanilla number of visual tokens is 294. The first line of each method is the raw performance of benchmarks, and the second line is the proportion relative to the upper limit. The average performance listed is calculated across all 10 benchmarks.

Method	VOT-RGBD22			LasHeR		RGBT234		VisEvent		TNL2K		OTB99		Avg Perf.(%)
Method	EAO	Acc	Rob	AUC	P	MSR	MPR	AUC	P	AUC	P	AUC	P	Avg Perf.(%)
Upper Bound, 294 Tokens (100%)
SUTrack 224	75.5	82.5	91.3	59.9	74.8	70.0	92.1	63.3	80.7	67.8	73.8	68.4	91.1	100%
SUTrack 224	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%	100%
Retain 71.4% Tokens ( $\approx$ 218 tokens, $\downarrow$ 25.6%)
SUTrack-CE 224	75.8	82.8	91.4	59.9	74.7	70.0	92.5	61.8	79.2	66.0	71.5	70.5	92.6	99.5% ( $\downarrow 0.5\%$ )

Appendix

Table 17: Performance comparisons under different vision token compression configurations across unified tracking on RGB-D/T/E/Lang tracking benchmark. The vanilla number of visual tokens is 294. The first line of each method is the raw performance of benchmarks, and the second line is the proportion relative to the upper limit. The average performance listed is calculated across all 10 benchmarks.

Figure 5: Visualization of the UTPTrack Pruning Process for RGB-based Tracking.

Figure 6: Visualization of the UTPTrack Pruning Process for RGB-Language Tracking.

Figure 7: Visualization of the UTPTrack Pruning Process for RGB-Depth Tracking.

Figure 8: Visualization of the UTPTrack Pruning Process for RGB-Thermal Tracking.

Figure 9: Visualization of the UTPTrack Pruning Process for RGB-Event Tracking.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 171 papers

An Efficient Token Compression Framework for Visual Object Tracking
May 8, 2026Weijing Wu, Qihua Liang, Bineng Zhong +3Visual Object Tracking Efficient ViTs
Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University, Guilin, 541004, China · Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin, 541004, China · University Engineering Research Center of Educational Intelligent Technology, Guangxi Normal University, Guilin, 541004, China +1
UETrack: A Unified and Efficient Framework for Single Object Tracking
Mar 2, 2026Ben Kang, Jie Zhao, Xin Chen +5Multimodal Visual Object Tracking Visual Object Tracking
Dalian University of Technology · City University of Hong Kong
ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance
May 24, 2025Duo Li, Zuhao Yang, Xiaoqin Zhang +2Efficient VLM Inference Large Vision-Language Models
CCDS, NTU, Singapore · CCST, ZJUT, China · Terminus AI Lab, UCAS, China

May 8, 2026J/K move · Enter open · S save

An Efficient Token Compression Framework for Visual Object Tracking

Weijing Wu, Qihua Liang, Bineng Zhong, Haiying Xia+2

Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University, Guilin, 541004, China · Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin, 541004, China · University Engineering Research Center of Educational Intelligent Technology, Guangxi Normal University, Guilin, 541004, China+1

Visual Object Tracking Efficient ViTs Visual Token Pruning Visual Token Compression

Open paper AskPDF arXiv

Refining visual representations by eliminating their internal feature-level redundancy is crucial for simultaneously optimizing the performance and computational cost of models in visual tracking. To enhance their performance, many contemporary Transformer-based trackers leverage a larger number of historical template frames to capture richer spatio-temporal cues. However, this strategy leads to a massive number of input visual tokens. This creates two critical issues: it imposes a quadratic computational burden and can also degrade the tracker's overall performance. To bridge this gap, we propose a compress-then-interact tracking framework, ETCTrack, that learns to efficiently compress template tokens from historical template frames into a robust target representation, moving beyond handcrafted rules. Our method first employs the Adaptive Token Compressor to dynamically construct compact yet highly discriminative template tokens by filtering out redundant visual tokens. These refined template tokens are then processed by our Hierarchical Interaction Encoder to achieve a deep, adaptive interaction with the search features. Refined search features ensure subsequent precise target localization. Experiments on seven benchmarks demonstrate that our method outperforms current state-of-the-art trackers. ETCTrack-B224 reduces the number of template tokens by 60%, leading to a 21.4% reduction in MACs with only a 0.4% drop in accuracy. The source code are available at https://github.com/PJD-WJ/ETCTrack.

Abstract

Visual Object Tracking

Explore similar work

An Efficient Token Compression Framework for Visual Object Tracking

UETrack: A Unified and Efficient Framework for Single Object Tracking

ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevance