MaST: Motion-aware Sparse Pipeline for Lightweight Object Tracking

Qingmao Wei; Fagui Liu; Dengke Zhang; Qingze He; Quan Tang

cs.CVAug 25, 2026

MaST: Motion-aware Sparse Pipeline for Lightweight Object Tracking

Authors: Qingmao Wei, Fagui Liu, Dengke Zhang, Qingze He, Quan Tang

Organizations: South China University of Technology, Guangzhou, China

Abstract

Transformer-based object trackers are renowned for their strong performance, yet dense token processing often leads to prohibitive computational cost, limiting real-time deployment on edge devices. While recent works explore token pruning to reduce computation, they often stop short of an end-to-end sparse pipeline, as early-layer token scores can be noisy without a motion prior, and many trackers ultimately fall back to dense reshaping to feed the dense prediction head that partially negates the savings. We introduce Motion-aware Sparse Tracker (MaST), a sparse tracking framework that makes sparsity effective from tokens to boxes. First, MaST injects a lightweight motion prior to refine cross-attention-based importance scores, enabling earlier and more stable token reduction in the search region. Second, we introduce a natively sparse prediction head that operates directly on the retained unstructured tokens with a score-first, regress-once design, eliminating dense padding/reshaping and reducing redundant computation. Extensive experiments on multiple benchmarks demonstrate that MaST establishes new state of the art among lightweight trackers, where MaST-tiny attains 63.8 AUC on LaSOT and 80.1 SUC on TrackingNet, surpassing the prior best AsymTrack-S by +1.0 AUC and +2.2 SUC while running at 152 FPS on Jetson Nano, nearly twice as fast as AsymTrack-S at 88 FPS. Code is available at https://github.com/TsingWei/MaST.

Figures & tables

Figure 1: Speed–accuracy trade-off on edge platforms. The left panel shows TrackingNet SUC vs. Orin Nano speed (FPS). The right panel shows LaSOT AUC vs. CPU speed (FPS). Bubble area is proportional to model parameter count. Our MaST-{small, tiny, nano} achieve Pareto-optimal trade-offs, delivering competitive or superior accuracy at significantly higher speeds compared to prior lightweight trackers.

Figure 2: Why early-stage token pruning fails. (a) Attention map visualization across encoder layers on three tracking sequences. Early layers produce diffuse, noisy attention that has not yet localized the target, explaining why existing methods conservatively defer token pruning to intermediate layers. (b) Effect of performing one-shot token reduction (33% retention align to OSTrack [ 54 ] ) at different encoder layers on GOT-10k val . Without spatial guidance ( w/o G.T. ), accuracy drops sharply as pruning moves to earlier layers. With ground-truth target locations guiding selection ( w/ G.T. ), accuracy remains nearly flat across all layers while GMACs decrease substantially with earlier pruning.

Figure 3: Overview of the MaST framework. The General Sparsification Module is inserted once at an early encoder layer, retaining the top- $K$ search tokens; all subsequent Transformer blocks operate on this reduced set. Token selection fuses cross-attention relevance with a motion prior from the previous prediction for early and stable pruning. The retained sparse tokens are then decoded by a lightweight MLP sparse head with a score-first, regress-once pipeline, eliminating dense reshaping and redundant computation.

Figure 4: Dense vs. sparse prediction head. (A) Dense head pads and reshapes sparse tokens into a full 2D grid, wasting computation on empty positions. (B) Our sparse head applies an MLP directly on retained tokens via sparse anchors, with no padding or reshape.

Model	MaST
Model	nano	tiny	small
Backbone	ViT-Tiny
Encoder Layers	8	12	12
Template Size	128	128	192
Search Size	256	256	384
Token Rate (%)	30	30	30

Table 1: Details of our MaST model variants.

	Method	Pub.	LaSOT			TrackingNet			GOT-10k			MACs	FPS
	Method	Pub.	AUC	P Norm	P	SUC	P Norm	P	AO	SR 0.5	SR 0.75	(G)	RPi5	CPU	Nano
	KCF [ 23 ]	TPAMI’14	21.1	22.8	18.4	-	-	-	27.9	26.3	9.9	-	142	946	203 1 1 1 KCF is tested on the CPU of Jetson Orin Nano with 256 input size.
$\sim$ 500M MACs
	MaST-nano	Ours	58.6	67.7	59.4	77.2	82.5	72.4	61.6	71.8	53.7	0.585	30.1	101	230
	AsymTrack-T [ 59 ]	AAAI’25	60.8	68.7	61.2	76.2	80.9	71.6	62.3	71.3	54.7	0.708	18.1	60	99
	FEAR-XS [ 5 ]	ECCV’22	53.5	-	54.5	-	-	-	61.9	72.2	-	0.532	19.5	59	146

Table 2: State-of-the-art comparisons of efficient tracking on three large-scale benchmarks, grouped by computational cost. Red and blue denote the best and second-best results within each group. Gray FPS values indicate speeds below VOT realtime setting (20 FPS).

Table 7

Sparsification By	LaSOT			Encoder MACs	FPS
Sparsification By	AUC	P Norm	P	Encoder MACs	RPi5	CPU	Nano
None	64.0	74.2	68.3	1752M	9.1	37	94
Attention	60.5	71.9	65.6	824M	22.9	73	157
Prediction	59.4	71.4	64.9	874M	21.5	66	138
Attention + Motion	63.8	73.6	67.9	824M	22.6	72	152
Motion	62.6	72.2	66.4	824M	23.5	78	169

Table 5: Comparison of performance on various sparisfication on encoder.

Global Localization
Prediction Head	AUC		MACs (G)		RPi FPS
Prediction Head	Dense	Sparse	Dense	Sparse	Dense	Sparse
Loc Tokens [ 13 ]	59.0	57.9	1.76	0.845	10.0	23.9
Trans. Decoder	61.4	60.1	1.88	0.833	9.4	22.7
Anchor-based Localization
3 $\times$ 3 Conv [ 54 ]	65.0	64.1	2.39	1.463	7.7	13.8

Table 6: Comparison of prediction heads on LaSOT with ViT-Tiny backbone.

Figure 5: AUC–FPS trade-off at different token retention rates on GOT-10k val

Sparsify at Layer	LaSOT AUC	RPi FPS	MACs (M)
1	63.82	22.6	836
2	63.87	17.6	942
3	63.65	14.0	1004
4	63.83	12.1	1080
5	63.91	11.3	1154
6	63.96	10.1	1238

Table 7: Effect of sparsification layer on tracking accuracy and efficiency.

Compression Method	From	MACs (M)	LaSOT			FPS
Compression Method	From	MACs (M)	AUC	P Norm	P	RPi5	CPU	Nano
Full model	-	1752	64.0	74.2	68.3	9.1	37	94
$\downarrow$ Token (Progressive)	OSTrack [ 54 ]	1293	64.0	74.1	68.1	10.5	41	106
$\downarrow$ Resolution	LoReTrack [ 17 ]	990	58.5	67.7	59.4	23.6	83	176
$\downarrow$ Layer	CompressTracker [ 24 ]	937	59.1	68.5	60.1	24.1	89	177
$\downarrow$ Token (One-shot)	Ours	836	63.8	73.6	67.9	22.6	76	152

Table 8: Comparison with alternative model compression strategies under the same computational budget (

\approx

1G MACs, ViT-Tiny backbone). “

\downarrow

Resolution” reduces the input image size to (48,96); “

\downarrow

Layer” compresses the encoder into 4 layers; our method applies motion-aware token sparsification at Layer 1 with 30% retention.

Figure 6: Token sparsification visualization. Left (baseline) shows Search Region, Raw Importance, and Tokens Retained. Right (ours) shows Previous Prediction, Raw Importance, Motion Window, Re-ranked Importance, and Tokens Retained. Three tracking sequences are shown per row.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Window	Importance Score	Dynamic	AUC	P	P Norm
Type	Importance Score	Dynamic	AUC	P	P Norm
Gaussian	✓	✓	79.9	71.7	91.5
Gaussian	✓	-	79.5	70.5	90.2
Cosine	✓	✓	79.8	71.1	91.2
Cosine	✓	-	79.1	70.2	89.9
-	✓	-	76.9	68.2	86.5

Appendix

Table 9: Comparison of different windowing strategies on GOT-10k val . We evaluate the impact of window type (Gaussian or Cosine), the use of importance scores, and whether the window dynamically adapts to the previous frame’s bounding box size and aspect ratio.

Model	Input	%Tokens Retained	LaSOT	GOT-10k	FPS
			AUC	AO	(GPU)
ViT-base	$224^{2}$	100%	71.3	71.9	116
	$224^{2}$	30%	69.8	70.2	215
	$378^{2}$	100%	72.1	73.3	51
		30%	70.4	71.5	104
		20%	68.9	70.6	119

Appendix

Table 10: Results of scaling to a ViT-Base model and increasing input resolution. Accuracy (Acc) and FPS are reported.

Method	Pub.	DTB70		UAVDT		UAV123		Param. (M)	MACs (G)	FPS
Method	Pub.	SUC	P	SUC	P	SUC	P	Param. (M)	MACs (G)	RPi	Nano
HiFT [ 6 ]	ICCV’21	59.4	80.2	47.5	65.2	59.0	78.7	15.9	5.48	5.5	57
TCTrack [ 7 ]	CVPR’22	62.2	81.2	53.0	72.5	60.5	80.0	9.7	8.8	-	-
SGDViT [ 53 ]	ICRA’23	60.4	78.5	48.0	65.7	57.5	75.4	23.3	11.3	-	-
Aba-ViTrack-DeiT [ 30 ]	ICCV’23	66.4	85.9	59.9	83.4	66.4	86.4	7.98	2.39	7.9	83
SMAT [ 22 ]	WACV’24	63.8	81.9	58.7	80.8	64.6	81.8	3.76	2.49	7.7	77

Appendix

Table 11: State-of-the-art comparison on three UAV-oriented tracking benchmarks. Methods above the double rule are traditional correlation-filter trackers; those below are deep trackers. Red and blue indicate the best and second-best results overall.

Method	Pub.	LaSOT	TrackingNet	GOT-10k	Params (M)	MACs (G)	FPS
Method	Pub.	LaSOT	TrackingNet	GOT-10k	Params (M)	MACs (G)	RPi5	M4	14900	Nano	3090
KCF [ 23 ]	TPAMI’14	21.1	-	27.9	-	-	142	946	679	203 2 2 2 KCF is tested on the CPU of Jetson Orin Nano with 256 input size.	-
LightTrack [ 52 ]	CVPR’21	53.8	72.5	61.1	1.96	0.483	13.4	44	260	124	398
FEAR-XS [ 5 ]	ECCV’22	53.5	-	61.9	6.21	0.532	19.5	59	453	146	650
E.T.Track [ 3 ]	WACV’23	59.1	75.0	-	6.98	1.56	8.8	24	100	71	194
HiT-Tiny [ 27 ]	ICCV’23	54.8	74.6	52.6	9.59	0.995	27.3	97	176	118	253

Appendix

Table 12: State-of-the-art comparisons of efficient tracking on three large-scale benchmarks, grouped by computational cost. Red and blue denote the best and second-best results within each group. Gray FPS values indicate speeds below VOT realtime setting (20 FPS).

Method	@30 fps			@240 fps			Nano
Method	AUC	P Norm	P	AUC	P Norm	P	FPS
OSTrack-Tiny [ 54 ]	65.1	81.2	78.4	65.8	81.3	78.5	83
MaST-tiny	66.2	81.8	78.7	68.7	85.3	82.5	152

Appendix

Table 13: Comparison on NFS [ 28 ] under different input frame-rate settings. We compare MaST-tiny with OSTrack-Tiny under the standard 30 FPS and 240 FPS protocols. Higher input frame rates provide a more stable motion prior, while lower tracker latency improves the ability to exploit this temporal continuity in practice. Please fill in the numeric entries.

Figure 7: Robustness analyses. Left: per-frame IoU on the LaSOT bus-2 sequence, with representative predictions around an occlusion. Right: AUC on challenging LaSOT attributes. Under the same $4\times$ search factor and 256-pixel search input as FERMT, MaST-tiny is less robust to severe occlusion and fast motion; increasing the candidate field to a $5\times$ factor and 384-pixel input allows MaST-small to surpass FERMT while using much less computation.

Figure 8: Token keep frequency on GOT-10k val . We align search crops to a common grid and accumulate binary keep/drop decisions. Warmer cells are retained more often. The product of appearance attention and the soft motion prior concentrates computation near the likely target while retaining off-center evidence.

Figure 9: Tracking a robot when it engage fast movement, leading to partially out-of-view. Sparsification without motion-aware fixing would quickly discard the features of the target, making the tracker focus on other objects in the view.

Figure 10: Handling small objects like UAVs under extremely bad situation. With or without sparsification or motion-aware sparsification, the tracker is difficult to keep tracking the target.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 174 papers

Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking
May 7, 2026Shiao Wang, Xiao Wang, Duoqing Yang +5Event-Based Vision Mixture of Experts
School of Computer Science and Technology, Anhui University, Hefei 230601, China · Beijing Institute of Technology, Beijing, China · Peng Cheng Laboratory, Shenzhen, China +2
SpikingMOT: A Spike-Driven Multi-Object Tracker
Jul 22, 2026Yiding Sun, Xiangyang Yang, Dongxu Zhang +7Activation Sparsity Spiking Neural Networks
1Xi’an Jiaotong University · 2Peking University · 3Guilin University of Technology
An Efficient Token Compression Framework for Visual Object Tracking
May 8, 2026Weijing Wu, Qihua Liang, Bineng Zhong +3Visual Object Tracking Efficient ViTs
Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University, Guilin, 541004, China · Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University, Guilin, 541004, China · University Engineering Research Center of Educational Intelligent Technology, Guangxi Normal University, Guilin, 541004, China +1

May 7, 2026J/K move · Enter open · S save

Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking

Shiao Wang, Xiao Wang, Duoqing Yang, Wenhao Zhang+4

School of Computer Science and Technology, Anhui University, Hefei 230601, China · Beijing Institute of Technology, Beijing, China · Peng Cheng Laboratory, Shenzhen, China+2

Event-Based Vision Mixture of Experts Visual Object Tracking Vision Transformer Adaptive Computation

Open paper AskPDF arXiv

Despite significant progress, RGB-based trackers remain vulnerable to challenging imaging conditions, such as low illumination and fast motion. Event cameras offer a promising alternative by asynchronously capturing pixel-wise brightness changes, providing high dynamic range and high temporal resolution. However, existing event-based trackers often neglect the intrinsic spatial sparsity and temporal density of event data, while relying on a single fixed temporal-window sampling strategy that is suboptimal under varying motion dynamics. In this paper, we propose an event sparsity-aware tracking framework that explicitly models event-density variations across multiple temporal scales. Specifically, the proposed framework progressively injects sparse, medium-density, and dense event search regions into a three-stage Vision Transformer backbone, enabling hierarchical multi-density feature learning. Furthermore, we introduce a sparsity-aware Mixture-of-Experts module to encourage expert specialization under different sparsity patterns, and design a dynamic pondering strategy to adaptively adjust the inference depth according to tracking difficulty. Extensive experiments on FE240hz, COESOT, and EventVOT demonstrate that the proposed approach achieves a favorable trade-off between tracking accuracy and computational efficiency. The source code will be released on https://github.com/Event-AHU/OpenEvTracking.

Abstract

Visual Object Tracking

Explore similar work

Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking

SpikingMOT: A Spike-Driven Multi-Object Tracker

An Efficient Token Compression Framework for Visual Object Tracking