Uni-MDTrack: Learning Decoupled Memory and Dynamic States for Parameter-Efficient Visual Tracking in All Modality

Wenrui Cai; Zhenyi Lu; Yuzhe Li; Yongchao Feng; Jinqing Zhang; Qingjie Liu; Yunhong Wang

cs.CVMar 15, 2026

Uni-MDTrack: Learning Decoupled Memory and Dynamic States for Parameter-Efficient Visual Tracking in All Modality

Authors: Wenrui Cai, Zhenyi Lu, Yuzhe Li, Yongchao Feng, Jinqing Zhang, Qingjie Liu, Yunhong Wang

Organizations: School of Computer Science and Engineering, Beihang University, Beijing 100191, China

Abstract

With the advent of Transformer-based one-stream trackers that possess strong capability in inter-frame relation modeling, recent research has increasingly focused on how to introduce spatio-temporal context. However, most existing methods rely on a limited number of historical frames, which not only leads to insufficient utilization of the context, but also inevitably increases the length of input and incurs prohibitive computational overhead. Methods that query an external memory bank, on the other hand, suffer from inadequate fusion between the retrieved spatio-temporal features and the backbone. Moreover, using discrete historical frames as context overlooks the rich dynamics of the target. To address the issues, we propose Uni-MDTrack, which consists of two core components: Memory-Aware Compression Prompt (MCP) module and Dynamic State Fusion (DSF) module. MCP effectively compresses rich memory features into memory-aware prompt tokens, which deeply interact with the input throughout the entire backbone, significantly enhancing the performance while maintaining a stable computational load. DSF complements the discrete memory by capturing the continuous dynamic, progressively introducing the updated dynamic state features from shallow to deep layers, while also preserving high efficiency. Uni-MDTrack also supports unified tracking across RGB, RGB-D/T/E, and RGB-Language modalities. Experiments show that in Uni-MDTrack, training only the MCP, DSF, and prediction head, keeping the proportion of trainable parameters around 30%, yields substantial performance gains, achieves state-of-the-art results on 10 datasets spanning five modalities. Furthermore, both MCP and DSF exhibit excellent generality, functioning as plug-and-play components that can boost the performance of various baseline trackers, while significantly outperforming existing parameter-efficient training approaches.

Figures & tables

Fig. 1: Comparison of existing methods for introducing spatio-temporal context features to one-stream trackers, including the parameter-efficient fine-tuning paradigms. (a) Introducing a memory bank, where the memory features are fused before the prediction head. (b) Introducing auxiliary templates and temporally propagated tokens. (c) Introducing memory-aware compressed tokens as prompts at the model input, and incorporating dynamic state features of the target into the multi-stage layers of the backbone.

Fig. 2: The overall architecture of Uni-MDTrack. Uni-MDTrack can uniformly process data from various modalities and consists of unified modality embedding layer, feature extraction network and prediction head.

Fig. 3: Detail structure of our proposed Memory-Aware Compression Prompt module (MCP) and Dynamic State Fusion module (DSF).

Method	Trainable	Params(M)	FLOPs(G)
Method	Params(M)	Params(M)	FLOPs(G)
Uni-MDTrack-B	27.1	88.2	27.9
HIPTrack [ 7 ]	34.1	120.4	66.9
LoRAT-B 384 [ 4 ]	13.0	99.1	97.0
SPMTrack-B [ 14 ]	29.2	115.3	-
Uni-MDTrack-L	54.9	287.4	257.4

TABLE I: Comparison of our method with other trackers using parameter-efficient training method in terms of total parameters, trainable parameters, and computational complexity.

Method	Source	LasHeR		VisEvent		DepthTrack
		SR(%)	PR(%)	AUC(%)	$P$ (%)	F-Score(%)	Re(%)	PR(%)
Uni-MDTrack-B	Ours	61.2	76.7	64.2	81.0	65.9	66.3	66.2
Uni-MDTrack-L	Ours	62.1	77.9	65.7	81.8	67.4	67.2	67.6
FlexTrack [ 36 ]	ICCV25	62.0	77.3	64.1	81.4	67.0	66.9	67.1
SUTrack-B 224 [ 35 ]	AAAI25	59.9	74.5	62.7	79.9	65.1	65.7	64.5
SUTrack-L 384 [ 35 ]	AAAI25	61.9	76.9	63.8	80.5	66.4	66.4	66.5

TABLE II: State-of-the-art comparison on RGB-T tracking dataset LasHeR, RGB-E tracking dataset VisEvent, and RGB-D tracking dataset DepthTrack. The best three results are highlighted in red , blue and bold , respectively.

Method	Source	LaSOT			TrackingNet			LaSOT ext
		AUC(%)	$P_{Norm}$ (%)	$P$ (%)	AUC(%)	$P_{Norm}$ (%)	$P$ (%)	AUC(%)	$P_{Norm}$ (%)	$P$ (%)
Unified Trackers
Uni-MDTrack-B	Ours	74.7	84.9	82.6	86.1	90.8	85.9	54.3	65.7	62.4
Uni-MDTrack-L	Ours	76.1	85.7	84.3	88.0	92.1	89.1	55.2	66.3	62.8
SUTrack-B 224 [ 35 ]	AAAI25	73.2	83.4	80.5	85.7	90.3	85.1	53.1	64.2	60.5
SUTrack-L 384 [ 35 ]	AAAI25	75.2	84.9	83.2	87.7	91.7	88.7	53.6	64.2	60.5

TABLE III: State-of-the-art comparison on RGB visual tracking datasets LaSOT, TrackingNet and LaSOT ext . The best three results are highlighted in red , blue and bold , respectively.

Method	AUC(%)	$P_{Norm}$ (%)	$P$ (%)
Uni-MDTrack-B	67.6	85.2	73.2
Uni-MDTrack-L	70.4	87.4	77.4
SUTrack-B 224 [ 35 ]	65.0	-	67.9
SUTrack-L 384 [ 35 ]	67.9	-	72.1
MCITrack-B [ 30 ]	62.9	-	-
LoRAT-B 378 [ 4 ]	59.9	-	63.7

TABLE IV: The performance of our method and other state-of-the-art trackers on RGB-Language Tracking dataset TNL2K. The best three results are highlighted in red , blue and bold .

Method	AO(%)	$SR_{0.5}$ (%)	$SR_{0.75}$ (%)
Uni-MDTrack-B	81.1	91.8	81.2
SUTrack-B 224 [ 35 ]	77.9	87.5	78.5
SPMTrack-B [ 14 ]	76.5	85.9	76.3
MCITrack-B [ 30 ]	77.9	88.2	76.8
ARPTrack 256 [ 41 ]	77.7	87.3	74.3
MambaLCT 384 [ 17 ]	76.2	86.7	74.3

TABLE V: The performance of our method and other state-of-the-art trackers on RGB-based Tracking dataset GOT-10k. The best three results are highlighted in red , blue and bold .

Method	UAV123	NfS	OTB2015
Uni-MDTrack-B	71.0	70.2	73.6
SUTrack-B 384 [ 35 ]	70.4	69.3	-
HIPTrack [ 7 ]	70.5	68.1	71.0
ARTrackV2-B [ 42 ]	69.9	67.6	-
ODTrack-L [ 13 ]	-	-	72.4
ARTrack 384 [ 43 ]	70.5	66.8	-

TABLE VI: The performance of our method and other state-of-the-art trackers on UAV123, NfS and OTB2015 in terms of AUC metrics. The best three results are highlighted in red , blue and bold .

Panel of Figure 10 — Fig. 4: Comparisons of our proposed Uni-MDTrack with other excellent trackers in the success curve on LaSOT test split, which includes eleven challenging scenarios such as Low Resolution, Motion Blur, Scale Variation, etc. We also provide the comparisons of the success and precision curves across the entire LaSOT test split. Zoom in for better view.

#	MCP	DSF	LaSOT	LasHeR	VisEvent	DepthTrack	$\bm{\Delta}$
1	✘	✘	73.2	59.9	62.7	65.1	0
2	✘	✔	73.8	60.4	63.6	65.7	+0.65
3	✔	✘	74.1	60.6	63.3	65.3	+0.6
4	✔	✔	74.7	61.2	64.2	65.9	+1.3

TABLE VII: Ablation studies on MCP and DSF modules. Experiments are conducted on LaSOT (evaluated by AUC), LasHeR (SR), VisEvent (AUC) and DepthTrack (F-Score).

Method	AUC(%)	$P_{Norm}$ (%)	$P$ (%)
DropTrack	71.8	81.8	78.1
DropTrack w/ Ours	73.1	82.7	79.7
DropTrack w/ HIP	72.7	82.9	79.5
DropTrack w/ Temporal Token	72.0	81.9	78.4

TABLE VIII: A performance comparison of existing trackers and their integration with our method on LaSOT test set.

Number	8	16	32	64
LaSOT	74.2	74.7	74.7	74.6
LasHeR	60.5	61.2	61.3	61.3
VisEvent	63.8	64.2	64.2	64.3
DepthTrack	65.5	65.9	65.9	66.1
$\bm{\Delta}$	-0.5	0	+0.03	+0.08

TABLE IX: Ablation study on different number of memory-aware compression tokens.

Panel of Figure 14 — Fig. 5: This figure presents a visual comparison among our proposed Uni-MDTrack, MambaLCT 256 [ 17 ] and SUTrack-B [ 35 ] in the challenges of target among similar objects, undergoes sudden movements, partial occlusion and scale variation. It demonstrates that our method achieves more effective and accurate tracking in the aforementioned challenging scenarios. Zoom in for better view.

Fig. 6: This figure illustrates cases of tracking failure under extremely complex scenarios. When the target undergoes severe occlusion and fast motion, while heavy background clutter are present, all methods experience tracking failures. Zoom in for better view.

Number	2	4	6	8
LaSOT	74.3	74.7	74.7	74.6
LasHeR	61.0	61.2	61.4	61.2
VisEvent	63.6	64.2	64.2	64.0
DepthTrack	65.4	65.9	65.9	65.7
$\bm{\Delta}$	-0.43	0	+0.05	-0.13

TABLE X: Ablation study on different number of DSF modules based on Uni-MDTrack-B.

#	Variants	LaSOT	LasHeR	VisEvent	DepthTrack	$\bm{\Delta}$
1	Output Search Feature	74.7	61.2	64.2	65.9	0
2	Output Whole Sequence	74.5	60.9	63.9	65.7	-0.25
3	Input Fusion Sequence	74.5	60.7	64.1	65.6	-0.28

TABLE XI: Ablation studies on which features to use for target state updates in DSF.

#	Variants	LaSOT	LasHeR	VisEvent	DepthTrack	$\bm{\Delta}$
1	SSM	74.7	61.2	64.2	65.9	0
2	LSTM	74.4	60.8	63.8	65.7	-0.33

TABLE XII: Ablation studies on replacing SSM with other model.

#	Variants	LaSOT	LasHeR	VisEvent	DepthTrack	$\bm{\Delta}$
1	Ours	74.7	61.2	64.2	65.9	0
2	Backbone Layers	74.0	60.5	63.5	65.4	-0.65

TABLE XIII: Ablation studies on leveraging SSMs as backbone layers.

Number	10	20	50	100
LaSOT	74.3	74.6	74.7	74.6
LasHeR	60.7	61.0	61.2	61.0
VisEvent	63.9	64.0	64.2	64.4
DepthTrack	65.9	66.0	65.9	65.8
$\bm{\Delta}$	-0.3	-0.1	0	-0.05

TABLE XIV: Ablation study on memory bank size of MCP module based on Uni-MDTrack-B.

Panel of Figure 21 — Fig. 7: This figure presents a visual comparison among our proposed Uni-MDTrack-B, STTrack [ 21 ] and SUTrack-B [ 35 ] in the challenges of different modalities including RGB-Depth, RGB-Event and RGB-Thermal tasks. It demonstrates that our method achieves more effective and accurate tracking in the aforementioned challenging scenarios. Zoom in for better view.

#	Variants	LaSOT	LasHeR	VisEvent	DepthTrack	$\bm{\Delta}$
1	ALiBi	74.7	61.2	64.2	65.9	0
2	w/o Position	74.6	61.2	64.0	65.9	-0.08
3	Absolute Position Encoding	74.4	61.0	64.1	65.9	-0.15

TABLE XV: Ablation studies on position bias added to attention score in MCP.

#	Variants	LaSOT	LasHeR	VisEvent	DepthTrack	$\bm{\Delta}$
1	Ours	74.7	61.2	64.2	65.9	0
2	FIFO + 5 Interval	74.1	61.1	63.8	65.8	-0.3
3	FIFO + 10 Interval	74.2	61.1	63.9	65.9	-0.23

TABLE XVI: Ablation studies on memory select strategy.

Fig. 8: Visualization of the attention of the Temporal Propagate Token on the template and search region in ODTrack, compared with the visualization of the attention between the search region and the Dynamic State Feature in DSF module of our method.

Fig. 9: Visualization of the attention map between the search region and the Dynamic State Feature output by DSF module when the target undergoes scale variation and fast motion.

Multimodal Visual Object Tracking

See all 19 papers

May 5, 2026

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

Multimodal visual object tracking can be divided into to several kinds of tasks (e.g. RGB and RGB+X tracking), based on the input modality. Existing methods often train separate models for each modality or rely on pretrained models to adapt to new modalities, which limits efficiency, scalability, and usability. Thus, we introduce OneTrackerV2, a unified multi-modal tracking framework that enables end-to-end training for any modality. We propose Meta Merger to embed multi-modal information into a unified space, allowing flexible modality fusion and robustness. We further introduce Dual Mixture-of-Experts (DMoE): T-MoE models spatio-temporal relations for tracking, while M-MoE embeds multi-modal knowledge, disentangling cross-modal dependencies and reducing feature conflicts. With a shared architecture, unified parameters, and a single end-to-end training, OneTrackerV2 achieves state-of-the-art performance across five RGB and RGB+X tracking tasks and 12 benchmarks, while maintaining high inference efficiency. Notably, even after model compression, OneTrackerV2 retains strong performance. Moreover, OneTrackerV2 demonstrates remarkable robustness under modality-missing scenarios.

Lingyi Hong, Jinglun Li, Xinyu Zhou +6

Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University

Cross-Modal Learning Multimodal Visual Object Tracking Multimodal Robustness Unified Multimodal Models Mixture of Experts Visual Object Tracking+2

Ask PDF

Feb 27, 2026

UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking

One-stream Transformer-based trackers achieve advanced performance in visual object tracking but suffer from significant computational overhead that hinders real-time deployment. While token pruning offers a path to efficiency, existing methods are fragmented. They typically prune the search region, dynamic template, and static template in isolation, overlooking critical inter-component dependencies, which yields suboptimal pruning and degraded accuracy. To address this, we introduce UTPTrack, a simple and Unified Token Pruning framework that, for the first time, jointly compresses all three components. UTPTrack employs an attention-guided, token type-aware strategy to holistically model redundancy, a design that seamlessly supports unified tracking across multimodal and language-guided tasks within a single model. Extensive evaluations on 10 benchmarks demonstrate that UTPTrack achieves a new state-of-the-art in the accuracy-efficiency trade-off for pruning-based trackers, pruning 65.4% of vision tokens in RGB-based tracking and 67.5% in unified tracking while preserving 99.7% and 100.5% of baseline performance, respectively. This strong performance across both RGB and multimodal scenarios underlines its potential as a robust foundation for future research in efficient visual tracking. Code will be released at https://github.com/EIT-NLP/UTPTrack.

Hao Wu, Xudong Wang, Jialiang Zhang +5

Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative · Shanghai Jiao Tong University +2

Visual Object Tracking Visual Token Pruning Efficient ViTs Multimodal Token Compression

Ask PDF

Sep 27, 2026

LoopTrack: A Simple Baseline for Parameter-Efficient Transformer Tracking

Current Transformer-based tracking methods typically stack multiple Transformer blocks with separate parameters to model interactions between the target template and the search region for target localization. These trackers often incur substantial parameter overhead from stacked blocks, making their deployment on resource-limited devices difficult. To address this, we propose a parameter-efficient Transformer tracking framework, dubbed LoopTrack, which repeatedly applies a set of Transformer blocks with shared parameters to interact features in a looped architecture for tracking, significantly reducing the number of parameters. To further exploit target cues, we present two lightweight designs, including target-aware looping (TAL) and gated target memory (GTM). The former applies intermediate target information generated by one loop to guide feature interaction in the subsequent loop, enabling progressive feature refinement, while the latter maintains a compact memory across frames, which is incorporated into the loop process to provide long-term information to the tracker, mitigating temporal drift in tracking. Compared to existing Transformer trackers, LoopTrack enables multiple rounds of feature interaction with fewer model parameters, making it resource-friendly for deployment. In extensive experiments on multiple datasets, LoopTrack shows a favorable accuracy-parameter trade-off. In particular, our LoopTrack

_{\rm One}

, with a single shared Transformer block, achieves 66.2% SUC score on LaSOT with only 3.4M parameters, while LoopTrack

_{\rm Three}

, using three shared blocks, achieves 69.3% SUC score with 6.4M parameters, surpassing existing parameter-efficient tracking methods with comparable or larger model size. With LoopTrack, we aim to establish a simple yet strong baseline for parameter-efficient Transformer tracking. Our code and models will be released.

Liang Peng, Chenxiao Li, Libo Zhang +2

School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University · University of North Texas · Institute of Software, Chinese Academy of Sciences

Visual Object Tracking Recurrent Transformers Efficient ViTs

Ask PDF

Abstract

Multimodal Visual Object Tracking

Explore similar work

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

UTPTrack: Towards Simple and Unified Token Pruning for Visual Tracking

LoopTrack: A Simple Baseline for Parameter-Efficient Transformer Tracking